Visual walkthrough — Causal language modeling objective

Before we start, three plain words we will use constantly:

• token — a chunk of text (a word or word-piece). Think of it as one bead on a string.
• sequence — the whole string of beads in order: bead 1, bead 2, bead 3, …
• model — a machine with knobs (we call all the knobs together $\theta$). Turning the knobs changes what the machine believes will come next.

---

Step 1 — Draw a sequence as beads on a string

WHAT. We lay the sentence out left-to-right as numbered positions.

WHY. Everything that follows depends on order: "which bead comes first". A language model reads like you do — left to right — so we must fix that direction before writing any math.

PICTURE.

Look at the string. Each bead is a token. We write them:

$\mathbf{x}=(\underset{\text{1st token}}{\underbrace{x_1}},\underset{\text{2nd token}}{\underbrace{x_2}},\dots ,\underset{\text{last token}}{\underbrace{x_T}})$

• $x_t$ ::: the token sitting at position $t$ (the $t$-th bead).
• $T$ ::: how many beads there are in total (the length).
• The bold $\mathbf{x}$ ::: the whole string bundled into one name.

---

Step 2 — The one honest question a language model asks

WHAT. At each bead, cover everything to the right with your hand. Ask: given only what I can see, what comes next?

WHY. This is the whole game. A next-token predictor answers exactly one kind of question, over and over: "what is bead $t$, given beads $1$ through $t-1$?" If it can answer that everywhere, it understands the language.

PICTURE.

The blue region is visible; the greyed-out region is hidden by the model's own rule. The answer to the question is not one guess — it is a whole probability spread over every possible token:

$P_{\theta }(x_t\mid x_{<t})$

• $P_{\theta }(\cdots )$ ::: a probability — a number between $0$ and $1$ — that the machine (with knobs $\theta$) assigns.
• the bar $\mid$ ::: reads "given". Left of the bar = the thing we score; right of the bar = what we already know.
• $x_t$ on the left ::: the token that actually came next in the real text.

---

Step 3 — Score a whole sentence: the chain rule

WHAT. To score the entire string, multiply the per-bead answers together.

WHY. We do not want a machine good at one word; we want one that assigns high probability to the whole real sentence. The chain rule of probability says any joint probability can be split into a product of "next given past" pieces — which is precisely the question our model already answers. That is the lucky coincidence that makes everything click.

PICTURE.

$P_{\theta }(\mathbf{x})=\underset{\text{1st, no past}}{\underbrace{P_{\theta }(x_1)}}\cdot \underset{\text{given bead 1}}{\underbrace{P_{\theta }(x_2\mid x_1)}}\cdot \underset{\text{given beads 1–2}}{\underbrace{P_{\theta }(x_3\mid x_1,x_2)}}\cdots \underset{\text{given all before}}{\underbrace{P_{\theta }(x_T\mid x_{<T})}}$

Written compactly with the "multiply-them-all" symbol $\prod$ (a capital Greek pi, meaning product):

$P_{\theta }(\mathbf{x})={\prod }_{t=1}^T{P}_{\theta }(x_t\mid x_{<t})$

• ${\prod }_{t=1}^T$ ::: "multiply the following for $t=1,2,\dots ,T$".
• Notice $x_1$ has empty context ($x_{<1}=\varnothing$) — see Step 6 for that edge case.

---

Step 4 — Turn multiply into add with a logarithm

WHAT. Take the logarithm of the product. This turns the long product into a plain sum.

WHY — two concrete reasons.

1. Tiny numbers. Each $P_{\theta }$ is below $1$. Multiply hundreds of them and you get a number like $0.0000\dots$ so small the computer rounds it to zero (underflow). We need a rescue.
2. The magic identity $\log (a\cdot b)=\log a+\log b$. A log converts every multiplication into an addition. Sums of moderate numbers never underflow.

That is why the log and not some other function: it is the one function that trades products for sums.

PICTURE.

$\log P_{\theta }(\mathbf{x})=\log {\prod }_{t=1}^T{P}_{\theta }(x_t\mid x_{<t})={\sum }_{t=1}^T\log P_{\theta }(x_t\mid x_{<t})$

• ${\sum }_{t=1}^T$ ::: "add the following for $t=1,\dots ,T$" (capital Greek sigma = sum).
• The product $\prod$ became a sum $\sum$ — that is the whole trick, nothing was lost.

---

Step 5 — Flip the sign to get a loss to minimise

WHAT. Attach a minus sign. Now we have a quantity that is small when the model is good.

WHY. We want to make $P_{\theta }(\mathbf{x})$ big (assign high probability to real text). But $\log$ of a probability is always $\le 0$ (log of a number below 1 is negative). Optimisers are built to push things down. So we negate: maximising a probability becomes minimising its negative log. Same goal, friendlier shape.

PICTURE.

The curve $-\log p$ falls from $+\infty$ (at $p=0$: "you were sure of the wrong thing, huge penalty") down to $0$ (at $p=1$: "you were certain and correct, no penalty").

$\boxed{{\mathcal{L}}_{\text{CLM}}=-\sum _{t=1}^T\log P_{\theta }(x_t\mid x_{<t})}$

• $\mathcal{L}$ ::: the loss — the thing training pushes toward $0$.
• the minus ::: turns "bigger probability is better" into "smaller loss is better".
• each term $-\log P_{\theta }(x_t\mid x_{<t})$ ::: the penalty at one bead. This per-token penalty is exactly cross-entropy against the true next token.

This boxed line is the parent formula. We built it from beads.

---

Step 6 — The edge case: the very first bead

WHAT. Position $1$ has no past. Does it get a loss?

WHY it matters. A common trap is to say "$x_1$ has nothing before it, so skip it". Skipping it is wrong — the machine must still learn which tokens like to start sentences ("The", "I", capital letters).

PICTURE.

The fix: prepend a fake starter bead called <BOS> (beginning-of-sequence). Now even $x_1$ has something to its left, and its penalty is the honest

$-\log P_{\theta }(x_1\mid \text{<BOS>})$

So every bead — first to last — contributes a term. No bead is free.

---

Step 7 — Sum over the whole dataset

WHAT. Do this for every sentence in the data and average.

WHY. One sentence is noise; we want the machine good on the whole corpus $\mathcal{D}$. Averaging (not just summing) keeps the number comparable across datasets of different sizes.

${\mathcal{L}}_{\text{CLM}}(\mathcal{D})=-\frac{1}{|\mathcal{D}|}{\sum }_{\mathbf{x}\in \mathcal{D}}{\sum }_{t=1}^T\log P_{\theta }(x_t\mid x_{<t})$

• $\mathcal{D}$ ::: the whole pile of training sequences.
• $|\mathcal{D}|$ ::: how many sequences are in the pile (we divide to average).

During training all positions are scored in parallel in one forward pass through a transformer decoder, with a mask hiding the future — this is teacher forcing. Parallel computation, still perfectly causal.

---

A worked number (so it isn't abstract)

Take "I love ML" with the parent's numbers:

bead	model's probability	penalty $-\ln p$
I ($\mid$<BOS>)	$0.1$	$2.30$
love ($\mid$ I)	$0.4$	$0.92$
ML ($\mid$ I love)	$0.05$	$3.00$

${\mathcal{L}}_{\text{CLM}}=2.30+0.92+3.00=6.22$

A random un-trained model over a vocabulary of size $|V|$ scores about $\ln |V|$ per token (it spreads probability evenly). For GPT-2's $|V|=50257$ that is $\ln (50257)\approx 10.82$ per token. Turning that per-token loss back into a "typical number of choices" gives perplexity $=e^{\mathcal{L}/T}$.

---

The one-picture summary

Read it top to bottom: beads → cover the future → one probability per bead → multiply (chain rule) → log turns multiply into add → minus turns "bigger is better" into "smaller is better" → average over the corpus.

Recall Retell it in plain words (Feynman check)

I write my sentence as beads in a row. At each bead I cover everything to its right with my hand and ask my machine: "what comes next?" The machine answers with a probability for the real next bead. I want that probability to be big for every bead, so I multiply all of them to score the whole sentence. But multiplying hundreds of small numbers vanishes to zero on a computer, so I take a logarithm — that magically turns the big multiplication into a plain addition. Logs of probabilities are negative and my optimiser only knows how to push things down, so I flip the sign. Now I have a loss: it's zero when the machine is perfectly right and huge when it's confidently wrong. I even score the very first bead by pretending a <BOS> marker sits before it, so nothing is free. Finally I average this over every sentence in my data. That whole story is the single line $\mathcal{L}=-{\sum }_t\log P_{\theta }(x_t\mid x_{<t})$.