Worked examples — Causal language modeling objective

This page is the "throw everything at it" companion to the parent objective note. We already know the loss:

${\mathcal{L}}_{\text{CLM}}=-{\sum }_{t=1}^T\log P_{\theta }(x_t\mid x_{<t})$

Here we use it on every kind of input a real training run meets. Before we start, two reminders in plain words so no symbol is a surprise:

The scenario matrix

Every causal-LM example falls into one of these cells. The 8 worked examples below cover all of them.

Cell	Case class	What's special	Covered by
A	Ordinary short sequence	all context present	Ex 1
B	First token, no context	$x_{<1}=\varnothing$ (degenerate context)	Ex 2
C	Perfect prediction	limiting value $P\to 1$, loss $\to 0$	Ex 3
D	Worst prediction	limiting value $P\to 0$, loss $\to \infty$	Ex 3
E	Uniform / untrained model	$P=1/|V|$ baseline	Ex 4
F	Padding & masking	tokens that must NOT count	Ex 5
G	Perplexity conversion	turning loss into an interpretable number	Ex 6
H	Real-world word problem	weather-report generator	Ex 7
I	Exam twist	which shift is correct? off-by-one trap	Ex 8

---

Example 1 — Cell A: an ordinary sequence

Forecast: guess before computing — is the total nearer $1$, $6$, or $60$? (The ML step is very surprising, so expect a chunky number.)

1. Turn each probability into surprise. ${\mathcal{L}}_t=-\log P_t$. Why this step? The objective is a sum of per-token surprises; each factor $P(w_t\mid w_{<t})$ in the product becomes one additive term after $-\log$.

   • ${\mathcal{L}}_1=-\log (0.10)\approx 2.3026$
   • ${\mathcal{L}}_2=-\log (0.40)\approx 0.9163$
   • ${\mathcal{L}}_3=-\log (0.05)\approx 2.9957$

2. Add them. Why this step? Chain rule: $P(\text{seq})=0.10\cdot 0.40\cdot 0.05$, and $-\log$ of a product is the sum of $-\log$s. $\mathcal{L}=2.3026+0.9163+2.9957\approx 6.2146$

Verify: compute directly. $P(\text{seq})=0.10\times 0.40\times 0.05=0.002$, and $-\log (0.002)\approx 6.2146$. ✓ Matches the sum — the two roads agree, as Cross-Entropy Loss promises.

---

Example 2 — Cell B: the first token has context after all

Forecast: zero, or the same $2.30$ as before?

1. Recognise the empty context is not empty. With <BOS>, we have $x_{<1}=(\text{<BOS>})$. Why this step? The model still predicts a distribution over first words; The, I, capitalised tokens are common. That is real learnable information, so the loss is not waived.

2. Apply the surprise formula. Why this step? Cell B is just Cell A with a one-token context; nothing about the loss changes. ${\mathcal{L}}_1=-\log (0.10)\approx 2.3026$

Verify: it equals ${\mathcal{L}}_1$ of Example 1 — consistent, since the probability is identical. Skipping this term would undercount the true negative log-likelihood. ✓

---

Example 3 — Cells C & D: the two extremes

Forecast: which is $0$ and which blows up?

1. Perfect case (C). $\mathcal{L}=-\log (1)=0$. Why this step? $-\log$ maps certainty to zero penalty — the floor of the loss.

2. Near-impossible case (D). $\mathcal{L}=-\log (10^{-6})=6\log (10)\approx 13.8155$. Why this step? As $P\to 0$, $-\log P\to +\infty$. This is the limiting behaviour that punishes over-confident wrong models catastrophically — the mathematical reason models learn to never assign exactly $0$.

Verify: $-\log (10^{-6})=6\times 2.302585\approx 13.8155$. ✓ And $-\log (1)=0$ exactly. The gap between best and worst on a single token is unbounded above, bounded below by $0$.

---

Example 4 — Cell E: the untrained-model baseline

Forecast: the parent note quoted "$\approx 10.8$". Can you derive it?

1. Write the uniform probability. $P=1/|V|$. Why this step? A random-init softmax over near-equal logits spreads probability evenly — no token is favoured.

2. Take the surprise. Why this step? Cell E is the calibration point: any trained model must beat this to have learned anything. $\mathcal{L}=-\log \left(\frac{1}{50257}\right)=\log (50257)\approx 10.8249$

Verify: $\log (50257)\approx 10.8249$, matching the parent's $\approx 10.8$. A trained GPT-2 scoring $\approx 3.5$ is genuinely far below chance. ✓ This baseline is exactly the Perplexity anchor $=|V|$ (see Ex 6).

---

Example 5 — Cell F: padding must not be graded

Forecast: should the two big $4.0$s drag the average up?

1. Identify which positions are real. Positions $1,2,3$ are content; positions $4,5$ are padding with ignore_index. Why this step? Padding exists only to make tensors rectangular; grading it would teach the model to "predict" filler, corrupting the objective.

2. Sum only the real terms. Why this step? The masked positions contribute $0$, not their raw surprise. $\text{sum}=1.0+0.5+2.0=3.5$

3. Average over the real count (3), not 5. Why this step? The mean must divide by the number of graded tokens. $\bar{\mathcal{L}}=3.5/3\approx 1.1667$

Verify: the wrong way, dividing $(3.5+8.0)/5=2.30$, is inflated by the padding — nearly double. Correct answer $1.1\overline{6}$. ✓ This is why real code passes ignore_index=pad_token_id.

---

Example 6 — Cell G: loss → perplexity

Forecast: perplexity is "effective number of choices" — expect a couple dozen for $3.5$.

1. Define perplexity. $\text{PPL}=e^{\bar{\mathcal{L}}}$. Why this step? Perplexity un-does the $\log$: it is the geometric-mean branching factor, "how many equally-likely tokens is the model effectively choosing among?" See Perplexity.

2. Plug in the trained loss. $\text{PPL}=e^{3.5}\approx 33.115$

3. Sanity-check the baseline. For Ex 4, $\bar{\mathcal{L}}=\log (50257)$, so $\text{PPL}=e^{\log (50257)}=50257.$ Why this step? Uniform-over-$V$ must give perplexity exactly $|V|$ — the model is genuinely torn between all $50257$ tokens.

Verify: $e^{3.5}\approx 33.115$; and $e^{\log (50257)}=50257$ exactly. ✓ A trained model with PPL $\approx 33$ vs. baseline $50257$ is a huge win.

---

Example 7 — Cell H: a real-world word problem

Forecast: with only 4 words, perfect uniform loss would be $\log 4\approx 1.39$; is this model better or worse than random?

1. Per-token surprises. Why this step? Same machinery, small vocabulary.

   • ${\mathcal{L}}_1=-\log (0.5)\approx 0.6931$
   • ${\mathcal{L}}_2=-\log (0.25)\approx 1.3863$

2. Total loss. $\mathcal{L}=0.6931+1.3863\approx 2.0794$. Why this step? Sum over the 2 graded positions.

3. Average loss. $\bar{\mathcal{L}}=2.0794/2\approx 1.0397$. Why this step? Two tokens, so divide by 2.

4. Perplexity. $\text{PPL}=e^{1.0397}\approx 2.8284$. Why this step? Effective branching factor; $2.83<4$ means the model beats a coin-flip-over-4 baseline. This connects directly to how Autoregressive Models chain conditionals during sampling.

Verify: $-\log (0.5)=\log 2\approx 0.6931$; $-\log (0.25)=\log 4\approx 1.3863$; sum $\approx 2.0794$; half $\approx 1.0397$; $e^{1.0397}\approx 2.8284$. Also $\text{PPL}=(0.5\cdot 0.25{)}^{-1/2}=(0.125{)}^{-1/2}=\sqrt{8}\approx 2.828$. ✓ Two roads agree.

---

Example 8 — Cell I: the off-by-one exam twist

Forecast: how many valid (context → target) pairs can length-5 actually produce?

1. State the causal rule. Position $t$ must be predicted from positions $<t$. Why this step? The whole objective forbids looking at $x_t$ when predicting $x_t$ — the no-shift code grades the model on copying its own input, which it can do trivially through the residual path. This is a fake near-zero loss.

2. Do the shift. Use logits[:, :-1, :] against labels[:, 1:]. Why this step? Logit at position $t-1$ (which has only seen $x_{<t}$) predicts label $x_t$. Dropping the last logit and the first label lines them up.

3. Count the pairs. Why this step? From $T=5$ tokens you get exactly $T-1=4$ prediction targets: $(x_1\to x_2),(x_2\to x_3),(x_3\to x_4),(x_4\to x_5).$

Verify: number of graded pairs $=T-1=5-1=4$. ✓ The no-shift version would spuriously "predict" $x_t$ from $x_t$, giving loss near $0$ — the classic sign of the Teacher Forcing shift being missing. (Note this is training-time; generation instead suffers Exposure Bias, and it's why Masked Language Modeling and causal decoders handle masking differently.)

---

Recall Quick self-test

Loss of a token predicted with $P=0.02$ ::: $-\log (0.02)\approx 3.912$ Perplexity from average loss $2.0$ nats ::: $e^2\approx 7.389$ Uniform baseline loss for a 1000-word vocab ::: $\log (1000)\approx 6.908$ Correct number of graded pairs for $T=10$ ::: $9$ Why don't padding tokens count? ::: they are filler; grading them corrupts the objective, so we mask with ignore_index