Exercises — Causal language modeling objective

Answer: (b). Why not (a)? That assumes the three words are independent — but "love" after "I" is not the same as "love" after "quantum", so they are not independent. Why not (c)? It is a valid chain-rule ordering mathematically, but it conditions on the future (predicting $w_1$ from $w_3$). "Causal" means left-to-right: position $t$ may only look at positions $<t$. Only (b) respects that direction.

Each term is $-\log (1.0)=0$. Summing zeros gives ${\mathcal{L}}_{\text{CLM}}=0$. Intuition: zero loss = the model was certain and correct at every step. Loss can never go below $0$ because probabilities are $\le 1$, and $-\log$ of anything $\le 1$ is $\ge 0$.

${\mathcal{L}}_1=-\ln (0.2)\approx 1.61$ ${\mathcal{L}}_2=-\ln (0.5)\approx 0.69$ ${\mathcal{L}}_3=-\ln (0.25)\approx 1.39$ Total $=1.61+0.69+1.39=3.69$. Notice position 1 does contribute — see Mistake 2 in the parent note.

Uniform means every token gets probability $1/100=0.01$. ${\mathcal{L}}_t=-\ln (0.01)=\ln (100)\approx 4.61$. Why this matters: $\ln |V|$ is the "know-nothing" baseline. If your trained model's per-token loss is not clearly below $\ln |V|$, it has learned nothing useful. (For GPT-2's $|V|=50257$, that baseline is $\ln (50257)\approx 10.82$.)

Why this tool (softmax)? Logits are raw scores in $(-\infty ,\infty )$; softmax turns them into positive numbers that sum to 1, i.e. a probability distribution. The formula above is just $-\log$ of the softmax probability of the true class, algebraically simplified. $\sum e^z=e^2+e^0+e^1=7.389+1+2.718=11.107$. $\log (11.107)\approx 2.408$. $\mathcal{L}=-2+2.408=0.408$. Check: softmax prob of class 0 $=e^2/11.107=7.389/11.107=0.665$, and $-\ln (0.665)=0.408$. ✓

Average per-token loss $=3.69/3=1.23$. $\text{PPL}=e^{1.23}\approx 3.42$. Interpretation: perplexity is the "effective branching factor" — as if, at each step, the model were choosing uniformly among $\approx 3.42$ equally likely tokens. Lower is better. Why exponentiate the average loss? Because loss is in log-units (nats); $\exp$ undoes the log and puts us back in "number of choices" units, which humans read more easily.

${\text{PPL}}_A=e^{2.0}\approx 7.389$, ${\text{PPL}}_B=e^{3.0}\approx 20.086$. Ratio $=20.086/7.389\approx 2.718=e^{3.0-2.0}=e^1$. Insight: a loss difference of $\Delta$ nats is a perplexity ratio of $e^{\Delta }$. A 1-nat improvement always multiplies perplexity down by $e\approx 2.72$, regardless of the starting point. This is why loss differences of "just 0.3" are quietly significant.

In self-attention, position $t$ computes a score against every other position, then softmaxes those scores into weights. To forbid looking ahead, we set the scores for positions $>t$ to $-\infty$ before the softmax. Why $-\infty$? Because softmax exponentiates: $e^{-\infty }=0$. So a future position receives weight exactly $0$ and contributes nothing to position $t$'s representation. See figure below. Key phrase: computationally parallel ≠ informationally non-causal. All positions are processed at once for speed, but each one is blind to its future because those weights are zeroed. See Transformer Decoder and Teacher Forcing.

Indexing from 0: positions hold <BOS>(0), I(1), love(2), ML(3). shift_logits uses positions 0,1,2 (drop the last, since there is no token after ML to predict). shift_labels uses positions 1,2,3 (drop the first, <BOS>, which we never predict). Pairs (context ⇒ target):

• <BOS> ⇒ I
• <BOS>, I ⇒ love
• <BOS>, I, love ⇒ ML That is 3 loss terms from 4 tokens. General rule: a length-$T$ sequence yields $T-1$ predictions when the first token is a given <BOS>.

Without: $-\ln (0.1)=2.303$. With: $-\ln (0.25)=1.386$. Improvement $=2.303-1.386=0.917$ nats. Why it works: <BOS> is a learnable anchor. Predicting "which token starts a sentence" from a consistent start symbol lets the model learn a real conditional distribution instead of a vaguer unconditional one — connects to how Autoregressive Models handle the boundary.

Causal LM: predicts each token from its left context ⇒ up to $3$ terms (or $T-1$ with <BOS>). Every position supervised. MLM: masks ~15% of tokens, here 1 token, and predicts only that ⇒ $1$ term per pass. MLM also lets predictions see both sides (bidirectional), so it is not left-to-right. Why causal for generation: generation produces text left-to-right, one token at a time. The training objective $P(x_t\mid x_{<t})$ is exactly the conditional you sample from at generation. MLM's bidirectional peeking cannot be run left-to-right, so it does not directly give a generation procedure. See GPT Family and Sampling Strategies.

Each step correct with prob $0.9$; steps' errors compound multiplicatively. $P(\text{all 5 correct})=0.9^5=0.59049\approx 0.590$, i.e. about $59\%$. Contrast: during teacher forcing the model always receives the ground-truth previous tokens, so a mistake at step 2 never corrupts step 3's context. At generation there is no ground truth to fall back on — errors feed forward and compound. That gap between training and inference conditioning is exactly exposure bias.

Expected per-token loss is the cross-entropy $H(P_{\text{data}},P_{\theta })=-{\sum }_v{P}_{\text{data}}(v)\log P_{\theta }(v)$. A standard identity: $H(P_{\text{data}},P_{\theta })=H(P_{\text{data}})+D_{\text{KL}}(P_{\text{data}}\Vert P_{\theta })$, where $H(P_{\text{data}})$ is the fixed data entropy and $D_{\text{KL}}\ge 0$ is the Kullback–Leibler divergence, equal to $0$ iff the two distributions match. Since $H(P_{\text{data}})$ does not depend on $\theta$, minimizing the loss means minimizing $D_{\text{KL}}$, whose minimum ($0$) is attained exactly when $P_{\theta }=P_{\text{data}}$. See Cross-Entropy Loss. Takeaway: the best possible loss is not zero — it is the data's own entropy $H(P_{\text{data}})$, the irreducible uncertainty of language. A model at that floor is perfect; it cannot do better because language itself is not fully predictable.

A perfect model outputs exactly $\{0.6,0.4\}$. Its expected loss equals the entropy: $H=-[0.6\ln 0.6+0.4\ln 0.4]$ $=-[0.6(-0.5108)+0.4(-0.9163)]$ $=-[-0.3065-0.3665]=0.673$ nats. Interpretation: even a flawless model pays $\approx 0.673$ nats here because the true next word is genuinely uncertain. No amount of training pushes this position's loss below $0.673$. That is the entropy floor from Exercise 5.2 made concrete.

One-nat loss improvement multiplies perplexity by which factor? ::: $e\approx 2.72$ Minimum achievable average loss equals what quantity? ::: the entropy of the true data distribution $H(P_{\text{data}})$ A length-$T$ sequence with a prepended <BOS> gives how many loss terms? ::: $T-1$ (predict each real token from its left context) What replaces future attention scores before softmax, and why? ::: $-\infty$, because $e^{-\infty }=0$ zeroes their weight, enforcing causality