Exercises — GPT family architecture evolution

GPT keeps only the decoder, and drops the encoder entirely (so there is no cross-attention). The mechanism that makes it autoregressive is the causal mask: a rule that forbids each token from looking at any token to its right (the future). "Autoregressive" simply means predict the next item using only the items before it — the mask is what enforces "only before it".

• (a) → 2: GPT-1 introduced generative pretraining then supervised fine-tuning per task.
• (b) → 4: GPT-2 showed zero-shot transfer and switched to pre-LN (LayerNorm placed before each sublayer).
• (c) → 1: GPT-3's headline was few-shot in-context learning emerging purely from scale.
• (d) → 3: GPT-3.5 / InstructGPT added RLHF, an alignment change on the same backbone. See RLHF and InstructGPT.

LayerNorm (LN) rescales a vector so its entries have mean 0 and variance 1 — it stabilizes the numbers flowing through the network.

• Post-LN (GPT-1): LN is applied after adding the sublayer output back into the residual stream.
• Pre-LN (GPT-2+): LN is applied before the sublayer, so the residual "highway" stays un-normalized. Switch reason: the un-normalized highway lets gradients flow cleanly through 48–96 stacked layers without exploding or vanishing.

Loss at $P=0.25$: $-\ln (0.25)=\ln 4\approx 1.386$ nats. Loss at $P=1.0$: $-\ln (1.0)=0$ nats. Reading it: loss $=0$ means the model was certain and correct. The lower the probability it puts on the truth, the bigger the penalty — and it blows up toward $+\infty$ as $P\to 0$. This is exactly the Cross-entropy loss penalizing confident wrong-ness.

The figure below plots this loss curve $-\ln (P)$ against the probability $P$ the model assigns to the true token. Follow the yellow curve: it is flat and near 0 on the right (confident and correct, blue dot at $P=1$) and rockets upward on the left (the pink dot at $P=0.25$ sits at loss $1.386$, and the curve keeps climbing toward $+\infty$ as $P\to 0$). The steep left tail is the visual meaning of "confident wrongness is punished hard".

Four matrices $\Rightarrow 4d^2$ parameters. For $d=12288$: $4\times 12288^2=4\times 150,994,944=603,979,776\approx 6.0\times 10^8$. So just the attention projections of one layer already hold ~0.6 billion parameters. Because it grows as $d^2$, doubling width quadruples this cost — that quadratic is why 175B is reached so quickly.

Variance $=d_k=64$, so standard deviation $=\sqrt{64}=8$. After dividing by $\sqrt{d_k}=8$: new variance $=64/64=1$, standard deviation $=1$. Why we do this: softmax turns scores into probabilities. If scores swing by $\pm 8$ or more, softmax saturates (one weight $\approx 1$, the rest $\approx 0$) and its gradient nearly vanishes. Rescaling to variance 1 keeps softmax in its responsive region so learning continues.

Rule: $M_{ij}=0$ if $j\le i$ (token $i$ may attend to itself and everything before it), and $M_{ij}=-\infty$ if $j>i$ (the future is blocked). The blocked region is the strict upper triangle ($j>i$). If we allowed token $i$ to see token $i+1$, then while predicting token $i+1$ the model could peek at the very answer it is supposed to guess. Training loss would collapse toward 0 by copying, and no real language ability would be learned. The strictness ($>$, not $\ge$) is essential.

The figure below draws the mask as a grid: rows are query positions $i$ (the token doing the predicting), columns are key positions $j$ (the token being looked at). Blue cells (value $0$) are allowed — they form a lower-triangular staircase where $j\le i$. Pink cells (value $-\infty$) are blocked — they form the strict upper triangle $j>i$, the "future" a token must never peek at. Reading any row left to right shows exactly how far into the past that token may attend.

Take $\log$ (natural log) of both sides: $\log L\approx {\alpha }_N\log N_c-{\alpha }_N\log N$. This is a straight line in the variables $(\log N,\log L)$ with slope $-{\alpha }_N$. A power law is the function whose log-log graph is a straight line. An exponential $L=e^{-cN}$ would instead be straight on a semi-log plot ($\log L$ vs plain $N$). Empirically, GPT losses fall on a line only in log-log — so the data say "power law". Physically: every time you multiply $N$ by 10, loss drops by the same fraction, not the same amount.

The gradient of the output with respect to the input has the form $\frac{\partial y}{\partial x}=I+\frac{\partial F}{\partial x}$, where $I$ is the identity (the "$+x$" term). Even if $\frac{\partial F}{\partial x}$ is tiny, the $I$ keeps the total near 1, so multiplying such factors across 96 layers does not drive the gradient to 0. Without the residual, each layer contributes a factor $\frac{\partial F}{\partial x}$; multiplying 96 small numbers gives a vanishingly small product — the classic vanishing-gradient failure. The identity "highway" is what makes very deep stacks trainable.

Ratio $N_3/N_2=175/1.5\approx 116.7$. Since $\log L=\text{const}-{\alpha }_N\ln N$ (natural log), moving from $N_2$ to $N_3$ changes it by $\Delta (\log L)=-{\alpha }_N\ln \left(\frac{N_3}{N_2}\right)=-0.076\times \ln (116.7)\approx -0.076\times 4.759\approx -0.362\text{nats}.$ Interpretation: the sign is negative — loss improves (drops). The size ~0.36 nats is meaningful but bounded: a 117× scale-up did not send loss to 0. That finite return is exactly what motivated the search for compute-optimal scaling (Chinchilla), where data is scaled alongside parameters.

Attention: $4d^2$. MLP: $W_1$ has $d\times 4d=4d^2$ params, $W_2$ has $4d\times d=4d^2$ params, total $8d^2$. Per layer: $4d^2+8d^2=12d^2$. With $d=12288$: $12\times 12288^2=12\times 150,994,944=1,811,939,328\approx 1.81\times 10^9$ parameters per layer. The MLP dominates ($8d^2$ vs $4d^2$) — it holds two-thirds of a layer's weights. This is why sparse alternatives like Mixture-of-Experts target the MLP for efficiency.

Architecture change: (a) pre-LN (moves a layer's LayerNorm), (c) causal masking (the core attention wiring), (e) Mixture-of-Experts (replaces the dense MLP with routed experts — Mixture-of-Experts). Training/alignment change: (b) RLHF (fine-tunes the same backbone with human feedback — RLHF and InstructGPT), (d) few-shot in-context learning. Justify the tricky one (d): in-context learning is not a wiring change and not a weight update at all — it happens at inference with frozen weights. It is placed under training/alignment change here because, of the two allowed buckets, it belongs on the "how the model is used/produced" side rather than the "how the model is wired" side. The deeper truth to remember: it is an emergent behaviour of a large trained model, but it definitively does not alter the architecture.

The power law $L\propto N^{-{\alpha }_N}$ with tiny ${\alpha }_N\approx 0.076$ means each 10× in parameters shaves only ~$0.076\times \ln (10)\approx 0.175$ nats — a shrinking absolute payoff as $L$ approaches its irreducible floor (the entropy of language itself, which no model can beat). Worse, a giant $N$ trained on too little data $D$ underfits: parameters have nothing new to learn from. The fix is compute-optimal scaling — the Chinchilla finding that $N$ and $D$ should grow together (roughly in proportion), not $N$ alone. See Scaling laws (Kaplan, Chinchilla). So mastery-level answer: scale works, but only balanced scale of parameters and tokens, bounded by the irreducible entropy floor.

Removing the mask makes attention bidirectional: while predicting token $i+1$, the model can attend to token $i+1$ (and beyond) directly.

• At training time: the target token becomes visible as an input, so the model "cheats" by copying rather than predicting. The loss collapses toward 0 without any genuine language learning.
• At generation time: those future tokens do not exist yet (you are producing them one at a time), so any skill that relied on seeing them is useless — the model was trained on information it will never have at inference. The family that does use bidirectional (unmasked) attention is BERT (encoder-only). But BERT is trained on a masked-token filling objective (hide random words, predict them from both sides) — not left-to-right generation — so bidirectionality is legitimate there. See BERT vs GPT (encoder vs decoder). Conclusion: for a generative left-to-right model the causal mask is not optional overhead — it is what keeps the training objective honest and matched to how the model is actually used.

(i) $\Delta (\log L)=-{\alpha }_N\ln (10)=-0.076\times 2.3026\approx -0.175$ nats. A modest, predictable gain — again bounded, reinforcing the diminishing-returns lesson. (ii) Attention projections $=4d^2=4\times 24576^2=4\times 603,979,776=2,415,919,104\approx 2.42\times 10^9$ per layer. Doubling $d$ quadruples the per-layer attention cost (from ~$6.0\times 10^8$ to ~$2.4\times 10^9$), the same $d^2$ scaling from L2.2 — showing why width is far more expensive than depth.

Every exercise here poked at one of four load-bearing ideas: (1) GPT is a decoder-only next-word guesser, (2) the causal mask keeps the guessing honest, (3) residuals + pre-LN let the stack go deep, and (4) scaling laws let you predict — but only bounded — improvements. Master those four and the whole GPT timeline is just "same skeleton, bigger, plus a couple of honest tweaks".