GPT family architecture evolution: Worked examples

The scenario matrix

Before working examples, let us list every kind of case the GPT-architecture math can throw at us. Each row is a "cell". Every cell is covered by at least one worked example below.

#	Cell (scenario class)	Degenerate / limiting version	Example
A	Counting attention params ( $4 d^{2}$ )	tiny $d$ vs GPT-3 $d$	Ex 1, Ex 2
B	Counting MLP params ( $8 d^{2}$ ) & full-layer total	when MLP dominates	Ex 2
C	Causal mask correctness	position 1 (sees only itself), last position (sees all)	Ex 3
D	Cross-entropy loss numeric	perfect model ( $L \to 0$ ), random model ( $L \to ln V$ )	Ex 4
E	The $d_{k}$ variance fix	$d_{k} = 1$ (no effect) vs large $d_{k}$	Ex 5
F	Scaling-law forecast (sign of $Δ L$ )	scale up (loss falls, $Δ L < 0$ ) vs scale down (loss rises, $Δ L > 0$ )	Ex 6, Ex 7
G	Limiting behaviour of the power law	$N \to \infty$ ( $L \to 0$ ) and $N \to 0$ ( $L \to \infty$ )	Ex 8
H	Real-world word problem	compute budget → predicted loss	Ex 9
I	Exam-style twist (spot the wrong claim)	encoder/fine-tune myth	Ex 10

The three "signs" that matter in this topic are:

the sign inside the mask ( $0$ = allowed to look, $- \infty$ = forbidden),
the sign of $Δ (lo g L)$ when you scale (negative when you grow, positive when you shrink),
the sign of the exponent in the power law (loss decreases as $N$ grows, so the useful form has $N$ in the denominator).

A & B — parameter counting (small and huge)

Worked example Example 1 — Attention params for a

tiny model (cell A, small- $d$ ) A toy GPT layer has model dimension $d = 4$ . How many parameters live in one causal self-attention block (ignore biases)?

Forecast: Guess first — is it closer to 16, 64, or 256?

Steps:

List the four weight matrices: $W_{Q}, W_{K}, W_{V}, W_{O}$ . Why this step? The parent formula $4 d^{2}$ counts these four; naming them stops you from forgetting $W_{O}$ (the output projection), the one people always drop.
Each matrix is $d \times d = 4 \times 4 = 16$ numbers. Why this step? A projection maps a $d$ -vector to a $d$ -vector, so it is a $d \times d$ grid.
Total $= 4 \times 16 = 64$ . Why this step? Four identical matrices, so multiply.

Verify: Plug into $4 d^{2} = 4 \times 4^{2} = 4 \times 16 = 64$ . ✓ Units: pure count (dimensionless). The middle guess (64) was right — the $d^{2}$ term makes even a tiny $d$ jump fast.

Worked example Example 2 — Full layer params, and does MLP dominate? (cells A + B, large-

$d$ ) For GPT-3, $d = 12288$ , and the MLP inner dimension is $4 d$ . Compute (a) attention params, (b) MLP params, (c) which dominates, per layer. We ignore all biases and LayerNorm scale/shift parameters (they add only $O (d)$ each, negligible against $O (d^{2})$ ).

Forecast: Will attention or the MLP hold more weights? Guess the ratio.

Steps:

Attention $= 4 d^{2} = 4 \times 1228 8^{2}$ . Why this step? Same $4 d^{2}$ rule as Ex 1, now at scale. $4 \times 1228 8^{2} = 603, 979, 776 \approx 6.04 \times 1 0^{8} .$
MLP has two matrices: $W_{1}$ of shape $(4 d) \times d$ and $W_{2}$ of shape $d \times (4 d)$ . Each is $4 d^{2}$ , total $8 d^{2}$ . Why this step? The MLP expands to $4 d$ then projects back — two rectangular matrices, each $4 \times d^{2}$ . $8 \times 1228 8^{2} = 1, 207, 959, 552 \approx 1.21 \times 1 0^{9} .$
Ratio MLP : Attention $= 8 d^{2} : 4 d^{2} = 2 : 1$ . Why this step? The $d^{2}$ cancels, leaving a clean constant — the MLP always carries twice the attention's weights, at any scale.

Verify: $8/4 = 2$ exactly, independent of $d$ . Full layer $= 12 d^{2} = 12 \times 1228 8^{2} \approx 1.81 \times 1 0^{9}$ . Multiply by GPT-3's 96 layers $\Rightarrow\approx 1.74 \times 1 0^{11}$ (the transformer-block parameters). The gap to the famous 175B is the token + position embedding matrices, which do NOT scale with layer count: the token embedding is $V \times d = 50257 \times 12288 \approx 6.2 \times 1 0^{8}$ , and the learned position embeddings are $T_{m a x} \times d = 2048 \times 12288 \approx 2.5 \times 1 0^{7}$ . Adding these to $1.74 \times 1 0^{11}$ recovers $\approx 1.75 \times 1 0^{11}$ . ✓

C — the causal mask at its boundaries

Worked example Example 3 — Mask at position 1 and at the last position (cell C, degenerate ends)

A sequence has $T = 4$ tokens. Write the mask entries $M_{ij}$ for (a) token $i = 1$ (the very first), (b) token $i = 4$ (the last).

Forecast: How many tokens can the first token look at? How many can the last look at?

Steps:

Rule from the parent note: $M_{ij} = 0$ if $j \leq i$ (allowed, look at the picture's blue cells), else $M_{ij} = - \infty$ (forbidden, red cells). Why this step? The mask is added before softmax; $- \infty$ becomes weight $0$ , so "forbidden" literally means "can't attend".
Row $i = 1$ : $M_{11} = 0$ ; $M_{12} = M_{13} = M_{14} = - \infty$ . So token 1 attends only to itself. Why this step? The first token has no past — it must predict token 2 from token 1 alone. This is the degenerate "sees only itself" end.
Row $i = 4$ : $M_{41} = M_{42} = M_{43} = M_{44} = 0$ ; none are $- \infty$ . Token 4 attends to all four. Why this step? The last token has the full history available — the other degenerate end.

Verify: Count allowed cells row by row: row 1 → 1 cell, row 2 → 2, row 3 → 3, row 4 → 4. Total $= 1 + 2 + 3 + 4 = 10$ . The formula for a $T \times T$ lower-triangular (including diagonal) count is $\frac{T ( T + 1 )}{2} = \frac{4 \cdot 5}{2} = 10$ . ✓ Look at the figure: the blue staircase has exactly 10 cells.

D — cross-entropy loss, from perfect to random

Worked example Example 4 — Loss when the model is perfect vs when it guesses uniformly (cell D, both limits)

Vocabulary size $V = 50257$ (GPT-2's). Compute the per-token loss $L = - lo g P_{θ} (x_{t} ∣ x_{< t})$ for: (a) a perfect model that assigns probability $1$ to the true token; (b) a random model that spreads probability uniformly, $P = 1/ V$ for every token. (Recall $lo g = ln$ , natural log.)

Forecast: One answer is $0$ . Guess the other — bigger or smaller than 10?

Steps:

Perfect model: $P = 1$ , so $L = - lo g 1 = 0$ . Why this step? This is the lower limit of the loss — you cannot do better than certainty about the truth. See Cross-entropy loss.
Random model: $P = 1/ V$ , so $L = - lo g (1/ V) = lo g V$ . Why this step? Uniform ignorance is the natural upper baseline; any trained model must beat it.
Compute $lo g V = ln 50257 \approx 10.82$ nats. Why this step? GPT uses natural log (nats); this number is the "loss of knowing nothing" and is exactly where training curves start.

Verify: $e^{10.82} \approx 50000 \approx V$ . ✓ So any real GPT-2 loss (typically $\approx 3$ nats) sits between $0$ and $10.82$ — sane. Units: nats (dimensionless log-probability).

E — the $d_{k}$ fix at two extremes

Worked example Example 5 — Variance of the dot product for

$d_{k} = 1$ vs $d_{k} = 64$ (cell E) Query and key entries are independent with mean $0$ , variance $1$ . The raw score is $q \cdot k = \sum_{m = 1}^{d_{k}} q_{m} k_{m}$ . Find its standard deviation for (a) $d_{k} = 1$ , (b) $d_{k} = 64$ , and state what $d_{k}$ -division does in each.

Forecast: For $d_{k} = 64$ , will the raw scores be roughly $\pm 1$ or roughly $\pm 8$ ?

Steps:

Each product $q_{m} k_{m}$ has mean $0$ and variance $1$ . Why this step? Let us actually derive it. By independence and zero mean, $E [q_{m} k_{m}] = E [q_{m}] E [k_{m}] = 0 \cdot 0 = 0$ . Then $Var (q_{m} k_{m}) = E [(q_{m} k_{m})^{2}] - (E [q_{m} k_{m}])^{2} = E [q_{m}^{2}] E [k_{m}^{2}] - 0$ . Since $E [q_{m}^{2}] = Var (q_{m}) + E [q_{m}]^{2} = 1 + 0 = 1$ (and likewise for $k_{m}$ ), we get $Var (q_{m} k_{m}) = 1 \cdot 1 = 1$ . This is the atom we sum. (See Self-attention and multi-head attention.)
Summing $d_{k}$ independent terms adds variances: $Var (q \cdot k) = d_{k}$ , so $std = d_{k}$ . Why this step? Variances (not standard deviations) add for independent sums — the reason the spread grows as $d_{k}$ , matching the parent's claim.
Case $d_{k} = 1$ : std $= 1 = 1$ . Dividing by $d_{k} = 1$ changes nothing — the degenerate case where the fix is invisible. Case $d_{k} = 64$ : std $= 64 = 8$ . Dividing by $8$ pulls scores back to std $1$ . Why this step? Shows the fix does nothing when small and everything when large — softmax of $\pm 8$ logits saturates (one weight $\approx 1$ , gradients die); $\pm 1$ logits stay soft.

Verify: After dividing by $d_{k}$ , new variance $= d_{k} / (d_{k})^{2} = d_{k} / d_{k} = 1$ for any $d_{k}$ . ✓ The orange curve in the figure (raw, wide) collapses onto the blue curve (scaled, unit-width) exactly.

F & G — scaling-law forecasts, both signs and both limits

Worked example Example 6 — Scaling

up: GPT-2 → GPT-3 (cell F, negative $Δ L$ ) Kaplan's law: $L (N) \approx (N_{c} / N)^{α_{N}}$ with $α_{N} = 0.076$ . Going $N_{1} = 1.5 B \to N_{2} = 175 B$ , find the change in log-loss $Δ (lo g L) = lo g L_{2} - lo g L_{1}$ . (All logs natural.)

Forecast: Will $Δ (lo g L)$ be positive or negative? By roughly how much?

Steps:

Take logs of the law: $lo g L = α_{N} (lo g N_{c} - lo g N) = const - α_{N} lo g N$ . Why this step? A power law is a straight line in log-log; taking $lo g$ turns multiplication into subtraction so the constant $N_{c}$ cancels in a difference.
Difference: $Δ (lo g L) = - α_{N} (lo g N_{2} - lo g N_{1}) = - α_{N} ln (N_{2} / N_{1})$ . Why this step? The $N_{c}$ term is identical in both, so it drops — we never need to know it.
Ratio $N_{2} / N_{1} = 175/1.5 \approx 116.7$ ; $ln (116.7) \approx 4.76$ ; $Δ = - 0.076 \times 4.76 \approx - 0.36$ nats. Why this step? The negative sign confirms scaling up lowers loss — matches the parent's worked Example 3.

Verify: Sign is negative (loss dropped) ✓, magnitude $\approx 0.36$ nats — bounded, not infinite, exactly the "capabilities improve but not without limit" message. ✓

Worked example Example 8 — The two limits of the power law (cell G,

$N \to \infty$ and $N \to 0$ ) Analyse $L (N) = (N_{c} / N)^{0.076}$ as (a) $N \to \infty$ and (b) $N \to 0^{+}$ .

Forecast: Does infinite scale give zero loss? Should it?

Steps:

As $N \to \infty$ : $N_{c} / N \to 0$ , and $0$ raised to a positive power $\to 0$ , so $L \to 0$ . Why this step? The clean power law predicts zero loss at infinite size — a known over-idealization: real data has irreducible entropy, so Chinchilla adds a floor term $L = E + (A / N)^{α}$ . Look at the figure: the naive curve (blue) dives to 0; the realistic curve (orange) flattens at a floor.
As $N \to 0^{+}$ : $N_{c} / N \to \infty$ , and $\infty^{0.076} = \infty$ , so $L \to \infty$ . Why this step? A model with (almost) no parameters is infinitely bad — the sanity boundary on the other end.

Verify: Both limits monotonic and correctly signed: $L$ strictly decreases as $N$ grows (derivative $\frac{d L}{d N} < 0$ ). Check the sign of the slope: $\frac{d}{d N} (N_{c} / N)^{0.076} = - 0.076 N_{c}^{0.076} N^{- 1.076} < 0$ for all $N > 0$ . ✓

H — a real-world word problem

Worked example Example 9 — "Given my compute budget, what loss can I expect?" (cell H)

A lab has compute for a model $N = 6 B$ params. A published $N_{0} = 1 B$ model reached loss $L_{0} = 2.20$ nats. Using $α_{N} = 0.076$ , predict the 6B model's loss — no retraining, just the law.

Forecast: Loss below or above $2.20$ ? By more or less than $0.2$ ?

Steps:

We don't know $N_{c}$ , but ratios kill it: $\frac{L}{L _{0}} = (\frac{N _{0}}{N})^{α_{N}}$ . Why this step? Anchoring to a known point $(N_{0}, L_{0})$ lets us predict a new point without any unknown constant — the practical use of the law.
$\frac{N _{0}}{N} = \frac{1}{6}$ ; raise to $0.076$ : $(1/6)^{0.076} = e^{0.076 l n (1/6)} = e^{0.076 \times (- 1.792)} = e^{- 0.1362} = 0.8727$ . Why this step? The exponent is small, so the multiplicative factor is close to (but below) 1 — big compute gains, modest loss gains, the recurring lesson.
$L = 2.20 \times 0.8727 = 1.92$ nats. Why this step? Convert the fractional factor into the actual predicted loss.

Verify: $1.92 < 2.20$ ✓ (bigger model, lower loss), and the drop $0.28$ nats is modest despite $6 \times$ compute — matches diminishing returns. Cross-check with the log-difference form of Ex 6: $Δ lo g L = - 0.076 ln 6 = - 0.136$ , and $e^{- 0.136} = 0.873$ . ✓ Same number.

I — the exam twist: spot the false claim

Worked example Example 10 — Which statement is wrong, and why? (cell I)

An exam gives four claims. Exactly one is false. Find it and correct it.

(i) GPT is decoder-only with causal masking.
(ii) GPT-3's headline result is few-shot in-context learning with frozen weights.
(iii) GPT-2 switched to pre-LN for training stability.
(iv) GPT-3 was fine-tuned separately on every downstream task.

Forecast: Which line smells wrong?

Steps:

Check (i) against the parent's BERT vs GPT (encoder vs decoder) mistake box: GPT keeps only the masked decoder, no encoder, no cross-attention. True. Why this step? Eliminate the classic encoder myth first.
Check (ii): GPT-3's star result is in-context/few-shot with frozen weights — you put examples in the prompt. True. Why this step? This is the emergent-at-scale property, not a bug.
Check (iii): GPT-2 indeed moved LayerNorm to the pre-activation position and added a final LayerNorm, which stabilised its 48-layer depth. True. Why this step? Confirms the one architectural tweak the parent note flagged as genuinely important.
Check (iv): this is the false one. GPT-1 fine-tuned per task, so the claim feels right by association — but GPT-3's whole point is that it needs no per-task fine-tuning. Correction: "GPT-3 requires no per-task fine-tuning; it performs the task from a few examples placed in the prompt, with all weights frozen." Why this step? The trap reuses GPT-1's true behaviour to lie about GPT-3 — exactly the parent note's second mistake box.

Verify: Exactly one false claim identified — (iv) — consistent with the parent note's "GPT-3 was fine-tuned on each task" mistake box. ✓

Recall Quick self-test

First token in a 4-length sequence attends to how many positions? ::: Exactly 1 (only itself). Ratio of MLP params to attention params per layer? ::: 2 : 1 ( $8 d^{2}$ vs $4 d^{2}$ ). Uniform-guess loss for vocab $V$ ? ::: $ln V$ nats ( $\approx 10.82$ for $V = 50257$ ). Sign of $Δ (lo g L)$ when you make the model bigger? ::: Negative (loss falls). What does dividing by $d_{k}$ do to the score variance? ::: Rescales it back to exactly 1 for any $d_{k}$ . In the scaling law, what does $N_{c}$ mean? ::: A fitted reference parameter-count (where extrapolated loss = 1 nat); it cancels in every ratio.