Foundations — GPT family architecture evolution

Before you can read the parent note you must be able to see every symbol it uses. This page builds each one from nothing, in an order where each idea rests on the one before it. Nothing is assumed. If the parent note wrote a symbol without explaining it, it gets explained here.

This is the foundation layer for GPT family architecture evolution.

1. A token — the atom of everything

Picture: the sentence "cats sleep a lot" becomes a row of boxes: [cats] [sleep] [a] [lot]. Each box is one token.

Why the topic needs it: the whole "predict the next word" game is really "predict the next token". Everything downstream — probabilities, attention — operates on this list of atoms.

We name the tokens with symbols: $x_1,x_2,x_3,\dots$

So in "cats sleep a lot": $T=4$, $x_2=$ sleep, and $x_{<3}=\{$cats, sleep$\}$.

2. A vector — turning a word into arrow-of-numbers

A computer cannot multiply the word cats. So each token is turned into a list of numbers called a vector.

Picture: in 2D, the vector $[3,2]$ is an arrow going 3 right and 2 up. In GPT the arrows live in thousands of dimensions — impossible to draw, but the idea is identical.

Why the topic needs it: attention, MLPs, everything — is matrix arithmetic on $X$. You cannot read one line of the parent's formulas without knowing that $X$ is "the stack of token arrows".

3. The dot product — measuring "how aligned are two arrows?"

The parent note says "attention score between token $i$ and $j$ is the dot product $q_i\cdot k_j$". Here is what that dot means.

Why this tool and not another? We want to ask "how similar are these two token-arrows?" The dot product answers exactly that: it is large and positive when the arrows point the same way, zero when they are perpendicular, negative when they point opposite ways. That is why attention uses it as a similarity score — a big score means "these two tokens are relevant to each other".

4. Probability, the product, and why we take a log

The parent's central formula is $P(x_1,\dots ,x_T)={\prod }_{t=1}^{T}P(x_t\mid x_{<t})$. Let's decode every piece.

Why multiply the probabilities? Because the chance of a whole sentence happening is the chance of the first word, times the chance of the second given the first, and so on — independent-ish choices chain by multiplication.

Why this tool? Multiplying thousands of tiny probabilities (like $0.001\times 0.002\times \dots$) gives a number so small the computer rounds it to zero — this is called underflow. Taking $\log$ turns that dangerous product into a safe sum, and (bonus) sums are far friendlier to the gradient-based learning GPT uses.

5. Softmax — turning scores into a probability list

The parent writes $\text{softmax}(\dots )$. What does it do?

Why the exponential $e^s$? Two reasons, both visual: (1) $e^s$ is always positive, so no probability comes out negative; (2) it exaggerates gaps — a slightly bigger score becomes a much bigger share, so softmax gently picks a "winner" while still giving everyone a nonzero slice.

Why the topic needs it: attention must decide how much weight to give each earlier token. Softmax converts the raw dot-product similarity scores into clean weights that add to 100%, so they behave like "attention shared out among tokens".

6. The causal mask — the "no peeking" rule

Why $-\infty$? Look at softmax: $e^{-\infty }=0$. So adding $-\infty$ to a score makes that token's weight exactly zero after softmax. It's a switch that turns off attention to the future.

Picture: a square grid where the upper-right triangle (future positions) is greyed out. Token 3 can look at 1, 2, 3 but the boxes for 4, 5, ... are blacked out.

Why the topic needs it: without this rule, when the model tries to predict token 4 it could just look at token 4 — cheating. The mask forces genuine prediction, which is the whole point of the game. (This is the difference between GPT and BERT.)

7. Variance and the $\sqrt{d_k}$ scaling

The parent divides scores by $\sqrt{d_k}$. One more foundation makes this click.

The key fact: a dot product of two random length-$d_k$ vectors has variance that grows with $d_k$. So in big models the raw scores get huge and scattered, which shoves softmax into a corner where one weight is ~1 and the rest ~0 — and there the gradient (learning signal) dies. Dividing by $\sqrt{d_k}$ shrinks the spread back to a sane size. ($d_k$ = the length of each key vector.)

Prerequisite map

Read it top-down: tokens become vectors, vectors combine via dot products into attention scores, softmax (with the mask) turns those into probabilities, and the log-loss trains the whole thing. That bottom node is the parent topic.

Equipment checklist

Self-test — can you answer each before revealing?

Once every reveal feels obvious, return to GPT family architecture evolution and the formulas will read like plain English.