GPT family architecture evolution: Visual walkthrough

We are building toward this one line — do not worry that it looks scary yet, we will earn every piece:

$L (θ) = - \frac{1}{T} \sum_{t = 1}^{T} lo g P_{θ} (x_{t} ∣ x_{< t})$

Step 1 — A sentence is just a list of guesses

WHAT. Take a sentence. Chop it into little pieces called tokens (roughly words or word-fragments). Number them left to right: the first piece is $x_{1}$ , the second is $x_{2}$ , and so on up to the last piece $x_{T}$ .

The letter $x$ means "a token" (a chunk of text).
The little number underneath — the subscript — is its position in the line. So $x_{3}$ means "the token sitting in slot 3".
$T$ is just how many tokens there are in total (the length of the sentence).

WHY. Before we can talk about "predicting the next word", we need a way to say which word and what comes before it. Numbering the slots gives us that language. Position matters: "dog bites man" and "man bites dog" are the same words in different slots.

PICTURE. Each slot is a box. A moving reader-arrow sits between slots — everything to its left is "seen", everything to its right is "hidden". The whole game is: given the boxes on the left, guess the next box.

Figure — GPT family architecture evolution

Step 2 — "Guess the next word" is a probability

WHAT. When you cover the rest of a sentence and guess the next word, you are never 100% sure. You have a hunch — "probably 'the', maybe 'a', unlikely 'zebra'". A number between 0 and 1 that measures a hunch is a probability. We write it $P (something)$ .

The object GPT cares about is:

$how likely P (the next token x_{t} given ∣ everything before it x_{< t})$

The vertical bar $∣$ reads "given" or "assuming we already know". It is a conditional probability: the chance of one thing once you already know another.
So $P (x_{t} ∣ x_{< t})$ = "how likely is the token in slot $t$ , given all the tokens before it".

WHY. Language is not deterministic — after "I poured myself a cup of ___" many words fit. A probability, not a single answer, is the honest description. GPT's whole job is to output a good probability for every possible next token.

PICTURE. A bar chart over the vocabulary. Tall bar on the likely word, tiny bars on the rest. That whole chart of bars is one $P (x_{t} ∣ x_{< t})$ .

Step 3 — Chaining guesses gives the whole sentence (the chain rule)

WHAT. How likely is the entire sentence $x_{1}, \dots, x_{T}$ ? We build it up guess by guess. Start with the plainest fact about probability, for just two tokens:

$P (x_{1}, x_{2}) = chance of the 1st P (x_{1}) \times chance of the 2nd, given the 1st P (x_{2} ∣ x_{1})$

WHY THIS TOOL. Why multiply? Because a joint event "A and B" happens only if A happens and then B happens given A. This "and then" is exactly multiplication of probabilities. To see it is not a guess but a definition, recall what a conditional probability formally means:

We use multiplication, not addition, because we want both tokens to be right simultaneously.

Now apply the same trick again to add $x_{3}$ , then $x_{4}$ , all the way to $x_{T}$ . Each new token is conditioned on all the ones before it. The pattern "telescopes":

$P (x_{1}, \dots, x_{T}) = \prod_{t = 1}^{T} P (x_{t} ∣ x_{< t})$

The big symbol $\prod$ (capital Greek "pi") means multiply everything together — the multiplication version of a sum $\sum$ .
The " $t = 1$ " underneath and " $T$ " on top say "let $t$ run from 1 up to $T$ , multiply all those factors".

This is exact — no approximation, no assumption. It is always true for any sequence.

PICTURE. A ladder. Each rung is one factor $P (x_{t} ∣ x_{< t})$ . Climbing the ladder and multiplying rung values reconstructs the probability of the full sentence.

Step 4 — Turn the product into a sum with a logarithm

WHAT. Multiplying many probabilities is dangerous: each is below 1, so their product shrinks toward zero fast. A 1000-token sentence gives numbers like $1 0^{- 800}$ — a computer rounds that to plain 0 and all information is lost. Fix: take the logarithm (base $e$ , as fixed at the top).

$lo g (\prod_{t = 1}^{T} P (x_{t} ∣ x_{< t})) = \sum_{t = 1}^{T} lo g P (x_{t} ∣ x_{< t})$

$lo g$ is a function whose one magic property is: the log of a product is the sum of the logs. It converts $\times$ into $+$ .
$\sum$ (capital Greek "sigma") means add everything up.

WHY THIS TOOL. Two reasons, both crucial. (1) Numerically: adding a thousand medium-sized negative numbers never underflows, whereas multiplying a thousand tiny ones does. (2) For learning: the training machine adjusts $θ$ by nudging in the direction that improves the score. Sums have simple, well-behaved slopes (gradients); products have slopes that explode or vanish. The log tames them.

PICTURE. Left: probabilities as shrinking areas that collapse to a dot. Right: their logs as tidy bars that stack (add) to a manageable height.

Step 5 — Flip the sign: from "reward" to "loss" (this is cross-entropy)

WHAT. A good model gives high probability to the true next token, so $lo g P$ is close to $0$ (log of something near 1). A bad model gives tiny probability, so $lo g P$ is a big negative number. Machines are built to minimize a cost, not maximize a reward, so we flip the sign and average:

$L (θ) = - \frac{1}{T} \sum_{t = 1}^{T} lo g P_{θ} (x_{t} ∣ x_{< t})$

The minus sign $-$ turns "big negative = bad" into "big positive = bad", so the machine can push it down.
$\frac{1}{T}$ averages over the $T$ tokens, so a long sentence and a short one are compared fairly (per-token loss).
$L (θ)$ is the loss: one number saying "how wrong is the model right now".

Definition Why this loss is called cross-entropy

The true next token is a fact — a distribution that puts probability $1$ on the word that actually appeared and $0$ on all others. The model outputs its own guessed distribution $P_{θ}$ . Cross-entropy measures how many extra nats it costs to describe the true outcome using the model's guesses instead of the truth (nats because our $lo g$ is base $e$ ; had we used $lo g_{2}$ it would count bits): $H (data, P_{θ}) = - \sum_{tokens v} 1 on the real token, else 0 P_{true} (v) lo g P_{θ} (v) .$ Because $P_{true}$ is $1$ on exactly the real token and $0$ elsewhere, every term drops out except the real one, leaving just $- lo g P_{θ} (x_{t} ∣ x_{< t})$ for each position. Averaging over positions gives our $L (θ)$ — so $L$ is literally the cross-entropy between the data and the model, which equals the average negative log-likelihood. This is exactly the Cross-entropy loss.

WHY. This is what "learning" optimizes. Every gradient step lowers $L$ , which means raising the probability of the actual next tokens in the training text — i.e. getting better at the guessing game.

PICTURE. A hilly landscape whose height is $L$ . A ball (the current $θ$ ) rolls downhill; low valleys = a model that predicts real text well.

Step 6 — The rule that makes it honest: the causal mask

WHAT — where do the scores come from? Inside the network, attention lets each token look at other tokens to decide the next one. Here is the minimal sketch of how one score $s_{ij}$ (how much token $i$ should look at token $j$ ) is born — enough for this derivation:

Every token starts as a list of numbers, its embedding (a vector) — think of it as an arrow in space describing that word's meaning.
From each token's arrow the network makes two smaller arrows: a query $q_{i}$ ("what am I looking for?") for the looking token $i$ , and a key $k_{j}$ ("what do I offer?") for the looked-at token $j$ .
The score is how well those two arrows agree, measured by the dot product — multiply matching entries and add them up: $s_{ij} = q_{i} \cdot k_{j}$ . Big when the arrows point the same way (a strong match), small or negative when they don't.

So a raw score $s_{ij}$ is just "does what token $i$ wants match what token $j$ offers?" (The full multi-head machinery lives in Self-attention and multi-head attention; we only need this one line.)

Turning scores into looking-weights. Attention converts the list of scores $s_{i 1}, \dots, s_{i T}$ into positive weights that sum to $1$ with a step called softmax:

$softmax (s_{ij}) = \frac{e ^{s_{ij}}}{\sum _{k} e ^{s_{ik}}}$

$e$ here is again Euler's number $\approx 2.718$ (the same constant that set our log base). Writing $e^{s}$ — " $e$ raised to the power $s$ ", the exponential — turns any score, even a negative one, into a positive number, because $e$ to any power is always above zero.
Dividing by the total $\sum_{k} e^{s_{ik}}$ (the exponential of every score in row $i$ added up) forces the weights to add to $1$ — so they behave like a probability of "how much to look at each token".

WHY exponential and not just the raw scores? Because we need weights that are (a) never negative and (b) sum to 1, and (c) let a clearly-winning score dominate smoothly. The exponential $e^{s}$ delivers all three: always positive, easily normalized, and it amplifies gaps between scores.

The mask. Now the causal mask forbids looking rightward before softmax is applied, by wrecking the score:

$M_{ij} = {0 - \infty j \leq i (slot j is at or before i : allowed) j > i (slot j is in the future: blocked)$

$i$ is "the token doing the looking", $j$ is "the token being looked at".
We add $M_{ij}$ to the score before softmax. Adding $- \infty$ to a future score, then taking $e^{- \infty} = 0$ , makes that looking-weight exactly $0$ : a hard "you may not see this".

WHY THIS SHAPE. The forbidden region is everything strictly to the right — a strictly upper-triangular block of $- \infty$ . This is the single mechanism that makes GPT autoregressive (predicting forward from the past only). Remove it and the objective in Step 5 becomes meaningless.

PICTURE. The $T \times T$ attention grid. Lower-left triangle (past & present) glows — allowed. Upper-right triangle (future) is blacked out — blocked. Read row $i$ to see exactly what token $i$ is permitted to attend to.

Step 7 — Every case, checked

WHAT. Let's make sure nothing breaks at the edges.

First token ( $t = 1$ ). There is nothing before it — $x_{< 1}$ is empty. Then $P_{θ} (x_{1} ∣ x_{< 1})$ is just the model's guess of how any text tends to start. The formula still works; the "condition" is simply blank. In the mask, row 1 can only attend to itself.
Last token ( $t = T$ ). It sees everything before it (the whole rest of the line) but nothing after — because there is nothing after. Row $T$ of the mask is fully lit except it never looks beyond $T$ .
A perfect model. If $P_{θ}$ hits $1$ on every true token, each $lo g 1 = 0$ , so $L = 0$ . This is the floor — you cannot do better. In practice language is not fully predictable, so real loss sits above 0 (that unbeatable residual is the language's intrinsic uncertainty, its entropy).
A model that assigns $0$ to a true token. Then $lo g 0 = - \infty$ , so $L = + \infty$ : infinitely punished for being certain and wrong. This is why models never output exactly $0$ — softmax keeps every probability strictly positive.

WHY. A derivation you can trust must survive its own boundaries. First token, last token, best case, worst case — the same objective handles them all with no special-casing.

PICTURE. The loss curve $- lo g p$ as $p$ slides from $0$ to $1$ : it dives from $+ \infty$ down to $0$ . Confidence in the right answer costs nothing; confidence in the wrong answer costs everything.

The one-picture summary

Everything above, in one frame: a sentence → per-slot conditional guesses (chain rule, ↑ Step 3) → masked so no slot peeks forward (↑ Step 6) → logged into a sum (↑ Step 4) → sign-flipped and averaged into the loss $L (θ)$ (↑ Step 5) → rolled downhill by training.

Recall Feynman retelling — the walkthrough in plain words

Picture reading a sentence with a card covering everything to the right of your finger. At each step you guess the next hidden word — you never know, you have a hunch, so we score the guess as a probability. To score the whole sentence we don't invent anything new; we just multiply all those step-by-step hunches together (that's the chain rule — "this AND then this AND then this"). Multiplying thousands of tiny numbers would shrink to zero on a computer, so we take logarithms (base $e$ , the number $2.718$ ), which magically turn the multiply-pile into an add-pile that's easy to work with and easy to improve. Then we flip the sign so that "bad guessing" becomes a big positive cost the machine can push downhill — that cost has a name, cross-entropy, because it measures how surprised the model is by the real text. Inside, each token makes a "what I want" arrow and a "what I offer" arrow, and how well two arrows agree (their dot product) becomes a score; the exponential $e$ -to-that-score turns scores into positive looking-weights that add to one. The one rule that keeps the whole thing honest is no peeking to the right — the causal mask — because during practice the real next word is literally printed in front of the model, and without the blindfold it would just copy instead of learn. Train by rolling that cost downhill, and a plain "guess the next word" game turns into GPT.

Recall Quick self-check

Why do we use $\prod$ (multiply) and not $\sum$ (add) for the sentence probability? ::: Because we need all tokens to be right simultaneously — "A and B" is a product of probabilities, not a sum. What formal fact justifies $P (x_{1}, x_{2}) = P (x_{1}) P (x_{2} ∣ x_{1})$ ? ::: The definition of conditional probability $P (A ∣ B) = P (A, B) / P (B)$ , rearranged. What does the $- \infty$ in the mask become after softmax, and why? ::: It becomes weight $0$ — softmax computes $e^{- \infty} = 0$ , hard-blocking any look into the future. Which log base does this page use, and what unit does that make the loss? ::: Base $e$ (natural log), so the loss is measured in nats; base $2$ would give bits. How is a single attention score $s_{ij}$ computed? ::: As the dot product of token $i$ 's query arrow with token $j$ 's key arrow, $s_{ij} = q_{i} \cdot k_{j}$ — big when they agree. Why is $L$ called cross-entropy? ::: Because it is the cross-entropy between the true (one-hot) next-token distribution and the model's distribution, which reduces to the average negative log-likelihood. What is the smallest possible value of $L (θ)$ and when? ::: $0$ , when the model assigns probability $1$ to every true next token (a perfect, impossible-in-practice model). Why take the log at all? ::: To stop tiny probability products from underflowing to 0, and to turn products into sums with well-behaved gradients.