Self-supervised pretraining objectives: Visual walkthrough

We will build the answer to one question: "How do we score a model that guesses the next word?"

Step 1 — What is a token, and what does "predict the next one" mean?

WHAT. Before any maths, we need the atoms. A machine does not read letters or words directly — it chops text into pieces called tokens (see Tokenization (BPE)). Think of tokens as beads on a string.

WHY. Every symbol we introduce later ( $x_{t}$ , $T$ , $lo g p$ ) refers to these beads. If we don't fix what a bead is, nothing downstream means anything.

PICTURE. Below, the sentence "the cat sat" becomes three beads $x_{1}, x_{2}, x_{3}$ . The little arrow says: given the beads so far, guess the next bead.

Figure — Self-supervised pretraining objectives

Step 2 — A guess is not a single word, it is a whole distribution

WHAT. When the model looks at the and tries to guess the next bead, it does not blurt out one word. It hands back a probability for every possible token in the vocabulary — a bar chart that sums to 1.

WHY. Language is uncertain: after the we could get cat, dog, sky… A single guess throws away that uncertainty; a distribution keeps it, and lets us measure how confident the model was in the word that actually came. That "how confident" is the seed of our loss.

PICTURE. The bar chart: the tall blue bar is the token that really appeared, the grey bars are the alternatives the model also considered. This bar chart is produced by the Transformer Architecture's final softmax.

Step 3 — The whole sentence's probability is a product (chain rule)

WHAT. We want the probability the model assigns to the entire true sentence, not just one bead. The rule of probability that lets us break a joint probability into a chain of one-step guesses is the chain rule: $p_{θ} (x_{1}, \dots, x_{T}) = \prod_{t = 1}^{T} p_{θ} (x_{t} ∣ x_{< t})$

WHY this tool and not another? We could imagine trying to model $p (x_{1}, \dots, x_{T})$ in one shot — but the number of possible sentences is astronomically large. The chain rule is exact (no approximation) and turns one impossible joint into $T$ easy next-bead guesses, which is exactly what Step 2 already gives us. That is why it is the natural tool here.

PICTURE. A relay of arrows: each token's bar-height is one link; multiply the links to get the sentence probability.

Step 4 — Multiplying tiny numbers is dangerous: take the log

WHAT. Each bar-height is between 0 and 1. Multiplying hundreds of them gives a number so small the computer rounds it to zero (underflow). The fix: take the logarithm, which turns a product into a sum: $lo g (\prod_{t} p_{θ} (x_{t} ∣ x_{< t})) = \sum_{t} lo g p_{θ} (x_{t} ∣ x_{< t})$

WHY the log and not, say, a square root? Two reasons, both essential. (1) Only the log has the magic identity $lo g (ab) = lo g a + lo g b$ , converting the fragile product into a stable sum. (2) $lo g$ is monotonic — it never changes which model is best — so maximizing the product and maximizing its log give the identical answer $θ$ .

PICTURE. The graph of $lo g$ on $(0, 1]$ : it is negative everywhere there (since probabilities are $\leq 1$ ), diving to $- \infty$ as the probability approaches 0. So a confident-but-wrong model (tiny $p$ on the true word) gets punished hard.

Step 5 — Flip the sign so "better = smaller", and average

WHAT. In Step 4 good models have log-prob near 0 and bad ones near $- \infty$ . Training machinery likes to minimize a number that bottoms out at 0. So multiply by $- 1$ (flip up) and divide by $T$ (average per token): $L_{CLM} = - \frac{1}{T} \sum_{t = 1}^{T} lo g p_{θ} (x_{t} ∣ x_{< t})$

WHY negate? So that a perfect model ( $p = 1$ everywhere, $lo g = 0$ ) gives $L = 0$ , and worse models give larger positive numbers — a natural thing to push downward. WHY average (÷T)? So a 10-word sentence and a 1000-word sentence are on the same scale; the loss becomes "average surprise per token," comparable across lengths.

PICTURE. The transformation in three panels: raw log-prob (negative) → negated (positive spikes) → averaged (one clean height). This final height is the cross-entropy (see Cross-entropy Loss).

Step 6 — The causal mask: why the model isn't allowed to peek

WHAT. For the guess at position $t$ to be honest, the model may only look at $x_{< t}$ — never at $x_{t}$ or beyond. Inside the Transformer Architecture this is enforced by a causal mask: a triangular wall that blocks attention to future positions.

WHY. Our loss in Step 5 assumes each factor is a genuine guess of an unseen bead. If position $t$ could see $x_{t}$ , it would just copy it, drive $p \to 1$ , $L \to 0$ — and learn nothing. The mask is what makes the "prediction" real.

PICTURE. The attention grid: green cells (allowed, on or below the diagonal) let information flow; red cells (future) are blocked. Notice the clean staircase — position 3 sees 1,2,3; position 1 sees only itself.

Step 7 — Edge & degenerate cases (so nothing surprises you)

WHAT / WHY / PICTURE — three corners of the derivation that people trip on.

The very first token $x_{1}$ . Its context $x_{< 1}$ is empty. Is $lo g p_{θ} (x_{1} ∣ \emptyset)$ valid? Yes — the model conditions on a special start marker (or the empty prefix), so the first factor is a plain "what word begins a sentence?" guess. Nothing breaks; it is just one more term.
The perfect model. If every true bead gets $p = 1$ , then every $lo g 1 = 0$ , so $L = 0$ and perplexity $e^{0} = 1$ — "certain of exactly one word." This is the theoretical floor.
The clueless model. With a vocabulary of $V$ words and uniform guessing, $p = 1/ V$ , so $L = - lo g (1/ V) = lo g V$ and $PPL = V$ — "as lost as choosing among all $V$ words." Every real model lives between these two extremes (see Perplexity).

The one-picture summary

Everything above, compressed: beads → bar charts → multiply (chain rule) → log (sum) → negate & average → cross-entropy loss, with the causal mask guarding honesty. This same skeleton powers MLM and span-corruption too — they only change which beads are hidden (contrast in GPT vs BERT), and the trained weights are then handed to Fine-Tuning LLMs.

Recall Feynman retelling — the whole walk in plain words

We cut the sentence into beads. For each bead, the model — instead of shouting one guess — hands us a bar chart of "how likely is each possible next bead." We peek at the bar sitting over the bead that really came next: tall bar = good, short bar = bad. To score the whole sentence we'd multiply all those bar-heights, but tiny numbers multiplied get lost, so we take logarithms and add instead. Logs of small numbers are big-negative, so we flip the sign (now small = good) and divide by the number of beads so long and short sentences compare fairly. That single number is the loss — the model's average surprise. Finally we build a wall so the model can't peek at the answer while guessing. Push that number down, and to do it the model is forced to actually understand language.