Self-supervised pretraining objectives: Exercises

Before we start, three symbols we lean on constantly:

Level 1 — Recognition

L1.1 For each description, name the objective (CLM, MLM, or Span Corruption): (a) hides ~15% of tokens with [MASK] and predicts them using both sides; (b) predicts the next token left-to-right; (c) blanks out contiguous chunks and generates them as a short output sequence.

L1.2 True or false: in Masked Language Modeling the loss is averaged over all positions in the sentence.

L1.3 Which single objective naturally powers text generation, and which is GPT's objective?

Recall Solution — L1

L1.1 (a) MLM (BERT-style, bidirectional). (b) CLM (causal / autoregressive, GPT-style). (c) Span Corruption (T5/BART-style denoising).

L1.2 False. MLM averages only over the masked set $M$ (the set of masked positions, defined above). Visible tokens are already given, so scoring them is trivial and adds no learning signal.

L1.3 Generation is powered by CLM, because you sample token-by-token left-to-right — exactly the shape of $p_{θ} (x_{t} ∣ x_{< t})$ . CLM is also GPT's objective.

Level 2 — Application

L2.1 A CLM scores the sentence the cat sat. The first token the has nothing before it to condition on, so there is no next-token prediction to score at position 1 — we only score a token once the model has seen something to its left. The model predicts $p (cat) = 0.5$ at position 2 (given the) and $p (sat) = 0.25$ at position 3 (given the cat). Compute the CLM loss $L_{CLM}$ and the perplexity.

L2.2 An MLM masks one token. The model gives probability $0.2$ to the true token at that masked spot. What is $L_{MLM}$ ?

L2.3 A CLM assigns each next token probability exactly $\frac{1}{50000}$ (uniform over a 50 000-token vocabulary). What is its perplexity? What does that number mean?

Recall Solution — L2

L2.1 The first token the is given (there is no left context, so nothing to predict there); we score only the two positions that have context. Average the per-token surprises $- lo g p$ over those two predicted positions: $L_{CLM} = - \frac{1}{2} (lo g 0.5 + lo g 0.25) = - \frac{1}{2} (- 0.6931 - 1.3863) = 1.0397.$ Perplexity $= e^{1.0397} \approx 2.828$ . Reading: the model is "as unsure as if it were guessing uniformly among $\approx 2.83$ equally likely words."

L2.2 Only one masked token, so the average is over one term: $L_{MLM} = - lo g (0.2) = 1.6094.$ Higher than L2.1's per-token surprise — the model was more surprised here.

L2.3 Uniform over $V = 50000$ means every $- lo g p = - lo g \frac{1}{50000} = lo g 50000$ . So $L = lo g 50000$ and $PPL = e^{l o g 50000} = 50000.$ Meaning: a model that has learned nothing is exactly as confused as picking uniformly among all 50 000 tokens — perplexity equals vocabulary size. Any useful model must score below this.

Level 3 — Analysis

L3.1 A CLM has cross-entropy loss $L = 1.6094$ . Show its perplexity, and explain in one sentence why $PPL = e^{L}$ is the right way to make the loss interpretable.

L3.2 Two models are evaluated on the same text. Model A has $PPL = 10$ ; Model B has $PPL = 40$ . By how many nats (natural-log units of loss) does A beat B? Which model would you deploy and why?

L3.3 Explain, using the chain-rule factorization $p (x_{1}, \dots, x_{T}) = \prod_{t} p (x_{t} ∣ x_{< t})$ , why removing the causal mask would let a CLM "cheat" and drive training loss to nearly zero without learning language.

Recall Solution — L3

L3.1 $PPL = e^{1.6094} = 5.0$ . It is interpretable because a raw loss like "1.61 nats" is abstract, but $e^{L}$ converts it into an effective branching factor: "the model is as confused as choosing uniformly among 5 words."

The figure below makes this concrete. The horizontal axis is the loss $L$ in nats; the vertical axis is perplexity $e^{L}$ . The purple curve is the mapping $PPL = e^{L}$ : notice it bends steeply upward, so equal steps in loss produce ever-larger jumps in perplexity. The three coloured dots are the worked numbers from this page — $(L = 1.04, PPL = 2.83)$ from L2.1, $(L = 1.61, PPL = 5.0)$ from this exercise, and $(L = 2.5, PPL = 12.2)$ from L5.2 — showing how a modest rise in loss (1.04 → 2.5) roughly quadruples perplexity. That is the visual reason we report perplexity: it turns an abstract nat count into a "number of equally-likely choices."

Figure — Self-supervised pretraining objectives

L3.2 Loss is $lo g (PPL)$ . So $L_{A} = lo g 10 = 2.3026$ , $L_{B} = lo g 40 = 3.6889$ . The gap is $L_{B} - L_{A} = lo g 40 - lo g 10 = lo g 4 = 1.3863$ nats. Deploy A: lower perplexity = lower average surprise = better predictions on held-out text.

L3.3 The chain rule is exact: to predict $x_{t}$ we are only allowed the past $x_{< t}$ . The causal mask enforces that by blocking attention to positions $\geq t$ . Without it, when scoring position $t$ the model can attend to $x_{t}$ itself (and beyond) — it simply reads the answer off the input and copies it. Loss collapses toward zero, but the model has learned to be a photocopier, not a language model. The learning signal requires that the target information be hidden.

Level 4 — Synthesis

L4.1 (80/10/10 accounting) BERT chooses 15% of tokens for the objective. Of a 200-token sequence, how many tokens are chosen? Of those, how many become [MASK], how many become a random token, and how many stay unchanged (80/10/10 split)?

L4.2 (why the split) In one crisp paragraph, tie the 10%-random and 10%-unchanged cases to the fine-tuning stage. What concrete failure does the split prevent?

L4.3 (span vs single token) A sentence has the phrase New York City (3 tokens). Compare, conceptually, what a single-token MLM mask versus a T5 span mask forces the model to do. Which demands more "genuine" reasoning and why?

Recall Solution — L4

L4.1 Chosen $= 0.15 \times 200 = 30$ tokens.

[MASK]: $0.80 \times 30 = 24$ tokens.
random token: $0.10 \times 30 = 3$ tokens.
unchanged: $0.10 \times 30 = 3$ tokens. (Check: $24 + 3 + 3 = 30$ . ✓)

L4.2 At fine-tuning time there is no [MASK] symbol in the input — real downstream text never contains it. A model trained only on [MASK] learns "produce a good vector only when I see the mask token", creating a train/test mismatch. Forcing 10% random and 10% unchanged means the model must build a useful representation of every position (because it can never be sure whether a token is original, corrupted, or masked), so its representations transfer cleanly to mask-free downstream tasks.

L4.3 A single-token mask on, say, York in New [MASK] City is often solvable from local n-gram statistics — New ___ City almost forces York. A T5 span mask replaces the whole New York City with one sentinel and asks the decoder to generate all three tokens in order. That removes the local crutch and forces genuine multi-token, coherent reasoning about what phrase fits — strictly harder, and closer to real generation.

Level 5 — Mastery

L5.1 (design a compression argument) You claim "predicting the next word forces the model to learn facts." Construct the tightest one-line argument connecting next-token prediction to world knowledge, and name the quantity that measures success.

L5.2 (perplexity of a perfect vs random model) Vocabulary $V = 32000$ . (a) What is the perplexity of a model that predicts the true next token with probability 1 every time? (b) What is the perplexity of a uniform-random model? (c) A real model reaches loss $L = 2.5$ ; between which two extremes does its perplexity fall, and what is its numeric value?

L5.3 (critique a proposal) A student proposes: "Mask 100% of tokens so the task is maximally hard, and the model will learn the most." Refute this rigorously in terms of what the loss $- lo g p_{θ} (x_{i} ∣ \tilde{x})$ can condition on.

Recall Solution — L5

L5.1 To assign high probability to the correct next token you must implicitly model everything that constrains it — grammar, coreference, arithmetic, and facts (e.g. finishing "The capital of France is ___" requires knowing it is Paris). Compressing text well = predicting it well, and the measurable score of that compression is perplexity (equivalently cross-entropy loss): lower perplexity ⇔ more structure captured in the weights $θ$ .

L5.2 (a) Perfect model: $p = 1$ each step, $- lo g 1 = 0$ , so $L = 0$ and $PPL = e^{0} = 1$ . Perplexity 1 = "never confused, always one choice." (b) Uniform-random: $PPL = V = 32000$ (from the argument in L2.3). (c) A trained model must satisfy $1 \leq PPL \leq 32000$ . With $L = 2.5$ , $PPL = e^{2.5} \approx 12.182$ — far below the random ceiling, so it has learned a great deal, but above the perfect floor of 1.

L5.3 The loss term $- lo g p_{θ} (x_{i} ∣ \tilde{x})$ conditions on the corrupted input $\tilde{x}$ — i.e. on the tokens that remain visible. If 100% of tokens are masked, $\tilde{x}$ contains no visible tokens: there is nothing to condition on, so the "context" is empty for every prediction. Each masked token must then be guessed from a prior with zero evidence, which is the uniform-random baseline — perplexity $\approx V$ , and no gradient signal that ties an output to any meaningful context. Difficulty is not the goal; learnable difficulty is. ~15% keeps enough visible context to make each guess informative while still non-trivial.

Recall

Recall One-screen summary

Loss vs perplexity ::: $L = - \frac{1}{N} \sum_{k = 1}^{N} lo g p_{k}$ where $N$ is the number of scored predictions and $p_{k}$ is the probability the model gave the correct token at prediction $k$ (average surprise, nats); $PPL = e^{L}$ (effective branching factor). Perplexity of a random model over vocab $V$ ::: exactly $V$ . Perplexity of a perfect model ::: $1$ . Which positions enter the MLM loss ::: only the masked set $M$ (the set of masked positions). BERT split on the chosen 15% ::: 80% [MASK], 10% random, 10% unchanged. Why not 100% masking ::: no visible context to condition on ⇒ unlearnable, PPL collapses to $V$ .