Self-supervised pretraining objectives: Worked examples

This page is a drill sheet. The parent note built the machinery; here we run that machinery through every kind of input it will ever meet — clean cases, edge cases, degenerate cases, a word problem, and an exam trap. Nothing new is assumed: every symbol we use is re-explained the first time it appears.

Before starting, three symbols you MUST have in hand (all from the parent):

The scenario matrix

Every worked example below is tagged with the cell of this matrix it covers. The goal: leave no cell empty.

#	Case class	What makes it tricky	Example
A	Clean CLM, multi-token	ordinary averaging	Ex 1
B	Perfect prediction ( $p \to 1$ )	limiting value, $lo g 1 = 0$	Ex 2
C	Worst prediction ( $p \to 0$ )	degenerate, loss $\to \infty$	Ex 2
D	Uniform / no-knowledge model	$p = 1/ V$ , sanity floor	Ex 3
E	MLM, count only masked	which positions enter the sum	Ex 4
F	MLM 80/10/10 corruption	random & unchanged sub-cases	Ex 5
G	Span corruption (T5)	multi-token target, one output seq	Ex 6
H	Word problem (real training run)	tokens ≠ words; Tokenization (BPE)	Ex 7
I	Exam twist: change log base / compare CLM vs MLM PPL	base conversion, fair comparison	Ex 8

Example 1 — Clean CLM, multi-token (cell A)

Worked example Loss on "I love NLP"

Tokens: I, love, NLP. Under GPT-style causal LM we score positions $2$ and $3$ (position $1$ has no left context to condition on). The model gives: $p (love ∣ I) = 0.4, p (NLP ∣ I love) = 0.1.$ Find the loss and the perplexity.

Forecast: will PPL be near $2$ , near $3$ , or bigger? (One prob is small — expect PPL a bit above $3$ .)

Take the negative log of each correct-token probability. $- lo g 0.4 = 0.916, - lo g 0.1 = 2.303.$ Why this step? $- lo g p$ is the surprise of the true token. A small $p$ (0.1) gives a big surprise (2.303).
Average over the $T = 2$ scored positions. $L = \frac{1}{2} (0.916 + 2.303) = 1.609.$ Why this step? CLM loss divides by the number of predicted tokens so sentences of different length are comparable.
Exponentiate to get perplexity. $PPL = e^{1.609} = 5.0.$ Why this step? $e^{L}$ converts "average surprise" into "effective number of choices."

Verify: $PPL = e^{1.609} = 5.0$ . Sanity: the two token-level perplexities are $1/0.4 = 2.5$ and $1/0.1 = 10$ ; their geometric mean is $2.5 \times 10 = 5$ . ✓ Perplexity is exactly the geometric mean of $1/ p$ — matches.

Example 2 — The two limits: perfect and worst prediction (cells B, C)

Worked example What happens at the extremes?

Same setup, one scored token. Compute the loss when (a) $p = 1$ (model is certain and right), (b) $p = 0.001$ (model is nearly sure of the wrong thing), (c) the true limit $p \to 0$ .

Forecast: which of these blows up to infinity?

Perfect case $p = 1$ . $L = - lo g 1 = 0$ , so $PPL = e^{0} = 1$ . Why this step? Zero surprise means "as unsure as picking among 1 word" — i.e. no uncertainty at all. This is the floor of the loss.
Bad case $p = 0.001$ . $L = - lo g 0.001 = 6.908$ , $PPL = e^{6.908} = 1000$ . Why this step? Confidently-wrong is punished hard: $1000$ effective choices.
Degenerate limit $p \to 0^{+}$ . $- lo g p \to + \infty$ . Why this step? This is why training frameworks clip probabilities away from exact $0$ — an exact- $0$ prob on the true token would make the loss infinite and gradients explode.

Verify: $- lo g 1 = 0$ ✓; $- lo g (0.001) = 6.9078$ and $e^{6.9078} = 1000$ ✓. The limit $lim_{p \to 0^{+}} (- lo g p) = + \infty$ ✓.

Figure — Self-supervised pretraining objectives

The red curve above is the per-token loss $- lo g p$ . Read off the two dots: at $p = 1$ it touches zero; as $p$ slides left toward $0$ it shoots up without bound. That single picture explains cells B and C at once.

Example 3 — The no-knowledge baseline (cell D)

Worked example What loss does an untrained model get?

A fresh model with vocabulary size $V = 50000$ hasn't learned anything, so it spreads probability uniformly: every token gets $p = 1/ V$ . What is its loss and perplexity?

Forecast: guess the perplexity before computing — is it near $V$ ?

Plug the uniform prob into the loss. $L = - lo g \frac{1}{V} = lo g V = lo g 50000 = 10.82.$ Why this step? Under uniform guessing every token is equally surprising, so the average surprise is just $lo g V$ .
Perplexity. $PPL = e^{l o g V} = V = 50000.$ Why this step? The exponential and the log cancel — the model is literally "as unsure as picking among all 50000 words," which is what "knows nothing" should mean.

Verify: $lo g 50000 = 10.819$ and $e^{10.819} = 50000$ ✓. This is the ceiling every real training run must beat: if your model's PPL $\geq V$ something is broken.

Example 4 — MLM: only masked positions count (cell E)

Worked example Which terms enter the sum?

BERT-style MLM on the cat sat on the mat (6 tokens). We mask positions ${2, 5}$ → the [MASK] sat on [MASK] mat. Targets: cat, the. The model outputs a distribution at all 6 positions, giving on the true tokens: $p_{1} = 0.9, p_{2} (cat) = 0.6, p_{3} = 0.8, p_{4} = 0.7, p_{5} (the) = 0.5, p_{6} = 0.95.$ Compute the MLM loss.

Forecast: how many terms are in the average — 6 or 2?

Keep only the masked positions $M = {2, 5}$ . Why this step? Positions $1, 3, 4, 6$ are visible to the model, so predicting them is trivial copying — zero learning signal. They must NOT enter the loss.
Average the surprises over $∣ M ∣ = 2$ . $L_{MLM} = - \frac{1}{2} (lo g 0.6 + lo g 0.5) = - \frac{1}{2} (- 0.511 - 0.693) = 0.602.$ Why this step? We divide by the count of masked tokens, not by 6.

Verify: $- \frac{1}{2} (lo g 0.6 + lo g 0.5) = 0.6019$ ✓. Common wrong answer: dividing by 6 gives $0.201$ — that would silently reward the trivial visible tokens.

Example 5 — MLM 80/10/10 corruption (cell F)

Worked example The three sub-cases of a chosen token

BERT chooses 15% of tokens, then splits them: 80% → [MASK], 10% → a random vocab token, 10% → left unchanged. Suppose we choose 10 tokens. How many of each sub-type do we expect, and — crucially — for how many do we still compute a loss?

Forecast: does the "unchanged 10%" contribute to the loss? (Trap: it looks visible.)

Split the counts. $80% \times 10 = 8$ masked, $10% \times 10 = 1$ random-replaced, $10% \times 10 = 1$ unchanged. Why this step? These are the definitional proportions of the corruption recipe.
Decide the loss set. All 10 chosen tokens contribute to the loss — including the unchanged one. Why this step? The label is always the original token. For the unchanged 10%, the input still shows the real token, yet we STILL ask the model to predict it. This forces the model to build a real representation of every token instead of only reacting to the [MASK] symbol — closing the train/test gap, since [MASK] never appears during fine-tuning.
Contrast with Example 4. In Ex 4 the non-chosen positions were excluded; here the chosen-but-unchanged token is NOT the same thing and is still scored.

Verify: $0.8 \times 10 = 8$ , $0.1 \times 10 = 1$ , $0.1 \times 10 = 1$ , and $8 + 1 + 1 = 10$ (all scored) ✓.

Example 6 — Span corruption, one output sequence (cell G)

Worked example T5-style multi-token span

Input Thank you for inviting me to your party last week. We mask the contiguous span for inviting me with a single sentinel <X>:

Encoder input: Thank you <X> to your party last week
Decoder target: <X> for inviting me

The decoder generates the span autoregressively, giving $p (for) = 0.5, p (inviting) = 0.4, p (me) = 0.5$ . Compute the span loss.

Forecast: is the span scored as one blob, or token-by-token like CLM?

Recognize the target is a short sequence, scored autoregressively. Why this step? The decoder emits the span left-to-right, so it's a mini-CLM over the 3 span tokens (see Transformer Architecture encoder–decoder).
Average the three surprises. $L = - \frac{1}{3} (lo g 0.5 + lo g 0.4 + lo g 0.5) = - \frac{1}{3} (- 0.693 - 0.916 - 0.693) = 0.767.$ Why this step? Same cross-entropy machinery, divided by the number of generated span tokens.
Why spans not single tokens? A single masked word is often guessable from local n-grams; forcing 3 tokens demands genuine multi-token reasoning.

Verify: $- \frac{1}{3} (lo g 0.5 + lo g 0.4 + lo g 0.5) = 0.7675$ ✓.

Example 7 — Word problem: tokens ≠ words (cell H)

Worked example Reporting a real training run

An engineer trains a CLM. After 1 epoch the average per-token loss over the validation set is $L = 3.2$ nats (natural log). Marketing asks for "perplexity per word." A tokenizer via Tokenization (BPE) splits the text into 1.3 tokens per word on average. Give (a) per-token PPL, (b) the more honest per-word PPL.

Forecast: is per-word perplexity higher or lower than per-token? (Fewer words than tokens → each word carries more surprise → higher.)

Per-token perplexity. $PPL_{tok} = e^{3.2} = 24.53.$ Why this step? Direct definition on the unit the loss was measured in (tokens).
Convert loss to per-word. Each word has $1.3$ tokens, so per-word loss $= 3.2 \times 1.3 = 4.16$ nats. Why this step? Surprise is additive across tokens; a word made of 1.3 tokens accumulates 1.3× the token surprise on average.
Per-word perplexity. $PPL_{word} = e^{4.16} = 64.07.$ Why this step? Reporting per-word makes cross-tokenizer comparisons fairer — a model that chops words into more tokens can look artificially "confident" per token.

Verify: $e^{3.2} = 24.53$ ✓; $3.2 \times 1.3 = 4.16$ ✓; $e^{4.16} = 64.07$ ✓.

Example 8 — Exam twist: log base & fair CLM-vs-MLM comparison (cell I)

Worked example The two classic traps

(a) A paper reports CLM loss $= 4.0$ bits ( $lo g_{2}$ ). What is the perplexity? (b) Model X (CLM) has PPL $30$ ; Model Y (MLM) has PPL $8$ . Is Y "3.75× better at language"?

Forecast: for (b), is comparing these two PPLs even fair?

(a) Match the base to the log. If loss was measured in bits, perplexity is $2^{L}$ , not $e^{L}$ . $PPL = 2^{4.0} = 16.$ Why this step? Perplexity's base must equal the log base of the loss, or the number is meaningless. (In nats it'd be $e^{4} = 54.6$ — a very different number for the same model.)
(b) Refuse the direct comparison. MLM predicts a token using both left and right context; CLM sees only the left. MLM's task is strictly easier, so its PPL is naturally lower — the $8$ vs $30$ gap does not mean Y "understands language" 3.75× better. Why this step? Perplexity is only comparable between models solving the same prediction task on the same tokenization. CLM-PPL and MLM-PPL (often called pseudo-perplexity) are different games.

Verify: $2^{4.0} = 16$ ✓; and $e^{4.0} = 54.598 \neq = 16$ — proving the base matters ✓.

Recall

Recall Did every cell get covered?

Which example handled the case where the model assigns probability $\to 0$ to the true token? ::: Example 2 (cell C) — loss $\to + \infty$ . Which positions enter the MLM loss sum? ::: Only masked positions $M$ (Example 4, cell E); but chosen-unchanged tokens are still scored (Example 5, cell F). Why can't you directly compare a CLM perplexity of 30 to an MLM perplexity of 8? ::: Different tasks — MLM sees future context, so its (pseudo)perplexity is easier and not comparable (Example 8, cell I). Perplexity of a model that knows nothing over vocab $V$ ? ::: Exactly $V$ (Example 3, cell D).

Flashcards

Per-token loss 1.609 on two tokens with probs 0.4 and 0.1 — perplexity?

e^{1.609} = 5.0

, the geometric mean of

2.5

and

10

Loss and PPL of a uniform model over vocab

V

L = lo g V

PPL = V

If loss is reported in bits, how do you get perplexity?

PPL = 2^{L}

, not

e^{L}

Does the 10% "unchanged" token in BERT contribute to the loss?

Yes — its label is the original token, forcing real representations even without a [MASK].

Per-token loss 3.2 nats, 1.3 tokens/word — per-word PPL?

e^{3.2 \times 1.3} = e^{4.16} \approx 64