Instruction tuning: Worked examples — AI-ML deep dive

This page is a drill. We take the response-masked cross-entropy loss from the parent note (Instruction tuning) and push it through every kind of situation it can face: normal cases, all-zero-mask degeneracies, tiny-probability blow-ups, multi-example batches, and a couple of exam-style traps. Nothing here contradicts the parent — we just go slower and cover the corners.

Before anything, two plain-words conventions we lean on the whole page.

Why $ln$ and not the raw probability? Two reasons we will see repeatedly below: (1) multiplying many probabilities under-flows to zero, and adding logs is the same thing but numerically safe — this is the logarithm-of-a-product identity $ln (ab) = ln a + ln b$ , which turns a fragile product into a stable sum (used all through Cross-Entropy Loss); (2) $- ln$ turns "probability $1$ = perfect" into "loss $0$ = perfect", and "probability $0$ = disaster" into "loss $= + \infty$ ", which is exactly the pain signal we want.

Look at Figure s01 below to lock in that shape. Its horizontal axis is the model's probability $p$ for the correct token (running $0.01 \to 1$ ); its vertical axis is the per-token loss $- ln p$ in nats. Follow the cyan curve from right to left: at the right edge ( $p \to 1$ ) it touches the floor at $0$ — a perfect, unsurprised model; as you move left ( $p \to 0$ ) it climbs without bound. The two amber dots mark exactly those extremes, and the small white squares sit on the checked numbers we compute in Example 4.

Figure s01 — the loss curve $- ln p$ : flat floor of $0$ at $p = 1$ (right), blow-up toward $+ \infty$ at $p \to 0$ (left).

The scenario matrix

Every case class this loss can throw at you, and which worked example covers it:

#	Case class	What's tricky about it	Covered by
A	Normal response, several tokens	the everyday case; compute + average	Ex 1
B	Confident vs. unsure model (high vs low $p$ )	how $- ln$ punishes low probability	Ex 2
C	Degenerate: mask all zero ( $\sum_{t} m_{t} = 0$ )	division by zero — an undefined loss	Ex 3
D	Limiting value: $p \to 0$	loss $\to + \infty$ ; why we clip	Ex 4
E	Limiting value: $p \to 1$	loss $\to 0$ ; the "already perfect" corner	Ex 4
F	Masked vs unmasked, same sequence	the number actually changes; by how much	Ex 5
G	Batch of multiple examples, unequal lengths	how to average across examples fairly	Ex 6
H	Real-world word problem (perplexity of an assistant)	translate loss ↔ perplexity	Ex 7
I	Exam twist: multi-token target word	one "word" = several tokens; product identity	Ex 8
J	Exam twist: which tokens get gradient?	conceptual sign check, no arithmetic	Ex 9

We now hit all ten cells.

Worked examples

Worked example Example 1 — Case A · the everyday multi-token response

Statement. A tiny response has 3 tokens. The model assigned probabilities $p_{1} = 0.5, p_{2} = 0.8, p_{3} = 0.25$ to the correct token at each response position. All three are response tokens ( $m_{t} = 1$ ). Prompt tokens are already masked out. Find $L$ .

Forecast: guess — will the average loss be closer to $0.5$ or to $2$ ? (Low probabilities hurt, so...)

Take $- ln$ of each response probability. Why this step? $- ln p$ is the "surprise" (in nats) of one token; we need each token's contribution before averaging. $- ln 0.5 = 0.6931, - ln 0.8 = 0.2231, - ln 0.25 = 1.3863.$
Sum them. Why this step? The formula's inner $\sum_{t} m_{t} (\cdot)$ adds up every response token's surprise. $0.6931 + 0.2231 + 1.3863 = 2.3026$ .
Divide by $\sum_{t} m_{t} = 3$ . Why this step? The $\frac{1}{\sum m _{t}}$ makes it a per-token average, so responses of different lengths are comparable. $2.3026/3 = 0.7675$ .

Result: $L \approx 0.768$ nats.

Verify: the smallest probability $0.25$ produced the biggest term ( $1.386$ ) — surprise grows as $p$ shrinks, as expected. The average sits between the best token's $0.223$ and worst's $1.386$ . ✓

Worked example Example 2 — Case B · confident model vs. unsure model

Statement. Same 3 response tokens, but now compare two models on the same data. Model X is confident-and-right: $p = 0.9, 0.9, 0.9$ . Model Y is unsure: $p = 0.4, 0.4, 0.4$ . Which has lower loss, and by how much?

Forecast: obviously X is lower — but is the gap small or large?

Loss of X. Why this step? Establish the good model's baseline. $- ln 0.9 = 0.10536$ ; average of three identical terms is just $0.10536$ .
Loss of Y. Why this step? Compare against the sloppy model. $- ln 0.4 = 0.91629$ .
Take the difference. Why this step? The loss gap is what gradient descent chases. $0.91629 - 0.10536 = 0.81093$ .

Result: $L_{X} \approx 0.105$ nats, $L_{Y} \approx 0.916$ nats; gap $\approx 0.811$ nats/token.

Verify: dropping confidence from $0.9$ to $0.4$ multiplies per-token loss by about $8.7 \times$ . This is exactly why IT works fast — the response-masked loss punishes an unsure assistant hard, steering weights toward "commit to the right answer." ✓

Worked example Example 3 — Case C · the degenerate all-zero mask

Statement. By a data bug, an example contains only prompt tokens — no response was included. So $m_{t} = 0$ for every $t$ . What does the formula give, and what should you do?

Forecast: guess — is the loss $0$ , or is it something worse than a number?

Compute the denominator. Why this step? Everything hinges on $\sum_{t} m_{t}$ . Here $\sum_{t} m_{t} = 0$ .
Look at the whole fraction. Why this step? The numerator $\sum_{t} m_{t} (\cdot)$ is also $0$ (every term multiplied by $0$ ). So the expression is $\frac{0}{0}$ — undefined, not zero.
The engineering fix. Why this step? You can't backprop through NaN. Real training loops skip examples with an empty response mask (or drop them at data-cleaning time). Never silently set the loss to $0$ — that would let malformed examples dilute the batch average.

Result: loss is undefined ( $0/0$ ); the correct action is to skip the example.

Verify (with a picture). Figure s02 below draws this exact trap. Read it left-to-right: seven cyan boxes each labelled "P" are the prompt tokens, and under every box the tag " $m = 0$ " reminds you they are all masked out. There is no response box at the end. The amber lines at the bottom then spell out the consequence: the mask sum $\sum_{t} m_{t} = 0$ , so the loss is $0/0$ , undefined — and the arrow points to the only safe action, skip. See also Prompt Templates and Chat Formats, where a missing ### Response: block is the usual cause.

Figure s02 — an all-prompt example: every token masked, denominator $0$ , loss undefined.

Worked example Example 4 — Cases D & E · the two limiting values

Statement. For a single response token, trace the loss as its probability $p$ ranges over $(0, 1]$ . What happens as (D) $p \to 0^{+}$ and (E) $p \to 1^{-}$ ?

Forecast: one end shoots to infinity, the other flattens to a floor. Which is which?

Case E, $p \to 1$ . Why this step? This is "the model is already perfect on this token." $- ln p \to - ln 1 = 0$ . Loss floor is exactly $0$ ; you cannot do better. (On Figure s01 this is the right-hand amber dot resting on the $x$ -axis.)
Case D, $p \to 0$ . Why this step? This is "the model gave the correct token essentially no chance." $- ln p \to + \infty$ . The loss is unbounded above. (On Figure s01 this is the left-hand amber dot, high up where the cyan curve is climbing steeply.)
Why we clip in practice. Why this step? A real $p$ can be $1 0^{- 30}$ , giving loss $\approx 69$ nats and a huge gradient that can blow up training. Libraries add a tiny $ε$ (e.g. clamp $p \geq 1 0^{- 12}$ ) so $- ln p$ stays finite. This is the same numerical-stability trick used in Cross-Entropy Loss.

Result: loss $\in [0, + \infty)$ nats; approaches $0$ as $p \to 1$ , diverges as $p \to 0$ .

Verify (the white squares on Figure s01): $- ln (0.999) = 0.001001$ (nearly the floor), $- ln (0.01) = 4.60517$ , $- ln (1 0^{- 6}) = 13.8155$ . The cyan curve passes through the two plotted white squares at $p = 0.999$ and $p = 0.01$ . ✓

Worked example Example 5 — Case F · masked vs. unmasked, the same sequence

Statement. A full example has 5 tokens: prompt = 3 tokens with model probs $0.6, 0.7, 0.9$ ; response = 2 tokens with probs $0.5, 0.5$ . Compute the loss (a) unmasked (grade all 5, the wrong way) and (b) masked (grade only the 2 response tokens, the right way).

Forecast: which number is smaller, and does the smaller one mean "better"? (Trap: smaller loss here is not the goal — it's the wrong metric.)

Per-token surprises. Why this step? We need each $- ln p$ once. Prompt: $- ln 0.6 = 0.5108$ , $- ln 0.7 = 0.3567$ , $- ln 0.9 = 0.1054$ . Response: $- ln 0.5 = 0.6931$ (twice).
Unmasked loss. Why this step? Show what the wrong recipe computes — average over all 5. Sum $= 0.5108 + 0.3567 + 0.1054 + 0.6931 + 0.6931 = 2.3591$ ; divide by $5 \Rightarrow 0.4718$ .
Masked loss. Why this step? The correct recipe: sum only response terms $= 0.6931 + 0.6931 = 1.3862$ ; divide by $\sum m_{t} = 2 \Rightarrow 0.6931$ .

Result: unmasked $\approx 0.472$ nats, masked $\approx 0.693$ nats.

Verify: the unmasked number is lower even though it grades the wrong thing — the easy-to-predict prompt tokens drag the average down and hide how badly the model does on the response. This is precisely why the parent note's "compute loss over the whole sequence" is a mistake: a flattering number that measures nothing you care about. See Supervised Fine-Tuning (SFT). ✓

Worked example Example 6 — Case G · a batch with unequal-length responses

Statement. A batch has two examples. Example 1 has a 1-token response with $p = 0.5$ . Example 2 has a 4-token response, each $p = 0.5$ . Compute the batch loss two ways: (a) per-example average then average the two, and (b) pool all response tokens then average once. Do they agree?

Forecast: guess — will these give the same number? (They usually don't.)

Per-example losses. Why this step? Each example is first normalized by its own response length. Ex 1: $- ln 0.5 = 0.6931$ over 1 token $= 0.6931$ . Ex 2: $4 \times 0.6931$ over 4 tokens $= 0.6931$ .
Method (a): average the two example-losses. Why this step? This treats each example as one unit, regardless of length. $(0.6931 + 0.6931) /2 = 0.6931$ .
Method (b): pool all 5 tokens. Why this step? This treats each token as one unit. Sum $= 5 \times 0.6931 = 3.4657$ ; divide by 5 $= 0.6931$ .

Result: here both give $0.693$ nats — because every token had the same probability.

Verify (the trap made visible): they only agreed because all $p$ were equal. Change Ex 1's single token to $p = 0.1$ ( $- ln 0.1 = 2.3026$ ), keeping Ex 2 as four tokens of $p = 0.5$ : method (a) $= (2.3026 + 0.6931) /2 = 1.4979$ ; method (b) pooled $= (2.3026 + 4 \times 0.6931) /5 = 1.0159$ . Different. Method (a) gives short responses equal weight; method (b) gives long responses more weight. Frameworks must pick one deliberately — token-level pooling (b) is common but silently favors long answers. ✓

Worked example Example 7 — Case H · real-world word problem (loss ↔ perplexity)

Statement. After instruction tuning, an assistant achieves an average response-token loss of $L = 1.6094$ nats/token on a validation set. Your PM asks "in plain terms, how uncertain is the model per word?" Report the perplexity.

Forecast: perplexity is "effective number of equally-likely choices per token." Guess: is it near 2, near 5, or near 50?

Recall the bridge. Why this step? Perplexity $= exp (L)$ when $L$ is average natural-log loss (nats) — this clean $exp$ is exactly why we chose base $e$ back in the conventions box. It converts "average surprise" back into "how many options it's effectively juggling."
Exponentiate. Why this step? Undo the $ln$ . $exp (1.6094) = 5.0$ .
Interpret. Why this step? Numbers need meaning for the PM: the model is, on average, about as uncertain as if it were choosing uniformly among 5 equally likely tokens at each step of the response.

Result: perplexity $= 5.0$ .

Verify: sanity by inversion — a uniform 5-way choice has per-token probability $1/5 = 0.2$ , and $- ln 0.2 = 1.6094 = L$ nats. The round trip closes exactly, so the " $5$ equally-likely tokens" story is faithful. (Lower loss → lower perplexity → more decisive assistant; ties back to FLAN and Zero-shot Generalization.) ✓

Worked example Example 8 — Case I · one target

word spanning several tokens Statement. The correct response is the single word "Paris", but the tokenizer splits it into two tokens: "Par" then "is". The model gives $p (Par ∣ prompt) = 0.4$ and $p (is ∣ prompt, Par) = 0.9$ . What is the model's probability for the whole word, and what is this example's masked loss?

Forecast: the word's probability is smaller than either piece — why?

Multiply the piece-probabilities for the word. Why this step? By the autoregressive factorization from the parent (Step 1), $p ("Paris") = p (Par) \cdot p (is ∣ Par) = 0.4 \times 0.9 = 0.36$ . Probabilities multiply along a sequence, so the whole is smaller than each part.
Loss = sum of the two token surprises. Why this step? By the logarithm-of-a-product identity ( $ln (ab) = ln a + ln b$ ), the log of that product becomes a sum: $- ln 0.4 - ln 0.9 = 0.9163 + 0.1054 = 1.0217$ .
Normalize by response length (2 tokens). Why this step? The mask sums to $2$ . $1.0217/2 = 0.5108$ .

Result: word-probability $= 0.36$ ; masked loss $\approx 0.511$ nats/token.

Verify: cross-check via the product: $- ln (0.36) = 1.0217 =$ the summed surprises — so "sum of log-losses" and "log of product" agree exactly. This is why token-level loss and word-level probability are the same statement written two ways. ✓

Worked example Example 9 — Case J · exam twist, no arithmetic (which tokens carry gradient?)

Statement. Given the templated example

### Instruction: Reverse the string
### Input: cat
### Response: tac

the tokens are, in order: [### Instruction: Reverse the string], [### Input: cat], [### Response:], [tac]. Which positions have $m_{t} = 1$ , and therefore push a gradient into the weights?

Forecast: guess how many blocks of this are graded — one, two, or three?

Identify the response span. Why this step? Only text the model must generate is graded. The user typed the instruction, the input, and even the literal ### Response: header — none of that is the model's job to produce.
Assign the mask. Why this step? Set $m_{t} = 0$ for every token up to and including the ### Response: header (that is, all of the instruction block, all of the input block, and the header itself). Set $m_{t} = 1$ only for the tokens of the answer tac. So exactly one block — the answer — is graded.
Conclude about gradient. Why this step? In the loss, each token's contribution is multiplied by $m_{t}$ ; a $0$ mask kills that term's gradient. Therefore only the tac tokens carry a nonzero gradient, and the model is steered purely toward producing the right answer after the header — never toward re-predicting text the user already supplied.

Result: exactly the answer tokens (tac) have $m_{t} = 1$ and get gradient; all instruction, input, and header tokens have $m_{t} = 0$ and are masked.

Verify: consistency with Example 1 of the parent note — there only the token 5 was scored, matching our "only the answer is graded" rule. Same trap, same fix: grading the header or input would waste capacity on text the user always supplies. See Self-Instruct and Alpaca for how these blocks are auto-generated. ✓

Recall

Recall Cover the answers

Does "log" in the loss mean base-10 or base- $e$ ? ⟶ base- $e$ (natural log); the unit is the nat.
What is a nat? ⟶ the surprise-unit from using $ln$ ; a uniform $e \approx 2.718$ -way guess costs $1$ nat.
Mask is all-zero — what's the loss? ⟶ undefined ( $0/0$ ); skip the example.
As $p \to 0$ the per-token loss does what? ⟶ diverges to $+ \infty$ (so we clip $p$ at a tiny $ε$ ).
As $p \to 1$ ? ⟶ approaches $0$ , the floor.
Unmasked loss vs masked on the same sequence — which is usually lower and is that good? ⟶ unmasked is lower (easy prompt tokens deflate it); it's the wrong metric.
Perplexity from a natural-log loss of $1.6094$ ? ⟶ $exp (1.6094) = 5$ .
Word "Paris" = tokens Par·is with $0.4, 0.9$ — word probability? ⟶ $0.4 \times 0.9 = 0.36$ .

Connections

Instruction tuning — parent; this page drills its masked-loss formula.
Cross-Entropy Loss — the $- ln p$ we averaged everywhere.
Pretraining Objective (Next-token prediction) — source of the chain-rule product in Ex 8.
Supervised Fine-Tuning (SFT) — where the masking convention lives.
Prompt Templates and Chat Formats — the ### Response: header that defines the mask boundary.
Self-Instruct and Alpaca — auto-built examples that must contain a non-empty response.
FLAN and Zero-shot Generalization — lower loss/perplexity → more decisive assistant.