Worked examples — Supervised fine-tuning (SFT)

This page is a case-by-case drill for the SFT loss. The parent SFT topic note built the loss from scratch; here we make sure you never meet a scenario you haven't already solved by hand. Every symbol used below was earned in the parent — we recall each one the moment it appears.

Recall The two formulas we will use over and over

Per-token loss for a target token $y_t$: the model outputs a probability $p_{y_t}$ (a number between $0$ and $1$) of the correct next token, and the loss is ${\ell }_t=-\log p_{y_t}$ The $\log$ here is the natural log (base $e\approx 2.718$). "$-\log$" turns a probability into a penalty: probability $1$ → loss $0$ (perfect), probability near $0$ → loss shoots to $+\infty$ (terrible).

Masked average loss over a sequence of $T$ tokens with mask $m_t\in \{0,1\}$ (1 = response token we grade, 0 = prompt token we ignore): $\mathcal{L}=-\frac{1}{{\sum }_t{m}_t}{\sum }_{t=1}^T{m}_t\log p_{y_t}$ The denominator ${\sum }_t{m}_t$ is just "how many response tokens we counted" — dividing by it gives a per-token average, so length doesn't inflate the loss.

---

The scenario matrix

Cross-entropy has fewer "quadrants" than trigonometry, but it has its own edge cases: probabilities near the two boundaries $0$ and $1$, the loss-mask bookkeeping, length normalization, and the "is loss going up or down?" reading of training curves. Here is every cell we must cover.

Cell	Case class	What makes it tricky	Example
A	Typical mid-confidence token	baseline arithmetic	Ex 1
B	Near-perfect prediction ($p\to 1$)	loss → 0, limiting behaviour	Ex 2
C	Near-zero prediction ($p\to 0$)	loss → $+\infty$, degenerate	Ex 2
D	Loss mask on prompt vs response	which tokens count	Ex 3
E	Length normalization (short vs long answer)	why we divide by token count	Ex 4
F	Reading a training curve (loss up vs down)	sign/direction interpretation	Ex 5
G	Real-world word problem (dataset choice)	applying imitation logic	Ex 6
H	Exam twist: multi-class softmax → probability	where $p_{y_t}$ comes from	Ex 7
I	Exam twist: log base & units (nats vs bits)	which log, converting	Ex 8

Every numeric answer below is machine-checked in the Verify block.

---

Example 1 — Cell A: a typical token

Forecast: Probability $0.25$ is "one-in-four" — not great, not disastrous. Guess: loss somewhere around $1$ to $1.5$.

1. Write the formula. $\ell =-\log p$. Why this step? Every per-token loss is just the negative log of the probability given to the true token — nothing else enters.
2. Plug in. $\ell =-\log (0.25)$. Why? $0.25$ is exactly the probability of the correct token, which is what $-\log$ eats.
3. Compute. $\log (0.25)=\log (1/4)=-\log 4\approx -1.386$, so $\ell \approx 1.386$. Why? $\log (1/4)$ is negative because $1/4<1$; the minus sign flips it positive so it's a valid penalty.

Verify: Sanity check the boundaries. $p=0.5$ gives $\ell =0.693$; $p=0.125$ gives $\ell =2.079$. Our $0.25$ sits between them at $1.386$. ✓ Monotone: lower probability → higher loss.

---

Example 2 — Cells B & C: the two limits ($p\to 1$ and $p\to 0$)

Forecast: B should be tiny (nearly perfect). C should be large. Guess: B ≈ $0.001$, C ≈ $7$.

1. Case B: ${\ell }_B=-\log (0.999)$. Why? Near-perfect prediction — we expect the penalty to vanish.
2. Compute B: $-\log (0.999)\approx 0.0010005$. Why it looks like $\approx 1-p$: write $p=1-ϵ$ with $ϵ=0.001$. The Taylor expansion of $-\log (1-ϵ)$ around $ϵ=0$ is $ϵ+\frac{{ϵ}^2}{2}+\dots$, so for tiny $ϵ$ the leading term $ϵ=1-p$ dominates — hence $-\log (0.999)\approx 0.001$. The curve is nearly flat and touches $0$ at $p=1$.
3. Case C: ${\ell }_C=-\log (0.001)=\log (1000)\approx 6.908$. Why? A confidently-wrong token is punished hard; as $p\to 0$ the loss $\to +\infty$.

Verify (limiting behaviour): Look at the loss curve below. At $p=1$ the blue curve hits $0$ (loss can never go negative — probabilities can't exceed $1$). As $p\to 0^{+}$ the curve rockets up with no ceiling. Both computed points ($0.001$ and $6.908$) lie on the curve. ✓

Figure s01 — the cross-entropy penalty curve $\ell =-\log p$. The horizontal axis is $p$, the probability the model gave the true token; the vertical axis is the per-token loss in nats. The blue curve falls to $0$ at the right edge ($p=1$, green point B: near-perfect) and climbs without bound toward the left edge ($p\to 0$, red point C: confidently wrong). The orange point A ($p=0.25$, loss $1.386$) is our typical-token baseline. Read left-to-right: the more probability you deny the correct token, the steeper the punishment.

---

Example 3 — Cell D: where the loss mask goes

Forecast: Only the last three tokens should count. Guess: loss around $0.4$.

1. Set the mask. $m=0$ for <user>, Hi, <asst> (prompt / template); $m=1$ for Hello, !, <eos>. Why this step? We only teach the model to produce the answer. The prompt is user-given input — grading the model for "predicting" it wastes capacity.
2. Keep only masked tokens. Response probabilities: $0.6,0.7,0.8$. Why? The ${\sum }_t{m}_t\cdot (\dots )$ in the formula zeroes out every $m=0$ term.
3. Per-token losses. $-\log 0.6=0.511$, $-\log 0.7=0.357$, $-\log 0.8=0.223$. Why? Same $\ell =-\log p$ applied to each surviving token.
4. Average over the $\sum m_t=3$ counted tokens. $\mathcal{L}=\frac{0.511+0.357+0.223}{3}=\frac{1.091}{3}=0.364$ Why divide by 3, not 6? The denominator is the number of masked-in tokens, not the total sequence length.

Verify: Include <eos> — it carries $m=1$ because "knowing when to stop" is a skill we teach. If we had (wrongly) trained on all 6 tokens, we'd have divided a larger sum by 6 and gotten a different, contaminated number. Our masked answer $0.364$ ignores <user>/Hi/<asst> entirely. ✓

---

Example 4 — Cell E: length normalization

Forecast: Summed loss should favour the short answer (fewer terms). Averaged loss should call them equal. That's the whole point of dividing.

1. Per-token loss is identical. Each token: $-\log 0.5=0.693$. Why? Same probability everywhere, so same per-token penalty.
2. Summed (un-normalized) loss. A: $2\times 0.693=1.386$. B: $8\times 0.693=5.545$. Why this matters: if we didn't divide, long answers would look "worse" purely for being long, and the optimizer would bias toward terseness.
3. Averaged loss. A: $1.386/2=0.693$. B: $5.545/8=0.693$. Why? Dividing by ${\sum }_t{m}_t$ (2 and 8 respectively) cancels the length. Now both equal exactly $0.693$.

Verify: Averaged losses match to the digit ($0.693=0.693$). ✓ This is precisely why the parent note's formula has $\frac{1}{{\sum }_t{m}_t}$ out front — loss magnitude must not depend on answer length.

---

Example 5 — Cell F: reading a training curve

Forecast: Loss falling means the model gets more probability right, so the effective probability should climb toward 1.

1. Invert the loss. Since $\ell =-\log p$, we recover $p=e^{-\ell }$. Why this tool — the exponential? $e^{-\ell }$ is the exact inverse of $-\log$; it "undoes" the loss transform to recover the probability the loss corresponds to. We use $e$ (not $10$) because SFT losses are in natural log.
2. Convert each.
   • $\ell =2.0\Rightarrow p=e^{-2.0}\approx 0.135$
   • $\ell =1.4\Rightarrow p=e^{-1.4}\approx 0.247$
   • $\ell =1.1\Rightarrow p=e^{-1.1}\approx 0.333$ Why? Each step maps the reported loss to the geometric-mean probability the model assigns to true tokens.
3. Read the direction. The effective probability rose $0.135\to 0.247\to 0.333$. Why this is "learning": gradient descent is doing its job — SFT is imitation, so rising probability on demonstration tokens = the model is copying the demos better.

Verify: Round-trip check: $-\log (0.135)=2.0$, $-\log (0.333)=1.1$. ✓ Loss down ⇔ true-token probability up. (Falling loss is good; a rising SFT loss would mean it's un-learning the demos.)

---

Example 6 — Cell G: real-world word problem

Forecast: SFT is imitation, and the loss rewards matching whatever you show it. So messy demos → messy behaviour. Guess: pick A.

1. Restate the loss's job. For a single example, the loss is our masked average $\mathcal{L}=-\frac{1}{{\sum }_t{m}_t}{\sum }_t{m}_t\log p_{y_t}$ — the same denominator (number of response tokens) from the recall box. It is minimized when the model assigns high probability to exactly the response tokens in your demonstration. Why this step? The math has no notion of "truth" — only "match the target." Good target → good habits; bad target → bad habits.
2. Trace what B teaches. In (B), many targets contain hallucinated facts, sloppy formatting, rude tone. Minimizing loss on them literally trains the model to reproduce those. Why? Every noisy demonstration is a token sequence the loss pushes the model toward.
3. Trace what A teaches. In (A), every target is clean, so the same loss now pushes toward clean behaviour. Fewer examples, but every gradient step points the right way. Why fewer can win: SFT mostly teaches format and behaviour, not facts (those come from pretraining). A few thousand excellent demos are enough to fix behaviour — the LIMA finding.
4. Decision: choose A.

Verify (consistency check): This matches the parent's Worked Example 3 (80/20: data quality is the 20% that matters) and its mistake box "SFT gives new knowledge" (false — it teaches behaviour). No contradiction. ✓

---

Example 7 — Cell H: where $p_{y_t}$ actually comes from (softmax)

Forecast: cat has the biggest logit, so it should get the biggest probability — maybe $\sim 0.6$ — and a modest loss.

1. Why softmax at all? Logits can be any real number (negative, huge), but a loss needs a probability in $[0,1]$ that sums to $1$ across the vocab. The softmax is the standard tool that turns arbitrary scores into a valid probability distribution: it exponentiates (making everything positive) then normalizes (making them sum to 1). $p_v=\frac{e^{z_v}}{{\sum }_u{e}^{z_u}}$
2. Exponentiate each logit. $e^{2.0}=7.389$, $e^{1.0}=2.718$, $e^{0.1}=1.105$. Why? Exponentiation guarantees positivity and amplifies gaps between scores.
3. Sum and normalize. Sum $=7.389+2.718+1.105=11.212$. So $p_{\text{cat}}=\frac{7.389}{11.212}=0.659$ Why? Dividing by the total makes the three probabilities add to exactly 1.
4. Loss on the true token. $\ell =-\log (0.659)=0.417$. Why? $p_{\text{cat}}$ is exactly the $p_{y_t}$ fed into the per-token loss.

Verify: The three probabilities $0.659,0.242,0.099$ sum to $1.000$. ✓ And $-\log 0.659=0.417$. ✓ Softmax is where every $p_{y_t}$ in this whole page is born.

Figure s02 — softmax turns three raw logits into three probabilities. Grey bars are the raw logits $z=(2.0,1.0,0.1)$ for cat, dog, eos; blue bars are the probabilities after exponentiating and normalizing ($0.659,0.242,0.099$, which sum to $1$). The biggest logit (cat) becomes the biggest probability, and the orange note shows its loss $-\log 0.659=0.417$. This bar-to-bar transform is the single step that produces the $p_{y_t}$ every other example on this page consumes.

---

Example 8 — Cell I: which log? nats vs bits

Forecast: Bits use ${\log }_2$, nats use ${\log }_e$. A bit is a "bigger" unit than a nat, so the nat number should be larger than $1.5$.

1. Why two units exist. Cross-entropy is $-\log p$, but which base? Base-$2$ gives bits (info-theory convention); base-$e$ gives nats (ML/optimizer convention, because $e^x$ and $\log$ have clean derivatives). They measure the same thing in different rulers.
2. Convert bits → nats. $1\text{ bit}=\ln 2$ nats $=0.693$ nats. So $1.5\text{ bits}=1.5\times 0.693=1.040\text{ nats}$ Why multiply by $\ln 2$? Because ${\log }_{2}p=\frac{\ln p}{\ln 2}$, so a nats-loss is $\ln 2$ times the bits-loss.
3. Recover the probability. From nats, $p=e^{-1.040}=0.354$. Why $e$? We're now in natural-log land, so the inverse is the natural exponential.
4. Cross-check via bits directly: $p=2^{-1.5}=0.354$. Same answer.

Verify: $2^{-1.5}=0.354$ and $e^{-1.040}=0.354$ agree. ✓ Always know whether your loss is in nats or bits before comparing to a paper — off-by-$\ln 2\approx 0.693$ is a classic confusion.

---

Recall Quick self-test

A token gets probability $p=0.2$; its loss? ::: $-\log 0.2=1.609$ Response has 3 tokens with $p=0.9,0.9,0.9$; averaged loss? ::: $-\log 0.9=0.105$ (all equal, so the average is just that) Loss drops from $1.6$ to $0.7$ — good or bad, and what happened to true-token probability? ::: Good; probability rose from $e^{-1.6}=0.202$ to $e^{-0.7}=0.497$ Why does <eos> get mask = 1? ::: So the model learns when to stop — a genuine response skill Confident-wrong ($p=0.001$) vs unsure-wrong ($p=0.3$): which costs more? ::: Confident-wrong, $6.9$ vs $1.2$ nats What happens when ${\sum }_t{m}_t=0$? ::: The example has no response tokens — drop it before averaging, else divide-by-zero

See also: Cross-entropy loss, Autoregressive language modeling, Chat templates & special tokens, and the preference-tuning follow-ups RLHF / Direct Preference Optimization (DPO).