Visual walkthrough — Supervised fine-tuning (SFT)

This is the visual companion to Supervised fine-tuning (SFT). If you have never seen probability multiplication or a logarithm used this way, start at line one — everything is earned.

Step 1 — A model is a machine that scores possible next words

WHAT. Feed the model the text "What is 2+2 ? ". It hands back a bar for every token in $V$. The bar for "4" is tall; the bar for "banana" is a sliver.

WHY. Before we can say "learn to answer," we need to see what the model actually produces. It never produces a word directly — it produces a confidence for every possible word. That confidence is the only thing we can push on.

PICTURE. Each bar's height is $p_v$ — the model's probability for token $v$. The heights sum to $1$ (they fill exactly one full unit of "belief").

In symbols, the distribution the model gives after prompt $x$ is written ${\pi }_{\theta }(\cdot \mid x)$

• $\pi$ — the model (a function from text to a bar chart).
• $\theta$ — the model's parameters (all its tunable weights). Changing $\theta$ reshapes the bars.
• $\cdot$ — a placeholder: "for every token $v$."
• $\mid x$ — "given that the text so far is $x$." The bars depend on what came before.

Step 2 — "Good answer" means: make the right bar tall

WHAT. Point at the one bar the human's demonstration says is correct. Call its height $p_{y_t}$ — the probability the model assigned to the actual next token $y_t$.

WHY. We are doing behavioral cloning: copy the human. The only way a next-token machine can "copy" is to route its probability mass onto the token the human actually chose. So the whole of SFT is: lift the correct bars.

PICTURE. One bar glows (the human's token). We want to grab its top and pull it toward $1$; the other bars shrink to compensate because the total is fixed at $1$.

• $y_t$ — the human's chosen token at position $t$.
• $p_{y_t}$ — its bar height, the number we push up.

Step 3 — An answer is a chain of these choices

WHAT. The response "4" then "<eos>" is two bar-charts in a row. Chart 1 (after the prompt) scores "4". Chart 2 (after prompt and "4") scores "<eos>" — the "I'm done" token.

WHY. We cannot judge an answer with a single bar; a good answer is a good path of choices. Every step's chart depends on the tokens already emitted — that's what "autoregressive" means.

PICTURE. A conveyor belt: each stop is a fresh bar chart, and the token chosen at one stop is fed forward as input to the next.

The probability of the whole response is the probability of taking every correct step. Independent-looking steps chained by "and" multiply — this is the chain rule of probability: ${\pi }_{\theta }(y\mid x)={\prod }_{t=1}^T{\pi }_{\theta }(y_t\mid x,y_{<t})$

• ${\prod }_{t=1}^T$ — multiply over all $T$ positions of the answer.
• $y_{<t}$ — "all response tokens before position $t$" (what the belt already emitted).
• The product is the height of the correct bar at step 1, times step 2, times… — the chance the model walks the entire human path.

Step 4 — Multiplying tiny bars is dangerous; take a log

WHAT. Apply $\log$ to both sides. The product becomes a sum: $\log {\pi }_{\theta }(y\mid x)={\sum }_{t=1}^T\log {\pi }_{\theta }(y_t\mid x,y_{<t})$

WHY. Two payoffs. (1) Numerical safety: sums of manageable numbers instead of a vanishing product. (2) $\log$ is monotonic (always increasing) — see the picture — so whatever makes the product biggest also makes its log biggest. Maximizing the log is the same problem, just easier.

PICTURE. The $\log$ curve climbs forever without ever going back down. So "highest product" and "highest log of that product" sit at the exact same input — swapping to log changes the scale, never the winner.

• ${\sum }_{t=1}^T$ — add up the per-step scores.
• $\log {\pi }_{\theta }(y_t\mid \dots )$ — the log-height of one correct bar. Since bar heights are between $0$ and $1$, their logs are $\le 0$ (negative or zero).

Step 5 — Flip the sign: from "reward" to "loss"

WHAT. Negate and average over the dataset $\mathcal{D}=\{(x^{(i)},y^{(i)})\}$: ${\mathcal{L}}_{\text{SFT}}(\theta )=-{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[{\sum }_{t=1}^T\log {\pi }_{\theta }(y_t\mid x,y_{<t})\right]$

WHY. $-\log p_{y_t}$ is exactly the Cross-entropy loss between "the truth is token $y_t$" (a single tall spike) and the model's bar chart. It is $0$ when the correct bar is at height $1$ (perfect) and shoots to $+\infty$ as that bar drops to $0$ (surprised, wrong). Minimizing it is lifting the correct bars — the goal from Step 2.

PICTURE. The curve $-\log p$: near $p=1$ it kisses zero (no penalty for confident-correct); near $p=0$ it rockets up (huge penalty for confident-wrong). This asymmetry is what makes the model care about not missing the right token.

• $-$ — the flip: maximize-likelihood becomes minimize-loss.
• ${\mathbb{E}}_{(x,y)\sim \mathcal{D}}$ — average over all example pairs in the dataset.
• Inside the brackets: the negated log-height sum — one number measuring "how surprised the model was by this human answer."

Step 6 — The mask: grade only the answer, never the question

WHAT. Slot $m_t$ into the sum and divide by the number of graded tokens: $\boxed{{\mathcal{L}}_{\text{SFT}}(\theta )=-{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[\frac{1}{{\sum }_t{m}_t}\sum _{t=1}^T{m}_t\log {\pi }_{\theta }(y_t\mid x,y_{<t})\right]}$

WHY. The prompt is given by the user — teaching the model to predict the question wastes capacity and can hurt answers. And dividing by ${\sum }_t{m}_t$ (the answer length) means a long answer and a short answer are scored on the same scale, so the loss doesn't just reward brevity or verbosity.

PICTURE. A strip of tokens: prompt cells greyed (switch off, $m=0$), answer cells lit (switch on, $m=1$). Only the lit cells send gradient back into $\theta$. The <eos> cell is lit too — that's how the model learns when to stop.

• $m_t$ — the on/off switch per token.
• ${\sum }_t{m}_t$ — count of answer tokens (the divisor = the average).
• Everything else is identical to Pretraining of LLMs; only the data and this mask differ.

Step 7 — Edge cases: does the formula survive the corners?

Case A — the model is already perfect ($p_{y_t}=1$ for every answer token). Then $-\log 1=0$ at every step, so $\mathcal{L}=0$. Nothing to fix — the valley floor. ✓

Case B — the model is catastrophically wrong ($p_{y_t}\to 0$). Then $-\log p_{y_t}\to +\infty$. One confidently-missed token can dominate the whole loss — which is correct: assigning zero probability to what actually happened is the worst possible mistake.

Case C — a one-token answer ($T=1$, e.g. just "Yes"). The sum has one term; ${\sum }_t{m}_t=1$; the average is just that single $-\log p$. No division-by-zero, formula holds. ✓

Case D — the whole prompt is masked ($m_t=0$ for prompt): those terms contribute $0\cdot (\dots )=0$. They vanish cleanly; the model never gets gradient for the question. ✓ (If you ever masked out every token, ${\sum }_t{m}_t=0$ and the average is undefined — so a valid SFT example must contain at least one response token.)

PICTURE. The $-\log p$ curve again, with A pinned at the far right ($p=1$, loss $0$) and B sliding off the left cliff ($p\to 0$, loss $\to \infty$). The middle worked example ($p=0.5\Rightarrow 0.693$) sits between them.

Worked check — the parent's tiny example, on the curves

Response "4" with $p=0.5$, then "<eos>" with $p=0.8$: ${\ell }_{\text{"4"}}=-\log 0.5=0.693,{\ell }_{\text{"<eos>"}}=-\log 0.8=0.223$ $\mathcal{L}=\frac{0.693+0.223}{2}=0.458$ Read it off Step 7's curve: $p=0.5$ lands at height $0.693$, $p=0.8$ at $0.223$; their average is the model's surprise per answer token. Gradient descent will push both bars up, shrinking both numbers next time.

The one-picture summary

Left to right, this single figure is the whole derivation: bars → chain → log-sum → sign-flip → mask. A raw score per token, chained across the answer, log-summed for stability, negated into a loss, and masked to the answer only.

Reveal checks