Exercises — Supervised fine-tuning (SFT)

What loss: the next-token cross-entropy (equivalently, the average negative log-likelihood of the correct next token), exactly as in Cross-entropy loss and Autoregressive language modeling. The difference: not the loss formula. Two things change:

1. Data — curated (prompt, response) demonstrations instead of raw internet text.
2. Loss mask — only response tokens contribute to the loss; prompt tokens are masked out. Everything else (the forward pass, the backprop, the cross-entropy) is identical.

Mask $=0$ for everything up to and including <|assistant|> (that is the prompt, given input). Mask $=1$ for the answer tokens and <|eos|> — so here Hello, !, and <|eos|>. Why include <|eos|>? So the model learns when to stop talking — a real assistant skill. See Chat templates & special tokens.

True. We show the model demonstrations of the desired output and train it to reproduce them; copying a demonstrated policy is exactly behavioral cloning.

$\ell =-\log (0.25)=-\ln (0.25)\approx 1.386$. Sanity check: $0.25$ is a "meh" probability, so the loss is well above $0$ but not huge. If $p$ were $1$, loss would be $0$; if $p\to 0$, loss $\to \infty$.

Step 1 — mask. Prompt tokens get $m=0$; only "Blue" and "" count. So ${\sum }_t{m}_t=2$. Step 2 — per-token NLL. ${\ell }_{\text{Blue}}=-\ln (0.4)\approx 0.916$ ${\ell }_{\text{<eos>}}=-\ln (0.9)\approx 0.105$ Step 3 — average. $\mathcal{L}=\frac{0.916+0.105}{2}\approx 0.511$ Why divide by 2? So a long answer isn't automatically penalized more than a short one — we average per response token.

Now ${\sum }_t{m}_t=6$ (4 prompt + 2 answer). Prompt contribution: $4\times (-\ln 0.1)=4\times 2.303=9.210$. Answer contribution: $0.916+0.105=1.021$. ${\mathcal{L}}_{\text{buggy}}=\frac{9.210+1.021}{6}\approx 1.705$ Interpretation: the loss is now dominated by how well the model predicts the user's own question — a skill we never wanted to teach. Compare to the correct $0.511$: the signal about the answer has been drowned out.

Unmasked, the loss $-\frac{1}{T}{\sum }_t\log p_{y_t}$ includes terms for predicting prompt tokens. Gradient descent then pushes the model to become a good question predictor, spending capacity on user input we already have. Two concrete harms:

1. Diluted signal — as Exercise 2.3 showed numerically, prompt terms can dominate the average and swamp the answer-learning signal.
2. Wasted capacity / worse answers — the model optimizes something irrelevant to the task, which can degrade answer quality. The mask $m_t$ zeroes out those prompt terms so only answer-generation is rewarded.

Loss $=-\overline{\ln p}$, so the geometric-mean per-token probability is $\bar{p}=e^{-\mathcal{L}}$.

• $\mathcal{L}=0.20\Rightarrow \bar{p}=e^{-0.20}\approx 0.819$
• $\mathcal{L}=0.80\Rightarrow \bar{p}=e^{-0.80}\approx 0.449$ Meaning: on the first response the model was already assigning ~82% average probability to the right tokens (nearly imitating it), while on the second only ~45% — much more to learn. Lower loss ⇔ closer to reproducing the demonstration.

Mostly refute. SFT teaches format and behavior — responding rather than autocompleting, tone, instruction-following, stopping at <|eos|>. The bulk of factual knowledge comes from pretraining. Why it looks like new knowledge: the model appears smarter because it now surfaces what it already knew in a helpful format. Danger: trying to cram fresh facts through a small SFT set often produces hallucination — the model learns the style of confidently stating facts without the underlying knowledge to back specific ones.

Choose the 2,000 clean demos (LIMA-style reasoning). Argument: the loss rewards matching whatever you show — good or bad. Noisy data teaches noisy behavior: hallucinated facts, sloppy formatting, inconsistent tone. The model becomes an excellent imitator of garbage. Failure mode of the big noisy set: the model's average behavior regresses to the noisy demonstrations; more tokens ≠ better if those tokens encode bad behavior. The 20% that matters here is data quality, not quantity. (This is the intuition behind Instruction Tuning datasets being carefully curated.)

Order: pretraining → SFT → preference optimization.

• Pretraining first: builds the raw language/world model from massive text; without it there's nothing to imitate with.
• SFT next: converts the autocomplete into an instruction-follower via demonstrations.
• Preference optimization last: refines which good answers to prefer using ranked comparisons — it assumes the model already produces reasonable candidates (which SFT gave it). Can't swap SFT/pretraining: SFT is continued training on a pretrained model; a from-scratch model has no linguistic competence to clone behavior onto, so a few thousand demos couldn't teach language and behavior at once.

Use LoRA / PEFT. It freezes the base weights and trains only small low-rank adapter matrices, so the optimizer stores gradients/states for a tiny fraction of parameters — the loss (masked cross-entropy) is unchanged, only which parameters receive updates differs.

Derivation. A response $y=(y_1,\dots ,y_T)$ has joint conditional probability factorizable by the chain rule of probability: ${\pi }_{\theta }(y\mid x)={\prod }_{t=1}^T{\pi }_{\theta }(y_t\mid x,y_{<t})$ Taking $\log$ (monotonic, so it doesn't move the maximizer) turns the product into a sum: $\log {\pi }_{\theta }(y\mid x)={\sum }_{t=1}^T\log {\pi }_{\theta }(y_t\mid x,y_{<t})$ Flipping sign gives a quantity to minimize; each term $-\log p_{y_t}$ is precisely the per-token Cross-entropy loss between the one-hot true token and the model's predicted distribution. So minimizing sequence NLL = summing per-token cross-entropies — the loss is forced by the autoregressive factorization, not chosen arbitrarily. Compute. Each token: $-\ln (0.5)\approx 0.693$. Average over 3 response tokens: $\mathcal{L}=\frac{3\times 0.693}{3}=0.693$. (Here the average equals the per-token value because all three are equal — a useful check.)

B's reported loss mixes prompt terms. With 3 prompt tokens at $p=0.2$: each contributes $-\ln (0.2)=1.609$. Let the single answer token contribute ${\ell }_a$. B averages over $3+1=4$ tokens: $0.36=\frac{3(1.609)+{\ell }_a}{4}\Rightarrow 3(1.609)+{\ell }_a=1.44\Rightarrow {\ell }_a=1.44-4.827=-3.387$ This is impossible — a per-token NLL can't be negative. So a reported ${\mathcal{L}}_B=0.36$ cannot arise from those prompt probabilities: the unmasked average would be dominated by the large prompt terms ($\approx 1.2$ from the prompt alone). The lesson: an unmasked loss is not comparable to a masked one — B's number is measuring a different quantity, and here the arithmetic even flags the mismatch. Never compare masked vs unmasked losses across runs.

Start from a strong base checkpoint (knowledge lives in pretraining, so pick one already fluent in code). Curate a small, high-quality demonstration set of (coding question → clean, correct, well-formatted answer) — quality bounds imitation quality (Instruction Tuning). Wrap every example in a consistent chat template so the model learns role boundaries and where answers begin/end, and mask everything up to <|assistant|>, keep <|eos|> masked-in so it learns to stop. Train with the standard masked next-token cross-entropy (unchanged from autoregressive pretraining); if memory is tight, use LoRA / PEFT to update only small adapters. Then, and only then, refine tone and correctness preferences with RLHF or Direct Preference Optimization (DPO), because those need a competent SFT policy to rank. The whole SFT phase is imitation — Demonstrate, Mask, Mimic.

SFT loss = masked next-token cross-entropy ::: average $-\log p_{y_t}$ over response tokens only Perfect-prediction per-token loss ::: $0$ (because $-\log 1=0$) Two things that differ from pretraining ::: the data and the loss mask Why average, not sum ::: so loss doesn't scale with answer length