Supervised fine-tuning (SFT): Question bank — AI-ML deep dive

Before you start, three words to keep in mind (the parent's mnemonic): Data, Mask, Mimic. Almost every trap below is someone forgetting one of those three.

True or false — justify

SFT invents a brand-new loss function specifically for teaching helpfulness.

False. It reuses the identical next-token cross-entropy from pretraining (see Cross-entropy loss); only the data and the loss mask change.

SFT changes the model from a next-token predictor into something that is no longer a next-token predictor.

False. It is still 100% autoregressive next-token prediction (see Autoregressive language modeling) — SFT just curates which next tokens it practices predicting.

Because SFT only trains on response tokens, the model never "sees" the prompt during training.

False. The prompt is fed through the network every forward pass so the model can condition on it; we only zero out the loss on prompt positions, not the input.

SFT is a form of imitation learning.

True. It is behavioral cloning: the model is rewarded purely for reproducing the demonstrated response, good or bad.

A model that scores low SFT loss on your data is guaranteed to be helpful to real users.

False. Low loss means it imitates your demonstrations well; if the demonstrations are narrow or biased, it faithfully imitates that narrowness.

Adding the end-of-sequence token to the loss mask is an optional nicety.

False. Masking <eos> in (setting

m = 1

for it) teaches the model when to stop; leave it out and the assistant tends to ramble past its answer.

SFT reliably injects new factual knowledge into the model.

False. New facts overwhelmingly come from Pretraining of LLMs; SFT mostly teaches format, tone, and instruction-following, and forcing new facts through it often breeds hallucination.

You must always do RLHF after SFT for the model to work at all.

False. SFT alone already yields a usable assistant; RLHF / Direct Preference Optimization (DPO) are refinement steps that add preference signal on top.

If you double the length of every response in your dataset, the averaged SFT loss scale roughly doubles.

False. The parent's loss divides by

\sum_{t} m_{t}

(the token count), so per-token loss is length-normalized and stays comparable across long and short answers.

Training on 100,000 noisy scraped Q&A pairs beats 1,000 clean hand-checked ones because more data is always better.

False. Since SFT clones behavior, noisy data teaches noisy behavior; the LIMA-style finding is that a few thousand excellent demos often win.

Spot the error

"We set the mask to 1 on the prompt so the model learns the questions really well."

Error: the mask should be 0 on the prompt. We don't teach the model to predict user input — that wastes capacity and can degrade answers.

"SFT and RLHF are the same thing, both just align the model."

Error: SFT is supervised imitation of fixed demonstrations; RLHF/DPO optimize against a preference/reward signal comparing better vs worse outputs. Different objective entirely.

"To do SFT we replace cross-entropy with a special supervised-imitation loss."

Error: SFT already is cross-entropy; imitation is the interpretation, not a new formula. Nothing about the loss equation changes.

"Because loss is only on the response, we can skip the chat template and special tokens."

Error: the mask boundary is defined by where the assistant turn begins, so you need Chat templates & special tokens to know which positions to unmask.

"We averaged the loss over all

T

tokens including the prompt tokens to be safe."

Error: you must divide by

\sum_{t} m_{t}

(response tokens only). Dividing by all

T

dilutes the true response loss and mis-weights long prompts.

"LoRA changes the SFT objective, so the loss math is different when using PEFT."

Error: LoRA / PEFT only changes which parameters get updated, not the loss. The masked cross-entropy is unchanged.

"Since the base model can already autocomplete, prompt-engineering makes SFT unnecessary."

Error: prompting is brittle and lives outside the weights; SFT bakes the assistant persona into

θ

so it's the default, not a fragile trick.

Why questions

Why do we take the log of the sequence probability before optimizing?

A product of many tiny per-token probabilities underflows numerically; log turns it into a stable sum and, being monotonic, keeps the same argmax.

Why do we flip the sign of the log-likelihood?

Optimizers minimize; maximizing log-likelihood equals minimizing its negative, which is exactly cross-entropy / negative log-likelihood.

Why is masking the prompt not just "throwing away free training signal"?

Predicting the user's question is not a skill we want; spending gradient there consumes capacity and can actively hurt answer quality.

Why does SFT usually come before preference tuning like DPO?

DPO/RLHF need a decent policy to sample and rank from; SFT first gives the model the basic assistant format so preference tuning has something coherent to refine.

Why does the same cross-entropy loss produce "helpful behavior" rather than just "plausible internet text"?

Because the data changed — the model is now imitating curated (prompt → ideal answer) demos, so its most-probable continuation becomes the helpful one.

Why can bad demonstrations actively harm the model even if the loss goes down nicely?

Falling loss means faithful imitation; if the target is hallucinated or sloppy, the model learns to hallucinate or be sloppy confidently.

Edge cases

What if a training example has an empty response (only a prompt, no answer tokens)?

Then

\sum_{t} m_{t} = 0

and the per-example loss is undefined (division by zero); such examples must be filtered out before training.

What if the same prompt appears with two contradictory "ideal" responses in the dataset?

The model can't satisfy both, so it learns a blurred average or picks the more frequent style — a sign your data needs deduplication or a consistency pass.

What happens if you accidentally unmask everything (train on prompt + response equally)?

You collapse SFT back toward plain Instruction Tuning-free continued pretraining on this data; the model spends effort mimicking questions and its answering quality typically drops.

What if a response is extremely long — does one huge example dominate the batch?

Not for its per-token loss (it's normalized), but its many tokens still contribute many gradient terms, so very long examples can still skew a batch; balanced batching helps.

What is the limiting behavior of the per-token loss

- lo g p_{y_{t}}

as the model becomes perfect on a token?

p_{y_{t}} \to 1

- lo g p_{y_{t}} \to 0

; and as

p_{y_{t}} \to 0

(confidently wrong), the loss

\to \infty

, which is why confident mistakes are punished hardest.

What if none of your demonstrations ever end a turn cleanly (no <eos> shown)?

The model never learns a stopping signal and will over-generate at inference; the fix is to include and unmask the end token in every example.

Recall One-line self-test

If someone hands you a "new SFT variant," ask: did the loss equation actually change, or only the data and the mask? Answer ::: Almost always only the data and the mask changed — the masked next-token cross-entropy is the same skeleton underneath.