Noesis
4.3.11Pretraining & Fine-Tuning LLMs

Supervised fine-tuning (SFT)

2,030 words9 min readdifficulty · medium4 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHY does SFT exist?

WHY not just prompt-engineer the base model? You can, but it's brittle. SFT bakes the desired behavior into the weights, so the assistant persona is the default, not a fragile trick.

WHAT is SFT, precisely?

Key nouns:

  • Base model πθ\pi_\theta: parameters θ\theta from pretraining.
  • Dataset D={(x(i),y(i))}\mathcal{D} = \{(x^{(i)}, y^{(i)})\}: xx = prompt, yy = demonstration response.
  • Loss masking: we do NOT train the model to predict the prompt tokens — only the answer.

HOW do we derive the SFT loss from first principles?

We want the model to assign high probability to the human's response given the prompt.

Step 1 — What "learning to answer" means probabilistically. The model defines a conditional distribution πθ(y∣x)\pi_\theta(y \mid x). We want πθ\pi_\theta to match the demonstrations, i.e. maximize the likelihood of the observed responses.

Why this step? Maximizing likelihood of the "good" answer is exactly saying "make this the text you'd most naturally continue."

Step 2 — Factorize the sequence probability. A response y=(y1,…,yT)y = (y_1, \dots, y_T) is generated token-by-token (autoregressively):

πθ(y∣x)=∏t=1Tπθ(yt∣x, y<t)\pi_\theta(y \mid x) = \prod_{t=1}^{T} \pi_\theta\big(y_t \mid x,\, y_{<t}\big)

Why this step? By the chain rule of probability, any joint distribution equals the product of conditionals. LLMs are literally built to output πθ(yt∣everything before)\pi_\theta(y_t \mid \text{everything before}).

Step 3 — Turn product into a sum (take log). Maximizing a product of tiny numbers is numerically unstable, so take the log (monotonic, so the argmax is unchanged):

log⁡πθ(y∣x)=∑t=1Tlog⁡πθ(yt∣x,y<t)\log \pi_\theta(y \mid x) = \sum_{t=1}^{T} \log \pi_\theta(y_t \mid x, y_{<t})

Step 4 — Flip sign to get a loss to minimize. Maximizing log-likelihood = minimizing negative log-likelihood. Average over the dataset:

  LSFT(θ)=− E(x,y)∼D[∑t=1Tlog⁡πθ(yt∣x,y<t)]  \boxed{\;\mathcal{L}_{\text{SFT}}(\theta) = -\,\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\sum_{t=1}^{T} \log \pi_\theta\big(y_t \mid x, y_{<t}\big)\right]\;}

Why this step? Optimizers minimize; the negative log-likelihood is the standard cross-entropy between the one-hot true token and the model's predicted distribution.

Figure — Supervised fine-tuning (SFT)

Worked Example 1 — Computing SFT loss on a tiny sequence

Prompt tokens: ["What","is","2+2","?"], response tokens: ["4","<eos>"]. Suppose after the prompt the model predicts:

  • P("4"∣prompt)=0.5P(\text{"4"} \mid \text{prompt}) = 0.5
  • P("<eos>"∣prompt,"4")=0.8P(\text{"<eos>"} \mid \text{prompt},\text{"4"}) = 0.8

Step 1 — mask. Prompt tokens get m=0m=0, response tokens get m=1m=1. Only "4" and "" count. Why? We only teach the answer.

Step 2 — per-token NLL. ℓ"4"=−log⁡0.5=0.693\ell_{\text{"4"}} = -\log 0.5 = 0.693, ℓ"<eos>"=−log⁡0.8=0.223\ell_{\text{"<eos>"}} = -\log 0.8 = 0.223.

Step 3 — average over response tokens. L=0.693+0.2232=0.458\mathcal{L} = \frac{0.693 + 0.223}{2} = 0.458 Why average? So loss magnitude doesn't depend on answer length.

Step 4 — interpret. Gradient descent will increase P("4")P(\text{"4"}) and P("<eos>")P(\text{"<eos>"}) next time. That's imitation.

Worked Example 2 — Chat template & where the mask goes

Modern SFT wraps data in a chat template:

<|user|> How do I sort a list in Python? <|assistant|> Use sorted(mylist). <|eos|>

Step 1 — tokenize the whole thing. Why? The model sees prompt+answer as one stream.

Step 2 — set mask = 0 up to and including <|assistant|>, mask = 1 for the answer + eos. Why include eos in the mask? So the model learns when to stop — a crucial assistant skill.

Step 3 — one forward pass, masked cross-entropy, backprop. Same machinery as pretraining, different bookkeeping.

Worked Example 3 — Why a small, clean dataset beats a huge messy one (80/20)

Given: 1,000 hand-checked high-quality demos vs 100,000 scraped noisy Q&A. Step 1 — recall SFT = imitation. The model copies style and behavior of whatever you show it. Step 2 — noisy data teaches noisy behavior (hallucinated facts, sloppy formatting). Why? Loss rewards matching the demonstration, good or bad. Step 3 — conclusion (LIMA-style finding): a few thousand excellent examples often outperform huge noisy sets. The 20% that matters: data quality.

Common Mistakes (Steel-manned)

Recall Feynman: explain it to a 12-year-old

Imagine a parrot that has read the entire library and can finish any sentence you start — but it doesn't know it's supposed to help you. SFT is like showing the parrot a stack of flashcards: on the front is a question, on the back is a perfect, polite answer. You make the parrot practice saying the back of each card. After enough cards, whenever someone asks a question, the parrot naturally gives a helpful answer instead of just rambling. We only grade the parrot on the answer part, never on repeating the question.

Flashcards

What is Supervised Fine-Tuning (SFT)?
Continued training of a pretrained LLM on curated (prompt, response) pairs, minimizing next-token cross-entropy loss on the response tokens only.
What loss function does SFT use?
The same next-token (autoregressive) cross-entropy / negative log-likelihood as pretraining.
What are the only two things that differ between SFT and pretraining?
The data (curated demonstrations) and the loss mask (response-only, prompt masked out).
Why do we mask the prompt tokens during SFT?
The prompt is user-given input; we only want to teach the model to generate the answer, not to predict the question.
Write the SFT loss for one example.
L=−∑tmtlog⁡πθ(yt∣x,y<t)\mathcal{L} = -\sum_t m_t \log \pi_\theta(y_t \mid x, y_{<t}) (normalized by ∑tmt\sum_t m_t), with mask mt=1m_t=1 on response tokens.
Why factorize πθ(y∣x)\pi_\theta(y\mid x) into a product of per-token conditionals?
Chain rule of probability; LLMs are autoregressive so they natively output πθ(yt∣x,y<t)\pi_\theta(y_t\mid x, y_{<t}).
SFT is an example of which learning paradigm?
Behavioral cloning / imitation learning.
Does SFT primarily add new knowledge or shape behavior?
It primarily shapes behavior and format; knowledge comes from pretraining. Injecting new facts via SFT tends to cause hallucination.
Why include the EOS token in the response mask?
So the model learns when to stop generating.
Which typically comes first, SFT or RLHF/DPO?
SFT first, then preference optimization (RLHF/DPO).
Per the LIMA-style insight, what matters most for SFT data?
Quality over quantity — a few thousand excellent demos can beat huge noisy datasets.

Connections

  • Pretraining of LLMs — provides the base model and the identical loss form.
  • Cross-entropy loss — the mathematical core of SFT.
  • Autoregressive language modeling — why we factorize token-by-token.
  • RLHF and Direct Preference Optimization (DPO) — alignment steps that usually follow SFT.
  • Instruction Tuning — SFT on instruction-style demonstrations.
  • LoRA / PEFT — parameter-efficient ways to run SFT cheaply.
  • Chat templates & special tokens — how prompt/response boundaries are marked for masking.

Concept Map

is only

creates

solved by

feeds

is a form of

bakes persona into

minimizes

derived via

then log turns product into

computed only on

via

yields

Pretrained base LLM

Plausible autocomplete

Gap: cannot answer

Supervised Fine-Tuning

Prompt-response pairs

Behavioral cloning

Model weights

Next-token cross-entropy

Chain rule factorization

Sum of log-probs

Response tokens

Loss masking

Helpful assistant

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek base LLM basically ek super-smart autocomplete hai — usne poora internet padh liya hai, par usko pata hi nahi ki usse "answer" dena hai. Tum poocho "bread kaise banaye?" to base model shayad aur questions likhna shuru kar de, kyunki internet pe aksar aisa hi pattern hota hai. SFT (Supervised Fine-Tuning) yahi gap bharti hai: hum model ko hazaaron (sawaal → perfect jawaab) examples dikhate hain aur bolte hain "next time aise hi likho." Isse assistant wala behavior model ke weights me baith jaata hai.

Sabse important baat: SFT koi naya jaadui loss nahi use karti. Wahi purana next-token cross-entropy loss hai jo pretraining me tha. Sirf do cheezein badalti hain — (1) data, jo ab curated demonstrations hai, aur (2) loss mask, jismein hum sirf response ke tokens pe loss lagate hain, prompt ke tokens pe nahi. Prompt user ne diya hai, usko predict karna sikhane ka koi fayda nahi. Isliye mask lagta hai. EOS token ko bhi mask me rakhte hain taaki model seekhe ki kab rukna hai.

Ek aur cheez yaad rakho: SFT mostly behavior aur format sikhati hai, naye facts nahi. Naya gyaan pretraining se aata hai. Agar tum SFT se naye facts thoosne ki koshish karoge, model hallucinate karega. Aur data ki quality quantity se zyada matter karti hai — 1000 badhiya examples aksar 1 lakh gande examples se better nikalte hain (yeh LIMA wali insight hai). Formula ek line me: "Demonstrate, Mask, Mimic" — bas yahi SFT hai.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,167 wordsD2Visual walkthroughThe derivation in pictures · 2,149 wordsD3Worked examplesEvery scenario covered · 2,834 wordsD4ExercisesGraded, full solutions · 2,469 wordsD5Question bankConcept traps, quick reveals · 1,320 words

Test yourself — Pretraining & Fine-Tuning LLMs

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Log-loss - binary cross-entropyAI-ML · 2.2.10Direct Preference Optimization (DPO)AI-ML · 4.4.4Instruction tuningAI-ML · 4.3.10Instruction tuningAI-ML · 4.3.10Reinforcement Learning from Human Feedback (RLHF)AI-ML · 4.4.1Proximal Policy Optimization for LLMsAI-ML · 4.4.3Direct Preference Optimization (DPO)AI-ML · 4.4.4