Supervised fine-tuning (SFT) — Hinglish notes

4.3.11 · AI-ML › Pretraining & Fine-Tuning LLMs

SFT exist kyun karta hai?

Sirf prompt-engineering kyun nahi karte base model par? Kar sakte ho, lekin yeh brittle hota hai. SFT chahiye behavior ko weights mein bake karta hai, toh assistant persona default ban jata hai, koi fragile trick nahi.

SFT precisely KYA hai?

Key nouns:

Base model $π_{θ}$ : parameters $θ$ pretraining se.
Dataset $D = {(x^{(i)}, y^{(i)})}$ : $x$ = prompt, $y$ = demonstration response.
Loss masking: hum model ko prompt tokens predict karne ki training NAHI dete — sirf answer ki.

SFT loss ko first principles se HOW derive karte hain?

Hum chahte hain ki model human ke response ko high probability assign kare prompt diya gaya ho toh.

Step 1 — "Jawab dena seekhna" probabilistically matlab kya hai. Model ek conditional distribution $π_{θ} (y ∣ x)$ define karta hai. Hum chahte hain $π_{θ}$ demonstrations se match kare, yaani observed responses ki likelihood maximize kare.

Yeh step kyun? "Acche" jawaab ki likelihood maximize karna exactly yeh keh raha hai "is text ko woh text banao jise tum naturally continue karte."

Step 2 — Sequence probability ko factorize karo. Ek response $y = (y_{1}, \dots, y_{T})$ token-by-token (autoregressively) generate hota hai:

π_{θ} (y ∣ x) = t = 1 \prod T π_{θ} (y_{t} ∣ x, y_{< t})

Yeh step kyun? Probability ke chain rule se, koi bhi joint distribution conditionals ke product ke barabar hoti hai. LLMs literally $π_{θ} (y_{t} ∣ sab kuch pehle)$ output karne ke liye bane hain.

Step 3 — Product ko sum mein convert karo (log lo). Chhote numbers ke product ko maximize karna numerically unstable hai, toh log lo (monotonic hai, toh argmax unchanged rehta hai):

lo g π_{θ} (y ∣ x) = t = 1 \sum T lo g π_{θ} (y_{t} ∣ x, y_{< t})

Step 4 — Sign flip karo ek loss banane ke liye jise minimize karein. Log-likelihood maximize karna = negative log-likelihood minimize karna. Dataset par average karo:

L_{SFT} (θ) = - E_{(x, y) \sim D} [t = 1 \sum T lo g π_{θ} (y_{t} ∣ x, y_{< t})]

Yeh step kyun? Optimizers minimize karte hain; negative log-likelihood standard cross-entropy hai one-hot true token aur model ki predicted distribution ke beech.

Worked Example 1 — Ek chhoti sequence par SFT loss compute karna

Prompt tokens: ["What","is","2+2","?"], response tokens: ["4","<eos>"]. Maano prompt ke baad model predict karta hai:

$P ("4" ∣ prompt) = 0.5$
$P ("<eos>" ∣ prompt, "4") = 0.8$

Step 1 — mask. Prompt tokens ko $m = 0$ milta hai, response tokens ko $m = 1$ . Sirf "4" aur "" count hote hain. Kyun? Hum sirf jawaab sikhate hain.

Step 2 — per-token NLL. $ℓ_{"4"} = - lo g 0.5 = 0.693$ , $ℓ_{"<eos>"} = - lo g 0.8 = 0.223$ .

Step 3 — response tokens par average karo. $L = \frac{0.693 + 0.223}{2} = 0.458$ Average kyun? Taaki loss magnitude answer ki length par depend na kare.

Step 4 — interpret karo. Gradient descent $P ("4")$ aur $P ("<eos>")$ ko increase karega agli baar. Yahi imitation hai.

Worked Example 2 — Chat template aur mask kahan jaata hai

Modern SFT data ko ek chat template mein wrap karta hai:

<|user|> How do I sort a list in Python? <|assistant|> Use sorted(mylist). <|eos|>

Step 1 — poori cheez tokenize karo. Kyun? Model prompt+answer ko ek stream ki tarah dekhta hai.

Step 2 — mask = 0 set karo <|assistant|> tak aur including, mask = 1 answer + eos ke liye. Eos ko mask mein kyun shamil karein? Taaki model seekhe kab rokna hai — yeh ek crucial assistant skill hai.

Step 3 — ek forward pass, masked cross-entropy, backprop. Pretraining jaisi wahi machinery, bas alag bookkeeping.

Worked Example 3 — Ek chhota, clean dataset ek bade messy wale se kyun behtar hota hai (80/20)

Diya gaya: 1,000 hand-checked high-quality demos vs 100,000 scraped noisy Q&A. Step 1 — yaad karo SFT = imitation. Model jo bhi dikhate ho uski style aur behavior copy karta hai. Step 2 — noisy data noisy behavior sikhata hai (hallucinated facts, sloppy formatting). Kyun? Loss demonstration se match karne par reward karta hai, accha ho ya bura. Step 3 — conclusion (LIMA-style finding): kuch hazaar excellent examples aksar bade noisy sets se behtar perform karte hain. Woh 20% jo matter karta hai: data quality.

Common Mistakes (Steel-manned)

Recall Feynman: ek 12-saal ke bachhe ko explain karo

Socho ek tota jisne poori library padh li hai aur jo bhi sentence tum start karo use finish kar sakta hai — lekin usse pata nahi ki use tumhari help karni hai. SFT aise hai jaise tote ko flashcards ka ek stack dikhao: aage sawaal hai, peeche ek perfect, polite jawaab. Tum tote ko har card ka peechi wala hissa bolne ki practice karate ho. Kaafi cards ke baad, jab bhi koi sawaal puchhe, tota naturally ek helpful jawaab deta hai sirf bakwaas karne ki jagah. Hum tote ko sirf jawaab wale hisse par grade karte hain, kabhi sawaal dobara repeat karne par nahi.

Flashcards

Supervised Fine-Tuning (SFT) kya hai?

Ek pretrained LLM ki continued training curated (prompt, response) pairs par, sirf response tokens par next-token cross-entropy loss minimize karke.

SFT kaunsa loss function use karta hai?

Wahi next-token (autoregressive) cross-entropy / negative log-likelihood jo pretraining mein use hoti hai.

SFT aur pretraining mein sirf kaunsi do cheezein alag hain?

Data (curated demonstrations) aur loss mask (response-only, prompt masked out).

SFT ke dauran prompt tokens ko kyun mask karte hain?

Prompt user-diya input hai; hum sirf model ko jawaab generate karna sikhana chahte hain, sawaal predict karna nahi.

Ek example ke liye SFT loss likhiye.

L = - \sum_{t} m_{t} lo g π_{θ} (y_{t} ∣ x, y_{< t})

(

\sum_{t} m_{t}

se normalize kiya), mask

m_{t} = 1

response tokens par.

π_{θ} (y ∣ x)

ko per-token conditionals ke product mein kyun factorize karte hain?

Probability ka chain rule; LLMs autoregressive hain toh woh natively

π_{θ} (y_{t} ∣ x, y_{< t})

output karte hain.

SFT kaunse learning paradigm ki example hai?

Behavioral cloning / imitation learning.

Kya SFT primarily naya knowledge add karta hai ya behavior shape karta hai?

Yeh primarily behavior aur format shape karta hai; knowledge pretraining se aata hai. SFT se naye facts inject karne se hallucination hoti hai.

EOS token ko response mask mein kyun shamil karte hain?

Taaki model seekhe ki generation kab rokni hai.

Pehle kya aata hai, SFT ya RLHF/DPO?

SFT pehle, phir preference optimization (RLHF/DPO).

LIMA-style insight ke anusaar, SFT data ke liye sabse zyada kya matter karta hai?

Quality over quantity — kuch hazaar excellent demos bade noisy datasets ko beat kar sakte hain.

Connections

Pretraining of LLMs — base model aur identical loss form provide karta hai.
Cross-entropy loss — SFT ka mathematical core.
Autoregressive language modeling — kyun hum token-by-token factorize karte hain.
RLHF aur Direct Preference Optimization (DPO) — alignment steps jo usually SFT ke baad aate hain.
Instruction Tuning — instruction-style demonstrations par SFT.
LoRA / PEFT — SFT ko saste mein chalane ke parameter-efficient tarike.
Chat templates & special tokens — masking ke liye prompt/response boundaries kaise mark ki jaati hain.