Knowledge distillation — Hinglish notes

4.3.14 · AI-ML › Pretraining & Fine-Tuning LLMs

Knowledge distillation KYA hai?

Soft targets zyada information kyun carry karte hain? {cat, dog, car} ke upar ek image classifier consider karo. Cat ki photo ke liye hard label hai $[1, 0, 0]$ . Lekin teacher output kar sakta hai $[0.9, 0.099, 0.001]$ . Yeh student ko woh baat bataata hai jo hard label nahi bata sakta: "ek cat ek car se kahin zyada ek dog jaisi dikhti hai." Yeh inter-class similarity structure dark knowledge kehlaata hai.

Mechanism: temperature ke saath softmax

Standard softmax logits $z_{i}$ ke upar: $p_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$

Temperature $T$ ke saath softmax: $p_{i} (T) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}$

$T = 1$ : normal softmax.
$T \to \infty$ : saare logits ek bade number se divide hote hain $\Rightarrow$ probabilities $\to$ uniform $1/ K$ .
$T \to 0$ : one-hot (argmax) ki taraf sharpen ho jaata hai.

Loss function — pehle principles se derive kiya hua

Hum chahte hain ki student ek saath do kaam kare:

(1) Teacher ke soft targets match kare. Teacher soft probs $p^{T} (T)$ aur student soft probs $q^{T} (T)$ ke beech cross-entropy (equivalently KL divergence) use karo, dono temperature $T$ par: $L_{soft} = - \sum_{i} p_{i}^{T} (T) lo g q_{i}^{T} (T)$

(2) Sahi answer phir bhi deta rahe. Hard labels $y$ ke saath $T = 1$ par standard cross-entropy: $L_{hard} = - \sum_{i} y_{i} lo g q_{i} (1)$

Inhe ek mixing weight $α$ ke saath combine karo:

$T^{2}$ factor kyun? (Yeh woh step hai jise sab bhool jaate hain.) Jab aap softmax-with-temperature ke through backprop karte ho, toh $L_{soft}$ ka gradient student logits ke w.r.t. $1/ T^{2}$ ki tarah scale karta hai (ek $1/ T$ student softmax se, ek logit division ke through chain rule se). Soft-target gradient ki magnitude ko hard-target gradient ke comparable rakhne ke liye, hum $T^{2}$ se multiply karte hain. Iske bina, $T$ badhane se soft loss silently irrelevance mein chali jaati hai.

Gradient sketch (WHY $1/ T^{2}$ ): Softmax cross-entropy ke liye, $\partial L / \partial z_{i} = q_{i} - p_{i}$ . Temperature ke saath, student prob $q_{i} (T)$ hai aur har logit $1/ T$ se scaled hai, isliye $\frac{\partial L _{soft}}{\partial z _{i}} = \frac{1}{T} (q_{i} (T) - p_{i} (T)) .$ Chhote logits ke liye (Taylor expand $e^{z / T} \approx 1 + z / T$ ), $q_{i} (T) - p_{i} (T) \approx \frac{1}{K T} (z_{i} - v_{i})$ jahan $v_{i}$ teacher logits hain, jo overall $\sim 1/ T^{2}$ deta hai. Loss ko $T^{2}$ se multiply karne se yeh cancel ho jaata hai. ✅

Worked example 1 — ek distribution ko soften karna

Teacher logits $z = [4, 1, - 2]$ {cat, dog, car} ke upar.

$T = 1$ par: $e^{4} = 54.6, e^{1} = 2.72, e^{- 2} = 0.135$ , sum $= 57.5$ . $p = [0.950, 0.047, 0.0023]$ . Yeh step kyun? Confidence "cat" par crush ho gayi hai; dog/car ki info almost invisible hai.

$T = 2$ par: logits ban jaate hain $[2, 0.5, - 1]$ . $e^{2} = 7.39, e^{0.5} = 1.65, e^{- 1} = 0.368$ , sum $= 9.40$ . $p = [0.786, 0.175, 0.039]$ . Yeh step kyun? Ab "dog" clearly non-trivial 17.5% hai — student seekh sakta hai ki cats dogs jaisi hoti hain. Yahi dark knowledge surface ho raha hai.

Worked example 2 — combined loss compute karna

Maano $α = 0.5$ , $T = 2$ . Student soft probs $q^{T} (2) = [0.70, 0.25, 0.05]$ , teacher $p^{T} (2) = [0.786, 0.175, 0.039]$ , true label = cat, student normal probs $q (1) = [0.80, 0.15, 0.05]$ .

Soft term: $L_{soft} = - (0.786 ln 0.70 + 0.175 ln 0.25 + 0.039 ln 0.05)$ $= - (0.786 (- 0.357) + 0.175 (- 1.386) + 0.039 (- 3.00)) = 0.640$ .

Hard term: $L_{hard} = - ln (0.80) = 0.223$ .

Total: $L = 0.5 \cdot 2^{2} \cdot 0.640 + 0.5 \cdot 0.223 = 1.280 + 0.112 = 1.392$ . Yeh step kyun? $T^{2} = 4$ scaling soft term ko yahan dominate karaata hai — yahi iska purpose hai: student ko teacher ki structure absorb karne par force karo jabki hard label use honest rakhta hai.

LLMs ke liye specifically KD

Common LLM KD variants:

Response-based (logit) KD: next-token logit distributions match karo. ← classic Hinton KD.
Feature-based KD: intermediate hidden states/attention maps match karo (student seekhta hai ki teacher kaise sochta hai, na sirf kya output karta hai).
Sequence-level KD: teacher text generate karta hai, student us generated text pe data ke roop mein train hota hai (instruction-following distillation ke liye heavily use hota hai, jaise Alpaca-style).

Common mistake Common errors ko steel-man karna

Mistake 1: "Bas $T = 1$ use karo, kyun complicate karo?" Kyun sahi lagta hai: $T = 1$ woh "real" probability hai jise teacher actually believe karta hai. Fix: $T = 1$ par teacher near one-hot hota hai, toh soft targets ≈ hard targets aur almost kuch gain nahi hota. Dark knowledge expose karne ke liye $T > 1$ (often 2–10) chahiye.

Mistake 2: $T^{2}$ factor bhool jaana. Kyun sahi lagta hai: Loss "theek lagta hai" aur phir bhi decrease hota hai. Fix: $T^{2}$ ke bina, soft-loss gradient $\sim 1/ T^{2}$ chhota hota hai, toh zyada temperatures accidentally distillation ko band kar dete hain. Hamesha scale karo.

Mistake 3: Ek aisi teacher se distill karna jo zaroorat se kamzor ho, ya galat data par. Kyun sahi lagta hai: Koi bhi teacher soft labels deta hai. Fix: Ek bekar teacher bekar dark knowledge transfer karta hai; aur agar aap sirf us data par distill karte ho jise teacher ne acche se kabhi nahi dekha, toh uske soft labels noise hain. Distillation data ko teacher competence se match karo.

Mistake 4: Soft targets aur label smoothing ko confuse karna. Kyun sahi lagta hai: Dono non-one-hot targets produce karte hain. Fix: Label smoothing probability uniformly spread karta hai (classes similar hain ya nahi iska koi info nahi). KD ke soft targets structured hote hain — woh structure hi poora point hai.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho ek genius chef (teacher) aur ek student cook. Agar chef sirf yeh kahe "yeh dish 'pizza' hai," toh student thoda seekhta hai. Lekin agar chef kahe "yeh zyaatar pizza hai, kuch had tak calzone jaisa hai, aur soup jaisa bilkul nahi," toh student kahin zyada seekhta hai — including yeh ki khaane kaise relate karte hain ek doosre se. Knowledge distillation chhote model ko bade model ki poori raay use karke sikhaana hai, na sirf uska final answer. "Temperature" aise hai jaise chef se kaho ki slow down karo aur subtle in-between thoughts share karo instead of seedha answer bolne ke.

Recall checkpoints

Flashcards

Knowledge distillation kya hai?

Ek chhote student model ko train karna taaki woh ek bade teacher model ki softened output distribution (soft targets) imitate kare, na ki sirf hard labels.

"Soft targets" kya hote hain?

Teacher ki classes ke upar poori probability distribution (temperature se soften ki hui), jo inter-class similarity info carry karta hai.

"Dark knowledge" kya hai?

Woh relational information jo teacher ki non-argmax probabilities mein chupi hoti hai (jaise cat dog se zyada milti hai car se nahi).

Temperature ke saath softmax likho.

p_{i} (T) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}

T \to \infty

hone par kya hota hai?

Distribution uniform ho jaata hai (har ek

1/ K

T \to 0

hone par kya hota hai?

Distribution one-hot (argmax) ki taraf sharpen ho jaata hai.

Distillation loss likho.

L = α T^{2} L_{soft} + (1 - α) L_{hard}

T^{2}

factor kyun?

Soft-loss gradient

1/ T^{2}

ki tarah scale karta hai;

T^{2}

se multiply karne se uski magnitude hard-loss gradient ke comparable rehti hai.

T = 1

kyun use nahi karte?

Ek trained teacher

T = 1

par near one-hot hota hai, isliye soft targets hard targets mein collapse ho jaate hain aur dark knowledge chupi rehti hai.

KD vs label smoothing?

Label smoothing uniform noise add karta hai (koi structure nahi); KD soft targets encode karte hain ki kaun si classes actually similar hain.

Teen LLM distillation variants batao.

Response/logit-based, feature-based (hidden states/attention), sequence-level (teacher-generated text par train karo).

Student ko factually correct kaun sa term rakhta hai?

Hard-label cross-entropy term

L_{hard}

(1 - α)

se weighted hota hai.

Connections

Softmax and Temperature
Model Compression
Pruning and Quantization
Cross-Entropy and KL Divergence
DistilBERT
Instruction Tuning
Transfer Learning