Instruction tuning — Hinglish notes

4.3.10 · AI-ML › Pretraining & Fine-Tuning LLMs

Instruction tuning KYA hai?

Key distinctions:

	Pretraining	Instruction tuning	RLHF
Data	raw web text	(instruction, response) pairs	preference rankings
Objective	next-token on everything	next-token on response only	reward-model / PPO / DPO
Seekhta hai	language, world facts	tasks follow karna	human preference alignment

KAISE: training objective, scratch se derive kiya

Hum uss cheez se shuru karte hain jo ek LLM kar sakta hai: token sequence ko probability assign karna.

Step 1 — base model. Parameters $θ$ wala ek model define karta hai $p_{θ} (x_{1}, \dots, x_{T}) = \prod_{t = 1}^{T} p_{θ} (x_{t} ∣ x_{< t}) .$ Yeh step kyun? Autoregressive factorization probability ke chain rule se exact hai — abhi tak koi assumption nahi.

Step 2 — training example split karo. Har IT example ek concatenation hai $instruction (+input) [prompt tokens P] target [response tokens R] .$ Hum $p_{θ} (R ∣ P)$ badhana chahte hain, yaani sahi jawab ki probability given the request. Yeh step kyun? Hume parwah nahi ki model user ki instruction predict kar sake — user khud likhta hai. Hum sirf response predict karne par reward dete hain.

Step 3 — loss. Sirf response tokens ka log-likelihood maximize karo: $L (θ) = - \sum_{i} \sum_{t : x_{t} \in R} lo g p_{θ} (x_{t} ∣ x_{< t}) .$ Yeh step kyun? Likelihood maximize karna = negative log-likelihood minimize karna = standard cross-entropy. Dataset $i$ par sum karke generalize hota hai.

Data recipe (jo 80/20 matter karta hai)

Ek single example template kiya jata hai taaki har task structurally similar lage:

### Instruction:
Summarize the paragraph in one sentence.

### Input:
The mitochondria is the powerhouse of the cell...

### Response:
Mitochondria produce most of the cell's energy.   ← only this is scored

Instruction data ke sources:

Human-written (FLAN, Super-NaturalInstructions): high quality, expensive.
Self-generated (Self-Instruct, Alpaca): ek strong model nayi instructions bootstrap karta hai → sasta, lekin teacher ki errors inherit karta hai.

Worked examples

Common mistakes (steel-manned)

Recall

Recall Active recall — answers cover karo

Loss mein kya mask out kiya jata hai, aur kyun? ⟶ prompt tokens; hum sirf response generation shape karna chahte hain.
IT behavior sikhata hai ya knowledge? ⟶ behavior (existing skills trigger karna).
Ek dataset property jo sabse zyada matter karta hai? ⟶ diversity + response quality.
Pipeline ka order? ⟶ pretrain → instruction tune (SFT) → RLHF/DPO.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho ek bachcha jisne poori library padh li hai aur bahut kuch jaanta hai, lekin jab bhi tum usse kuch poochho woh tumhe jawab dene ki jagah aur book sentences bolne lagta hai. Instruction tuning aise hai jaise use kuch sau flashcards dikhao: "Jab koi sawaal poochhe, jawab do aur ruk jao." Use jawab pehle se pata the — ab use pata hai ki use actually reply karna chahiye. Hum use sirf har flashcard ke jawab wale part par grade karte hain, sawaal dobarana repeat karne par nahi.

Connections

Pretraining Objective (Next-token prediction) — woh skills provide karta hai jo IT unlock karta hai.
Supervised Fine-Tuning (SFT) — instruction tuning ek specialized SFT hai.
RLHF and DPO — IT ke baad wala alignment stage.
Cross-Entropy Loss — woh loss jo IT minimize karta hai.
Self-Instruct and Alpaca — bootstrapped instruction data.
FLAN and Zero-shot Generalization — scaled IT ka empirical origin.
Prompt Templates and Chat Formats — structural glue.

Instruction tuning kis tarah ke data par train karta hai?

Diverse (instruction, optional input, target response) pairs across many tasks.

IT loss mein kaun se tokens mask out hote hain?

Prompt/instruction tokens — loss sirf response tokens par compute hoti hai.

Instruction tuning primarily knowledge add karta hai ya behavior shape karta hai?

Behavior shape karta hai — yeh woh skills trigger karta hai jo pretraining ke dauran already store ho gayi hain; SFT se facts add karne par hallucination hoti hai.

Response-masked cross-entropy objective likho.

L = - \frac{1}{\sum _{t} m _{t}} \sum_{t} m_{t} lo g p_{θ} (x_{t} ∣ x_{< t})

with

m_{t} = 1

on response tokens.

Typical alignment pipeline ka order kya hai?

Pretraining → Instruction tuning (SFT) → RLHF/DPO.

~1000 curated examples (LIMA) itne achhe kyun kaam karte hain?

Kyunki IT behavior steer karta hai facts sikhane ki jagah; diversity aur response quality quantity se zyada matter karti hai.

Instruction tuning aur RLHF mein kya fark hai?

IT diye gaye answers ki supervised imitation hai; RLHF answers compare karke ek preference/reward signal optimize karta hai.

Self-Instruct kya hai?

Ek method jisme ek strong model apna khud ka instruction–response data generate karta hai taaki fine-tuning set cheaply bootstrap ho sake.

Prompt par bhi loss compute kyun nahi karte?

Yeh capacity waste karta hai model ko woh text (user ki instruction) predict karna sikhane mein jo use kabhi generate nahi karni, aur alignment hurt kar sakta hai.