We start from the only thing an LLM can do: assign probability to a sequence of tokens.
Step 1 — the base model. A model with parameters θ defines
pθ(x1,…,xT)=∏t=1Tpθ(xt∣x<t).Why this step? Autoregressive factorization is exact by the chain rule of probability — no assumption yet.
Step 2 — split a training example. Each IT example is a concatenation
instruction (+input)[prompt tokens P]target[response tokens R].
We want to raise pθ(R∣P), the probability of the right answer given the request.
Why this step? We don't care whether the model can predict the user's instruction — the user writes it. We only reward predicting the response.
Step 3 — the loss. Maximize log-likelihood of the response tokens only:
L(θ)=−∑i∑t:xt∈Rlogpθ(xt∣x<t).Why this step? Maximizing likelihood = minimizing negative log-likelihood = standard cross-entropy. Summing over the dataset i generalizes it.
A single example is templated so every task looks structurally similar:
### Instruction:
Summarize the paragraph in one sentence.
### Input:
The mitochondria is the powerhouse of the cell...
### Response:
Mitochondria produce most of the cell's energy. ← only this is scored
Sources of instruction data:
Human-written (FLAN, Super-NaturalInstructions): high quality, expensive.
Self-generated (Self-Instruct, Alpaca): a strong model bootstraps new instructions → cheap, but inherits the teacher's errors.
What is masked out of the loss, and why? ⟶ the prompt tokens; we only want to shape generation of the response.
IT teaches behavior or knowledge? ⟶ behavior (triggering existing skills).
One dataset property that matters most? ⟶ diversity + response quality.
Order of the pipeline? ⟶ pretrain → instruction tune (SFT) → RLHF/DPO.
Recall Feynman: explain to a 12-year-old
Imagine a kid who has read the entire library and knows tons of stuff, but whenever you ask him something he just keeps reciting more book sentences instead of answering you. Instruction tuning is like showing him a few hundred flashcards: "When someone asks a question, give the answer and then stop." He already knew the answers — now he knows he's supposed to actually reply. We only grade him on the answer part of each flashcard, not on repeating the question.
Dekho, ek base LLM sirf next token predict karta hai — usne internet ka poora text padha hai, to usme knowledge to hai, par usko yeh pata hi nahi ki "jab koi command de, to command follow karni hai." Agar tum use "Translate to French: Hello" doge, to woh aage aur English examples likhna shuru kar sakta hai, kyunki web par aisa pattern common hai. Instruction tuning yahi problem fix karta hai: hum use hazaaron (instruction, sahi jawaab) pairs par fine-tune karte hain, taaki woh behaviour seekh le — "task karo aur ruk jao."
Important baat: instruction tuning naya gyaan nahi sikhata, woh sirf pehle se andar padi skills ko trigger karta hai. Isiliye kuch hazaar achhe examples se hi model ka behaviour badal jaata hai (LIMA paper isi ka proof hai). Agar tum galat facts thoosne ki koshish karoge to model hallucinate karega — kyunki woh "confidently answer do" seekh leta hai bina fact ke.
Training ka asli trick hai masking: loss sirf response tokens par compute hota hai, prompt (instruction) par nahi. Kyun? Kyunki user khud instruction likhta hai — model ko woh generate thodi karni hai. Model ko sirf answer wale part par grade karo. Formula simple cross-entropy hai bas response tokens par restricted.
Pipeline yaad rakho: pehle pretraining, phir instruction tuning (SFT), phir RLHF/DPO. Instruction tuning helpful assistant banata hai, RLHF uski tone aur preferences ko human ke hisaab se tune karta hai. Data ka rule: quality aur diversity > quantity.
