Exercises — Instruction tuning

Answer: (b).

• (a) is pretraining data — raw text, no task structure.
• (c) is preference data used by RLHF and DPO.
• (b) is (instruction, desired response), exactly the format instruction tuning consumes.

False. The loss is computed only over response tokens (the ones with $m_t=1$). The prompt is masked out because the user supplies it — we never need the model to generate it.

Pretraining → Instruction tuning → RLHF. First learn language + world facts, then learn to follow instructions, then refine preferences/tone/safety.

There are 11 tokens. Only the final token "5" is the response. $m=[0,0,0,0,0,0,0,0,0,0,1].$ The loss uses only the bright position: $\mathcal{L}=-\log p_{\theta }(\text{"5"}\mid \text{all 10 preceding tokens}).$ Everything before "5" is context the model reads but is not graded on generating.

Only the $4$ response tokens count, so ${\sum }_t{m}_t=4$. $\mathcal{L}=-\frac{1}{4}(\log 0.5+\log 0.5+\log 0.5+\log 0.5)=-\log 0.5=\ln 2\approx \mathrm{0.693.}$ Notice the normalizer is 4, not 24 — the prompt length is irrelevant to the averaged loss because masked terms vanish.

${\mathcal{L}}_{\text{unmasked}}=-\frac{1}{24}(20\log 0.1+4\log 0.5).$ $=-\frac{1}{24}(20(-2.302585)+4(-0.693147))=-\frac{1}{24}(-46.0517-2.7726)=\frac{48.8243}{24}\approx \mathrm{2.034.}$ Compare to the masked value $0.693$: the unmasked number is dominated by the $20$ prompt tokens the model will never need to generate. Its gradient is wasted on echoing the user.

The three task families share a common surface structure (the template) even though their content differs. During training the gradient repeatedly rewards the pattern:

"whatever tokens appear right after ### Response: must satisfy the request in the instruction, then stop."

That habit is abstract — it does not depend on the task being translation vs. summarization. At test time the haiku prompt has the same surface structure, so the learned habit fires: the model produces content that satisfies the new instruction. This is the zero-shot generalization result — generalization across instruction types, driven by shared template structure.

Right: for behavior tuning, quality + diversity dominate raw count. The skills already live in the pretrained weights; you are steering, not stuffing (from the parent's "80/20"). A few clean, diverse examples can flip behavior. Wrong: "bigger is useless" overgeneralizes. Scale still helps if the added data adds diversity or coverage of new instruction types. What LIMA actually shows is diminishing returns from near-duplicate, low-quality data — not that data is worthless. Ten thousand paraphrases of the same instruction add little; one thousand genuinely different tasks add a lot.

Because the loss makes the model imitate the given responses, it learns the style ("state an answer confidently, then stop") independent of correctness. It will produce fluent, assertive answers that are often false — the failure mode is hallucination. Instruction tuning shapes form, and here the form being taught is "sound confident regardless of truth."

A defensible design — one example each from 6 distinct families, all in one shared template:

1. Translation — Translate to French: Hello → Bonjour
2. Summarization — Summarize in one sentence: <para> → <1 sentence>
3. Arithmetic/QA — What is 12 × 7? → 84
4. Rewriting — Make this polite: "give me the file" → "Could you please share the file?"
5. Classification — Sentiment of "I loved it": → Positive
6. Generation — Write a two-line poem about rain → <2 lines>

Why diversity matters: if all 6 were translation, the model would tie "obey-mode" to the translation content and fail to fire on a haiku. Spreading across families forces the gradient to attribute the shared behavior to the shared template, not the content — which is what enables zero-shot transfer. This is exactly the Self-Instruct philosophy of covering many instruction types cheaply.

Example A ($\sum m=2$): ${\mathcal{L}}_A=-\frac{1}{2}(\log 0.5+\log 0.5)=-\log 0.5=\ln 2\approx \mathrm{0.6931.}$ Example B ($\sum m=3$): ${\mathcal{L}}_B=-\frac{1}{3}(\log 0.5+\log 0.25+\log 0.5)=-\frac{1}{3}(-0.6931-1.3863-0.6931)=\frac{2.7726}{3}\approx \mathrm{0.9242.}$ Batch loss: $\mathcal{L}=\frac{1}{2}(0.6931+0.9242)\approx \mathrm{0.8087.}$ Note each example is normalized by its own response length first — a long-response example does not silently dominate a short one.

Recall the autoregressive factorization from the parent: $p_{\theta }(x_1,\dots ,x_T)={\prod }_{t=1}^T{p}_{\theta }(x_t\mid x_{<t}).$ Nothing about this equation changed — the model is still a next-token predictor. What changed is the conditional distribution $p_{\theta }(x_t\mid \text{prompt})$ for the first response token after the instruction.

• Before IT: conditioned on that prompt, the highest-probability next tokens were English continuation tokens (because "list of examples" is a common web pattern from pretraining).
• After IT: the masked cross-entropy repeatedly raised $p_{\theta }(\text{"Bonjour"}\mid \text{prompt})$ and lowered the English-continuation mass, so the argmax flipped.

Was French vocabulary newly learned? No. The token Bonjour and its association with "Hello"/"French" were already in the weights from multilingual pretraining. Instruction tuning only re-weighted which continuation the prompt should trigger. It changed behavior (what to output), not knowledge (what the model knows). This is the parent's "FORM over FILL."

Ranking: (ii) > (i) > (iii).

• (ii) best. Diversity forces the shared behavior onto the template, not the content (L3.1, L4.1), maximizing zero-shot transfer — the FLAN result. Clean responses teach correct form.
• (i) middle. Clean and high-quality, so it won't teach hallucination, but the model ties obey-mode to translation; it may fail to fire on unseen task types (poor generalization). Quantity without diversity gives diminishing returns (L3.2 / LIMA).
• (iii) worst. Even though it's diverse, imitating confidently-wrong answers teaches the model to hallucinate confidently (L3.3). Bad responses actively teach bad behavior — worse than narrow-but-clean, because the damage transfers to all task types.

The unifying principle: instruction tuning imitates responses, so it copies both the good habit (obey the template) and any bad habit (be confidently wrong). Maximize diversity, guarantee response quality, then hand off to RLHF and DPO to refine preferences on top.

• L2.2 masked loss with all response probs $=0.5$? ::: $\ln 2\approx 0.693$
• Why is the normalizer 4 and not 24? ::: masked prompt terms are multiplied by $m_t=0$ and vanish
• What does IT change in $p_{\theta }(x_t\mid x_{<t})$? ::: the conditional after the prompt — behavior, not the factorization
• Best 2000-example strategy? ::: diverse + clean across many task families