Noesis
4.3.10 · D5Pretraining & Fine-Tuning LLMs

Question bank — Instruction tuning

1,532 words7 min read↑ Back to topic

Before you start, keep three anchors in mind:

  • The loss mask — only response tokens are graded (see Cross-Entropy Loss).
  • IT changes behavior, not stored knowledge (skills come from Pretraining Objective (Next-token prediction)).
  • The pipeline order: pretrain → instruction tune (Supervised Fine-Tuning (SFT)) → RLHF and DPO.

True or false — justify

A raw pretrained model already "knows" how to follow instructions, it just needs a longer prompt.
False — a base model is a next-token predictor; a longer prompt may nudge it, but it has no learned behavior of "do the task then stop." IT installs that behavior with (instruction, response) pairs.
Instruction tuning and supervised fine-tuning are the same objective under the hood.
True in mechanism (both minimize cross-entropy on target tokens), but IT is the special case of SFT where targets are responses to diverse natural-language instructions — that data diversity is what buys zero-shot generalization.
The instruction-tuning loss is just standard next-token prediction.
True in form, false in scope — it is next-token cross-entropy but computed only over response tokens via a mask ; pretraining computes it over every token.
Because IT is supervised imitation, it can only reproduce responses it literally saw.
False — it learns the abstract habit "text after the response marker satisfies the request," so it generalizes to unseen instruction types (FLAN and Zero-shot Generalization).
More instruction examples always lower validation loss and improve behavior.
False — LIMA showed ~1,000 curated examples beat far larger noisy sets; for behavior tuning, diversity and response quality dominate raw quantity.
RLHF must be run before instruction tuning to make the model helpful.
False — order is reversed: IT (imitate good answers) comes first, then RLHF and DPO refines preference, tone, and safety on top.
Masking the prompt just saves compute; the final model is the same either way.
False — masking changes the gradient signal, not merely cost. Grading prompt tokens spends capacity re-learning to predict user text the model never generates, which can degrade alignment.
If two responses are equally correct, IT has no way to prefer one.
True — IT imitates the single target it was given; ranking two correct answers is a preference problem handled by RLHF and DPO, not by cross-entropy imitation.
Self-Instruct data is strictly worse than human-written data.
False as a blanket claim — it is cheaper and scales, and can be competitive; but it inherits the teacher model's errors (Self-Instruct and Alpaca), so quality control matters.

Spot the error

"We averaged the loss over all 24 tokens (20 prompt + 4 response) for cleaner gradients."
Error — you must normalize by the number of response tokens (), not all 24. Dividing by 24 dilutes the gradient with prompt tokens you'll never generate.
"We added 300 fresh factual examples so the model would learn a new API released last week."
Error — a handful of SFT examples can't reliably install new facts; it teaches "confidently answer" without the underlying knowledge, causing hallucination. Use retrieval or continued pretraining for new facts.
"We set for the instruction and for the response, to teach the model good questions."
Error — the mask is backwards. The user writes the instruction; we only grade the model on generating the response, so response tokens get .
"We trained on 50k paraphrases of the same 'summarize this' instruction to be safe."
Error — that is near-duplicate low-diversity data. It over-fits one task and hurts the abstract "obey any instruction" habit that generalization depends on.
"Since IT uses cross-entropy like pretraining, we reused the exact pretraining data pipeline."
Error — the pipeline must apply the Prompt Templates and Chat Formats structure and the response mask; raw web text has neither, so the model never learns the instruction→response mapping.
"The model still rambles after answering, so we need more data."
Likely a formatting error, not a data-quantity one — if targets don't include a clear stop/end token in a consistent template, the model never learns when to stop. Fix the template first.

Why questions

Why do we mask the prompt instead of just weighting it down slightly?
Because predicting the user's own text is not a behavior we ever need; even small weight wastes gradient capacity and can pull the model toward echoing prompts. A hard mask concentrates all learning on generation.
Why can a few thousand examples change behavior so dramatically?
Because IT is steering, not stuffing — the skills already live in the pretrained weights; we only need to trigger the surface-form mapping "instruction → do the task," which is a low-dimensional behavioral change.
Why does sharing one template across translation, QA, and summarization help?
A common ### Instruction … ### Response … structure lets the model learn a single abstraction — "satisfy whatever follows the instruction marker" — so it transfers to instruction types never seen in training.
Why is response quality emphasized over quantity?
Because IT imitates its targets exactly; low-quality responses teach the model to produce confident-but-wrong outputs. The model copies the style and correctness it's shown.
Why does IT come before RLHF rather than replacing it?
IT gives a competent imitator of good answers, which RLHF then optimizes against a preference signal (which of two answers is better) for tone, safety, and alignment — a signal cross-entropy imitation cannot express.
Why does the autoregressive factorization let us "grade only the response"?
Because the product splits cleanly into prompt factors and response factors; keeping only the response-token log-terms () is exactly optimizing , the answer given the request.

Edge cases

What if a training example's response is empty (zero response tokens)?
Then and the masked loss is undefined (division by zero). Such examples must be filtered — there is no behavior to imitate.
What happens at test time when the instruction is a type never seen in training?
The model still switches to "obey mode" because it learned the abstract habit, not memorized task list — this is precisely the zero-shot generalization result of FLAN and Zero-shot Generalization.
What if the prompt is extremely long and the response is one token?
Masking still saves you — loss normalizes over the single response token, so the gradient isn't drowned by hundreds of ignored prompt tokens. Without masking, alignment signal would be nearly zero.
If the base model was never pretrained on a language, can IT teach it that language?
No — IT unlocks existing skills; a language absent from pretraining isn't in the weights to trigger. That requires continued pretraining, not instruction tuning.
What if you instruction-tune but never include a stop/end-of-response marker?
The model may follow instructions but not know when to stop, rambling past the answer. The "follow-and-stop" behavior depends on consistent end markers in the Prompt Templates and Chat Formats.
What if all instructions are diverse but all responses are low quality?
Diversity alone fails — the model learns to attempt any task but produce garbage, since it imitates the (bad) targets. Both diversity and response quality are required.
Can instruction tuning on correct answers still increase hallucination?
Yes — if examples reward "always give a confident answer" even when the correct behavior is "I don't know," the model generalizes overconfidence to questions it can't answer. Include abstention examples.

Recall

Recall One-line self-test
  • Masked out of the loss? ⟶ prompt tokens; we only shape response generation.
  • IT installs behavior or knowledge? ⟶ behavior (triggers existing skills).
  • Data property that dominates? ⟶ diversity + response quality, not quantity.
  • Pipeline order? ⟶ pretrain → IT (SFT) → RLHF/DPO.

Connections

  • Instruction tuning — the parent topic this bank drills.
  • Cross-Entropy Loss — the masked objective every item leans on.
  • Supervised Fine-Tuning (SFT) — IT is a specialized SFT.
  • Pretraining Objective (Next-token prediction) — source of the skills IT unlocks.
  • RLHF and DPO — the preference stage after IT.
  • FLAN and Zero-shot Generalization — the generalization results referenced.
  • Self-Instruct and Alpaca — bootstrapped instruction data trade-offs.
  • Prompt Templates and Chat Formats — the structural glue and stop markers.