4.3.14 · D5Pretraining & Fine-Tuning LLMs
Question bank — Knowledge distillation
The vocabulary here is built in the parent note: teacher (big pretrained model), student (small model being trained), soft targets (teacher's full softened probability list), hard targets (one-hot ground-truth), dark knowledge (the similarity structure hiding in the small non-winning probabilities), and temperature (the knob in softmax with temperature that flattens the distribution).
True or false — justify
True or false: The soft targets and hard targets always agree on which class is most likely.
True in the normal case — softmax preserves ranking, so the teacher's argmax equals the one-hot class when the teacher is correct. But a teacher can be wrong, and then its argmax disagrees with the hard label; that is exactly why we keep the hard-label term.
True or false: Raising temperature changes which class the teacher considers most probable.
False. Dividing every logit by the same shrinks the gaps but never reorders them, so the argmax is unchanged; only the confidence spread flattens.
True or false: At the soft targets become maximally informative.
False. As every probability approaches the uniform , erasing all inter-class structure — the dark knowledge disappears into a flat line. Usefulness peaks at a moderate (often 2–10), not at the extreme.
True or false: If the teacher is already near one-hot, distillation at still beats plain training.
False (or barely true). At a confident teacher's soft targets the hard labels, so the student gains almost nothing new — you must warm it up with to expose the buried similarities.
True or false: The factor changes the location of the loss minimum.
False. is a constant multiplier on the soft term; it rescales the gradient magnitude to match the hard-target gradient, keeping the two objectives balanced, but a constant scale does not move where the loss is minimised.
True or false: Distillation can only compress a model, never improve accuracy over training from scratch.
False. Because soft targets carry richer signal per example, a distilled student often generalises better than the same architecture trained on hard labels alone — compression is the motive, but accuracy gains are a common bonus.
True or false: A student can be distilled from a teacher of a completely different architecture.
True for response-based (logit) distillation — you only need matching output vocabularies/classes. Feature-based distillation is stricter because it aligns internal hidden states, which usually assumes compatible layer shapes.
True or false: KD requires ground-truth labels for the training data.
False. Pure soft-label distillation needs only the teacher's outputs, so you can distill on unlabelled data. The hard-label term is optional — it just keeps the student honest when labels exist.
Spot the error
"We set for both training and inference, so the deployed student runs at ."
Error: temperature is a training-time device to soften targets. At inference the student uses (normal softmax); leaving would give an artificially flat, less confident deployment distribution.
"We forgot the factor but the loss still went down, so it doesn't matter."
Error: without the soft-loss gradient is smaller, so at high distillation is silently turned off — the loss decreases mostly from the hard term while the teacher's knowledge is ignored.
"Soft targets are just label smoothing, both spread probability off the true class."
Error: label smoothing spreads mass uniformly (no relational info), while KD's soft targets are structured — they say which classes are similar. That structure is the entire point.
"We distill the teacher's soft targets at but score the hard-label term also at ."
Error: the hard-label cross-entropy must be computed at , using the student's real probabilities against the true class. Softening the hard term would blur the ground-truth signal.
"Higher temperature is always better because it reveals more dark knowledge."
Error: past a point over-flattens toward uniform and destroys the similarity ranking as it shrinks all gaps. There is a sweet spot; monotonically raising eventually hurts.
"Our teacher never saw this domain, but any teacher gives valid soft labels, so it's fine."
Error: an out-of-domain teacher's soft labels are essentially noise — its confidence structure is meaningless there. Distillation data must match the teacher's competence, or you transfer garbage.
"We use KL divergence one way and cross-entropy the other, so our two loss terms are inconsistent."
Not necessarily an error: minimising KL over the student differs from cross-entropy only by the teacher's entropy, a constant w.r.t. the student. So they give the same gradients — the soft term is well-defined either way.
Why questions
Why do soft targets carry more information than hard labels?
A one-hot label says only "cat"; the teacher's soft distribution says "mostly cat, somewhat dog, nothing like car" — encoding inter-class similarity (dark knowledge) that the student can learn from.
Why divide logits by rather than multiply the probabilities directly?
Because softmax is exponential: dividing logits inside the exponent uniformly compresses the gaps in a mathematically clean, ranking-preserving way. Rescaling probabilities afterward would break normalisation and distort the shape.
Why does the gradient of the soft loss scale like ?
One factor of comes from the student softmax's temperature, and a second from the chain rule through the logit division ; together they give , which the multiplier cancels.
Why keep the hard-label term at all if soft targets are richer?
The soft term chases the teacher, which can be wrong; the hard term anchors the student to the actual ground truth, keeping it honest and correcting teacher mistakes.
Why do LLMs benefit especially from distillation?
An LLM emits a distribution over ~50k vocabulary tokens at every position — an enormously rich soft target per token, so the student inherits far more structure per training step than in a small-class problem.
Why does sequence-level KD (training on teacher-generated text) work for instruction following?
The teacher's generated sequences implicitly encode its whole policy, so training a student on that text via ordinary instruction tuning transfers behaviour without needing per-token logits.
Why might a distilled student generalise better than one trained on hard labels?
Soft targets act like a rich, per-example regulariser — they tell the student the confidence and similarity structure, which smooths its decision boundaries and reduces overfitting to noisy one-hot labels.
Edge cases
Edge case: what are the soft targets when ?
The distribution sharpens to a one-hot on the argmax logit — identical to hard labels, so all dark knowledge vanishes and distillation degenerates into ordinary training.
Edge case: two classes have exactly equal teacher logits — what does raising do to them?
Nothing to their relative balance; equal logits stay equal at every . Temperature only rescales gaps, so tied classes remain tied.
Edge case: the teacher outputs a perfectly uniform distribution — is distillation useful?
No. A uniform teacher carries zero similarity structure (maximum entropy, no dark knowledge), so the soft term teaches nothing beyond "all classes equally likely."
Edge case: teacher and student have different vocabularies/output spaces — can you do logit KD?
Not directly — response-based KD needs a shared output space to match distributions. You would need a mapping, or fall back to feature-based or sequence-level KD instead.
Edge case: in the distillation loss — what training is this?
Pure soft-target distillation with no hard-label anchor; the student imitates the teacher entirely, which is fine if the teacher is reliable but offers no correction for teacher errors.
Edge case: but — what have you effectively built?
You match the teacher's sharp (near one-hot) probabilities with no hard labels — almost equivalent to training on the teacher's argmax, so most dark knowledge stays hidden and the benefit over hard-label training is minimal.
Edge case: the student is larger than the teacher — is this still distillation?
Yes, mechanically (it's called "self-distillation" or reverse distillation), and it can still help via the regularising soft targets — but it no longer serves the usual compression goal, since the student isn't smaller.
Recall One-line summary of the traps
Temperature reveals structure but never reorders classes; the factor keeps gradients balanced; soft targets are structured (unlike label smoothing); a bad or out-of-domain teacher poisons the whole process; and both extremes of (zero and infinity) destroy dark knowledge.