Knowledge distillation: Exercises — AI-ML deep dive

Throughout, we reuse one running vocabulary of size $K$ (number of classes/tokens). A logit $z_{i}$ is just the raw, un-normalised score the network assigns to class $i$ before softmax. Nothing below uses a symbol we have not named.

Level 1 — Recognition

Exercise 1.1

State, in one sentence each, what hard targets, soft targets, and dark knowledge are.

Recall Solution

Hard targets: the one-hot ground-truth label, e.g. $[1, 0, 0]$ — it says only the correct class.
Soft targets: the teacher's full softened probability vector, e.g. $[0.786, 0.175, 0.039]$ — every class gets a share.
Dark knowledge: the relational information carried by the non-largest probabilities — "a cat looks more like a dog than like a car," which the hard label $[1, 0, 0]$ cannot express.

Exercise 1.2

Write the temperature-softmax formula and say what happens in the two limits $T \to \infty$ and $T \to 0$ .

Recall Solution

$p_{i} (T) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}$

$T \to \infty$ : every logit is divided by a huge number, so all $z_{i} / T \to 0$ , so $e^{z_{i} / T} \to 1$ , and every class gets $1/ K$ — the uniform distribution.
$T \to 0$ : the largest logit's exponent dominates everything else, so the distribution sharpens to a one-hot at the argmax.

Level 2 — Application

Exercise 2.1

Teacher logits $z = [3, 1, 0]$ over ${$ cat, dog, car $}$ . Compute the softmax at $T = 1$ .

Recall Solution

$e^{3} = 20.086, e^{1} = 2.718, e^{0} = 1$ . Sum $= 23.804$ . $p = [\frac{20.086}{23.804}, \frac{2.718}{23.804}, \frac{1}{23.804}] = [0.844, 0.114, 0.042] .$ Cat dominates; dog and car are already partly visible because these logits are not extreme.

Exercise 2.2

Now soften the same logits at $T = 3$ . Compare with 2.1 — did the ordering change? Did the gaps?

Recall Solution

Divide logits by $3$ : $[1, 0.333, 0]$ . $e^{1} = 2.718, e^{0.333} = 1.395, e^{0} = 1$ . Sum $= 5.113$ . $p (3) = [0.5316, 0.2729, 0.1956] .$ Ordering: cat > dog > car — unchanged (it always is; temperature never reorders). Gaps: hugely reduced — cat fell from $0.844$ to $0.53$ , car rose from $0.042$ to $0.20$ . The similarity structure is now loud.

Exercise 2.3

Given student softmax $q (1) = [0.60, 0.30, 0.10]$ and true label = cat, compute the hard cross-entropy loss $L_{hard} = - \sum_{i} y_{i} lo g q_{i} (1)$ .

Recall Solution

The one-hot label is $y = [1, 0, 0]$ , so only the cat term survives: $L_{hard} = - ln (0.60) = 0.5108.$

Level 3 — Analysis

Exercise 3.1

Using the softened teacher $p (3) = [0.5316, 0.2729, 0.1956]$ from 2.2 and student soft probs $q (3) = [0.50, 0.30, 0.20]$ , compute the soft cross-entropy $L_{soft} = - \sum_{i} p_{i} lo g q_{i}$ .

Recall Solution

$L_{soft} = - (0.5316 ln 0.50 + 0.2729 ln 0.30 + 0.1956 ln 0.20) .$ $ln 0.50 = - 0.6931, ln 0.30 = - 1.2040, ln 0.20 = - 1.6094$ . $= - (0.5316 (- 0.6931) + 0.2729 (- 1.2040) + 0.1956 (- 1.6094))$ $= - (- 0.3685 - 0.3286 - 0.3148) = 1.0119.$

Exercise 3.2

With $α = 0.5$ , $T = 3$ , $L_{soft} = 1.0119$ (Ex 3.1) and $L_{hard} = 0.5108$ (Ex 2.3), compute the total distillation loss $L = α T^{2} L_{soft} + (1 - α) L_{hard}$ . Then recompute it wrongly without the $T^{2}$ and state how badly the soft term shrinks.

Recall Solution

Correct: $L = 0.5 \cdot 3^{2} \cdot 1.0119 + 0.5 \cdot 0.5108 = 4.5 \cdot 1.0119 + 0.2554 = 4.5536 + 0.2554 = 4.8090.$ Without $T^{2}$ (the bug): $L_{bug} = 0.5 \cdot 1.0119 + 0.2554 = 0.5060 + 0.2554 = 0.7614.$ The soft contribution collapses from $4.55$ to $0.51$ — a factor of $T^{2} = 9$ smaller. The optimiser would then almost ignore the teacher and just fit hard labels. This is exactly why the $T^{2}$ factor exists.

Exercise 3.3

Explain why the soft-loss gradient scales like $1/ T^{2}$ , referencing the softmax cross-entropy gradient $\partial L / \partial z_{i} = q_{i} - p_{i}$ .

Recall Solution

Two independent factors of $1/ T$ appear:

From the student softmax argument. Each logit enters as $z_{i} / T$ , so differentiating $q_{i} (T)$ with respect to the raw logit $z_{i}$ brings down a chain-rule factor $1/ T$ . This gives $\frac{\partial L _{soft}}{\partial z _{i}} = \frac{1}{T} (q_{i} (T) - p_{i} (T))$ .
From the difference $q_{i} (T) - p_{i} (T)$ itself. For small logits, Taylor-expand $e^{z / T} \approx 1 + z / T$ . Then $q_{i} (T) - p_{i} (T) \approx \frac{1}{K T} (z_{i} - v_{i})$ where $v_{i}$ are teacher logits — a second $1/ T$ . Multiplying the two: overall gradient $\sim 1/ T^{2}$ . Scaling $L_{soft}$ by $T^{2}$ cancels this shrinkage so the soft and hard gradients keep comparable magnitude. See Cross-Entropy and KL Divergence for the base gradient identity.

Level 4 — Synthesis

Exercise 4.1

A giant LLM predicts over a vocabulary of $K = 50000$ tokens at every position of a 512-token sequence. Argue, with numbers, why sequence-level logit distillation is enormously more informative per training example than hard next-token labels.

Recall Solution

Hard labels: each position gives one integer (the true next token) — $512$ scalars of signal per sequence.
Soft logit targets: each position gives a full distribution over $50000$ tokens. That is $512 \times 50000 = 25600000$ probability values per sequence.
So the soft signal is $\sim 50000 \times$ richer per position (before compression). Even after the teacher concentrates most mass on a few tokens, the tail encodes which alternative continuations are plausible — grammatical, semantic, and stylistic neighbours. This is why DistilBERT and instruction-distilled models learn so efficiently: one forward pass of a teacher yields a dense target where a hard label yields one bit of "correct token."

Exercise 4.2

Design a distillation recipe for compressing a 7B-parameter chat model into a 1.3B student that must follow instructions. Which KD variant, temperature, and loss mix would you choose, and why? Name one prerequisite topic you rely on.

Recall Solution

A defensible recipe (many valid answers — grade on justification):

Variant: sequence-level KD. Let the teacher generate high-quality instruction–response pairs, then train the student on that generated corpus (Alpaca-style). Rationale: instruction following is about producing coherent sequences, so imitating the teacher's actual outputs transfers behaviour, not just per-token marginals. Ties directly to Instruction Tuning.
Optionally add response-based logit KD on the same data: match next-token distributions at $T \approx 2$ so the student inherits calibrated token uncertainty, not just the greedy path.
Loss mix: $L = α T^{2} L_{soft} + (1 - α) L_{hard}$ with $α \approx 0.5$ , $T = 2$ – $4$ . Keep the hard term so the student stays anchored to real targets and doesn't drift into teacher hallucinations.
Prerequisite leaned on: Transfer Learning — the student is warm-started from a pretrained checkpoint, then distilled, rather than trained from scratch.

Level 5 — Mastery

Exercise 5.1

Prove that as $T \to \infty$ , the softened softmax of any finite logit vector $z$ converges to the uniform distribution $1/ K$ , and interpret what this means for the usefulness of distillation at extreme temperature.

Recall Solution

Write $p_{i} (T) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}$ . As $T \to \infty$ , every $z_{i} / T \to 0$ , so $e^{z_{i} / T} \to e^{0} = 1$ . Hence the numerator $\to 1$ and the denominator $\to \sum_{j = 1}^{K} 1 = K$ . Therefore $lim_{T \to \infty} p_{i} (T) = \frac{1}{K} for every i .$ Interpretation: at extreme $T$ the teacher's distribution is indistinguishable from a coin flip over all classes — it carries zero relative information. The student learns nothing about class structure. This bounds how high you may push $T$ : dark knowledge lives at moderate softening, then evaporates. Confirms the L3 trap's warning quantitatively.

Exercise 5.2

Show that KD's soft target and label smoothing produce different targets even when they happen to share the same top-1 probability, and explain why only one of them transfers dark knowledge. Use logits $z = [3, 1, 0]$ (from Ex 2.1) at $T = 1$ as the KD target and a label-smoothing target with smoothing $ε = 0.156$ over $K = 3$ classes.

Recall Solution

KD soft target (Ex 2.1): $p_{KD} = [0.844, 0.114, 0.042]$ — cat highest, then dog, then car. The ordering $0.114 > 0.042$ encodes "cat resembles dog more than car." Label smoothing target: $y_{i} = (1 - ε) 1 [i = cat] + ε / K$ . $y = [(1 - 0.156) + 0.156/3, 0.156/3, 0.156/3] = [0.896, 0.052, 0.052] .$ Both put most mass on cat, but label smoothing gives dog and car the identical $0.052$ — it is uniform over the wrong classes and says nothing about similarity. KD gives them $0.114$ vs $0.042$ — a structured split. Conclusion: only KD carries dark knowledge; label smoothing is a flat regulariser. Matching top-1 mass is not enough; the shape of the tail is what teaches relationships.

Recall Final self-check

Did $T$ ever change the ranking of classes in any exercise? ::: No — temperature only changes the spread/confidence, never the order. In Ex 3.2, by what factor did dropping $T^{2}$ shrink the soft loss? ::: By $T^{2} = 9$ . What single property distinguishes KD's soft targets from label smoothing? ::: KD's tail is structured (encodes class similarity); label smoothing's tail is uniform.