Knowledge distillation: Worked examples — AI-ML deep dive

Everything here builds on three tools from the parent, restated in plain words so no symbol is unearned:

Definition The symbols we reuse

Logit $z_{i}$ — a raw, unnormalised score the model assigns to option $i$ before it is turned into a probability. Think "vote count," can be any real number, positive or negative.
Temperature $T$ — a knob (a positive number) we divide every logit by. Large $T$ = "flatten the votes"; small $T$ = "exaggerate the winner." See Softmax and Temperature.
Mixture weight $α$ — a number between $0$ and $1$ that decides how much the student listens to the teacher vs the true label. $α = 1$ means "pure imitation of the teacher"; $α = 0$ means "ignore the teacher, learn only from ground truth"; in between is a blend. It weights the soft loss by $α$ and the hard loss by $(1 - α)$ .
Softmax with temperature — the machine that turns logits into probabilities: $p_{i} (T) = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}$ Here $e$ is Euler's number ( $\approx 2.718$ ); $e^{x}$ is "how big $x$ grows when exponentiated" — it is always positive, which is why every probability comes out positive and they sum to 1.

The scenario matrix

Every distillation calculation lives in one of these cells. The examples below are labelled with the cell they cover.

#	Case class	What's special	Example
A	Baseline $T = 1$	normal softmax, no softening	Ex 1
B	Warming $T > 1$	soft targets surface (positive & negative logits)	Ex 2
C	Limit $T \to \infty$	distribution $\to$ uniform	Ex 3
D	Limit $T \to 0^{+}$	distribution $\to$ one-hot (argmax)	Ex 3
E	Degenerate teacher — all logits equal	zero dark knowledge to transfer	Ex 4
F	Full loss, teacher-agrees	$L_{soft}$ + $L_{hard}$ , both point same way	Ex 5
G	Full loss, teacher-disagrees	teacher's argmax $\neq =$ true label	Ex 6
H	$α$ extremes $α = 0, α = 1$	pure hard / pure soft training	Ex 7
I	The $T^{2}$ audit	show gradient shrinks without $T^{2}$	Ex 8
J	Word problem (LLM vocab)	huge $K$ , next-token distribution	Ex 9
K	Exam twist — negative-logit shift invariance	trap: does adding a constant change $p$ ?	Ex 10

Case A & B — softening across signs

Look at the s01 figure: the burnt-orange bars ( $T = 1$ ) are spiky; the teal bars ( $T = 3$ ) are flattened — same order, softer spread.

Cases C & D — the two limits

Worked example Example 3 (Cells C & D) — pushing

$T$ to both extremes Same logits $z = [3, 0, - 3]$ . Evaluate $T \to \infty$ and $T \to 0^{+}$ .

Forecast: guess the two limiting distributions before computing.

$T$ very large ( $T = 100$ ). Why? To see the $T \to \infty$ behaviour numerically. Effective logits $[0.03, 0, - 0.03]$ ; $e^{0.03} = 1.0305, 1, e^{- 0.03} = 0.9704$ . Sum $= 3.0009$ . $p = [0.3434, 0.3332, 0.3234]$ .
Interpret. As $T \to \infty$ every logit $/ T \to 0$ , so $e^{z_{i} / T} \to 1$ for all $i$ , giving $p_{i} \to \frac{1}{K} = \frac{1}{3} = 0.3333$ . The $T = 100$ numbers already sit within $0.01$ of uniform. ✅
$T$ very small ( $T = 0.1$ ). Why? To see $T \to 0^{+}$ . Effective logits $[30, 0, - 30]$ ; the largest exponent dominates completely: $p \approx [1, 0, 0]$ (numerically $e^{30}$ swamps the rest).
Interpret. As $T \to 0^{+}$ softmax collapses onto the argmax — a one-hot vector. This is why training usually uses $T = 1$ and distillation uses $T > 1$ .

Verify: uniform limit $= 1/3 = 0.3333$ ; argmax limit picks index 0 (logit 3, the largest). Both match the parent's stated limits. ✅

The s02 figure shows three panels — cold ( $T = 0.1$ , spike), warm ( $T = 3$ ), hot ( $T = 100$ , flat) — the whole temperature spectrum on one strip.

Case E — the degenerate teacher

Cases F & G — the full combined loss

Recall the parent's distillation loss (with $α$ the mixture weight defined above): $L = α T^{2} L_{soft} + (1 - α) L_{hard}$ where $L_{soft} = - \sum_{i} p_{i} lo g q_{i}$ (teacher $p$ vs student $q$ , both at temperature $T$ ) and $L_{hard} = - lo g q_{true} (1)$ . See Cross-Entropy and KL Divergence for why cross-entropy is the right "distance."

Worked example Example 5 (Cell F) — teacher and true label AGREE

$α = 0.5, T = 2$ . Teacher soft probs $p = [0.70, 0.20, 0.10]$ (argmax = cat), true label = cat, student soft probs $q = [0.60, 0.30, 0.10]$ , student normal prob for cat $q_{cat} (1) = 0.75$ .

Forecast: will the $T^{2} = 4$ factor make the soft term dominate?

Soft cross-entropy. Why? It measures how far student's soft distribution is from teacher's. $L_{soft} = - (0.70 ln 0.60 + 0.20 ln 0.30 + 0.10 ln 0.10)$ . Compute: $0.70 (- 0.5108) = - 0.3576$ , $0.20 (- 1.2040) = - 0.2408$ , $0.10 (- 2.3026) = - 0.2303$ ; negate the sum $\Rightarrow 0.8287$ .
Hard cross-entropy. Why? Keeps the student honest on the ground truth. $L_{hard} = - ln 0.75 = 0.2877$ .
Combine. $L = 0.5 \cdot 4 \cdot 0.8287 + 0.5 \cdot 0.2877 = 1.6574 + 0.1438 = 1.8013$ .

Verify: soft contribution $1.657$ vs hard $0.144$ — the $T^{2}$ scaling makes soft dominate, exactly as designed, and both terms pull the student toward cat. ✅

Worked example Example 6 (Cell G) — teacher and true label DISAGREE

Same setup but teacher argmax is dog: $p = [0.20, 0.70, 0.10]$ , true label still cat, student $q = [0.60, 0.30, 0.10]$ , $q_{cat} (1) = 0.75$ , $α = 0.5, T = 2$ .

Forecast: which term "wins" — does the student trust the (wrong) teacher or the true label?

Soft term. $- (0.20 ln 0.60 + 0.70 ln 0.30 + 0.10 ln 0.10) = - (0.20 (- 0.5108) + 0.70 (- 1.2040) + 0.10 (- 2.3026)) = - (- 0.1022 - 0.8428 - 0.2303) = 1.1752$ .
Hard term. Unchanged: $- ln 0.75 = 0.2877$ .
Combine. $L = 0.5 \cdot 4 \cdot 1.1752 + 0.5 \cdot 0.2877 = 2.3505 + 0.1438 = 2.4943$ .

Verify: the soft loss is larger than in Ex 5 ( $1.175$ vs $0.829$ ) because the student's cat-leaning output now conflicts with the dog-leaning teacher. This is why Mistake 3 warns against bad teachers: a confidently-wrong teacher fights the true label with weight $α T^{2} = 2$ against $(1 - α) = 0.5$ . ✅

Case H — the $α$ extremes

Case I — the $T^{2}$ audit

Worked example Example 8 (Cell I) — proving the gradient shrinks without

$T^{2}$ Take a tiny 2-class problem, teacher logits $v = [1, - 1]$ , student logits $z = [0.5, - 0.5]$ . Compare the soft-loss gradient magnitude at $T = 1$ vs $T = 4$ , with and without the $T^{2}$ prefactor.

Forecast: at $T = 4$ , without $T^{2}$ , how many times smaller is the gradient than at $T = 1$ ?

Gradient formula. From the parent: $\frac{\partial L _{soft}}{\partial z _{1}} = \frac{1}{T} (q_{1} (T) - p_{1} (T))$ . Why this form? One $1/ T$ comes from the logit division inside the student softmax.
$T = 1$ . Teacher $p_{1} = \frac{e ^{1}}{e ^{1} + e ^{- 1}} = 0.8808$ ; student $q_{1} = \frac{e ^{0.5}}{e ^{0.5} + e ^{- 0.5}} = 0.7311$ . Gradient $= \frac{1}{1} (0.7311 - 0.8808) = - 0.1497$ .
$T = 4$ . $p_{1} = \frac{e ^{0.25}}{e ^{0.25} + e ^{- 0.25}} = 0.6225$ ; $q_{1} = \frac{e ^{0.125}}{e ^{0.125} + e ^{- 0.125}} = 0.5622$ . Gradient $= \frac{1}{4} (0.5622 - 0.6225) = \frac{1}{4} (- 0.0603) = - 0.01508$ .
Ratio without $T^{2}$ . $∣ - 0.1497 ∣ / ∣ - 0.01508 ∣ = 9.93$ . The soft gradient is $\sim 10 \times$ weaker at $T = 4$ — distillation nearly switched off. This is the $\sim 1/ T^{2}$ shrink beginning to bite (the ratio grows toward $T^{2} = 16$ as logits get smaller and the small-logit Taylor regime is reached).
Restore with $T^{2}$ . Multiply the $T = 4$ gradient by $T^{2} = 16$ : $- 0.01508 \times 16 = - 0.2413$ , now larger in magnitude than the $T = 1$ value $- 0.1497$ — the prefactor has fully compensated (and here slightly over-compensated because these logits are not yet in the deep small-logit limit).

Verify: raw ratio $9.93$ confirms the soft gradient collapses as $T$ rises; rescaling by $T^{2}$ lifts it back from $0.015$ to $0.24$ , the same order as the $T = 1$ gradient. This is Mistake 2 made quantitative. ✅

Case J — the LLM word problem

Worked example Example 9 (Cell J) — next-token distillation over a huge vocabulary

A 50,000-token vocabulary teacher predicts the next token after "The capital of France is". Its top logits: Paris $= 12$ , London $= 6$ , Berlin $= 5$ , and the remaining 49,997 tokens each have logit $0$ . Use $T = 1$ then $T = 5$ and ask how much mass leaks to the 49,997 "background" tokens.

Forecast: at $T = 1$ what fraction is on Paris? At $T = 5$ , does the background grab a big share?

$T = 1$ numerator sum. Why sum background? All 49,997 zero-logit tokens contribute $e^{0} = 1$ each. Denominator $= e^{12} + e^{6} + e^{5} + 49997 \cdot e^{0} = 162754.79 + 403.43 + 148.41 + 49997 = 213303.6$ .
$T = 1$ probs. $P (Paris) = 162754.79/213303.6 = 0.7630$ ; background total $= 49997/213303.6 = 0.2344$ .
$T = 5$ . Effective logits: Paris $2.4$ , London $1.2$ , Berlin $1.0$ , background $0$ . $e^{2.4} = 11.023, e^{1.2} = 3.320, e^{1.0} = 2.718$ . Denominator $= 11.023 + 3.320 + 2.718 + 49997 \cdot 1 = 50014.06$ . $P (Paris) = 11.023/50014.06 = 0.0002204$ .
Interpret. Why did Paris collapse at $T = 5$ ? Because 49,997 background tokens, each tiny, together swamp everything when the sharp winners are flattened. Lesson: in large-vocab LLM KD you cannot naively crank $T$ — you typically distill on the top- $k$ logits or use moderate $T$ (2–4), or the uniform background drowns the signal.

Verify: at $T = 1$ , Paris $= 0.7630$ dominates as expected; at $T = 5$ the background mass $= 49997/50014.06 = 0.99966$ confirms the flattening pathology. ✅

Case K — the exam twist

Worked example Example 10 (Cell K) — shift-invariance trap

Exam question: "A colleague adds a constant $c = 100$ to every teacher logit before softmax, claiming it will change the soft targets. Are they right?" Teacher logits $z = [3, 0, - 3]$ , $c = 100$ , $T = 1$ .

Forecast: guess yes/no before computing.

Shifted logits. $z + c = [103, 100, 97]$ . Why check softmax? Softmax has a known invariance we can test.
Factor the shift. $\frac{e ^{z_{i} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} e ^{z_{i}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$ . The $e^{c}$ cancels top and bottom.
Numeric confirmation. Original $p = [0.9506, 0.0473, 0.0024]$ (Ex 1). Shifted gives the identical $[0.9506, 0.0473, 0.0024]$ .

Verify: softmax is invariant to adding a constant to all logits — the colleague is wrong. (This is why implementations subtract $max_{i} z_{i}$ for numerical stability without changing the answer.) But note the twist: dividing by $T$ is not shift-invariant in effect on spread — $T$ changes gaps, $c$ does not. ✅

Recall Matrix coverage self-test

Which cell hides the biggest real-world danger for LLM distillation? ::: Cell J — a huge uniform "background" of low-logit tokens; over-warming ( $T$ too large) lets those tokens collectively steal almost all probability mass, destroying the useful signal, so real LLM KD uses moderate $T$ or top- $k$ logits. Why is Cell E (equal logits) untrainable? ::: The teacher's distribution is uniform for every $T$ , carrying zero relational dark knowledge. Which cell proves you must keep the $T^{2}$ factor? ::: Cell I — the soft gradient collapses as $T$ rises (about $10 \times$ weaker at $T = 4$ here, trending toward $T^{2} = 16$ ); the $T^{2}$ prefactor restores it. What does $α = 0$ correspond to? ::: Pure hard-label training with no teacher signal — ordinary supervised learning.

Connections

Parent: Knowledge distillation
The softening knob: Softmax and Temperature
The loss metric: Cross-Entropy and KL Divergence
Where students come from: Model Compression, Pruning and Quantization, DistilBERT
Downstream uses: Instruction Tuning, Transfer Learning