Visual walkthrough — Knowledge distillation

We assume you know nothing except: a model reads an input and outputs some numbers, one per possible answer. Everything else we construct.

---

Step 1 — What a model actually outputs: logits

WHAT. When a model looks at a picture and must choose between three labels — say cat, dog, car — it does not immediately output probabilities. It outputs three raw scores called logits. For the teacher we write them $v=[v_1,v_2,v_3]$; for the student, $z=[z_1,z_2,z_3]$. A logit is just "how strongly the model leans toward this answer", any real number, positive or negative.

WHY. We start here because logits are the rawest thing the model produces. Probabilities are built from logits, so to understand distillation we must begin before the probabilities exist.

PICTURE. Look at the three bars below. Their heights are the teacher's logits $v=[4,1,-2]$. Notice the car bar dips below zero — a negative logit is allowed; it just means "I lean away from this".

---

Step 2 — Turning logits into probabilities: the softmax

WHAT. We need those three bars to become three probabilities that are all $\ge 0$ and add up to 1 (a valid opinion). The tool that does this is the softmax. Applied to the teacher's logits $v_i$: $p_i=\frac{\stackrel{\begin{array}{c}\text{make it}\\ \text{positive}\end{array}}{\overbrace{e^{v_i}}}}{\underset{\begin{array}{c}\text{divide so they}\\ \text{sum to }1\end{array}}{\underbrace{\sum _j{e}^{v_j}}}}$

Term by term: $e^{v_i}$ takes each logit and makes it a positive number (the exponential of anything is positive — this kills negative scores). The denominator ${\sum }_j{e}^{v_j}$ adds up all those positive numbers so that dividing forces the total to be exactly $1$. (The exact same formula with $z_i$ in place of $v_i$ gives the student's probabilities — we will use that in Step 6.)

WHY this tool and not another? We could just clip negatives and divide, but softmax has two magic properties: (a) it always gives positive numbers no matter how negative a logit is, and (b) the exponential amplifies gaps — a slightly bigger logit becomes a much bigger probability. That amplification is exactly what makes a trained model confident.

PICTURE. The left panel is the teacher's logits again; the right panel is softmax applied. Watch the cat bar balloon to $0.95$ while dog and car get crushed toward the floor.

See Softmax and Temperature for softmax on its own.

---

Step 3 — The problem: confidence hides the interesting part

WHAT. The teacher just told us cat $=0.950$, dog $=0.047$, car $=0.0023$. The interesting fact is buried: dog is 20× more likely than car, meaning "a cat looks kind of like a dog but nothing like a car". That is dark knowledge. But at $0.047$ vs $0.0023$ those numbers are tiny — a student learning from them barely feels the difference.

WHY. A hard one-hot label $[1,0,0]$ throws this away completely. Even the teacher's real softmax nearly throws it away because it is overconfident. We need a knob to surface the small numbers without changing which one wins.

PICTURE. Same probabilities, but now on a log scale so you can see the buried structure. The gap between dog and car is huge in information terms — it is just visually invisible on a normal scale.

---

Step 4 — The knob: temperature $T$

WHAT. We introduce a single number $T>0$ called the temperature, and divide every logit by it before softmax. For the teacher: $p_i(T)=\frac{e^{v_i/T}}{{\sum }_j{e}^{v_j/T}}$

Term by term: the new thing is the $/T$ sitting on each teacher logit $v_i$. Dividing every logit by the same $T>1$ shrinks the gaps between them, so after softmax the probabilities come out flatter and gentler.

WHY this tool? We want to flatten confidence without reordering it. Dividing by a constant $T$ does exactly that: it scales all logits equally, so the biggest logit stays the biggest (ranking preserved), but the spread between them shrinks.

PICTURE. Three panels — the same teacher at $T=1$, $T=2$, $T=5$. As $T$ climbs, watch the bars level out toward a flat line. The dog bar rises from an invisible $0.047$ to a clearly-learnable $0.175$.

---

Step 5 — The two limiting cases (never skip these)

WHAT. A knob is only understood once you turn it all the way both directions.

• $T\to \infty$: every logit divided by a huge number $\Rightarrow$ all exponents $\to 0\Rightarrow$ every $e^{v_i/T}\to 1\Rightarrow$ probabilities become perfectly uniform, $p_i\to 1/K$ (here $1/3$). Total forgetting of who wins.
• $T\to 0^{+}$: the largest logit divided by a tiny number blows up far above the rest $\Rightarrow$ softmax collapses to a one-hot on the argmax. Maximum confidence, all structure lost.

WHY show both? So you know the safe zone is in between. Too hot → uniform noise (no answer). Too cold → hard label (no dark knowledge). Practice lives at $T\approx 2$–$10$.

PICTURE. The dog-class probability plotted as $T$ sweeps from near $0$ to large. It rises from $\approx 0$ (cold, one-hot) up toward $1/3$ (hot, uniform), passing through the useful middle.

---

Step 6 — Two goals → two losses

WHAT. We now want the small student to learn. First, we must define the student's probabilities, using the very same softmax from Step 2/4 but on the student's logits $z_i$: $\boxed{q_i(T)=\frac{e^{z_i/T}}{{\sum }_j{e}^{z_j/T}}}\text{(student, warm)},q_i(1)=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}\text{(student, cold)}.$ So $q$ is to the student exactly what $p$ is to the teacher. Now the student must do two jobs at once:

1. Copy the teacher's warm opinion. Compare student soft probs $q_i(T)$ to teacher soft probs $p_i(T)$ using cross-entropy: ${\mathcal{L}}_{\text{soft}}=-{\sum }_i\underset{\begin{array}{c}\text{teacher, warm}\\ =\text{softmax}(v_i/T)\end{array}}{\underbrace{p_i(T)}}\log \underset{\begin{array}{c}\text{student, warm}\\ =\text{softmax}(z_i/T)\end{array}}{\underbrace{q_i(T)}}$

2. Still get the true answer right. Compare the student's normal ($T=1$) probs to the real one-hot label $y$: ${\mathcal{L}}_{\text{hard}}=-{\sum }_i\underset{\text{true label}}{\underbrace{y_i}}\log \underset{\text{student, cold}}{\underbrace{q_i(1)}}$

WHY cross-entropy? Because it measures "how surprised is my prediction by the target". Minimising it drags the student distribution toward the target distribution. (Cross-entropy differs from KL divergence only by a constant that doesn't depend on the student — see Cross-Entropy and KL Divergence — so minimising either gives the same student.)

PICTURE. Two side-by-side comparisons: student $q$ vs teacher $p$ (soft, warm) on the left, student $q(1)$ vs truth $y$ (hard, cold) on the right, with arrows showing each loss "pulling" the student bars.

---

Step 7 — The gradient reveals a hidden shrinkage

Everything below is about differentiating ${\mathcal{L}}_{\text{soft}}$ with respect to a student logit $z_i$ — because that is what gradient descent nudges. The teacher logits $v_i$ are fixed constants (the teacher is frozen).

WHAT — factor A (the chain rule through $/T$). Start from the classic fact for softmax cross-entropy: if $u_i$ are the numbers fed into the softmax and $t_i$ the target, then $\partial \mathcal{L}/\partial u_i=q_i-t_i$ (prediction minus target — clean, no $T$). But here the softmax is not fed $z_i$; it is fed $u_i=z_i/T$. By the chain rule, $\frac{\partial {\mathcal{L}}_{\text{soft}}}{\partial z_i}=\frac{\partial {\mathcal{L}}_{\text{soft}}}{\partial u_i}\cdot \underset{=1/T}{\underbrace{\frac{\partial u_i}{\partial z_i}}}=(q_i(T)-p_i(T))\cdot \frac{1}{T}.$ That $\frac{1}{T}$ is the first factor of $1/T$ — it comes purely from the fact that we divided the student logit by $T$ before the softmax saw it.

WHAT — factor B (the difference itself shrinks). Now zoom in on the term $q_i(T)-p_i(T)$. Watch what happens to it as $T$ grows. When $T$ is large, both $v_i/T$ and $z_i/T$ are small numbers, and for small $x$ the exponential is nearly a straight line: $e^x\approx 1+x$ (first-order Taylor — accurate whenever $|x|\ll 1$, i.e. whenever the logits are small compared to $T$). Substituting $e^{z_i/T}\approx 1+z_i/T$ into the softmax and simplifying: $q_i(T)-p_i(T)\approx \frac{1}{KT}(z_i-v_i),$ where $K$ is the number of classes (here $3$) and the $z_i-v_i$ is "student logit minus teacher logit". The point: this difference carries its own hidden $1/T$ — because warming flattens both distributions toward the same uniform $1/K$, so their gap physically closes as $T$ grows. This is the second factor of $1/T$. (Validity: exact in the limit of large $T$; a good approximation once $T$ exceeds the logit spread, which is exactly the regime distillation uses.)

WHY — multiply the two factors. Factor A is $\frac{1}{T}$; factor B hides another $\frac{1}{T}$ inside $q-p$. Together: $\frac{\partial {\mathcal{L}}_{\text{soft}}}{\partial z_i}\sim \frac{1}{T}\cdot \frac{1}{T}=\frac{1}{T^2}.$ So raising $T$ to expose dark knowledge secretly divides the learning signal by $T^2$. At $T=5$ the soft loss teaches $25\times$ more weakly — it silently switches off.

PICTURE. The soft-loss gradient magnitude plotted against $T$: a curve that plummets like $1/T^2$. Right beside it, the same curve multiplied by $T^2$ — flat, restored.

---

Step 8 — Assemble the loss

WHAT. Rescale the soft loss by $T^2$ to cancel the shrinkage, then mix the two goals with a dial $\alpha \in [0,1]$:

$\boxed{\mathcal{L}=\underset{\begin{array}{c}\text{how much}\\ \text{to imitate}\end{array}}{\underbrace{\alpha }}\underset{\begin{array}{c}\text{undo the}\\ 1/T^2\text{ shrink}\end{array}}{\underbrace{T^2}}\underset{\text{copy teacher}}{\underbrace{{\mathcal{L}}_{\text{soft}}}}+\underset{\begin{array}{c}\text{how much}\\ \text{to be correct}\end{array}}{\underbrace{(1-\alpha )}}\underset{\text{stay right}}{\underbrace{{\mathcal{L}}_{\text{hard}}}}}$

WHY $T^2$ and not $T$? Because Step 7 showed the shrink is $1/T^2$ (two factors of $1/T$ — factor A from the chain rule, factor B from the closing gap), so exactly $T^2$ restores it. One factor would leave a residual $1/T$; three would over-boost.

PICTURE. A gentle two-branch flow: student logits $z$ split into a warm branch (soft loss vs teacher $p$, $\times \alpha T^2$) and a cold branch (hard loss vs label $y$, $\times (1-\alpha )$), summing to $\mathcal{L}$.

---

The one-picture summary

Everything on one canvas: teacher logits $v$ → softmax → warm with $T$ (dark knowledge appears) → student logits $z$ produce $q$ → two losses pulling the student → rescale soft by $T^2$ → mix with $\alpha$ → final loss.

Recall Feynman retelling of the whole walkthrough

A big teacher model outputs raw scores (its logits $v$) for each answer. Softmax turns them into a confident opinion $p$ — but confidence hides the good stuff, like "a cat resembles a dog but not a car". So we gently warm the opinion with a temperature $T$: this flattens it so the small differences become visible and learnable. Too warm and everything blurs to equal; too cold and it snaps back to a plain answer — so we pick something in the middle. The student has its own logits $z$ giving its own probabilities $q$, and two jobs: imitate the warm teacher $p$ (soft loss) and still name the right answer (hard loss). But warming secretly weakens the imitation signal by a factor of $T^2$ — one $1/T$ because we divided the student logit by $T$, and a second $1/T$ because warming closes the gap between $q$ and $p$. So we multiply the soft loss by $T^2$ to undo that. Finally a dial $\alpha$ balances "imitate" against "be correct". Soft, Warm, Squared.

---

Related: Model Compression · Pruning and Quantization · DistilBERT · Transfer Learning · Instruction Tuning