Knowledge distillation: Foundations — AI-ML deep dive

This page assumes you have seen nothing. Before you can read the parent note Knowledge Distillation, every letter and squiggle it uses is built here, one at a time, each earning its place before the next arrives.

0. The very first picture: what is a "score" and a "probability"?

A model looking at a photo of {cat, dog, car} produces three raw numbers — one per option. These raw, unbounded numbers are called logits.

Figure s01 — reading the logits. Three bars, one per option: magenta = cat, violet = dog, orange = car. Their heights are the logit values ( $z = 4, 1, - 2$ ), printed above each bar. Notice the car bar dips below the zero line drawn in navy — logits are allowed to be negative, as the arrow annotation points out. But we cannot use raw scores as probabilities, because probabilities must (a) never be negative and (b) add up to exactly $1$ . That is the job of the next tool.

1. Why we need $e^{x}$ (the exponential) — the "make-it-positive" machine

To turn any score into something never-negative, we need a function that:

takes any real number in, and
always spits out a positive number.

The exponential function $e^{x}$ does exactly this. Here $e \approx 2.718$ is a fixed constant (just a number, like $π$ ).

Figure s02 — the exponential curve. The magenta curve is $e^{x}$ ; the violet dots mark the sample values above. Read two things off it: (1) the curve never touches the navy zero line — output is always positive, highlighted by the orange "always > 0" arrow; (2) it climbs steeply on the right — small increases in $x$ cause large increases in $e^{x}$ .

2. The sum symbol $\sum$ — "add up all of them"

The parent note writes $\sum_{j} e^{z_{j}}$ . That big Greek letter $Σ$ (sigma) is just a compact way to say "add up a list".

Why do we need it? Because to make probabilities add to $1$ , we must divide each positive score by the total of all positive scores — and $\sum$ is how we write "the total".

3. Putting it together: the softmax — the "make-it-a-probability" machine

Now we can read the parent note's central formula: $p_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}$

Read it left to right in plain words:

$p_{i}$ = the probability the model assigns to option $i$ (a number between $0$ and $1$ ).
Top ( $e^{z_{i}}$ ): make this option's score positive.
Bottom ( $\sum_{j} e^{z_{j}}$ ): the total of all options' positive scores.
Divide → each option's share of the total. By construction they now add to $1$ .

Figure s03 — the softmax pipeline. For each option (cat, dog, car) three bars stand side by side: orange = the raw logit $z$ , violet = its exponential $e^{z}$ (now all positive), magenta = the final probability (scaled ×60 just so the tiny values are visible). Follow one option left-to-right and you see the machine work: a possibly-negative score becomes positive, then the navy arrow "divide by the sum → adds to 1" turns the violet bars into the magenta shares printed above them ( $p = 0.950, 0.047, 0.0023$ ). That exponential middle step is why a trained model looks overconfident — the tiny probabilities on "dog" and "car" get crushed. Distillation's whole trick, temperature, is about un-crushing them.

Recall Why is the biggest logit still the biggest probability?

Because $e^{x}$ is increasing: a bigger input always gives a bigger output. So softmax preserves the ranking of the logits — it only re-scales how confident each one looks, never who wins. ::: Softmax preserves ranking; only the confidence spread changes.

3b. A practical edge case: numerical stability (log-sum-exp)

4. The word "distribution", and $K$ = how many options

We keep saying distribution. It just means "the full list of probabilities across all options", e.g. $[0.9, 0.099, 0.001]$ . The parent uses $K$ for how many options there are (3 for {cat,dog,car}; ~50,000 for an LLM's vocabulary). When the parent writes $1/ K$ for the uniform case, that is "everyone gets an equal slice".

5. The $lo g$ and "cross-entropy" — measuring "how wrong"

To train, we need a single number that says how far apart two distributions are. The building block is the logarithm $lo g$ (specifically $ln$ , the natural log — the inverse of $e^{x}$ ).

Now the parent's loss line reads cleanly. Recall from §4 that $q_{i}$ is the student's probability for option $i$ : $L_{soft} = - \sum_{i} p_{i} lo g q_{i}$ means: for every option $i$ , take the teacher's opinion $p_{i}$ , multiply by the student's surprise $- lo g q_{i}$ , and add them up. Small when the student's distribution matches where the teacher put its weight.

6. The leftover symbols: $y$ , $α$ , $T$ , and $\partial$

$y$ — the hard target / true label as a list with a single $1$ : for a real cat, $y = [1, 0, 0]$ . This is a "one-hot" vector.
$α$ (alpha) — a mixing weight between $0$ and $1$ : how much to trust soft targets vs. hard labels. $α = 0.5$ means "half and half".
$T$ (temperature) — the star of the show, defined and pictured in Softmax and Temperature. Here just know: $T$ is a positive number you divide every logit by before softmax, to flatten the distribution.
$\partial L / \partial z_{i}$ — the gradient: "if I nudge logit $z_{i}$ up a tiny bit, how much does the loss $L$ change?" This is what training uses to improve the student. The curly $\partial$ just means "rate of change with respect to one variable while holding the others still".

Prerequisite map

Every node above is a symbol the parent uses. Follow the arrows: scores become probabilities, probabilities feed a "how-wrong" measure, temperature reshapes them, and the mixing weight and gradient assemble the final distillation loss.

Connections

Next tool up: Softmax and Temperature — the $T$ knob in full.
The loss's measuring stick: Cross-Entropy and KL Divergence.
Where distillation sits among cousins: Model Compression, Pruning and Quantization.
Real distilled model: DistilBERT.
How teachers generate training data: Instruction Tuning, and the reuse-of-knowledge idea in Transfer Learning.

Equipment checklist

Recall Are you ready for the parent note?

A logit is... ::: a raw, possibly-negative score for one option, before it becomes a probability. $e^{x}$ is used because... ::: it is always positive and it stretches gaps between scores (causing overconfidence). $\sum_{j} e^{z_{j}}$ means... ::: add up the exponentials of every option's logit — the normalising total. Softmax guarantees... ::: all outputs positive and summing to $1$ , while preserving the ranking of logits. Softmax is unchanged when you... ::: add the same constant to every logit (shift-invariance); only gaps matter. The log-sum-exp trick is for... ::: computing softmax/log-probs stably by subtracting the max logit to avoid overflow/underflow. $p_{i}$ vs $q_{i}$ are... ::: the teacher's probability for option $i$ and the student's probability for option $i$ . $K$ is... ::: the number of options (classes / vocabulary size). $- lo g q$ measures... ::: the surprise of an outcome the model gave probability $q$ ; big when confident-and-wrong. Cross-entropy is... ::: average surprise of one distribution measured under another; same gradient as KL divergence. $y = [1, 0, 0]$ is... ::: a hard one-hot label, all weight on the true class. $α$ controls... ::: the mix between soft-target loss and hard-label loss. The gradient $\partial L / \partial z_{i} = q_{i} - p_{i}$ means... ::: push each logit by how far the student overshot the teacher there.