Multinomial - softmax regression: Visual walkthrough

We will use a running example of three contestants with raw scores $z_{1} = 2$ , $z_{2} = 1$ , $z_{3} = 0$ . Watch these same three numbers travel through every step.

Step 1 — What we start with: raw scores that can be anything

WHAT. A machine looks at an input (a picture of a digit, say) and, for each possible answer, spits out one number. We call these numbers logits — a fancy word for "raw scores before we tidy them up." We write them $z_{1}, z_{2}, \dots, z_{K}$ , where $K$ is how many possible answers there are.

$score for answer k z_{k} = the rules for answer k w_{k}^{⊤} the input x$

$z_{k}$ — the raw score for class $k$ . It can be any real number: big, small, positive, or negative.
$w_{k}$ — the machine's "rulebook" for class $k$ (one per class).
$x$ — the input, written as a list of numbers.

WHY. Before we can talk about probabilities (which must behave nicely), we must be honest that the machine's raw output is unconstrained. Nothing stops $z_{2}$ from being $- 40$ .

PICTURE. Three bars, one per contestant, sitting on a number line that runs into negative territory. Notice the bars can dip below zero — that is the whole problem we must fix.

Figure — Multinomial - softmax regression

Step 2 — Problem 1: probabilities can't be negative

WHAT. A probability is a fraction of certainty. You cannot be " $- 30%$ sure." So any raw score that is negative is illegal as a probability. We need a machine that takes any real number and returns a positive number, while keeping the order (a bigger score should stay bigger).

WHY this tool — the exponential $e^{(\cdot)}$ . Why not just "add a big number to make everything positive," or "take absolute value"? Absolute value would make $- 5$ and $+ 5$ collide — order lost. Adding a constant fails if a score is very negative. We want a function that is:

always positive (never touches zero, never dips below),
increasing (bigger input → strictly bigger output, so order is preserved),
smooth (so we can take gradients later).

The exponential $e^{z}$ does exactly this. $e$ is just a fixed magic number ( $\approx 2.718$ ). Raising it to any power gives a positive result: $e^{- 5} \approx 0.0067$ (tiny but positive), $e^{0} = 1$ , $e^{2} \approx 7.39$ .

$e^{z_{1}} = e^{2} = 7.389, e^{z_{2}} = e^{1} = 2.718, e^{z_{3}} = e^{0} = 1$

PICTURE. The left panel shows the negative-capable line $z$ ; the curve $y = e^{z}$ bends every input up into positive territory. The right panel shows our three scores after the lift — all bars now above zero, and the tallest score got stretched the most.

Step 3 — Problem 2: they must add up to one whole

WHAT. Probabilities of all the possible answers must add up to $1$ (i.e. $100%$ — something must be the answer). Right now our lifted scores sum to $7.389 + 2.718 + 1 = 11.107$ , which is not $1$ .

To fix any pile of positive numbers so it sums to 1, divide each one by the total. We give the total a name, the normalizer:

$Z = \sum_{j = 1}^{K} e^{z_{j}} (here Z = 11.107)$

$\sum_{j = 1}^{K}$ — "add up, letting $j$ run over every class $1$ to $K$ ."
$e^{z_{j}}$ — the lifted score of class $j$ .
$Z$ — one single number: the grand total of all lifted scores.

Then each share is:

$p_{k} = \frac{e ^{z_{k}}}{Z} = \frac{e ^{z_{k}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$

$p_{k}$ — the fraction of the whole pie that class $k$ gets. Guaranteed positive (top and bottom both positive) and guaranteed to sum to 1 (every slice divided by the same total).

WHY. Dividing by the total is the only operation that forces the pieces to sum to exactly 1 while keeping their ratios intact. It's the same as slicing a pizza: everyone's slice is "their portion ÷ the whole pizza."

PICTURE. The three lifted bars get poured into one pie. Each contestant's slice angle is proportional to their bar height. Slices fill the pie completely — no gaps, no overlap.

$p = (\frac{7.389}{11.107}, \frac{2.718}{11.107}, \frac{1}{11.107}) = (0.665, 0.245, 0.090)$

Step 4 — Edge case: what if two scores are equal? What if one is huge?

WHAT. We must check the machine behaves sensibly at the corners, not just the "nice" example.

All scores equal ( $z_{1} = z_{2} = z_{3}$ ): every lifted score is the same, so every slice is $1/ K$ . With $K = 3$ that's a perfectly fair three-way split of $0.333$ each. Softmax says "I have no idea, so I'll guess evenly" — exactly right.
One score dominates ( $z_{1} \to + \infty$ while others stay fixed): $e^{z_{1}}$ swamps the sum, $p_{1} \to 1$ and the rest $\to 0$ . Softmax turns into a confident "it's class 1."
One score very negative ( $z_{3} \to - \infty$ ): $e^{z_{3}} \to 0$ , so $p_{3} \to 0$ but never exactly 0. This is important — softmax never fully rules a class out, which keeps $lo g p_{k}$ finite later.

WHY. These three limits confirm the three properties we demanded: positivity (nothing ever hits 0), sum-to-one (holds in every limit), and monotonicity (bigger score → bigger slice, all the way to certainty).

PICTURE. Three mini-pies side by side: the fair split, the near-certain split, and the "almost-ignored" split. Watch how the slices move as one score grows.

Step 5 — The hidden freedom: shifting all scores changes nothing

WHAT. Add the same constant $c$ to every score. Then $z_{k} \to z_{k} + c$ , so

$p_{k} = \frac{e ^{z_{k} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} e ^{z_{k}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$

$e^{z_{k} + c} = e^{c} \cdot e^{z_{k}}$ — the rule that "adding in the exponent = multiplying outside."
The $e^{c}$ appears in both the top and bottom, so it cancels. Same slices.

This is called shift-invariance. Our scores $(2, 1, 0)$ and $(5, 4, 3)$ (all shifted by $+ 3$ ) give identical probabilities $(0.665, 0.245, 0.090)$ .

WHY it matters practically. Because we can shift freely, we shift by $- max_{k} z_{k}$ so the largest score becomes $0$ before exponentiating. Then no $e^{(\cdot)}$ can overflow to infinity — this is the Log-Sum-Exp Trick. Same answer, safe arithmetic.

PICTURE. Two pies from two shifted score-sets. Bars move up by $+ 3$ on the left; the two pies on the right are pixel-for-pixel identical.

Step 6 — Grading the machine: cross-entropy loss

WHAT. Now the machine predicts slices $p = (p_{1}, \dots, p_{K})$ , but we know the true answer. We encode the truth with a one-hot vector $t$ (see One-Hot Encoding): a list of zeros with a single $1$ at the true class. If the truth is class 1: $t = (1, 0, 0)$ .

We want a score that is small when the machine put a big slice on the true class, and big (bad) when it didn't. The natural choice from Maximum Likelihood Estimation is the negative log of the true class's slice:

$J = - \sum_{k = 1}^{K} t_{k} lo g p_{k} = - lo g p_{true}$

$t_{k}$ — 1 only for the true class, 0 elsewhere. So the sum picks out one term: the true class.
$lo g p_{true}$ — as $p_{true} \to 1$ , $lo g \to 0$ (no loss); as $p_{true} \to 0$ , $lo g \to - \infty$ (huge loss). The minus sign flips this into a positive penalty.

WHY $lo g$ ? Multiplying probabilities across many examples gives tiny unstable numbers; taking $lo g$ turns products into sums and stretches the "you were almost sure and wrong" region into a heavy penalty. This is Cross-Entropy Loss.

PICTURE. The curve of $- lo g p$ : flat and near zero when the true slice is fat, rocketing upward as the true slice shrinks. A dot marks our example ( $p_{true} = 0.665$ , loss $\approx 0.408$ ).

Step 7 — The beautiful cancellation: the gradient is just (predicted − true)

WHAT. To learn, we ask: if I nudge score $z_{m}$ , how does the loss $J$ change? That rate of change is $\partial J / \partial z_{m}$ . The clean result is:

$\frac{\partial J}{\partial z _{m}} = p_{m} - t_{m}$

$p_{m}$ — the slice the machine gave class $m$ (what it predicted).
$t_{m}$ — 1 if $m$ is the true class, else 0 (the truth).
$p_{m} - t_{m}$ — prediction minus truth: the error on class $m$ .

WHY it's this clean. The $lo g$ in $J$ contributes a $\frac{1}{p _{k}}$ ; the softmax derivative contributes a $p_{k}$ factor; they cancel. Because the shift-freedom (Step 5) and the sum-to-one (Step 3) glue everything together, all the messy exp/normalizer terms collapse and only the error survives. Chaining to the weights via $z_{m} = w_{m}^{⊤} x$ :

$\frac{\partial J}{\partial w _{m}} = (p_{m} - t_{m}) x$

PICTURE. For our example ( $p = (0.665, 0.245, 0.090)$ , truth class 1) the error vector $(- 0.335, + 0.245, + 0.090)$ is drawn as arrows: the true class's arrow points to raise its score, the wrong classes' arrows point to lower theirs. This is Gradient Descent in one glance.

The update (with input $x = (1, 2)$ , true class 1) nudges $w_{1}$ by $- η (- 0.335) (1, 2)$ , i.e. it increases $w_{1}$ so class 1's score climbs next time — the machine learns to trust the right answer more.

The one-picture summary

The whole pipeline in a single strip: raw scores (can be negative) → exponentiate (all positive) → divide by the total (slices summing to 1) → compare to the one-hot truth → error $= p - t$ flows back to fix the rules.

Recall Feynman retelling: explain the whole walkthrough to a friend

Three judges hand you three raw scores, and some can be negative — useless as probabilities. First you make every score positive without messing up the order, by using it as a power of the magic number $e$ ; big scores stretch bigger, negatives become tiny positives, none reach zero. Second you add up all these boosted scores and give each judge a slice of the pie equal to their share of that total — now every slice is positive and all slices add to one whole pie. Then you check who actually won using a one-hot flag (a single 1 at the true answer). You grade with $- lo g$ of the true winner's slice: tiny loss if their slice was fat, huge loss if it was thin. Finally the fix is astonishingly simple: for each class, subtract truth from prediction ( $p - t$ ), multiply by the input, and step the rules that way — raise the true class's score, lower the impostors'. Exponentiate, share the pie, then push predicted toward true.

Connections

Parent: Softmax Regression — this page is its picture-by-picture derivation.
Logistic Regression — Step 3 at $K = 2$ becomes the sigmoid of the score difference.
Cross-Entropy Loss — the grading rule built in Step 6.
Maximum Likelihood Estimation — why $- lo g p_{true}$ is the right grade.
Gradient Descent — the Step 7 update $w_{m} \leftarrow w_{m} - η (p_{m} - t_{m}) x$ .
Log-Sum-Exp Trick — the safe way to run Steps 2–3 (Step 5's shift).
Neural Network Output Layer — softmax as the standard multiclass head.
One-Hot Encoding — the truth vector $t$ used in Step 6.