Multinomial - softmax regression: Worked examples

This page is a case gym. The parent softmax note built the machinery; here we throw every kind of input at it — negatives, ties, zeros, huge numbers, a word problem, an exam twist — and grind each one by hand. If you can do all of these, nothing about softmax can surprise you.

Before we start, a one-line refresher of the only two operations you need, in plain words:

Every example below is just: EXP, add up, SHARE. Watch how each weird input still obeys these two moves.

The scenario matrix

Softmax + its cross-entropy loss can be hit with a small, finite family of situations. Here is the full grid, and which worked example nails each cell. One example per cell, no gaps.

Cell	Situation	Why it's tricky	Example
A	All logits positive, distinct	baseline warm-up	Ex 1
B	Mix of negative and positive logits	$e^{z}$ of a negative number	Ex 2
C	All logits equal (degenerate tie)	must give uniform $1/ K$	Ex 3
D	Huge logits (overflow risk)	$e^{1000} = \infty$ on a computer	Ex 4
E	Limiting case: one logit $\to + \infty$	softmax $\to$ one-hot (arg-max)	Ex 5
F	$K = 2$ collapse to sigmoid	shows softmax ⊇ sigmoid	Ex 6
G	Loss + gradient for a correct-ish prediction	$(p - t)$ sign check	Ex 7
H	Loss + gradient for a badly wrong prediction	large loss, large push	Ex 8
I	Real-world word problem (3-class classifier)	reading logits from a story	Ex 9
J	Exam twist: shift-invariance + stability proof by numbers	conceptual trap	Ex 10
K	Very negative logits (underflow risk)	$e^{- 800} \approx 0$ on a computer	Ex 11

Figure — Multinomial - softmax regression

How to read the map above. Each colored dot is one cell of the table — its bold letter (A–K) is the cell label, the word underneath is the situation in shorthand, and the italic label below that (Ex1…Ex11) is the worked example that solves it. The gray dashed line is the "study path": it visits the dots in order A→B→C→…→K, so if you follow the dashes you have covered every scenario softmax can present exactly once. Nothing is plotted numerically here — it is a legend/index, a picture of the checklist, not a graph of data.

Ex 1 — Cell A: all-positive, distinct logits

Ex 2 — Cell B: negative logits mixed in

Worked example Statement

Compute $softmax (z)$ for $z = (- 1, 0, 2)$ .

Forecast: the first logit is negative — will its probability be negative too? (No — watch why.)

EXP. $e^{- 1} = 0.3679, e^{0} = 1.000, e^{2} = 7.389$ . Why this step? $e^{- 1}$ is not negative — a negative power means "one over": $e^{- 1} = 1/ e$ . Still positive. This is exactly why we use $e^{(\cdot)}$ and not, say, the identity: it repairs negatives.
Sum. $S = 0.3679 + 1.000 + 7.389 = 8.757$ . Why this step? $S$ is the whole pie — the normalizer we divide every slice out of. Even a tiny term like $e^{- 1} = 0.3679$ contributes to it.
SHARE. $p = (0.0420, 0.1142, 0.8438)$ . Why this step? Normalize — divide each positive exp by $S$ so the slices add to $1$ .

Verify: all three slices are positive (even the one from the negative logit), and $0.0420 + 0.1142 + 0.8438 = 1.000$ ✅. The class with logit $2$ dominates.

Ex 3 — Cell C: the degenerate tie (all logits equal)

Ex 4 — Cell D: huge logits (overflow) and the fix

Worked example Statement

Compute $softmax (z)$ for $z = (1000, 1001, 1002)$ — the way a computer must.

Forecast: $e^{1000}$ is astronomically large. Can we even compute this? (Yes — by cheating with a shift.)

Subtract the max. Let $c = max_{k} z_{k} = 1002$ . Form $z^{'} = z - c = (- 2, - 1, 0)$ . Why this step? Softmax is shift-invariant: adding the same constant to every logit multiplies every $e^{z_{k}}$ by $e^{c}$ , which cancels in the ratio. So $softmax (z) = softmax (z^{'})$ — identical answer, but now the biggest exponent is $e^{0} = 1$ , no overflow. This is the Log-Sum-Exp Trick (Log-Sum-Exp Trick).
EXP the shifted logits. $e^{- 2} = 0.1353, e^{- 1} = 0.3679, e^{0} = 1.000$ .
Sum. $S = 0.1353 + 0.3679 + 1.000 = 1.5032$ . Why this step? $S$ is the normalizer (the whole pie) built from the shifted exps — all safely near $1$ instead of near $\infty$ .
SHARE. $p = (0.0900, 0.2447, 0.6652)$ .

Verify: compare to the naive $softmax (1000, 1001, 1002)$ : it is mathematically the same distribution $(0.0900, 0.2447, 0.6652)$ , sums to $1$ ✅ — but the naive route hits $e^{1000} = \infty$ and returns garbage. Same numbers as $softmax (- 2, - 1, 0)$ because the two logit vectors differ by a constant.

Ex 5 — Cell E: limiting case, one logit runs to $+ \infty$

Worked example Statement

Track $softmax (z)$ for $z = (t, 0, 0)$ as $t$ grows: $t = 2$ , then $t = 10$ , then $t \to \infty$ .

Forecast: as one class becomes infinitely more confident, what shape does the pie approach?

$t = 2$ . $e^{2} = 7.389, e^{0} = 1, e^{0} = 1$ ; $S = 9.389$ ; $p = (0.787, 0.107, 0.107)$ .
$t = 10$ . $e^{10} = 22026.5$ ; $S = 22028.5$ ; $p = (0.99991, 0.0000454, 0.0000454)$ . Why this step? One exponential now dwarfs the others; its slice approaches the whole pie.
$t \to \infty$ . $p \to (1, 0, 0)$ — a one-hot vector. Why this step? $e^{t} / (e^{t} + 2) \to 1$ and $1/ (e^{t} + 2) \to 0$ . This limit is the arg-max: softmax is a soft, smooth version of "pick the biggest." That's literally where the name comes from.

Verify: at $t = 10$ , $0.99991 + 0.0000454 + 0.0000454 \approx 1$ ✅. And the limit $(1, 0, 0)$ is a legal probability vector, matching the target One-Hot Encoding we'd want for a perfectly confident correct prediction.

The figure shows $p_{1}$ climbing toward $1$ as $t$ increases — the "temperature" of the decision cooling into certainty.

Ex 6 — Cell F: $K = 2$ collapses to the sigmoid

Ex 7 — Cell G: loss + gradient, prediction mostly right

Worked example Statement

True class $y = 1$ , one-hot $t = (1, 0, 0)$ . Model predicts $p = (0.665, 0.245, 0.090)$ (from Ex 1). Input $x = (1, 2)$ , learning rate $η = 0.1$ . Compute the cross-entropy loss, the error, and the update to all three weight vectors.

Forecast: the correct class already has the biggest slice ( $0.665$ ). Should the loss be small, and the push gentle?

Cross-entropy loss (using $J = - lo g p_{true}$ from the loss callout above). Only the true class survives: $J = - lo g (0.665) = 0.408$ . Why this step? $\sum_{k} t_{k} lo g p_{k}$ zeroes every term except $k = 1$ .
Error vector $p - t = (0.665 - 1, 0.245 - 0, 0.090 - 0) = (- 0.335, 0.245, 0.090)$ . Why this step? The clean gradient wrt logits is $p_{m} - t_{m}$ (parent derivation). The true class has a negative error → we must raise its logit; the two wrong classes have positive errors → we must lower theirs.
Gradients wrt every class weight. The rule $\nabla_{w_{m}} J = (p_{m} - t_{m}) x$ applies to every class $m$ , not just class 1: $\nabla_{w_{1}} J = (- 0.335) (1, 2) = (- 0.335, - 0.670),$ $\nabla_{w_{2}} J = (0.245) (1, 2) = (0.245, 0.490), \nabla_{w_{3}} J = (0.090) (1, 2) = (0.090, 0.180) .$ Why this step? Softmax couples all classes through the shared normalizer $S$ , so one training example nudges all $K$ weight vectors at once — we do not update only the true class. Earlier drafts that touched just $w_{1}$ were incomplete.
Updates (Gradient Descent): $w_{m} \leftarrow w_{m} - η \nabla_{w_{m}} J$ . For class 1: $w_{1} + (0.0335, 0.0670)$ (raises $z_{1}$ ). For classes 2,3: $w_{2} - (0.0245, 0.0490)$ and $w_{3} - (0.0090, 0.0180)$ (lowers $z_{2}, z_{3}$ ). Why this step? Subtracting a negative gradient adds to $w_{1}$ (bigger $p_{1}$ ); subtracting a positive gradient shrinks $w_{2}, w_{3}$ (smaller $p_{2}, p_{3}$ ). The pie is pushed toward the true class from every side.

Verify: loss $0.408$ is modest (already fairly confident); the errors sum to $- 0.335 + 0.245 + 0.090 = 0$ ✅ — always true, since $p$ and $t$ each sum to $1$ , so $(p - t)$ sums to $0$ . Because the errors sum to zero, the total nudge across all classes is balanced — mass is moved, not created.

Ex 8 — Cell H: loss + gradient, prediction badly wrong

Worked example Statement

Same setup but the model is confidently wrong: true class $y = 1$ , $t = (1, 0, 0)$ , prediction $p = (0.02, 0.90, 0.08)$ , $x = (1, 2)$ , $η = 0.1$ . Compute loss, error, and updates to all three weight vectors.

Forecast: the correct class got only $2%$ . Compared to Ex 7, should the loss be much bigger and the correction much stronger?

Loss. $J = - lo g (0.02) = 3.912$ . Why this step? $- lo g$ blows up as the true-class probability approaches $0$ : being confidently wrong is punished heavily.
Error. $p - t = (0.02 - 1, 0.90, 0.08) = (- 0.98, 0.90, 0.08)$ . Why this step? Class 1 error $- 0.98$ is huge (nearly $- 1$ ) → strong push to raise $z_{1}$ ; class 2 error $+ 0.90$ → strong push to lower $z_{2}$ .
Gradients wrt all classes (same $(p_{m} - t_{m}) x$ rule for every $m$ ): $\nabla_{w_{1}} J = (- 0.98) (1, 2) = (- 0.98, - 1.96),$ $\nabla_{w_{2}} J = (0.90) (1, 2) = (0.90, 1.80), \nabla_{w_{3}} J = (0.08) (1, 2) = (0.08, 0.16) .$ Why this step? Again all $K$ weight vectors move on this one example, because they share the normalizer. The wrongly-favored class 2 gets the biggest downward push.
Updates (Gradient Descent): apply $w_{m} \leftarrow w_{m} - η \nabla_{w_{m}} J$ with $η = 0.1$ . Class 1: $w_{1} + (0.098, 0.196)$ (raise $z_{1}$ ). Class 2: $w_{2} - (0.090, 0.180)$ (drop the bad favorite $z_{2}$ ). Class 3: $w_{3} - (0.008, 0.016)$ . Why this step? We take the gradient-descent step here for the same reason as Ex 7 — move each weight vector against its gradient to reduce $J$ — but now the correction to $w_{1}$ is roughly $3 \times$ larger because the mistake (and hence the error $- 0.98$ ) is roughly $3 \times$ larger. Subtracting the negative class-1 gradient adds to $w_{1}$ , raising $z_{1}$ next time; subtracting the positive class-2 gradient shrinks $w_{2}$ , dropping the wrongly-confident $z_{2}$ .

Verify: loss $3.912 ≫ 0.408$ from Ex 7 ✅ (confidently wrong ≫ mostly right), and errors sum to $- 0.98 + 0.90 + 0.08 = 0$ ✅. The class-1 update magnitude $0.098$ here vs $0.0335$ in Ex 7 confirms the "bigger mistake, bigger step" behaviour that Maximum Likelihood Estimation intends.

Ex 9 — Cell I: real-world word problem

Worked example Statement

A weather model looks at today's data and produces three logits for tomorrow: Sunny $= 1.2$ , Cloudy $= 2.0$ , Rainy $= - 0.5$ . (1) What probability does it give each? (2) If it actually rains, what's the loss and which way does the Rainy weight move?

Forecast: Cloudy has the top logit. Guess the Cloudy probability before computing.

EXP. $e^{1.2} = 3.3201, e^{2.0} = 7.3891, e^{- 0.5} = 0.6065$ . Why this step? Turn scores into positive un-normalized weights.
Sum. $S = 3.3201 + 7.3891 + 0.6065 = 11.3157$ . Why this step? $S$ is the whole pie — the normalizer built from all three positive exps, including the small Rainy term $0.6065$ ; we cut every slice out of it.
SHARE. $p = (Sunny 0.2934, Cloudy 0.6530, Rainy 0.0536)$ . Why this step? Normalize into a forecast distribution that sums to $1$ .
It rains → true class is Rainy, $t = (0, 0, 1)$ . Loss $J = - lo g (0.0536) = 2.926$ (again using $J = - lo g p_{true}$ ). Why this step? We only paid for the truth's probability; the model gave rain just $5.4%$ , so it's penalised.
Error for Rainy $= p_{Rainy} - t_{Rainy} = 0.0536 - 1 = - 0.9464$ → strongly negative → push the Rainy weights up so next time rain scores higher.

Verify: $0.2934 + 0.6530 + 0.0536 = 1.000$ ✅; the model's top pick (Cloudy, $65.3%$ ) was reasonable but wrong, so the loss $2.926$ is large and the corrective push on Rainy is near its maximum magnitude ( $- 0.9464$ , close to $- 1$ ). Units check: probabilities are dimensionless slices summing to a whole ✅.

Ex 10 — Cell J: exam twist (shift-invariance + why we subtract the max)

Worked example Statement

An exam asks: "Student A computes $softmax (3, 1, - 1)$ . Student B computes $softmax (0, - 2, - 4)$ . Both claim the same answer. Prove or disprove — without a calculator if you can — and explain how this proves the numerical-stability trick is loss-free."

Forecast: guess whether the two vectors give the same distribution before computing.

Spot the shift. $(3, 1, - 1) - (0, - 2, - 4) = (3, 3, 3)$ : the two logit vectors differ by the constant $+ 3$ in every slot. Why this step? Shift-invariance says a constant offset can't change the answer — recognise it and you're done.
Algebra of why. $\frac{e ^{z_{k} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} e ^{z_{k}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}$ . The $e^{c}$ cancels top and bottom. Why this step? This is the entire justification for subtracting $max_{k} z_{k}$ before exponentiating (Ex 4): subtracting the max is just choosing the constant $c = - max_{k} z_{k}$ , which is a shift, so by the cancellation above the probabilities are provably unchanged — the stability trick costs us nothing in accuracy, it only rescues the arithmetic from overflow.
Confirm numerically. $softmax (0, - 2, - 4)$ : $e^{0} = 1, e^{- 2} = 0.1353, e^{- 4} = 0.0183$ ; $S = 1.1536$ ; $p = (0.8668, 0.1173, 0.0159)$ . And $softmax (3, 1, - 1)$ gives the identical $(0.8668, 0.1173, 0.0159)$ .

Verify: both distributions equal $(0.8668, 0.1173, 0.0159)$ , sum to $1$ ✅. Student A and B are both right — the twist is that they look different but differ only by a shift. This is why the Log-Sum-Exp Trick is exact, not an approximation.

Ex 11 — Cell K: very negative logits (underflow) and the same fix

Worked example Statement

Compute $softmax (z)$ for $z = (- 800, - 801, - 802)$ — the way a computer must, avoiding underflow.

Forecast: every logit is huge and negative, so $e^{- 800}$ rounds to $0$ on a machine. If all three round to $0$ , what happens when we divide by $S = 0$ ? (Disaster — unless we shift.)

Diagnose the danger. Naively $e^{- 800} \approx 0, e^{- 801} \approx 0, e^{- 802} \approx 0$ , so $S = 0$ and every slice is $0/0$ — a fatal NaN. This is underflow, the mirror image of Ex 4's overflow. Why this step? Recognising the failure mode tells us we need the same shift-by-max rescue.
Subtract the max. $c = max_{k} z_{k} = - 800$ , so $z^{'} = z - c = (0, - 1, - 2)$ . Why this step? Shift-invariance again: the answer is unchanged, but now the largest exponent is $e^{0} = 1$ — nothing underflows to zero.
EXP the shifted logits. $e^{0} = 1.000, e^{- 1} = 0.3679, e^{- 2} = 0.1353$ .
Sum. $S = 1.000 + 0.3679 + 0.1353 = 1.5032$ . Why this step? $S$ is the normalizer, now safely near $1.5$ instead of $0$ .
SHARE. $p = (0.6652, 0.2447, 0.0900)$ .

Verify: $0.6652 + 0.2447 + 0.0900 = 1.000$ ✅. Note this is exactly the Ex 4 distribution reversed in size ordering (there the biggest logit was last; here the biggest is first), confirming the shift trick fixes both overflow (top too big) and underflow (everything too small) with the single move "subtract the max."

Active Recall

Recall (Ex 1 monotonicity) Why does a bigger logit always get a bigger probability?

Because $e^{(\cdot)}$ is increasing, so $z_{a} > z_{b} \Rightarrow e^{z_{a}} > e^{z_{b}}$ , and dividing both by the same total $S$ preserves the order.

Recall (Cell B) A logit is

$- 3$ . Is its softmax probability negative? No. $e^{- 3} = 0.0498 > 0$ . Softmax outputs are always in $(0, 1)$ ; negative logits give small probabilities, never negative ones.

Recall (Cell C) All

$K$ logits are equal. What is each probability? Exactly $1/ K$ — the uniform distribution. The equal exponentials cancel, leaving only the count.

Recall (Cell E) One logit

$\to + \infty$ , the rest fixed. What does softmax approach? A one-hot vector (all mass on that class). Softmax is the smooth version of arg-max.

Recall (Cell D & K) Subtracting

$max_{k} z_{k}$ fixes which two numerical failures? Overflow (a huge logit sends $e^{z} \to \infty$ ) and underflow (all logits hugely negative send every $e^{z} \to 0$ so $S = 0$ ). The shift forces the top exponent to be $e^{0} = 1$ , curing both.

Recall (Cell G/H) Why do the entries of

$(p - t)$ always sum to zero, and does one example update all $K$ weight vectors? They sum to zero because $\sum_{k} p_{k} = 1$ and $\sum_{k} t_{k} = 1$ . Yes — all $K$ weight vectors update on each example, since $\nabla_{w_{m}} J = (p_{m} - t_{m}) x$ is nonzero for every class (they share the normalizer $S$ ).

Recall (Cell J) Two logit vectors differ by a constant in every slot. Same softmax?

Yes — shift-invariance. The common $e^{c}$ factor cancels in the ratio. This is exactly why subtracting the max is loss-free.

Connections

Multinomial - softmax regression (index 2.2.12) — the parent machinery these cases exercise.
Logistic Regression — the $K = 2$ special case (Ex 6).
Cross-Entropy Loss — the loss computed in Ex 7–9.
Maximum Likelihood Estimation — why $- lo g p_{true}$ is the right penalty.
Gradient Descent — the $(p - t) x$ updates.
Log-Sum-Exp Trick — the stability proof in Ex 4, Ex 10, and Ex 11.
One-Hot Encoding — the target $t$ and the limit of Ex 5.
Neural Network Output Layer — where all this lives in practice.