Softmax for output layers: Visual walkthrough

Before we begin, let us fix the vocabulary in the plainest possible words.

Our whole job: take a list of logits and turn it into a distribution. Let us call the logits $z_{1}, z_{2}, z_{3}$ (the letter $z$ is standard for "raw score"; the little number is which class). Throughout we use the running example

$z_{1} = 2.0, z_{2} = 1.0, z_{3} = 0.1.$

Step 1 — Draw the raw logits and see the problem

WHAT. Plot the three raw scores as bars on a number line.

WHY. Before fixing anything we must see what is broken. We want probabilities (positive, summing to $1$ ). Look at the bars and ask: do these numbers already behave like probabilities?

PICTURE. In the figure below, one bar dips below zero territory in spirit (logits are allowed to be negative — imagine $z_{3}$ were $- 0.5$ instead), and their total is $2.0 + 1.0 + 0.1 = 3.1$ , not $1$ .

$biggest score z_{1} = 2.0, middle z_{2} = 1.0, smallest, but still positive here z_{3} = 0.1$

Two problems, stated exactly:

Problem A — sign: a logit is allowed to be negative, and a probability can never be negative.
Problem B — total: the bars add to $3.1$ , not $1$ .

We will kill Problem A in Step 2 and Problem B in Step 4.

Step 2 — Why we bend the numbers through $e^{z}$

WHAT. Pass every logit through the exponential function $e^{z}$ . Here $e \approx 2.718$ is a fixed number, and $e^{z}$ means " $e$ raised to the power $z$ ."

WHY this tool and not another? We need a function that:

turns any real input into a strictly positive output (kills Problem A — the sign problem), and
preserves order — a bigger logit must stay bigger — so the winner stays the winner, and
is smooth (differentiable) so we can later train by gradient descent.

The exponential does all three. Look at its graph: it lives entirely above the horizontal axis (always positive), it only ever climbs (order preserved), and it has no corners (smooth). A straight line $f (z) = z$ fails #1 (goes negative); squaring $f (z) = z^{2}$ fails #2 (it maps $- 3$ and $+ 3$ to the same $9$ , scrambling order). Exponential is the natural pick.

PICTURE. The red curve is $e^{z}$ . Trace each logit up to the curve and across: negatives land just above zero, positives shoot up high.

$e^{z_{1}} = e^{2.0} = 7.389, e^{z_{2}} = e^{1.0} = 2.718, e^{z_{3}} = e^{0.1} = 1.105$

Read that as: the input $z_{i}$ (a raw score) becomes $e^{z_{i}}$ (a positive weight). Every weight is now above zero — Problem A is gone.

Step 3 — See how $e^{z}$ amplifies the gap

WHAT. Compare the spacing of the bars before and after exponentiating.

WHY. Softmax is meant to be a soft argmax — a smooth way of "picking the biggest." For that we want the leader to pull ahead, not just barely lead. The exponential stretches gaps: equal steps in $z$ become multiplied steps in $e^{z}$ , because going from $z$ to $z + 1$ multiplies the output by $e$ .

PICTURE. On the left, the raw gaps between $2.0, 1.0, 0.1$ are modest. On the right, $7.389$ towers over $1.105$ — the same gaps, exaggerated. The tallest bar (red) is now dramatically dominant.

$\frac{e ^{z_{1}}}{e ^{z_{3}}} = \frac{7.389}{1.105} = 6.69 versus the raw ratio \frac{2.0}{0.1} = 20 — but the raw one is meaningless because logits can be negative.$

The point: exponentiating gives a confident-but-still-smooth winner. Nobody is crushed to exactly zero (every $e^{z} > 0$ ), so the loser still gets some share — crucial for learning.

Step 4 — Normalize: divide by the total

WHAT. Add up all the positive weights to get the normalizer $S$ , then divide each weight by $S$ .

WHY. After Step 2 we have positive weights, but they still total $S = 7.389 + 2.718 + 1.105 = 11.212$ , not $1$ (Problem B). Dividing every weight by the same total $S$ shrinks the whole pile so it sums to exactly $1$ , without changing any ratios — the winner stays the winner, just measured as a slice of the whole.

PICTURE. Think of a pizza. The total weight $S$ is the whole pizza; each $e^{z_{i}}$ is one person's claim; $p_{i}$ is the slice they actually get. Slices are drawn as a pie — they must fill the circle exactly once.

$p_{i} = \frac{e ^{z_{i}} this class’s positive weight}{total of ALL weights = S \sum _{j = 1}^{K} e ^{z_{j}}}$

Term by term: the top $e^{z_{i}}$ is how loud class $i$ shouted; the bottom $\sum_{j} e^{z_{j}}$ is how loud everyone shouted together; the fraction is class $i$ 's share of the noise. For our numbers:

$p_{1} = \frac{7.389}{11.212} = 0.659, p_{2} = \frac{2.718}{11.212} = 0.242, p_{3} = \frac{1.105}{11.212} = 0.099.$

Check: $0.659 + 0.242 + 0.099 = 1.000$ . ✔️ Both problems solved. This fraction is the softmax.

Step 5 — Edge case: what if all logits are equal?

WHAT. Set $z_{1} = z_{2} = z_{3} = c$ for any constant $c$ (say all zero, or all $5$ ).

WHY. A good formula must behave sensibly at its degenerate inputs. If the network has no preference — all scores equal — the only fair answer is "totally unsure": every class equally likely. Does softmax do that?

PICTURE. Three equal bars in, three equal slices out — a perfectly even pie of $1/3$ each.

$p_{i} = \frac{e ^{c}}{e ^{c} + e ^{c} + e ^{c}} = \frac{e ^{c}}{3 e ^{c}} = \frac{1}{3} .$

The $e^{c}$ cancels top and bottom, leaving $1/ K$ (here $K = 3$ ). So equal logits $\Rightarrow$ uniform distribution — exactly the "maximally uncertain" answer. This is the maximum-entropy behaviour: with no information distinguishing the classes, spread the probability evenly.

Step 6 — Edge case: giant logits and the shift trick

WHAT. Take huge logits $z = [1000, 1001, 1002]$ . Computing $e^{1002}$ directly overflows to infinity on a real computer.

WHY. The formula is mathematically fine but numerically dangerous. Rescue: subtract the largest logit from every logit before exponentiating. This works because softmax is shift-invariant — adding (or subtracting) the same constant $c$ to every logit changes nothing.

PICTURE. The three bars slide left together (a rigid shift). Their relative heights — and therefore the pie slices — are untouched. Only the danger of overflow moves away.

Why the shift is free, shown symbol by symbol:

$\frac{e ^{z_{i} + c}}{\sum _{j} e ^{z_{j} + c}} = \frac{e ^{c} e ^{z_{i}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{c} e ^{z_{i}}}{e ^{c} \sum _{j} e ^{z_{j}}} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}} .$

The common factor $e^{c}$ pulls out of every term and cancels. Choose $c = - max_{k} z_{k}$ so the biggest shifted logit is $0$ and $e^{0} = 1$ — the largest exponential is now a safe $1$ , never infinity. For our example, shifted logits are $[- 2, - 1, 0]$ , giving $p = [0.090, 0.245, 0.665]$ — identical to the honest answer, but with no overflow.

Step 7 — The payoff: the gradient becomes $p - t$

WHAT. Pair softmax with the cross-entropy loss $L = - \sum_{i} t_{i} lo g p_{i}$ , where $t_{i}$ is the target: $1$ for the true class, $0$ otherwise (a "one-hot" list). The gradient of the loss with respect to each logit collapses to a stunningly simple form.

WHY show it here. This is the reason softmax is the standard output layer. When you push the loss back through softmax and the log (see Backpropagation), all the messy pieces cancel and you are left with just prediction minus target.

PICTURE. For each class, an arrow shows which way its logit is pushed. True class: prediction below $1$ , so the arrow points up (raise this logit). Wrong classes: prediction above $0$ , arrows point down.

$\frac{\partial L}{\partial z _{k}} = what we predicted p_{k} - what was true t_{k}$

Take $p = [0.7, 0.2, 0.1]$ with true class $0$ so $t = [1, 0, 0]$ :

$p - t = [0.7 - 1, 0.2 - 0, 0.1 - 0] = [- 0.3, 0.2, 0.1] .$

Read the signs: negative on the true class means "push its logit up"; positive on the wrong classes means "push them down." Exactly what learning should do — and there is no vanishing factor hiding in the softmax, because the $e^{z}$ and the $lo g$ perfectly cancel. That clean cancellation is the whole magic.

The one-picture summary

The entire pipeline in one flow: raw logits → exponentiate (positive + gap-amplified) → sum & divide (normalize to a pie of total $1$ ) → probabilities; and when trained, the backward signal is simply $p - t$ .

Recall Feynman: the whole walkthrough in plain words

Three friends shout how much they want their restaurant, on a "loudness" scale where the number can even go negative (grumbling). Step 1: those raw shouts are not fair shares yet — some are negative, and they don't add to a whole. Step 2–3: we run each shout through a machine ( $e^{z}$ ) that (a) makes every number positive, so nobody has a negative claim, and (b) makes loud shouts much louder, so the leader clearly leads. Step 4: we bake one pizza whose size is the total of all shouts, and give each friend a slice equal to their share — now the slices fill exactly one pizza. Step 5: if everyone shouts the same, everyone gets an equal slice — perfect fairness under total ignorance. Step 6: if everyone shouts insanely loudly, we quietly subtract the loudest shout from all of them first — the slices don't change, but the computer stops choking. Step 7: to teach the network, we compare each predicted slice $p$ to the true answer $t$ ; the correction is just "slice minus truth," nudging the true class up and the rest down.

Active Recall

Two problems with raw logits?

They can be negative (probabilities cannot) and they don't sum to

1

Why exponential and not squaring?

e^{z}

is positive AND order-preserving AND smooth;

z^{2}

scrambles order (maps

- 3

and

3

to the same value).

Effect of dividing by the sum?

Normalizes the positive weights so they total exactly

1

, without changing their ratios.

Softmax of all-equal logits?

The uniform distribution

1/ K

per class — maximally uncertain.

Why is the max-subtraction free?

Shift-invariance: the common factor

e^{c}

cancels top and bottom, leaving the ratio unchanged.

Gradient of softmax + cross-entropy?

\partial L / \partial z_{k} = p_{k} - t_{k}

— raise the true logit, lower the wrong ones.

Connections

Parent: Softmax for output layers — the topic this page derives visually.
Cross-Entropy Loss — the partner giving the clean $p - t$ gradient in Step 7.
Sigmoid Activation — the 2-class / multi-label sibling.
Logits and Log-Odds — the raw inputs of Step 1.
Maximum Entropy Distributions — why the exponential form (Steps 2, 5).
Backpropagation — where $p - t$ flows backward.
Temperature Scaling — sharpening/softening the pie by scaling logits.