Softmax for output layers

WHAT is softmax?

WHY these three design choices?

WHY exponentiate? We need every output positive. Raw logits can be negative; $e^{z_i}$ is always positive, so no probability is ever negative. Exponentiation also amplifies gaps — a slightly larger logit becomes a much larger probability, giving a confident-but-differentiable "winner."

WHY divide by the sum? Normalization. After exponentiating we have positive numbers, but they don't sum to 1. Dividing by their total forces a valid distribution — like sharing a pie so all slices add up to the whole pie.

WHY not just normalize the raw logits directly (i.e. $z_i / \sum z_j$ )? Because raw logits can be negative, giving negative "probabilities," and the sum could be zero (division blows up). The exponential fixes both.

HOW to derive it from first principles

We want a function $\mathbf{z} \to \mathbf{p}$ with these demands:

$p_i \ge 0$
$\sum_i p_i = 1$
Larger $z_i \Rightarrow$ larger $p_i$ (monotonic)
Differentiable everywhere (so we can train with gradient descent)

Step 1 — force positivity. Apply a positive, increasing function $f$ . The exponential $f(z)=e^{z}$ is the canonical choice (satisfies 1, 3, 4).

Why exp specifically? It arises naturally from the maximum-entropy / Gibbs distribution: if you assume the log-probability is linear in the score, $\log p_i \propto z_i$ , then $p_i \propto e^{z_i}$ . It's the least-biased distribution consistent with given expected scores.

Step 2 — enforce sum-to-1. We have unnormalized weights $e^{z_i}$ . Divide by their total: $p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$ Now $\sum_i p_i = \frac{\sum_i e^{z_i}}{\sum_j e^{z_j}} = 1$ . ✔️ Done — this is softmax.

Numerical stability trick (HOW to compute safely)

Why this is safe: the ratio is unchanged, so predictions are identical — we only fixed floating-point overflow.

The gradient (WHY softmax + cross-entropy is magic)

Softmax is almost always paired with the cross-entropy loss. For a true class $y$ (one-hot $t_i$ ), loss $L = -\sum_i t_i \log p_i$ .

The beautiful result — derive it:

$\frac{\partial p_i}{\partial z_k} = p_i(\delta_{ik} - p_k)$ where $\delta_{ik}=1$ if $i=k$ else $0$ . Combined with cross-entropy, the gradient w.r.t. the logits collapses to: $\boxed{\frac{\partial L}{\partial z_k} = p_k - t_k}$

Why this matters: the gradient is just (prediction − target). Clean, cheap, no vanishing pieces from the softmax and log cancelling. This is the reason they're used together.

Common mistake Steel-manned common errors

Mistake 1: "Softmax outputs are confidences/certainties." Why it feels right: they sum to 1 and look like probabilities. The fix: they're relative scores, easily overconfident and uncalibrated. A wrong prediction can still show $p=0.99$ . Softmax gives a distribution, not calibrated truth.

Mistake 2: "Use softmax for multi-label problems." Why it feels right: it's the go-to output layer. The fix: softmax forces classes to compete (sum to 1). If an image can be BOTH "beach" AND "sunny," use independent sigmoids per class instead.

Mistake 3: "Softmax then softmax again," or applying it twice / to already-normalized values. Why it feels right: seems harmless. The fix: applying softmax twice flattens the distribution toward uniform and destroys information. Apply it exactly once, to raw logits.

Mistake 4: forgetting the max-subtraction and getting NaN/inf. Why it feels right: the math formula has no subtraction. The fix: always subtract $\max z$ in code — mathematically identical, numerically safe.

Recall Feynman: explain to a 12-year-old

You have three friends voting on where to eat, and each shouts a "loudness score." Softmax turns those loudness scores into slices of a pizza. Louder shout ⇒ bigger slice, but every slice is positive and all slices together make one whole pizza. The loudest gets the most, but everyone gets some. That's how the computer says "I'm 66% sure it's a cat, 24% dog, 10% bird" — all adding up to one whole "sure."

Active Recall

Flashcards

What does softmax convert logits into?

A valid probability distribution — values in

(0,1)

that sum to 1.

Write the softmax formula for class

i

p_i = e^{z_i} / \sum_j e^{z_j}

Why exponentiate the logits?

e^{z}

is always positive (no negative probabilities) and amplifies gaps between scores, while staying differentiable.

Why divide by the sum of exponentials?

To normalize so all outputs sum to 1, forming a proper distribution.

What is softmax's shift-invariance property?

Adding a constant

c

to all logits leaves outputs unchanged:

\text{softmax}(z+c)=\text{softmax}(z)

What numerical trick prevents overflow?

Subtract

\max_k z_k

from every logit before exponentiating.

What is the gradient of cross-entropy+softmax w.r.t. logits?

\partial L/\partial z_k = p_k - t_k

(prediction minus target).

Softmax vs sigmoid output layer — when each?

Softmax for mutually-exclusive single-label classification; independent sigmoids for multi-label (classes not competing).

What's the derivative

\partial p_i/\partial z_k

p_i(\delta_{ik}-p_k)

Why is applying softmax twice wrong?

It flattens the distribution toward uniform and discards information; apply once to raw logits.

Connections

Cross-Entropy Loss — softmax's natural partner; together give gradient $p-t$ .
Sigmoid Activation — the 2-class / multi-label sibling of softmax.
Logits and Log-Odds — the raw inputs softmax consumes.
Maximum Entropy Distributions — theoretical justification for the exponential form.
Backpropagation — where the clean $p-t$ gradient flows.
Temperature Scaling — dividing logits by $T$ to sharpen/soften softmax (calibration).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho tumhare neural network ke last layer se teen raw numbers nikalte hain — inko logits kehte hain, jaise $[2.0, 1.0, 0.1]$ . Ye numbers negative bhi ho sakte hain, bade bhi — ye probabilities nahi hain. Softmax ka kaam hai in raw scores ko ek proper probability distribution mein badalna: har value $0$ se $1$ ke beech, aur sabka total exactly $1$ . Formula simple hai: pehle har logit ka $e^{z_i}$ nikaalo (exponentiate), phir sabke sum se divide karo (normalize). Bas — "Exponentiate, then Equalize".

Exp kyun? Kyunki $e^{z}$ hamesha positive hota hai, toh koi bhi probability negative nahi aayegi, aur bade-chote scores ke beech gap bhi badh jaata hai (confident winner milta hai). Divide by sum kyun? Taaki total $1$ ho jaaye — jaise ek poori pizza ko slices mein baant diya. Ek important practical trick: computation se pehle sabse bada logit subtract kar do (max subtraction). Isse answer bilkul same rehta hai (shift-invariance), par $e^{1000}$ jaisa overflow nahi hota.

Softmax ka best partner hai cross-entropy loss. Dono saath mein use karo toh gradient super clean nikalta hai: $p_k - t_k$ , yaani prediction minus target. Sahi class ka logit upar push hota hai, galat waale neeche — exactly jo learning ko chahiye. Isiliye classification networks mein ye combo standard hai.

Do galtiyan avoid karo: (1) softmax ko multi-label problem mein mat use karo — wahan classes compete karti hain (sum = 1), toh agar image "beach" aur "sunny" dono ho sakti hai toh alag-alag sigmoids lagao. (2) Softmax outputs ko "100% certain truth" mat samjho — ye relative scores hain, kabhi-kabhi overconfident aur miscalibrated hote hain.

WHAT is softmax?

WHY these three design choices?

HOW to derive it from first principles

Numerical stability trick (HOW to compute safely)

The gradient (WHY softmax + cross-entropy is magic)

Active Recall

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Neural Network Fundamentals

Connections