Softmax for output layers: Worked examples — AI-ML deep dive

This is the hands-on companion to Softmax for output layers. The parent note built the machine; here we run it on every kind of input it could ever see — big numbers, tiny numbers, negatives, ties, a single class, extreme gaps, a word problem, and an exam twist. Nothing here contradicts the parent; we only go deeper and cover the corners.

Before line one, a promise: you never need to have seen $e^{z}$ , a "distribution," or a "gradient" before. We rebuild each as it appears.

What we are actually computing (from zero)

The scenario matrix

Every case softmax can be handed. Each later example is tagged with the cell it covers.

Cell	Input class	What is special	Example
A	All small positive logits	ordinary, no tricks	Ex 1
B	Mixed signs (negative logits)	must not give negative "probabilities"	Ex 2
C	Huge logits (overflow)	needs the max-subtraction shift	Ex 3
D	An exact tie between logits	equal slices — symmetry	Ex 4
E	Extreme gap (one logit dominates)	limiting behaviour → argmax	Ex 5
F	Single class ( $K = 1$ )	degenerate: must output $1$	Ex 6
G	Word problem (real-world)	translate a story into logits	Ex 7
H	Exam twist: temperature scaling	dividing by $T$ sharpens/softens	Ex 8
I	Gradient sanity ( $p - t$ )	check the learning signal's sign	Ex 9

Recall Why these cells and not others?

Cell E ::: The limiting case: as gaps grow, softmax approaches the hard argmax — proves it is a soft max. Cell F ::: The degenerate case $K = 1$ : one class must always get probability $1$ ; a formula that failed here would be broken. Cell D ::: The symmetry case: identical inputs must give identical outputs, or the function is unfair.

Cell A — ordinary small logits

Cell B — negative logits (mixed signs)

Cell C — overflow, needs the shift

Cell D — an exact tie (symmetry)

Cell E — extreme gap, the limiting case

Cell F — single class, the degenerate case

Cell G — a real-world word problem

Worked example Ex 7 · Cell G

A weather model scores tomorrow: sunny gets logit $2.2$ , cloudy $2.0$ , rainy $- 0.5$ . Report the forecast probabilities.

Forecast: sunny and cloudy are close — will their probabilities be close too?

Exponentiate: $e^{2.2} = 9.025$ , $e^{2.0} = 7.389$ , $e^{- 0.5} = 0.607$ . Why this step? Convert the model's raw preference scores into positive weights.
Sum: $9.025 + 7.389 + 0.607 = 17.021$ .
Divide: $p_{sun} = 0.530$ , $p_{cloud} = 0.434$ , $p_{rain} = 0.036$ .

Verify: $0.530 + 0.434 + 0.036 = 1.000$ ✔ (units: probabilities, dimensionless). A small logit gap ( $0.2$ ) gives a small probability gap — sunny barely beats cloudy, exactly the "uncertain call" a human would make. Note: because these are mutually exclusive outcomes we use softmax, not independent Sigmoid Activation (that would be for "could be BOTH sunny and cloudy").

e^{- 1} = 0.368

), so no probability is ever negative.

Is max-subtraction exact or an approximation?

Exact — shift-invariance means the ratio is unchanged.

Softmax of

K

equal logits gives what?

The uniform distribution

1/ K

on every class.

Does softmax ever output exactly

1

for one class (with

K > 1

No — only in the limit; losers keep an infinitesimal positive slice.

Higher temperature

T

does what to the distribution?

Softens it toward uniform.

Connections

Softmax for output layers — the parent this page drills into.
Cross-Entropy Loss — supplies the $p - t$ gradient of Ex 9.
Temperature Scaling — the $T$ trick of Ex 8.
Maximum Entropy Distributions — explains the uniform tie of Ex 4.
Logits and Log-Odds — the raw inputs across all cells.
Sigmoid Activation — the multi-label alternative flagged in Ex 7.
Backpropagation — where Ex 9's gradient flows.
3.1.06 Softmax for output layers (Hinglish) — same ideas in Hinglish.

Worked examples — Softmax for output layers