Image classification pipeline: Visual walkthrough

The heart of the image classification pipeline is the moment a stack of raw scores becomes a set of confidences you can trust. This page builds that moment from absolute zero. No symbol appears before we draw it.

Step 1 — What the network hands us: a row of "scores"

WHAT. The last layer of the network gives us a short list of numbers, one per possible class. If there are 3 classes (say cat, dog, fox), we get three numbers. We call each number a logit and write the whole row as $z$ :

z = (z_{1}, z_{2}, z_{3})

$z$ — the whole row of raw scores (a list).
$z_{1}$ — the score for class 1. The subscript $1$ just means "first slot".
The comma-list means these are separate numbers sitting side by side.

WHY. These numbers are the network's gut feeling, but they are useless as-is: a score of $- 2$ or $+ 5$ means nothing on its own. We cannot say "the network is 5 sure it's a dog." We need to compare them and convert to something bounded.

PICTURE. Look at the three bars below. Their heights are the raw logits — one dips below zero (negative score allowed!), and they clearly do not add to anything meaningful.

Step 2 — Try the naïve fix, and watch it break

WHAT. The obvious idea: to make numbers add to 1, divide each by their total. That is

p_{i} = \frac{z _{i}}{z _{1} + z _{2} + z _{3}}

$p_{i}$ — our attempted probability for class $i$ .
$z_{i}$ — the score of that class (top).
$z_{1} + z_{2} + z_{3}$ — the running total of all scores (bottom).

WHY (it fails). Two disasters. First, if any $z_{i}$ is negative, we get a negative "probability" — impossible. Second, if the scores sum to zero (e.g. $z = (2, - 2, 0)$ ), we divide by zero — the formula explodes.

PICTURE. The red bar is a negative score. Dividing it by the total produces a bar pointing below the zero line — a negative probability. That is our warning sign: raw division is not allowed.

Step 3 — The tool that fixes "must be positive": the exponential

WHAT. We pass every score through the exponential function $e^{(\cdot)}$ . Here $e \approx 2.718$ is a fixed number, and $e^{z}$ means " $e$ raised to the power $z$ ". We create new numbers

t_{i} = e^{z_{i}}

$t_{i}$ — the exponentiated score for class $i$ (always positive).
$e^{z_{i}}$ — $e$ raised to that class's logit.

WHY this tool and not another? We need a function that (a) turns any real number — even negative — into a positive one, and (b) keeps the order: a bigger score must stay bigger. The exponential is the natural choice because:

$e^{z} > 0$ for every $z$ , including negatives (e.g. $e^{- 2} \approx 0.135$ , still positive).
$e^{z}$ is strictly increasing, so bigger logit → bigger $t_{i}$ . Order preserved.

PICTURE. The black curve is $e^{z}$ . Follow the red guide line: a negative input at the bottom still lands on a positive output. The curve never touches zero and never goes below it.

Step 4 — Now divide safely: the softmax formula appears

WHAT. Every $t_{i} = e^{z_{i}}$ is positive, so their sum is positive too — no more divide-by-zero, no more negatives. So now we do the "divide by the total" step from Step 2, but on the safe positive numbers:

\overset{y}{^}_{i} = \frac{t _{i}}{t _{1} + t _{2} + t _{3}} = \frac{e ^{z_{i}}}{j = 1 \sum C e ^{z_{j}}}

Term by term:

$\overset{y}{^}_{i}$ — the final probability for class $i$ (the hat means "the network's estimate").
$e^{z_{i}}$ (top) — the positive, exponentiated score of the class we're asking about.
$\sum_{j = 1}^{C} e^{z_{j}}$ (bottom) — the sum of the same transform over all $C$ classes. The symbol $\sum$ ("sigma") means "add up"; $j$ runs from $1$ to $C$ ; $C$ is the number of classes.
The bottom is the same for every $i$ — it is the shared normaliser that guarantees the outputs add to 1.

This is the softmax function, and it feeds straight into Softmax activation.

WHY it works now. Top is positive → each $\overset{y}{^}_{i} > 0$ . Bottom is the sum of the tops → dividing forces $\sum_{i} \overset{y}{^}_{i} = 1$ . Both probability laws satisfied, and no input can break it.

PICTURE. Left: the positive exponentials as tall bars. Right: after dividing by their total, they become the red probability bars whose heights stack to exactly 1 (the dashed line at the top).

Step 5 — Why exponential amplifies the winner (the secret bonus)

WHAT. Softmax does more than normalise — it sharpens. Compare two classes by dividing their probabilities:

\frac{y ^ _{a}}{y ^ _{b}} = \frac{e ^{z_{a}}}{e ^{z_{b}}} = e^{z_{a} - z_{b}}

$z_{a} - z_{b}$ — the gap between the two scores.
$e^{z_{a} - z_{b}}$ — that gap, exponentiated: the ratio depends only on the difference, and it grows fast.

WHY this matters. If $z_{a} = 10$ and $z_{b} = 8$ , the gap is $2$ , so class $a$ is $e^{2} \approx 7.4 \times$ more likely than $b$ — a confident vote. Plain division (Step 2) would give only $10/8 = 1.25 \times$ — a wishy-washy vote. The exponential turns small score gaps into decisive preferences, exactly what a classifier wants.

PICTURE. The black line is naïve division's ratio (nearly flat as the gap grows). The red curve is softmax's ratio $e^{gap}$ , shooting upward — small gaps become strong decisions.

Step 6 — The degenerate cases: ties, huge numbers, one giant

WHAT & WHY — three edge cases every reader must see.

All scores equal ( $z = (4, 4, 4)$ ). Then every $e^{z_{i}}$ is the same, so $\overset{y}{^}_{i} = 1/ C$ for all $i$ . With $C = 3$ that's $(0.333, 0.333, 0.333)$ — pure "I have no idea", spread evenly. Correct behaviour.
One score dominates ( $z = (1, 1, 20)$ ). The bottom is basically $e^{20}$ , so the big class gets $\approx 1$ and the rest $\approx 0$ . Softmax approaches a hard "pick the max" — but stays smooth (never exactly 0 or 1), which backprop needs to keep learning.
Enormous scores overflow ( $z = (1000, 1001)$ ). A computer cannot store $e^{1000}$ . The fix: subtract the biggest logit from all of them before exponentiating. Since Step 5 showed only differences matter, this changes nothing about the answer:

\overset{y}{^}_{i} = \frac{e ^{z_{i} - m}}{\sum _{j} e ^{z_{j} - m}}, m = j max z_{j}

$m$ — the largest logit in the row.
$z_{i} - m$ — every score shifted down so the biggest becomes $0$ ; now $e^{0} = 1$ is the largest term, safe to compute.

PICTURE. Three tiny bar charts side by side: the flat tie, the single-spike winner, and the shifted-then-safe version. Same shape before and after shifting — proof the shift is free.

The one-picture summary

The whole journey: raw logits (can be negative, no rules) → exponentiate (all positive, order kept, winner amplified) → divide by the total (heights now sum to 1) → a valid probability row.

Recall Feynman retelling — say it like you're explaining to a friend

The network ends by shouting out a few raw numbers, one per class — like judges holding up scorecards, and some scores are even negative. We want to turn those scorecards into "how sure are we, in percent." The dumb way — just divide each score by the total — breaks the moment a score is negative or the total is zero. So first we run every score through $e^{(\cdot)}$ , which does two lovely things at once: it makes any number positive (negatives included), and it keeps the ranking (bigger stays bigger) while blowing up the gaps so the leader really pulls ahead. Now that all our numbers are safely positive, we finally divide each by their total — and because they're all positive, the results are guaranteed between 0 and 1 and add up to exactly 1. If two scores tie, we get an even split ("no idea"). If one score towers over the rest, that class gets nearly all the probability but never exactly 1, so learning can continue. And if the scores are astronomically large, we quietly subtract the biggest one first — which the math proves changes nothing — so the computer doesn't choke. That's softmax: exponential to be safe and sharp, divide to be fair.

Recall

What is a logit? ::: A raw, unbounded score the network's final layer outputs for each class, before any conversion to probability. Why exponentiate before normalizing instead of dividing raw scores? ::: $e^{z}$ is always positive (handles negative scores) and keeps ordering, so the divide can't produce negatives or blow up; it also amplifies score gaps into confident decisions. Why does softmax always sum to 1? ::: Each output divides a term by the sum of all those same terms, so adding the outputs reconstructs the whole sum over itself, which is 1. How do we prevent numerical overflow with huge logits? ::: Subtract the maximum logit $m$ from every score before exponentiating; since only differences matter, the probabilities are unchanged. What does softmax output when all logits are equal? ::: A uniform distribution, $1/ C$ for each of the $C$ classes.

See also: Cross-entropy loss (the partner that scores these probabilities), Backpropagation (learns the logits), ResNet architecture and Transfer learning (produce the logits in the parent pipeline), and Multi-label classification (where softmax is replaced because classes are no longer mutually exclusive).