Worked examples — Image classification pipeline

This page is the stress test for the image classification pipeline. The parent note showed you the "happy path": a clean dog image flowing from pixels to an 85% prediction. But real pipelines meet ugly inputs — all-black images, logits that are all equal, one giant logit that explodes the exponential, a tie between two classes. If you have never seen these cases, you will misread the output the day they appear.

Below we first lay out every case class the pipeline can throw at you, then work through each one from zero.

The scenario matrix

Everything downstream of the CNN body reduces to three tiny functions acting on a logit vector $z=(z_1,z_2,\dots ,z_C)$ (the raw scores before probabilities). Those functions are softmax (scores → probabilities) and cross-entropy (probabilities vs truth → one loss number). The matrix enumerates the corner cases of these plus preprocessing.

We now cover every cell with a worked example.

[!definition] The three functions we keep reusing

Here $e\approx 2.718$ is Euler's number and $\log$ is the natural logarithm (base $e$), so that $\log (e^x)=x$ exactly — this pairing is why softmax uses $e$ and loss uses $\log$: they undo each other cleanly in the gradient.

[!example] Example A — Baseline: one clear winner (cell A)

Statement. Three classes, logits $z=(2,0,-1)$. Find the probabilities and the prediction.

Forecast: class 1 should win. Guess: roughly how confident — 60%? 80%? Write your guess down.

1. Exponentiate each logit. $e^2=7.389,e^0=1,e^{-1}=\mathrm{0.368.}$ Why this step? Softmax works on $e^{z_i}$, not $z_i$, because $e^x$ is always positive (probabilities can't be negative) and monotone (bigger score ⇒ bigger share).
2. Sum them. $S=7.389+1+0.368=\mathrm{8.757.}$ Why this step? The sum is the "whole pie" we divide into.
3. Divide. $\hat{y}=(7.389/8.757,1/8.757,0.368/8.757)=(0.844,0.114,0.042).$ Why this step? Dividing by $S$ forces the parts to sum to $1$ — the probability axiom.
4. Argmax. Largest is ${\hat{y}}_1=0.844\Rightarrow$ predict class 1.

Verify: $0.844+0.114+0.042=1.000$ ✓ (probabilities sum to one). The winner (84%) is more confident than the naive "2 out of 3" you might have guessed — softmax amplifies gaps.

[!example] Example B — All logits equal (degenerate, cell B)

Statement. $z=(0,0,0)$. What does the network say?

Forecast: the model has no reason to prefer any class. What must the probabilities be?

1. Exponentiate. $e^0=1$ for all three. Why this step? Same input ⇒ same exponential; symmetry is preserved.
2. Sum. $S=1+1+1=3.$
3. Divide. $\hat{y}=(1/3,1/3,1/3)=(0.333,0.333,0.333).$ Why this step? Equal shares of the pie.

Verify: exactly $1/C=1/3$ each; sum $=1$ ✓. This is the maximum-uncertainty output. Note it does not require zero logits — any equal vector like $(5,5,5)$ gives the same, which motivates Example D.

[!example] Example C — Huge logits & the overflow trick (cell C)

Statement. $z=(1000,999,998)$. Compute softmax without your calculator exploding.

Forecast: $e^{1000}$ is astronomically large (overflows to inf on a computer). Will the answer be extreme, or is the naive computation just numerically unsafe?

1. Subtract the max logit from every entry. $m=\max (z)=1000$, so $z-m=(0,-1,-2).$ Why this step? Softmax is unchanged when you shift all logits by a constant (proved in Example D). Shifting by $-m$ makes the biggest exponent $e^0=1$ — no overflow.
2. Exponentiate the shifted logits. $e^0=1,e^{-1}=0.368,e^{-2}=\mathrm{0.135.}$
3. Sum. $S=1+0.368+0.135=\mathrm{1.503.}$
4. Divide. $\hat{y}=(0.665,0.245,0.090).$

Verify: sum $=1.000$ ✓. The spacing of logits (gaps of $1$) — not their absolute size — sets the probabilities. Compare Example A's gaps: here the top class gets 66%, matching what evenly-spaced logits should give. This max-subtraction is exactly what every deep-learning library does internally.

[!example] Example D — Shift invariance (cell D)

Statement. Prove softmax gives the same answer for $z$ and $z+k$ (add the same constant $k$ to every logit). Check with $z=(2,0,-1)$ and $k=7$.

Forecast: does adding a bias to all scores change the ranking or the probabilities?

1. Write the shifted softmax. $\frac{e^{z_i+k}}{{\sum }_j{e}^{z_j+k}}=\frac{e^{z_i}{e}^k}{{\sum }_j{e}^{z_j}{e}^k}.$ Why this step? $e^{a+b}=e^a{e}^b$ — the property that lets us factor out $e^k$.
2. Cancel the common factor $e^k$ from top and bottom: $=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}.$ Identical to the unshifted softmax. ∎
3. Numeric check. $z+7=(9,7,6)$: $e^9=8103.08,e^7=1096.63,e^6=403.43$, sum $=9603.14$, giving $(0.844,0.114,0.042)$ — same as Example A.

Verify: the shifted probabilities equal Example A's to three decimals ✓. This is why the overflow trick in C is legal, and why a network's logit bias can drift freely without changing predictions.

[!example] Example E — Near-tie and the confidence threshold (cell E)

Statement. $z=(3.0,2.9,-4)$, three classes. The pipeline outputs "uncertain" if the top probability $<0.7$. What does it output?

Forecast: the top two logits are almost equal. Will confidence clear the 70% bar?

1. Overflow-safe: subtract $3.0$. $z-m=(0,-0.1,-7).$
2. Exponentiate. $e^0=1,e^{-0.1}=0.905,e^{-7}=\mathrm{0.000912.}$
3. Sum. $S=1+0.905+0.000912=\mathrm{1.9059.}$
4. Divide. $\hat{y}=(0.5247,0.4749,0.00048).$
5. Apply threshold. Top probability $0.5247<0.7\Rightarrow$ output "uncertain" (even though class 1 technically wins).

Verify: sum $=1.000$ ✓. A logit gap of only $0.1$ between the leaders produces a near-coin-flip (52% vs 47%). The reject-option (see Multi-label classification for a related many-answers view) exists precisely for this cell.

[!example] Example F — Wrong and confident: loss blows up (cell F)

Statement. True class is class 2, but the model gave $\hat{y}=(0.98,0.01,0.01)$. Compute the cross-entropy loss. Compare to a model that gave the truth $80\%$.

Forecast: the model is confidently wrong about class 2. Should the loss be tiny or huge?

1. Pick out the true-class probability. $t=2\Rightarrow {\hat{y}}_t=\mathrm{0.01.}$ Why this step? Cross-entropy $-\log ({\hat{y}}_t)$ only reads the probability of the correct answer.
2. Take negative log. $\mathcal{L}=-\log (0.01)=\mathrm{4.605.}$ Why this step? $\log$ of a tiny number is a large negative number; the minus sign flips it to a large positive loss — a heavy penalty for confident mistakes.
3. Compare to a decent model giving ${\hat{y}}_2=0.8$: $\mathcal{L}=-\log (0.8)=\mathrm{0.223.}$

Verify: $4.605$ vs $0.223$ — being confidently wrong costs about 20× more loss. This asymmetry is what pushes Backpropagation to fix egregious errors first.

[!example] Example G — The two limits of loss (cell G)

Statement. What is the loss as ${\hat{y}}_{\text{true}}\to 1$ (perfect) and as ${\hat{y}}_{\text{true}}\to 0$ (utterly wrong)? Illustrate the curve.

Forecast: best case and worst case — sketch the $-\log$ shape before reading.

1. Perfect prediction. ${\hat{y}}_t\to 1:\mathcal{L}=-\log (1)=0.$ Why this step? $\log (1)=0$; a perfectly confident correct answer incurs zero loss — the floor.
2. Confidently wrong. ${\hat{y}}_t\to 0^{+}:-\log ({\hat{y}}_t)\to +\infty .$ Why this step? $\log$ of numbers approaching $0$ heads to $-\infty$; the minus makes loss $\to +\infty$. There is no ceiling — the curve (red in the figure) shoots up vertically at ${\hat{y}}_t=0$.
3. Read off two waypoints. $-\log (0.5)=0.693,-\log (0.1)=\mathrm{2.303.}$

Verify: the figure's red curve passes through $(1,0)$, $(0.5,0.693)$, $(0.1,2.303)$ ✓ and grows without bound near $0$. This unbounded penalty is exactly why we clamp $\hat{y}$ away from $0$ in code (add a tiny $\epsilon$).

[!example] Example H — Degenerate preprocessing: a dead channel (cell H)

Statement. A calibration frame is a flat grey patch: every pixel in the red channel equals $128$. You try to normalize with the patch's own statistics $\mu =128,\sigma =0$. What happens, and what's the fix?

Forecast: every value equals the mean. What does subtracting the mean give — and then dividing by $\sigma$?

1. Center. $x^{\prime }=x_{\text{raw}}-\mu =128-128=0$ for every pixel. Why this step? Subtracting the mean is fine; a constant channel just becomes all zeros.
2. Scale — and hit the wall. $x_{\text{norm}}=x^{\prime }/\sigma =0/0$, which is undefined (division by zero). Why this step? $\sigma =0$ means "no spread"; there is nothing to rescale, so the formula is degenerate.
3. The standard fix: use fixed dataset stats, not per-image stats. With ImageNet red-channel $\mu =0.485,\sigma =0.229$ and $x_{\text{raw}}/255=128/255=0.502$: $x_{\text{norm}}=\frac{0.502-0.485}{0.229}=\mathrm{0.0742.}$ Why this step? Using global constants (or adding $\epsilon$ to $\sigma$) guarantees a finite, meaningful output even for flat inputs. This is also why Batch normalization adds a small $\epsilon$ inside its own denominator.

Verify: $128/255=0.50196$ and $(0.50196-0.485)/0.229=0.07405$ ✓ — finite. The lesson: never divide by an input's own $\sigma$ without a guard.

[!example] Example I — Real-world word problem: a QC reject camera (cell I)

Statement. A factory camera classifies each part as good, scratched, or cracked. For one part the logits are $z=(1.2,1.0,2.5)$. Policy: flag cracked (send to human) if $P(\text{cracked})>0.5$; otherwise auto-pass as good only if $P(\text{good})>0.6$; else route to manual review. What happens to this part?

Forecast: the cracked logit is the biggest — will it clear the 0.5 flag bar?

1. Overflow-safe: subtract $2.5$. $z-m=(-1.3,-1.5,0).$
2. Exponentiate. $e^{-1.3}=0.2725,e^{-1.5}=0.2231,e^0=1.$
3. Sum & divide. $S=1.4956\Rightarrow \hat{y}=(0.1822,0.1492,0.6686).$ Order: (good, scratched, cracked).
4. Apply policy. $P(\text{cracked})=0.6686>0.5\Rightarrow$ flag as cracked, send to human.

Verify: sum $=0.1822+0.1492+0.6686=1.000$ ✓, and $0.6686>0.5$ triggers the safety route. The good rule never even fires — correct, because we check the danger class first. Deciding on multiple thresholds like this is the bridge to Object detection confidence gating.

[!example] Example J — Exam twist: temperature scaling (cell J)

Statement. Take $z=(2,0,-1)$ (same as Example A). "Temperature" scaling replaces $z$ by $z/T$ before softmax. Compute the class-1 probability for a cold $T=0.5$ (sharpen) and a hot $T=2$ (soften), and state the limits as $T\to 0^{+}$ and $T\to \infty$.

Forecast: dividing by a small $T$ blows the gaps up; dividing by a large $T$ shrinks them. Which way does confidence move?

1. Cold, $T=0.5$: scale logits. $z/0.5=(4,0,-2).$ Why this step? Small $T$ multiplies every gap by $1/T=2$, exaggerating differences.
2. Softmax. $e^4=54.598,e^0=1,e^{-2}=0.1353$, sum $=55.734\Rightarrow {\hat{y}}_1=\mathrm{0.9796.}$
3. Hot, $T=2$: scale logits. $z/2=(1,0,-0.5).$ Why this step? Large $T$ shrinks gaps, flattening toward uniform.
4. Softmax. $e^1=2.7183,e^0=1,e^{-0.5}=0.6065$, sum $=4.3248\Rightarrow {\hat{y}}_1=\mathrm{0.6286.}$
5. Limits. As $T\to 0^{+}$: winner $\to 1$ (a hard argmax). As $T\to \infty$: all logits $\to 0$ ⇒ uniform $1/3$ (Example B).

Verify: cold ${\hat{y}}_1=0.9796$ (more confident than A's $0.844$), hot ${\hat{y}}_1=0.6286$ (less confident) ✓, and the two limits bracket everything between one-hot and uniform. Temperature is the standard knob for calibrating an overconfident classifier — often paired with Transfer learning models that ship over-sharp.

[!recall]- Quick self-test

Where to go next

• Softmax activation — the function stressed here, in full.
• Cross-entropy loss — the loss whose limits Examples F & G explored.
• Batch normalization — how the $\epsilon$-guard from cell H reappears mid-network.