Exercises — Image classification pipeline

WHAT we do: plug numbers into the normalization formula from the parent note. WHY: normalization re-centers data on $0$ and rescales spread to $1$ so no single feature dominates the loss. $x_{\text{normalized}}=\frac{200-100}{50}=\frac{100}{50}=2$ The value $2$ means "this pixel sits $2$ standard deviations above the mean" — bright relative to typical pixels.

WHAT: apply softmax to three identical logits. WHY: when all logits are equal, exponentiating and dividing must treat every class the same — symmetry. ${\hat{y}}_i=\frac{e^1}{e^1+e^1+e^1}=\frac{e^1}{3e^1}=\frac{1}{3}\approx 0.333$ Every class gets $1/3$. The pie is split into three equal slices.

• (a) Preprocessing → resize + normalize the raw pixels so all images share one scale.
• (b) Feature extraction → convolution + pooling learn edges → textures → parts.
• (c) Classification → fully connected layers + softmax map features to class probabilities.
• (d) Postprocessing → apply the argmax and a confidence threshold to produce a final label.

WHAT: two-class softmax. WHY exponentials and not raw division? $e^x$ is always positive (so probabilities can never go negative) and it amplifies gaps — a logit gap of $2$ becomes a probability ratio of $e^2\approx 7.4$. ${\hat{y}}_0=\frac{e^2}{e^2+e^0}=\frac{7.389}{7.389+1}=\frac{7.389}{8.389}\approx 0.881$ ${\hat{y}}_1=\frac{e^0}{8.389}=\frac{1}{8.389}\approx 0.119$ Check: $0.881+0.119=1.000$. ✓

WHAT: feed the correct-class probability into the surprise meter. WHY only one term? The label is one-hot: $y_0=1$, $y_1=0$. Every term $y_c\log {\hat{y}}_c$ with $y_c=0$ vanishes, so only the true class survives. $\mathcal{L}=-\log (0.881)\approx 0.127$ Small loss → the network was confidently correct. Good.

WHAT: scale to $[0,1]$ first (divide by $255$), then normalize — this is the exact recipe Transfer learning with ImageNet weights expects. $\frac{180/255-0.485}{0.229}=\frac{0.7059-0.485}{0.229}=\frac{0.2209}{0.229}\approx 0.965$ Matching this distribution is why Batch normalization-trained backbones transfer cleanly.

WHAT: four-class softmax, top class. $e^{5.7}=298.87,e^{3.2}=24.53,e^{0.8}=2.23,e^{-1.5}=0.223$ Sum $=298.87+24.53+2.23+0.223=325.85$. ${\hat{y}}_0=\frac{298.87}{325.85}\approx 0.917$ WHY dominance: because $e^x$ grows exponentially, a logit that is only $2.5$ larger than the next produces a term $e^{2.5}\approx 12$ times bigger. The largest term alone is $\approx 92\%$ of the sum, so the denominator is approximately the top term — this is the "$85\%$ confidence" behaviour the parent example described.

WHAT: prove $\text{softmax}(z+k)=\text{softmax}(z)$. ${\hat{y}}_i=\frac{e^{z_i+k}}{{\sum }_j{e}^{z_j+k}}=\frac{e^k{e}^{z_i}}{e^k{\sum }_j{e}^{z_j}}=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}$ The common factor $e^k$ cancels top and bottom. WHY it matters: implementations subtract $k={\max }_j{z}_j$ before exponentiating to avoid overflow (computing $e^{1000}$ would crash) — and this proof shows that trick changes nothing. Check with $z=[2,0]$, $k=10\Rightarrow z=[12,10]$: ${\hat{y}}_0=\frac{e^{12}}{e^{12}+e^{10}}=\frac{e^2}{e^2+1}\approx 0.881$ Identical to Ex 2.1. ✓

${\mathcal{L}}_{\text{good}}=-\log (0.9)\approx 0.105,{\mathcal{L}}_{\text{bad}}=-\log (0.1)\approx 2.303$ Ratio $=2.303/0.105\approx 21.9$. WHY the huge gap: $-\log (p)$ shoots to $+\infty$ as $p\to 0$. Being confidently wrong is punished far more than being confidently right is rewarded — this asymmetry is what pushes gradient descent (via Backpropagation) away from disastrous mistakes first.

WHAT: apply the convolution output-size formula with padding $P=3$ on each side, so effective input $=W+2P=224+6=230$. $\text{out}=\lfloor \frac{230-7}{2}\rfloor +1=\lfloor \frac{223}{2}\rfloor +1=111+1=112$ WHY padding: without $P=3$ the map would shrink below $112$ and lose border information; padding preserves the intended $112\times 112$ used by ResNet-50. Full batch output tensor: $32\times 112\times 112\times 64$.

Sigmoid: $\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.1353}=\frac{1}{1.1353}\approx 0.881$. Softmax ${\hat{y}}_0$ for $[2,0]$ (Ex 2.1) $\approx 0.881$ too — numerically equal here because two-class softmax is a sigmoid of the logit difference. Conceptual difference: softmax forces the probabilities to compete (they must sum to $1$ — "exactly one class"). Sigmoid treats each class independently (each answer is its own yes/no — "cat AND outdoors AND sunny" can all be true). WHY it matters: choose softmax + cross-entropy for mutually exclusive labels; choose sigmoids + per-class loss for multi-label tasks like Object detection attributes.

Expected flips $=10,000\times 0.5=5,000$. WHY it helps: a horizontally flipped dog is still a dog. Showing both orientations teaches the network that class identity is invariant to left-right mirroring, enlarging the effective dataset without collecting new photos — reducing overfitting exactly as pooling's spatial invariance does inside the network.

WHAT: compute how loss changes as we nudge one logit — the signal Backpropagation sends backward. Setup: $\mathcal{L}=-{\sum }_c{y}_c\log {\hat{y}}_c$, with ${\hat{y}}_c=\frac{e^{z_c}}{{\sum }_j{e}^{z_j}}$. Step 1 — softmax derivative. For the same index: $\frac{\partial {\hat{y}}_i}{\partial z_i}={\hat{y}}_i(1-{\hat{y}}_i)$. For a different index $c\ne i$: $\frac{\partial {\hat{y}}_c}{\partial z_i}=-{\hat{y}}_c{\hat{y}}_i$. WHY these two cases: raising $z_i$ makes class $i$'s slice grow but steals probability from every other slice, hence the negative cross-terms. Step 2 — chain rule. $\frac{\partial \mathcal{L}}{\partial z_i}=-{\sum }_c\frac{y_c}{{\hat{y}}_c}\frac{\partial {\hat{y}}_c}{\partial z_i}=-\frac{y_i}{{\hat{y}}_i}{\hat{y}}_i(1-{\hat{y}}_i)-{\sum }_{c\ne i}\frac{y_c}{{\hat{y}}_c}(-{\hat{y}}_c{\hat{y}}_i)$ $=-y_i(1-{\hat{y}}_i)+{\hat{y}}_i{\sum }_{c\ne i}{y}_c=-y_i+y_i{\hat{y}}_i+{\hat{y}}_i{\sum }_{c\ne i}{y}_c$ Step 3 — collapse the label sum. Since ${\sum }_c{y}_c=1$ (one-hot), $y_i+{\sum }_{c\ne i}{y}_c=1$, so $y_i{\hat{y}}_i+{\hat{y}}_i{\sum }_{c\ne i}{y}_c={\hat{y}}_i$. $\boxed{\frac{\partial \mathcal{L}}{\partial z_i}={\hat{y}}_i-y_i}$ Beautifully simple: the gradient is just "predicted minus true". Evaluate for $z=[2,0]$, true class $0$: ${\hat{y}}_0\approx 0.881,y_0=1\Rightarrow \partial \mathcal{L}/\partial z_0=0.881-1=-0.119$; and ${\hat{y}}_1\approx 0.119,y_1=0\Rightarrow 0.119-0=0.119$. Sign check: $z_0$ should increase (negative gradient → gradient descent moves $z_0$ up), $z_1$ should decrease. Exactly what we want.

$e^{1.5}=4.4817,e^{1.2}=3.3201,e^{1.0}=2.7183,\text{sum}=10.5201$ ${\hat{y}}_{\max }=\frac{4.4817}{10.5201}\approx 0.426$ Since $0.426<0.7$, the pipeline outputs "uncertain" and abstains. WHY thresholding exists: on out-of-distribution or ambiguous images the logits are close together, softmax spreads probability thin, and a low peak is an honest signal that no class is trustworthy.

(a) From Ex 3.1: ${\hat{y}}_0=298.87/325.85\approx 0.917$. (b) $\mathcal{L}=-\log (0.917)\approx 0.0866$ — low loss, confidently correct. (c) $\partial \mathcal{L}/\partial z_0={\hat{y}}_0-y_0=0.917-1=-0.083$ — a small negative nudge, because the network is already nearly right; little learning left on this example. (d) ${\hat{y}}_{\max }=0.917\ge 0.7\Rightarrow$ output "dog" with $91.7\%$ confidence. This is the whole pipeline in four numbers: normalize → softmax → loss → gradient → decision.

Softmax over equal logits $[1,1,1]$ ::: each probability is $1/3\approx 0.333$ Loss when correct-class probability is $0.9$ (natural log) ::: $-\ln (0.9)\approx 0.105$ The softmax+cross-entropy logit gradient ::: ${\hat{y}}_i-y_i$ ("predicted minus true") Why subtract $\max$ before softmax ::: shift-invariance means it changes nothing but prevents overflow Softmax vs sigmoid head choice ::: softmax for mutually exclusive classes (sum to 1); sigmoids for independent multi-label