Worked examples — LeNet and AlexNet

This page is a workout. The LeNet and AlexNet parent note gave you the two architectures and one master formula. Here we use that formula until it can no longer surprise you. We will hit every kind of case a convolution/pooling problem can throw at you — clean divisions, floor-truncation, padding tricks, degenerate $1\times 1$ outputs, parameter counting, and a couple of exam-style traps.

Before any numbers, let us re-earn the single tool that runs this whole page.

Let us say what each piece means, because we will lean on all of them.

• $W$ = input side length — how wide the picture is, in pixels.
• $K$ = kernel side length — the width of the little sliding stamp.
• $P$ = padding — a border of zeros we optionally wrap around the picture so the stamp can reach the edges.
• $S$ = stride — how far the stamp hops between stamps. $S=1$ means "touch every pixel"; $S=4$ means "skip 3, land on the 4th".
• $\lfloor \cdot \rfloor$ = the floor, meaning "throw away the fractional part, round down". $\lfloor 3.9\rfloor =3$.

---

The scenario matrix

Every convolution/pooling question you will ever meet falls into one of these cells. The six examples that follow are each tagged with the cell(s) they cover, so you can see the whole space is filled.

Cell	Case class	What makes it tricky	Covered by
A	Clean divide ($W-K$ divisible by $S$)	Nothing — warm-up	Ex 1
B	Floor actually bites (remainder $\ne 0$)	Must truncate down	Ex 2
C	Padding used to preserve size	"same" convolution	Ex 3
D	Degenerate output $1\times 1$	Conv acts like fully-connected	Ex 4
E	Zero-stride-1 pooling chain	Repeated shrinking	Ex 4
F	The "224 vs 227" trap	Hidden padding	Ex 5
G	Parameter counting (weights + biases)	Channels multiply in	Ex 6
H	Activation edge case (ReLU/tanh sign)	Sign of input decides gradient	Ex 7
I	Real-world word problem	Translate English → $W,K,P,S$	Ex 8

---

Example 1 — Cell A: the clean divide (LeNet C1)

Forecast: guess the number before reading on. (Hint: the stamp is 5 wide, so it loses a 4-pixel margin.)

1. Identify the four numbers. $W=32$, $K=5$, $P=0$, $S=1$. Why this step? The formula is useless until every symbol has a value; naming them prevents mixing up kernel and stride.
2. Plug into the ruler. $O=\lfloor \frac{32-5+2(0)}{1}\rfloor +1=\lfloor 27\rfloor +1=28$ Why this step? $W-K$ measures how far the stamp's left edge can travel before its right edge falls off; dividing by $S$ counts the hops.
3. Attach the depth. 6 filters ⇒ output is $28\times 28\times 6$. Why this step? Each filter produces one 2-D map; depth equals the filter count, never the input channels.

Verify: count fence posts directly. Positions the stamp's left edge can occupy: $1,2,\dots ,28$ — that is $32-5+1=28$ slots. Matches. This is Cell A because $27/1$ divides exactly; floor changed nothing.

---

Example 2 — Cell B: when the floor actually bites (AlexNet Conv1, naive)

Forecast: will $224-11=213$ divide evenly by 4? Guess the leftover.

1. Numbers: $W=224$, $K=11$, $P=0$, $S=4$.
2. Numerator first. $224-11+0=213$. Why this step? Isolating the numerator lets us see the remainder before dividing.
3. Divide and floor. $\frac{213}{4}=53.25\Rightarrow \lfloor 53.25\rfloor =53$ Why this step? $213=4\times 53+1$; that stray $+1$ pixel is a strip too thin for another hop, so it is discarded — this is exactly the case where floor matters.
4. Add one. $53+1=54$. Output $54\times 54\times 96$.

Verify: $4\times 53=212\le 213$, but $4\times 54=216>213$, so 53 hops is the most that fit. The remainder $213-212=1$ pixel is lost. This is Cell B: had we used $S=1$ the answer would be $214$, but the coarse stride throws away information — which is precisely why the parent note warns the "224" number is a simplification. (Ex 5 fixes it.)

---

Example 3 — Cell C: padding to preserve the size (AlexNet Conv2)

Forecast: the parent note claims Conv2 keeps $27\times 27$. Can padding really cancel the kernel's shrinkage?

1. Numbers: $W=27$, $K=5$, $P=2$, $S=1$.
2. Why $P=2$ specifically? A $5\times 5$ stamp loses $K-1=4$ pixels of margin total (2 on each side). Adding $P=2$ zeros to each side donates back exactly those 4 pixels. Why this step? This is the "same-padding rule": choose $P=(K-1)/2$ for stride 1 to keep size fixed.
3. Plug in. $O=\lfloor \frac{27-5+2(2)}{1}\rfloor +1=\lfloor \frac{26}{1}\rfloor +1=27$

Verify: $27$ in, $27$ out — confirmed. Look at the figure: the violet zero-border lets the stamp centre reach the corner pixel, which without padding it never could. This is Cell C. Contrast with VGGNet, which uses this trick on every $3\times 3$ layer so only pooling ever changes the size.

---

Example 4 — Cells D & E: shrinking to a single pixel (LeNet C3 → S4 → C5)

Forecast: predict all three output sizes before computing.

1. C3 convolution. $W=14,K=5,P=0,S=1$: $O=\lfloor \frac{14-5}{1}\rfloor +1=9+1=10\Rightarrow 10\times 10\times 16.$ Why this step? Same ruler as always — a conv layer never gets special treatment.
2. S4 pooling. Pooling uses the same formula with $K=S=2$, $P=0$ (Cell E, repeated shrink): $O=\lfloor \frac{10-2}{2}\rfloor +1=4+1=5\Rightarrow 5\times 5\times 16.$ Why this step? A $2\times 2$ window hopping by 2 tiles the image into non-overlapping blocks — it halves each side. See Pooling Layers.
3. C5 convolution. $W=5,K=5,P=0,S=1$: $O=\lfloor \frac{5-5}{1}\rfloor +1=0+1=1\Rightarrow 1\times 1\times 120.$ Why this step? When $K=W$ the stamp fits in exactly one spot — the degenerate Cell D. The convolution now touches every input pixel at once, so it behaves identically to a fully-connected layer. That is why the parent note writes "(effectively FC)".

Verify: Chain $14\to 10\to 5\to 1$. Each transition satisfies its formula, and the final $1\times 1$ is the smallest legal output ($O\ge 1$ always, since $W\ge K$ here). Note $O=0$ would mean the kernel is bigger than the input — an illegal layer; the design avoids it exactly.

---

Example 5 — Cell F: the "224 vs 227" trap

Forecast: where could one extra pixel of output come from?

1. State the discrepancy. Formula on $224$: output $54$ (Ex 2). Paper: $55$. Off by one.
2. Hypothesise hidden padding. Suppose the true effective input is $227$, i.e. an implicit $P$ that makes $W+2P=227$. Why this step? One extra output pixel needs the numerator to grow by $S=4$ (since each hop of stride 4 adds one output). $227-11=216$, and $216-213=3$; combined with the border this pushes past the next multiple of 4.
3. Recompute with $227$. $O=\lfloor \frac{227-11}{4}\rfloor +1=\lfloor \frac{216}{4}\rfloor +1=54+1=55.$ Why this step? $216=4\times 54$ exactly — no floor loss this time — so we cleanly reach $55$.

Verify: $4\times 54=216\le 216$ and $4\times 55=220>216$, so $54$ hops, $+1=55$. Cell F resolved: the "$224$" is the crop size before an implicit 1.5-pixel-per-side pad rounds the effective field to $227$. Moral: always ask what the effective, post-padding input is.

---

Example 6 — Cell G: counting parameters (weights + biases)

Forecast: guess the order of magnitude — thousands? hundreds of thousands?

1. Weights of one filter. A single filter spans all input channels, so its weight count is $K\times K\times C_{\text{in}}=5\times 5\times 96=2400.$ Why this step? A filter is a 3-D block: it sees every input channel at each of its $5\times 5$ positions. Forgetting the $\times C_{\text{in}}$ is the classic error.
2. All filters. $256$ filters ⇒ $2400\times 256=614,400$ weights. Why this step? Each output channel has its own independent filter.
3. Biases. One bias per output channel: $+256$. $\text{Total}=614,400+256=614,656.$

Verify: dimensional sanity — weights scale as $C_{\text{in}}\cdot C_{\text{out}}\cdot K^2=96\cdot 256\cdot 25$, which is indeed $614,400$; biases equal $C_{\text{out}}$. Cell G complete. This is why the FC layers (see Ex, and Dropout) dominate AlexNet's ~60M total — a single $6\times 6\times 256\to 4096$ layer alone has $6\cdot 6\cdot 256\cdot 4096\approx 37.7$M weights.

---

Example 7 — Cell H: the activation sign edge case

Forecast: where does each function stop learning?

1. tanh gradient rule: $\frac{d}{dx}\tanh (x)=1-{\tanh }^2(x)$.
   • $x=-3$: $\tanh (-3)\approx -0.995$, gradient $\approx 1-0.990=0.0099$ — nearly dead (saturated).
   • $x=0$: gradient $=1-0=1$ — maximal.
   • $x=5$: $\tanh (5)\approx 0.9999$, gradient $\approx 0.0002$ — dead. Why this step? Large $|x|$ pushes $\tanh$ flat, so its slope $\to 0$ — the vanishing gradient the parent note warns about.
2. ReLU gradient rule: $1$ if $x>0$, else $0$.
   • $x=-3$: gradient $0$ (dead, but only on the negative side).
   • $x=0$: gradient $0$ (boundary, conventionally 0).
   • $x=5$: gradient $1$ — full-strength, no matter how large $x$ grows. Why this step? ReLU never saturates on the positive side; this is the whole speed advantage.

Verify: the figure overlays both slopes. Notice tanh dies at both extremes while ReLU stays alive for all $x>0$ — this is Cell H, the sign-dependent case. The negative-side death of ReLU is the "dying ReLU" trade-off; see Activation Functions and Backpropagation for how these gradients chain.

---

Example 8 — Cell I: a real-world word problem

Forecast: roughly, to shrink 256 → 64 you divide by 4, so stride ≈ 4. Let us pin it exactly.

1. Translate English to symbols. $W=256$, $K=7$, $P=0$, target $O=64$. Unknown: $S$. Why this step? Word problems fail when you skip this translation; write the four slots and mark the blank.
2. Rearrange the ruler (ignore floor for a first estimate): $O-1=\frac{W-K}{S}\Rightarrow S=\frac{W-K}{O-1}=\frac{256-7}{63}=\frac{249}{63}\approx \mathrm{3.95.}$ Why this step? Solving for $S$ tells us which stride would land on 64 if the divide were clean.
3. Test the nearest integer $S=4$. $O=\lfloor \frac{256-7}{4}\rfloor +1=\lfloor \frac{249}{4}\rfloor +1=\lfloor 62.25\rfloor +1=63.$ Why this step? $S=4$ undershoots to 63 because floor eats the remainder — we do not get 64.
4. Fix it with padding. We need one more output pixel, i.e. numerator up to $\ge 4\times 63=252$. Add $P=2$ (border of 2 each side): numerator $=256-7+4=253$, $\lfloor 253/4\rfloor +1=63+1=64$. ✓ Why this step? Padding is the honest lever when stride alone can't hit the target — this mirrors the 227-trick from Ex 5.

Verify: with $W=256,K=7,P=2,S=4$: $\lfloor (256-7+4)/4\rfloor +1=\lfloor 253/4\rfloor +1=63+1=64$. Target met. Cell I closed: a plain-English requirement became four numbers and one padding tweak.

---

Recall Self-test (reveal after guessing)

LeNet C1 output side for $32\times 32$, $5\times 5$, stride 1, pad 0 ::: $28$ AlexNet Conv1 naive output on literal $224$ input ::: $54$ AlexNet Conv1 output on effective $227$ input ::: $55$ LeNet C5 spatial output when $K=W=5$ ::: $1\times 1$ Parameter count of AlexNet Conv2 ($96\to 256$, $5\times 5$) ::: $614,656$ Stride+padding to turn $256\to 64$ with $7\times 7$ ::: stride $4$, padding $2$ tanh gradient magnitude at $x=0$ ::: $1$

---

See also: Convolutional Layers · Pooling Layers · Batch Normalization · ResNet · ImageNet Dataset · Image Classification · Transfer Learning · Object Detection · 3.4.06 LeNet and AlexNet (Hinglish)