CNN architecture design: Worked examples — AI-ML deep dive

Before any numbers, let us re-earn the four formulas so no symbol is a mystery.

Definition The four tools, in plain words

Output size of one conv/pool layer. A filter of side $k$ slides over an input of side $W$ , stepping $s$ pixels each time ("stride"), after padding $p$ zeros on each edge. The number of positions it lands in is $W_{out} = ⌊ \frac{W + 2 p - k}{s} ⌋ + 1$ The $- k$ says "the filter needs $k$ pixels to sit on"; dividing by $s$ says "we only keep every $s$ -th landing spot"; $+ 1$ counts the very first landing spot; $⌊ ⌋$ (floor) throws away a fractional last step because you cannot land half a filter. The same formula with $W$ replaced by the height $H$ gives $H_{out}$ ; from here on $H_{out}$ means "apply this identical formula to the height."

Parameters of one conv layer. Each output channel needs one $k \times k$ patch of weights for every input channel, and there are $C_{out}$ output channels: $Params = k^{2} \cdot C_{in} \cdot C_{out} (+ C_{out} biases)$

FLOPs (multiply-adds) of one conv layer. The output map has height $H_{out}$ and width $W_{out}$ (each from the size formula above). Each of the $H_{out} W_{out}$ output pixels, in each of $C_{out}$ channels, is a sum over $k^{2} C_{in}$ products: $FLOPs = H_{out} W_{out} \cdot C_{out} \cdot k^{2} \cdot C_{in}$

Receptive field growth. How many input pixels one deep neuron ultimately "sees". Stacking layers, the RF grows by $(k - 1)$ scaled by the product of all strides so far: $R F_{new} = R F_{old} + (k - 1) \cdot \prod s_{i}$

Cloze checkpoints:

The floor in the output-size formula exists because

you cannot fit a fractional filter — a partial last step is discarded.

1 \times 1

conv has kernel side

k = 1

, so its receptive field contribution

(k - 1)

0

— it mixes channels but sees only one pixel of space.

The scenario matrix

Every CNN-design question is one (or a blend) of these cells. Each example below is tagged with its cell.

Cell	What makes it tricky	Covered by
A. Vanilla forward size	plain stride-1, no padding	Ex 1
B. Stride > 1	division + floor, size shrinks fast	Ex 2
C. "Same" padding	choose $p$ so size is preserved	Ex 3
D. Non-divisible / floor bites	$(W + 2 p - k)$ not a multiple of $s$	Ex 4
E. Degenerate $k = 1$	$1 \times 1$ conv, RF unchanged, channel mixer	Ex 5
F. Parameter comparison	two designs, same RF, which is cheaper	Ex 6
G. FLOP balance limiting law	halve $H, W$ + double $C$ ⇒ FLOPs constant	Ex 7
H. Receptive-field chain	multi-layer RF with mixed strides	Ex 8
I. Real-world word problem	pick depth to cover an object	Ex 9
J. Exam twist / zero-degenerate	pooling to $1 \times 1$ , global average pool	Ex 10

Ex 1 — Cell A: vanilla forward pass

Forecast: guess — is the output bigger, smaller, or the same size as $32$ ? How many weights, roughly thousands or millions?

Output side. Plug into the size formula. $W_{out} = ⌊ \frac{32 + 0 - 5}{1} ⌋ + 1 = 27 + 1 = 28$ Why this step? A $5 \times 5$ filter with no padding cannot reach the outer 2-pixel rim, so the map shrinks by $k - 1 = 4$ pixels total. Output is $28 \times 28 \times 6$ .
Weights. $Params = k^{2} \cdot C_{in} \cdot C_{out} = 25 \cdot 3 \cdot 6 = 450$ . Why this step? Each of 6 output channels holds a $5 \times 5$ stack across all 3 input channels.

Verify: $28 = 32 - 4$ matches "shrink by $k - 1$ " ✓. This is exactly LeNet-5's first layer (parent note), giving $28 \times 28 \times 6$ — our numbers agree with the classic. Weights $450$ is tiny, as expected for a small kernel. See 3.401-convolution-operation.

Ex 2 — Cell B: stride greater than one

Forecast: stride $4$ keeps only every 4th position — expect the output to be roughly a quarter of $227$ . Guess a number near $55$ .

Numerator first. $W + 2 p - k = 227 + 0 - 11 = 216$ . Why this step? This is "how much room the filter's top-left corner has to travel."
Divide by stride, floor, add one. $W_{out} = ⌊ \frac{216}{4} ⌋ + 1 = 54 + 1 = 55$ Why this step? Stride $4$ means only every 4th travel-position is a real output pixel; $+ 1$ counts the starting position.

Verify: $216/4 = 54$ exactly (no floor needed), giving $55 \times 55$ — the published AlexNet number. Sanity: $55 \approx 227/4$ ✓.

Ex 3 — Cell C: "same" padding

Forecast: to undo the $- (k - 1)$ shrink, you must add pixels back. Guess $p = 1$ .

Set output = input and solve for $p$ (stride $1$ ): $W = W + 2 p - k + 1 \Rightarrow 0 = 2 p - k + 1 \Rightarrow p = \frac{k - 1}{2}$ Why this step? We demand $W_{out} = W$ and see what padding that forces.
Plug $k = 3$ : $p = \frac{3 - 1}{2} = 1$ . Why this step? One ring of zeros exactly compensates the one-pixel rim a $3 \times 3$ filter cannot reach on each side.
Edge case — even $k$ . The clean symmetric answer $p = \frac{k - 1}{2}$ is a whole number only when $k$ is odd (then $k - 1$ is even and divides by $2$ ). For $k = 4$ , $p = \frac{3}{2} = 1.5$ is not an integer, so you cannot pad the same amount on both sides and still preserve size exactly — frameworks resort to asymmetric padding (e.g. $1$ on the left, $2$ on the right). This is exactly why almost every real CNN uses odd kernels ( $1, 3, 5, 7$ ): odd kernels have a well-defined centre pixel and admit integer symmetric "same" padding.

Verify: with $k = 3, p = 1$ : $⌊(W + 2 - 3) /1 ⌋ + 1 = (W - 1) + 1 = W$ ✓. With $k = 4$ : $\frac{k - 1}{2} = 1.5 \in / Z$ , confirming the even-kernel obstruction. This is why VGG uses odd $3 \times 3$ kernels with $p = 1$ everywhere and its spatial size only changes at pooling — see 3.4.02-pooling-layers.

Ex 4 — Cell D: the floor actually bites

Forecast: stride $2$ roughly halves $28$ → guess $14$ . But watch the floor.

Numerator: $28 + 0 - 3 = 25$ . Why this step? This is the travel room for the filter's top-left corner; we compute it first because everything else divides this number by the stride.
Divide + floor: $W_{out} = ⌊ \frac{25}{2} ⌋ + 1 = 12 + 1 = 13$ Why this step? $25/2 = 12.5$ ; the filter's last would-be landing spot falls off the edge, so the floor discards it — the rightmost column of input is never centred on. This is a real information loss, not a rounding cosmetic.

Verify: naive (no floor) gives $25/2 + 1 = 13.5$ , but a real map must have a whole number of pixels, so we floor to $⌊ 13.5 ⌋ = 13$ . The $0.5$ we drop is the pixel column the design silently discards — exactly the "loses information, use sparingly" warning about stride in the parent note.

Ex 5 — Cell E: degenerate $1 \times 1$ conv

Forecast: $k = 1$ so spatial size should be untouched; the " $- (k - 1)$ " shrink is zero. Guess $28 \times 28$ , RF unchanged.

Spatial size. $⌊(28 + 0 - 1) /1 ⌋ + 1 = 27 + 1 = 28$ . Unchanged ✓. Why this step? A $1 \times 1$ filter sits on a single pixel — no rim to lose.
Channels & params. Output $28 \times 28 \times 64$ . Params $= 1^{2} \cdot 256 \cdot 64 = 16384$ . Why this step? It's a per-pixel linear mix of the $256$ input channels into $64$ new channels — a channel compressor, not a spatial filter.
Receptive field. $(k - 1) \prod s = 0$ , so RF does not grow. Why this step? It sees no new spatial context; this is why Inception/ResNet put $1 \times 1$ convs before the expensive $3 \times 3/5 \times 5$ ones purely to cut channel count.

Verify: $16384 = 256 \cdot 64$ ✓. This $256 \to 64$ squeeze is the dimensionality-reduction trick from the parent's Inception section.

Ex 6 — Cell F: parameter comparison, same receptive field

Forecast: parent note claimed ~28% saving. Guess (b) is cheaper.

Design (a) params: $5^{2} \cdot C \cdot C = 25 C^{2}$ . Why this step? Straight from the parameter formula $k^{2} C_{in} C_{out}$ with $k = 5$ and both channel counts equal to $C$ , so $C_{in} C_{out} = C^{2}$ .
Design (b) params: two layers, each $3^{2} \cdot C \cdot C$ : $9 C^{2} + 9 C^{2} = 18 C^{2}$ . Why this step? Each $3 \times 3$ layer also has $C$ in and $C$ out, giving $9 C^{2}$ ; there are two such layers stacked, so we add their counts.
Saving fraction: $\frac{25 - 18}{25} = \frac{7}{25} = 0.28 = 28%$ . Why this step? $C^{2}$ cancels — the saving is independent of width.

Verify: with $C = 64$ : (a) $= 25 \cdot 4096 = 102400$ , (b) $= 18 \cdot 4096 = 73728$ , and $1 - 73728/102400 = 0.28$ ✓. Bonus: (b) also inserts an extra ReLU, adding nonlinearity for free — the VGG insight.

Ex 7 — Cell G: the FLOP-balance limiting law

Forecast: parent note says "halve $H, W$ , double $C$ ⇒ FLOPs constant." Guess: equal.

Stage-1 FLOPs. $H_{out} W_{out} \cdot C_{out} \cdot k^{2} \cdot C_{in} = 5 6^{2} \cdot 128 \cdot 9 \cdot 128$ . $= 3136 \cdot 128 \cdot 9 \cdot 128 = 4.62 \times 1 0^{8}$ Why this step? We plug directly into the FLOP formula: one multiply-add per output pixel ( $H_{out} W_{out} = 5 6^{2}$ ), per output channel ( $128$ ), per weight in the patch ( $k^{2} C_{in} = 9 \cdot 128$ ).
Stage-2 FLOPs. $2 8^{2} \cdot 256 \cdot 9 \cdot 256 = 784 \cdot 256 \cdot 9 \cdot 256$ . $= 4.62 \times 1 0^{8}$ Why this step? Spatial term dropped by $4 \times$ ( $5 6^{2} \to 2 8^{2}$ ), but the two channel counts each doubled, multiplying by $4 \times$ — the factors cancel exactly.

Verify: ratio $= \frac{784 \cdot 256 \cdot 256}{3136 \cdot 128 \cdot 128} = \frac{784}{3136} \cdot \frac{25 6 ^{2}}{12 8 ^{2}} = \frac{1}{4} \cdot 4 = 1$ ✓. This is why "double on pool" is the default channel schedule.

Ex 8 — Cell H: receptive-field chain with mixed strides

Forecast: each stride-2 layer roughly doubles how fast RF grows. Guess final RF somewhere near $30$ .

Use $R F_{new} = R F_{old} + (k - 1) \cdot \prod s_{before}$ , where $\prod s_{before}$ is the product of strides of earlier layers.

Layer 1 (first layer, prior stride product $= 1$ ): $R F = 1 + (7 - 1) \cdot 1 = 7$ . Running stride $= 2$ . Why this step? We start at $R F = 1$ because a single output neuron of layer 0 (an input pixel) sees exactly one input pixel — that is the base case. The first layer has no earlier strides, so its scale factor is $\prod s_{before} = 1$ .
Layer 2 (pool, prior stride product $= 2$ ): $R F = 7 + (3 - 1) \cdot 2 = 11$ . Running stride $= 2 \cdot 2 = 4$ . Why this step? Because layer 1 had stride 2, one step of this pool now jumps 2 input pixels, so its $(k - 1)$ growth is scaled by the accumulated $2$ ; then we update the running product to $4$ .
Layer 3 (conv, prior stride product $= 4$ ): $R F = 11 + (3 - 1) \cdot 4 = 19$ . Why this step? Two stride-2 layers came before, so the running product is $2 \cdot 2 = 4$ ; each of this layer's $(k - 1) = 2$ pixel jumps now covers $4$ input pixels, adding $8$ to reach $19$ .

Verify: monotonically increasing $1 \to 7 \to 11 \to 19$ ✓, and every added term is a positive multiple of $(k - 1)$ ✓. Deep enough stems reach the $100 \times 100$ target of the parent's "required depth" example. See 4.2.03-vanishing-gradient for why we can afford such depth via skips.

The figure below turns this chain into a picture. How to read it: the horizontal axis lists the four states left-to-right (input, then after each of the three layers); the vertical axis is the receptive field measured in input pixels. Each coloured bar's height is the RF after that layer, so you watch the bar climb $1 \to 7 \to 11 \to 19$ . The red arrows between consecutive bars are the added term $(k - 1) \times \prod s_{before}$ — notice the arrow lengths are $+ 6$ , then $+ 4$ , then $+ 8$ : the middle jump is smaller than the last even though both use $k = 3$ , because the running stride product grew from $2$ to $4$ , doubling the reach of every filter step. That growing-stride effect is the whole point of the figure.

Ex 9 — Cell I: real-world word problem

Forecast: each $3 \times 3$ layer adds $2$ to RF, and we need $R F \geq 50$ . Guess ~25 layers.

RF law for $n$ stacked $3 \times 3$ , stride 1: $R F = 1 + n \cdot (3 - 1) = 2 n + 1$ . Why this step? With stride 1 everywhere, the running stride product stays $1$ at every layer, so the general growth $R F + (k - 1) \prod s$ simplifies: each of the $n$ layers just adds a flat $(k - 1) = 2$ on top of the base case $R F = 1$ , giving $1 + 2 n$ .
Solve $2 n + 1 \geq 50$ : $2 n \geq 49 \Rightarrow n \geq \frac{49}{2} = 24.5$ , so $n = 25$ . Why this step? $n$ counts whole layers, so a fractional requirement of $24.5$ must be rounded up: $24$ layers give only $R F = 49 < 50$ (one pixel short of the nodule), while $25$ layers give $R F = 51$ , the first depth that fully contains a $50$ -pixel feature.

Verify: $n = 25 \Rightarrow R F = 2 (25) + 1 = 51 \geq 50$ ✓; $n = 24 \Rightarrow R F = 2 (24) + 1 = 49 < 50$ ✗, so $25$ is indeed the minimum. Interpretation: pure $3 \times 3$ stacks are RF-inefficient — this is exactly why real designs insert pooling/stride (Ex 8) or dilations to grow RF faster, and why transfer learning from a deep pretrained backbone is often the practical answer.

Ex 10 — Cell J: exam twist / degenerate pooling to $1 \times 1$

Forecast: global average pool collapses each $7 \times 7$ map to one number → $1 \times 1 \times 512$ . Guess the FC-vs-GAP head is far cheaper.

GAP step. Average each of the $512$ channels over its whole $7 \times 7$ grid: output $1 \times 1 \times 512$ . Why this step? Averaging over all spatial positions throws away where a feature fired, keeping only whether it fired — full translation invariance (parent Principle 3).
$1 \times 1$ conv (1000 filters). Params $= 1^{2} \cdot 512 \cdot 1000 = 512000$ . Output $1 \times 1 \times 1000$ → softmax. Why this step? A per-"pixel" (here single pixel) linear map from $512$ features to $1000$ class scores.
Compare to flatten→FC. Flatten gives $7 \cdot 7 \cdot 512 = 25088$ inputs; FC(1000) needs $25088 \cdot 1000 = 25088000$ params. Why this step? Flattening bakes in the $7 \times 7$ grid, so it needs $49 \times$ more weights and locks the input size.
(b) The twist. On $448 \times 448$ the backbone outputs, say, $14 \times 14 \times 512$ . GAP still averages the whole grid down to $1 \times 1 \times 512$ — its output shape is independent of grid size. A flatten head would instead produce $14 \cdot 14 \cdot 512 = 100352$ numbers and crash the fixed FC layer, which expects exactly $25088$ inputs. Why this step? Zero spatial position encoding = works at any input size; GAP's output dimension depends only on channel count, never on $H, W$ .

Verify: GAP head params $512000$ vs FC head $25088000$ ; ratio $512000/25088000 \approx 0.0204$ , i.e. GAP uses about $2.0%$ of the weights ✓ — a $\sim 49 \times$ saving matching the $7 \times 7$ collapse. Degenerate check: on a $7 \times 7$ input GAP already yields $1 \times 1$ , the smallest non-trivial map, so no further pooling is possible — the boundary case is handled. See 6.3.01-model-compression.

Recall Self-test (reveal after guessing)

Conv $k = 5, s = 1, p = 0$ on side $32$ gives output side ::: $28$ . AlexNet layer 1 ( $227, k = 11, s = 4, p = 0$ ) output side ::: $55$ . Padding to keep size for $k = 3, s = 1$ ::: $p = 1$ (integer only because $k$ is odd). $k = 3, s = 2, p = 0$ on side $28$ gives ::: $13$ (floor drops half a step). Two $3 \times 3$ vs one $5 \times 5$ param saving ::: $28%$ . Layers of $3 \times 3$ (stride 1) to reach $R F \geq 50$ ::: $25$ . GAP+ $1 \times 1$ (1000) head params on $7 \times 7 \times 512$ ::: $512000$ .

Connections

Parent: 3.4.5 CNN architecture design
Prerequisites: 3.401-convolution-operation, 3.4.02-pooling-layers, 3.4.03-activation-functions, 3.4.04-batch-normalization
Onward: 4.2.03-vanishing-gradient, 5.1.02-hyperparameter-tuning, 3.5.01-transfer-learning, 6.3.01-model-compression