CNN architecture design: Visual walkthrough — AI-ML deep dive

Before we start, three words we will earn with pictures, not assume:

Convolution — sliding a small window (a "kernel") across a grid and computing one number per position. Built in the convolution operation note; here we only need the sliding window picture.
Kernel size $k$ — the width (and height) of that window. A $3 \times 3$ kernel is a $3$ -wide square.
Receptive field ( $R F$ ) — for one neuron deep in the network, the number of original input pixels that can influence its value.

Step 1 — What can a single output neuron "see"?

WHAT. We place one $3 \times 3$ kernel on the input and produce one output number.

WHY. Before we stack anything, we must nail the base case: a single convolution. Everything deeper is just this, repeated. If we get the base case wrong, every later count is wrong.

PICTURE. In the figure, the input is the peach grid. The magenta $3 \times 3$ square is the kernel. Every input pixel under the square feeds into the one violet output neuron. Count them: a $3 \times 3$ window covers $3$ pixels across. So this neuron's receptive field is $3$ wide.

Step 2 — Stack a second conv: overlap, don't add

WHAT. We feed the output grid of layer 1 into a second $3 \times 3$ conv, and ask: how many original input pixels does one layer-2 neuron see?

WHY. The tempting wrong answer is " $3 + 3 = 6$ ." That double-counts the pixels shared between neighbouring windows. We need to see the overlap to get the true count.

PICTURE. One layer-2 neuron (orange) reads $3$ neighbouring layer-1 neurons. But those $3$ layer-1 neurons come from overlapping $3 \times 3$ input windows — they share pixels in the middle. Slide your eye along the input: neuron A covers pixels 1–3, neuron B covers 2–4, neuron C covers 3–5. Total distinct pixels = 1 through 5 = 5, not 6.

Step 3 — The recurrence: how RF grows per layer

WHAT. We turn the Step-2 observation into a rule you can apply layer after layer.

WHY. A single formula beats re-drawing overlaps forever. We want a machine we can crank $n$ times.

PICTURE. The figure shows three stacked layers as three coloured bands. Each arrow up adds a $k - 1$ "reach" on each side. The receptive field is the widening cone at the bottom.

Check against Steps 1–2: $R F_{1} = 3$ . Then $R F_{2} = 3 + (3 - 1) = 5$ . ✅ Matches the picture.

Step 4 — Close the recurrence into $2 n + 1$

WHAT. We unroll the recurrence for $n$ layers of $3 \times 3$ convs and get a closed formula.

WHY. Architects don't want to iterate by hand; they want "how deep for RF = X?" in one line. This is the exact result the parent note stated.

PICTURE. The figure stacks the $+ 2$ increments as a staircase: start at $1$ (a lone pixel), climb $+ 2$ per layer, land at the top. The height of the staircase is the formula.

Start from the recurrence with $k = 3$ (so $k - 1 = 2$ ): $R F_{1} = 3, R F_{l} = R F_{l - 1} + 2.$

Unrolling: each of the $n$ layers contributes $+ 2$ , but layer 1 already gave us $3 = 1 + 2$ . Group it as one seed pixel plus $n$ steps of $2$ :

Step 5 — The pooling case (the big jumps)

WHAT. We add pooling to the picture: it halves spatial size, which multiplies how far each later pixel reaches back.

WHY. Convolutions grow $R F$ slowly ( $+ 2$ each). To see a whole object in a $224 \times 224$ image, that's too slow. Pooling gives the big leaps. Architects mix both; we must show both.

PICTURE. Below the conv, a $2 \times 2$ pool merges 2 input columns into 1 output column. Now every pixel above the pool, when it reaches back "one step," lands on two input pixels. The reach scale doubles.

Step 6 — The stride case (jumps with a cost)

WHAT. We show stride: instead of sliding the kernel one pixel at a time, jump $s$ pixels.

WHY. Stride is the other way to grow $R F$ fast (parent Principle 1). But it skips pixels, so it loses information — we must show why and when it's risky.

PICTURE. Left panel: stride-1, windows touch every pixel. Right panel: stride-2, windows leap over pixels (the greyed-out gaps are never sampled by this layer). The reach multiplies by $s$ , but the pale gaps are the lost information.

Step 7 — Degenerate & edge cases (never leave the reader stranded)

WHAT. We test the formula at its boundaries.

WHY. The contract: cover every degenerate input. A formula you only trust in the "nice" middle is a bug waiting to happen.

PICTURE. Four mini-panels: (a) $n = 0$ , (b) $k = 1$ , (c) huge $n$ , (d) $1 \times 1$ conv.

The one-picture summary

The figure compresses the whole derivation: a seed pixel at the bottom, conv layers each widening the cone by $k - 1$ , a pooling layer that doubles the cone's spread, and the final neuron at the top whose cone width is the receptive field. Three growth engines — conv ( $+ (k - 1)$ ), pool ( $\times 2$ ), stride ( $\times s$ ) — and one seed.

Recall Feynman retelling — say it back in plain words

Imagine one deep neuron and ask: which original pixels could I nudge to change this neuron? Start with a single conv window — it sees $k$ pixels across. Stack another conv on top: it doesn't see $k + k$ pixels, because neighbouring windows overlap by one column, so each new layer only reaches out by $k - 1$ more pixels. Do that $n$ times starting from one seed pixel and you get $R F = 1 + n (k - 1)$ , which for $3 \times 3$ kernels is $2 n + 1$ . That growth is slow and linear, so to see a whole object we add pooling (which halves the grid, so every later reach counts double: $\times 2$ ) or stride (jump $s$ pixels: $\times s$ , but you skip information). Sanity checks: no layers → you see just yourself ( $R F = 1$ ); $1 \times 1$ convs → $R F$ never grows (they only mix channels); overcounting by multiplying $n \cdot k$ is the classic mistake. That's the entire logic behind why VGG stacks tiny kernels, why every architecture pools, and why $1 \times 1$ bottlenecks are "free" context-wise.

Recall Quick self-test

One 5×5 conv vs two stacked 3×3 convs — same receptive field? ::: Yes, both give $R F = 5$ ; the two 3×3s use fewer params and add an extra nonlinearity. After $n$ stacked $3 \times 3$ convs, what is $R F$ ? ::: $2 n + 1$ . Why does a $1 \times 1$ conv not grow the receptive field? ::: Because $k - 1 = 0$ , so it adds $0$ reach per layer — it only mixes channels. Why does pooling grow $R F$ faster than stacking convs? ::: Pooling multiplies the reach ( $\times 2$ for $2 \times 2$ ) while convs only add $(k - 1)$ ; multiply beats add.

Related: ↑ Parent: CNN architecture design · deeper kernels feed transfer learning, and receptive-field budgeting interacts with hyperparameter tuning and model compression.