Convolutional neural networks — convolution operation, pooling: Visual walkthrough

Step 1 — A picture is just a grid of numbers

WHAT. Before any maths, we agree what an "image" is to a computer. It is a grid. Each little cell is a pixel, and each pixel holds one number — how bright it is. Bigger number = brighter.

WHY. Everything that follows is arithmetic on this grid. If you can read the grid, you can read the whole derivation. No calculus, no probability — just numbers in boxes.

PICTURE. Below, a $5 \times 5$ grid. We label the rows with an index $i$ (which goes down) and the columns with an index $j$ (which goes right). So $I (i, j)$ means "the number in row $i$ , column $j$ ". The corner cell is $I (0, 0)$ — we count from zero, like most code does.

Figure — Convolutional neural networks — convolution operation, pooling

Step 2 — The tiny stamp (the kernel)

WHAT. Now take a much smaller grid — say $2 \times 2$ — and fill it with its own numbers. Call it $K$ , the kernel (also called a filter, or the "stamp" from the parent's Feynman story).

WHY. A fully-connected net would give every pixel its own private weight — millions of them, each blind to its neighbours (this is exactly problem 1 and 2 in the parent note). Instead we want one small pattern of weights that we will reuse everywhere. That small pattern is $K$ . Its numbers say: "when I sit on a patch of image, how much do I care about each cell under me?"

PICTURE. The kernel $K$ has its own tiny indices $m$ (row, down) and $n$ (column, right), also from zero. So $K (m, n)$ is one weight of the stamp. Here $K$ is $2 \times 2$ , so $m, n \in {0, 1}$ .

Step 3 — Lay the stamp down and multiply cell-by-cell

WHAT. Place the stamp so its top-left corner sits on image cell $(i, j)$ . Now every kernel cell $K (m, n)$ lies directly on top of image cell $I (i + m, j + n)$ . Multiply each overlapping pair together.

WHY. Multiplying pairs is how you ask "does the pattern under the stamp match the stamp?" A big kernel weight sitting on a big pixel gives a big product; a big weight on a dark (small) pixel gives almost nothing. The products are the "agreement" between the stamp and the patch, cell by cell.

PICTURE. Look at the arrows: each kernel cell (yellow) points to the image cell it multiplies (blue). The offset $(m, n)$ is the same inside the stamp no matter where on the image the stamp sits — that is the shifting rule $I (i + m, j + n)$ .

image cell under kernel cell (m, n) I (i + m, j + n) \times the kernel weight there K (m, n)

The $i + m$ says "start at image row $i$ , then step down $m$ rows into the stamp".
The $j + n$ says "start at image column $j$ , then step right $n$ columns into the stamp".

Step 4 — Add up all the products (+ a nudge)

WHAT. Sum every one of those $k^{2}$ products into a single number, then add a constant $b$ (the bias). That single number is the output $S (i, j)$ .

WHY — the sum. Each product measured agreement on one cell. Adding them turns many small "does this cell match?" answers into one overall "did the whole pattern appear here?" score. One number per stamp position.

WHY — the bias $b$ . The sum can only ever be zero when the whole patch is zero. Real detectors need a baseline you can shift up or down — a threshold. $b$ is that adjustable nudge, learned like the kernel weights.

PICTURE. The two nested sums are just "walk over every stamp cell" — the outer sum over rows $m$ , the inner sum over columns $n$ . The figure animates the four products of a $2 \times 2$ stamp collapsing into one output cell.

Step 5 — Slide the stamp: one output cell becomes a whole map

WHAT. Move the stamp right by one column, repeat Step 3–4 → that fills $S (0, 1)$ . Keep sliding right, then drop to the next row and sweep again. Every landing spot produces one output number, and together they form a new grid $S$ — the feature map.

WHY. This is where the parent's "weight sharing" lives: the same four kernel numbers made every output cell. We did not learn a new stamp per location — we reused one. That is why a wing-edge detected top-left is detected bottom-right too (Translation equivariance vs invariance), and why the parameter count is tiny.

PICTURE. Watch the stamp (yellow) sweep across the blue image, each stop dropping one green number into the output map. The step between stops is the stride — here $s = 1$ .

Step 6 — Why the output is smaller: counting the landing spots

WHAT. The $3 \times 3$ image with a $2 \times 2$ stamp gave a $2 \times 2$ output — it shrank. Let's count why exactly, so we can predict the size before computing anything.

WHY. The stamp's left edge can only sit where the right edge still fits inside the image. Count those legal left-edge positions and you have counted the output cells. See the parent's Padding and stride for the knobs $p$ and $s$ .

PICTURE. The last legal start index is $W_{in} - k$ (here $3 - 2 = 1$ : columns $0$ and $1$ ). Along one row that is positions $0, 1$ → two cells. The figure marks the legal starts in green and the forbidden overhang in red.

Step 7 — Edge case: a stride too big, and the empty map

WHAT. What if the stride is so large the stamp can only land once — or the kernel is bigger than the image? The formula must still make sense.

WHY. A correct derivation covers degenerate inputs, not just the pretty ones. Two dangers:

PICTURE. Three panels: (a) $s$ equal to the whole image → exactly one landing → $1 \times 1$ output; (b) $k > W_{in}$ → numerator negative → the floor gives a non-positive count, which means no valid output (you must pad); (c) padding chosen so $W_{o u t} = W_{in}$ ("same" convolution).

(a) $W_{in} = 4, k = 2, p = 0, s = 4$ : $⌊(4 - 2) /4 ⌋ + 1 = ⌊ 0.5 ⌋ + 1 = 1$ . One cell.
(b) $W_{in} = 2, k = 3, p = 0, s = 1$ : $⌊(2 - 3) /1 ⌋ + 1 = ⌊ - 1 ⌋ + 1 = 0$ . Empty — kernel cannot fit; the only cure is padding.
(c) $W_{in} = 32, k = 5, p = 2, s = 1$ : $⌊(32 - 5 + 4) /1 ⌋ + 1 = 32$ . Size preserved.

Step 8 — Pooling: shrink the map, keep the meaning

WHAT. After convolution we often downsample the feature map: chop it into non-overlapping $p \times p$ windows and replace each window with one summary number — its max (max-pool) or its average (average-pool). No stamp of weights here; pooling has zero learnable parameters.

WHY. (1) If a feature wobbles by a pixel, the max over its window usually does not change — that is local translation invariance (Translation equivariance vs invariance). (2) A $2 \times 2$ stride- $2$ pool quarters the map, cutting downstream cost. (3) Each later neuron then sees a larger patch of the original image — a bigger receptive field.

PICTURE. The $4 \times 4$ map splits into four coloured $2 \times 2$ blocks; each block's loudest cell (max) survives on the left output, the block's average survives on the right.

The one-picture summary

Everything above in a single diagram: image → slide the shared stamp → feature map (sized by the formula) → pool → smaller, wobble-proof map. This is the whole pipeline a CNN stacks over and over to find cracks in a wing panel or runways from orbit (Image classification for aerospace inspection).

Recall Feynman: the whole walkthrough in plain words

We started with a photo, which is really a grid of brightness numbers. We made a tiny stamp of our own numbers. We pressed the stamp on a patch, multiplied each stamp number by the photo number under it, added the four products, and added one little nudge — that gave us one answer number. Then we slid the same stamp one step over and did it again, and again, all across the photo, laying down a new smaller grid of answers: the feature map. We could predict how much smaller by counting where the stamp is allowed to land — that's the "IN minus K, plus pad twice, over stride, plus one" rule, and it even tells us honestly when the stamp is too big to fit at all. Finally we chopped the answer grid into little squares and kept only the loudest number in each (max-pool), so the grid got smaller and stopped caring if a feature wobbled a pixel. Stack these stamps and shrinks many times and the machine learns to spot a crack in a wing.

Connections

Parent topic
Fully-connected neural networks — the unconstrained layer convolution specialises.
Padding and stride — the knobs $p$ and $s$ in Steps 6–7.
Feature maps and receptive fields — what the sliding stamp produces.
Translation equivariance vs invariance — why sharing (Step 5) and pooling (Step 8) matter.
Backpropagation — how $K$ and $b$ get learned.
Image classification for aerospace inspection — the payoff.