Pooling layers (max, average): Visual walkthrough

This is the visual walkthrough child of the parent pooling note. There we stated the pooling formulas. Here we build them from a single grid of numbers, one picture at a time, never using a symbol before you can see it.

By the end you will be able to draw max pooling, average pooling, the shrinking of the feature map, and even the way gradients flow backwards — all from scratch.

Step 1 — Meet the grid and the window

WHAT. We start with a $4 \times 4$ grid of numbers. " $4 \times 4$ " means 4 rows and 4 columns, so 16 numbers total. On top of it we place a small $2 \times 2$ window — a square that covers exactly 4 cells at a time.

WHY. Before any math, you need to see the two objects the whole operation involves: the big grid (the input) and the little window (the thing that slides). Every later step is "where is the window now, and what number does it produce?"

PICTURE. The cream grid below holds our numbers. The orange square is the window sitting on the top-left corner — its home position.

Everything past this point is: slide the window, summarise the patch it covers.

Step 2 — One window, one summary number (the max)

WHAT. For the four numbers the window currently covers, we keep only the largest one and throw the other three away. That largest number becomes a single output cell.

WHY. In a CNN a large activation means "this feature is strongly present right here." Keeping the maximum keeps the loudest evidence and discards the quieter background — this is why it is called max pooling.

PICTURE. The window covers ${1, 3, 5, 6}$ . We circle the biggest, $6$ , in teal; the arrow carries it out to the first cell of a new, smaller grid.

Y_{0, 0} = max (X_{0, 0} 1, X_{0, 1} 3, X_{1, 0} 5, X_{1, 1} 6) = 6

Reading the equation term by term:

$Y_{0, 0}$ — the output cell we are filling: output row 0, column 0.
$max (\dots)$ — the word "max" is a function that answers one question: "which of these is biggest?" We pick it over "sum" or "average" because we want the peak, not the total.
each $X_{r, c}$ under the brace — one of the four input cells the window is sitting on right now.

Step 3 — Slide the window: stride

WHAT. We move the window to the right by 2 columns, land on the next $2 \times 2$ patch, and take its max. That "2" is called the stride $s$ — how far the window jumps each move.

WHY. If we jumped by only 1, consecutive windows would overlap and share cells. Jumping by the full window width ( $s = p = 2$ ) makes the patches tile the grid with no overlap — the clean, most common setup. The stride is the single dial that controls overlap.

PICTURE. The window is now on the top-right patch ${2, 4, 7, 8}$ ; its max, $8$ , drops into output cell $Y_{0, 1}$ . Notice the window's left edge moved from column 0 to column $s = 2$ .

Step 4 — Cover every patch: the full max-pool

WHAT. Repeat the slide for the two bottom patches. Four patches → four output numbers → a complete $2 \times 2$ output grid.

WHY. We must show all windows, not just the first, so no patch is left un-summarised. The bottom-left patch ${9, 10, 3, 4}$ gives $10$ ; the bottom-right ${1, 2, 5, 6}$ gives $6$ .

PICTURE. All four windows and their four teal outputs, assembled into the shrunken grid.

Y_{0, 0} Y_{1, 0} = max (1, 3, 5, 6) = 6 = max (9, 10, 3, 4) = 10 Y_{0, 1} Y_{1, 1} = max (2, 4, 7, 8) = 8 = max (1, 2, 5, 6) = 6

The $4 \times 4$ grid became $2 \times 2$ . That shrinkage is not luck — it comes from a formula.

Step 5 — Why the output is exactly this small

WHAT. We derive the output height. The window's last valid position is when its right edge just reaches the grid's edge — no hanging off. Count how many jumps fit.

WHY. You need to predict output size for any $p$ and $s$ , not just guess "half." A window of width $p$ must start no later than $H_{in} - p$ ; jumps of size $s$ between $0$ and $H_{in} - p$ give the count.

PICTURE. A number line of possible start positions; dots mark the legal window starts, and the count of dots is the output height.

Step 6 — Change the summary rule: average pooling

WHAT. Same sliding window, same tiling — but instead of "biggest," each patch reports its average: add the four numbers, divide by 4.

WHY. Sometimes you don't want the single loudest cell; you want "how much of this feature is here overall." Averaging keeps every cell's contribution, giving a smoother result. We pick division-by- $p^{2}$ because there are $p^{2}$ cells in the window, and an average is total over count.

PICTURE. The top-left patch ${1, 3, 5, 6}$ : all four highlighted in plum (not just one), summed to $15$ , divided by $4 = 3.75$ .

Y_{i, j} = \frac{1}{p ^{2}} m = 0 \sum p - 1 n = 0 \sum p - 1 X_{i \cdot s + m, j \cdot s + n}

$\sum\sum$ — add up all cells in the window (not pick one — that is the whole difference from max).
$\frac{1}{p ^{2}}$ — divide by the cell count $p^{2} = 4$ to turn a sum into a mean.

Full average output:

\frac{1 + 3 + 5 + 6}{4} = 3.75 \frac{9 + 10 + 3 + 4}{4} = 6.5 \frac{2 + 4 + 7 + 8}{4} = 5.25 \frac{1 + 2 + 5 + 6}{4} = 3.5

Compare Step 4's ${6, 8, 10, 6}$ (spiky) with these ${3.75, 5.25, 6.5, 3.5}$ (gentle). Same window, different personality.

Step 7 — Running the film backwards: gradients

WHAT. During training, an error signal $\frac{\partial L}{\partial Y}$ arrives at each output cell and must be pushed back onto the input cells. For max pooling it goes only to the cell that won (was the max); for average it is split equally.

WHY. A gradient measures "if I nudge this input, how much does the loss change?" In max pooling only the winner touched the output, so only the winner can affect the loss — every other cell had zero influence, hence zero gradient. In average pooling every cell contributed $\frac{1}{p ^{2}}$ , so each gets that same slice back.

PICTURE. Left panel: incoming gradient $0.5$ routed to the single winner cell (value $6$ ), zeros elsewhere. Right panel: the same $0.5$ spread as $0.125$ across all four cells.

max: \frac{\partial L}{\partial X _{i, j}} = ⎩ ⎨ ⎧ \frac{\partial L}{\partial Y _{k, l}} 0 X_{i, j} was the winner otherwise avg: \frac{\partial L}{\partial X _{i, j}} = \frac{1}{p ^{2}} \frac{\partial L}{\partial Y _{k, l}}

winner cell keeps the whole incoming gradient — it was the only path.
$\frac{1}{p ^{2}}$ — each of the 4 cells gets an equal quarter, mirroring how each contributed a quarter forward.

This backwards routing (sparse vs dense) is why max pooling trains a few strong detectors while average pooling nudges everyone gently — it also underlies Global Average Pooling and appears in ResNet Architecture and VGG Network.

The one-picture summary

Everything above in a single frame: the grid, the sliding window, the two summary rules diverging, the shrunk outputs, and the gradient flowing back.

Recall Feynman retelling — say it back in plain words

Start with a grid of numbers where big means "feature is loud here." Lay a tiny square window on the top-left corner. Max pooling keeps only the loudest number in that square; average pooling keeps the square's average instead. Now hop the window across by its stride — hop by the full width and the squares tile perfectly with no overlap. Each hop makes one output number, so a $4 \times 4$ grid collapses to $2 \times 2$ . How small exactly? Count the legal window start-positions: $⌊(H - p) / s ⌋ + 1$ . When we later train the net, the error signal walks backwards — for max it goes only to the cell that won, for average it splits into equal pieces. No weights are ever learned; the window is a fixed rule, and that simplicity is the point.

Recall Quick self-test

Max of the patch ${1, 3, 5, 6}$ ::: $6$ Average of ${2, 4, 7, 8}$ ::: $5.25$ Output size for $H_{in} = 7, p = 3, s = 2$ ::: $⌊(7 - 3) /2 ⌋ + 1 = 3$ Where does the max-pool gradient go ::: only to the cell that held the maximum Number of learnable parameters in a pooling layer ::: zero

Prerequisites revisited: Convolutional Layers, Feature Maps, Receptive Field, Translation Invariance, Stride and Padding.