Convolution operation and filters: Visual walkthrough

Step 1 — What is an image, really?

WHAT. An image is a grid of numbers. Each little square (a pixel) holds one number: how bright that spot is. Dark = small number (near 0), bright = large number.

WHY start here. Before we can "match a pattern", we need the thing patterns live in. Everything below is just arithmetic on these numbers — so we must see them as numbers, not as a photo.

PICTURE. Below, a $5 \times 5$ image. The colour of each cell is just its number painted in. We label rows top-to-bottom and columns left-to-right, both starting at $0$ — that is the convention we keep the whole page.

Figure — Convolution operation and filters

Step 2 — The question a filter asks

WHAT. A filter (also called a kernel) is a tiny grid of numbers — here $3 \times 3$ . It is a template: a little picture of the pattern we want to find.

WHY a small grid and not the whole image? Real patterns are local. "Is there a vertical edge right here?" depends only on a handful of neighbouring pixels, not on a pixel a thousand rows away. So the filter is small on purpose — it asks a local question.

PICTURE. Our filter is the vertical-edge template: negative numbers on the left column, zero in the middle, positive on the right. Read it as: "I reward images that are dark on the left and bright on the right."

Step 3 — Overlay: line the template up on one patch

WHAT. Place the filter on top of the image's top-left corner. Now every filter cell $K [m, n]$ sits directly over one image pixel $I [m, n]$ .

WHY overlay? To compare the template to the image, they must be aligned cell-for-cell. Overlaying is how we decide which pixel each weight is talking about.

PICTURE. The yellow box is the $3 \times 3$ window the filter currently covers. Only those $9$ pixels are in play right now — the rest of the image is ignored for this one measurement. The red dashed cell is the window's centre (Step 3's definition below).

The nine overlapping pairs are $(I [m, n], K [m, n])$ for each of the $9$ cells.

Step 4 — Multiply, cell by cell

WHAT. For each of the $9$ overlapping cells, multiply the image pixel by the filter weight sitting on it.

WHY multiply and not, say, subtract? Multiplication is a vote of agreement:

Bright pixel ( $+$ big) under a positive weight ( $+$ ) → big positive product = "yes, this matches!"
Bright pixel under a negative weight → big negative product = "no, wrong place for brightness."
Dark pixel (near $0$ ) → product near $0$ = "this cell has no opinion."

So each product is one cell's signed vote for how well image matches template there.

PICTURE. Each cell shows its own little equation $I \times K$ . Watch the signs: a bright pixel under a red (negative) weight becomes a strong negative vote.

For our top-left window each product is:

bright, left, punished 1 \cdot (- 1) middle, ignored 2 \cdot 0 bright, right, rewarded 3 \cdot (+ 1) \dots

$I [m, n]$ ::: the pixel's brightness
$K [m, n]$ ::: how much this position should count, and with what sign
their product ::: this cell's signed vote

Step 5 — Add up the votes into one number

WHAT. Sum all nine products. That single number is the filter's answer for this location.

WHY sum? Voting only means something when you tally it. A large positive total = "the whole $3 \times 3$ patch strongly matches my template." Near zero = "no clear match." Large negative = "the opposite pattern is here (bright-left, dark-right)."

PICTURE. The nine votes flow into one box. For our corner patch the tally is $+ 3$ : a mild vertical edge, right side a touch brighter than the left.

Now — and only now — we have earned the notation. The two nested $\sum$ signs just say "add over all rows $m$ and all columns $n$ ", each running $0$ to $k - 1$ :

The $i + m, j + n$ is the whole trick: $i, j$ says where the window is, and $m, n$ walks inside the window.

Step 6 — Slide the window, fill the map

WHAT. Move the window one column to the right ( $j \to j + 1$ ) and repeat Steps 3–5. When you run out of columns, drop one row down ( $i \to i + 1$ ) and sweep again. Each landing produces one number.

WHY slide? We asked "is the pattern here?" — but the pattern could be anywhere. Sliding asks the same question at every location, reusing the same nine weights. That reuse is why one small filter can scan a whole image (this is the parameter sharing the parent praised, and it powers receptive fields).

PICTURE. Three snapshots of the window at $(0, 0)$ , $(0, 1)$ , $(0, 2)$ , each dropping its tally into the growing output grid — the feature map.

For a $5 \times 5$ image and $3 \times 3$ filter, the window can start at only $3$ columns and $3$ rows, so the map is $3 \times 3$ :

3 - 1 - 3 021 - 2 11

The output is smaller than the input — the window cannot hang off the edge.

Step 7 — The edge problem, and padding

WHAT. Because the window must stay fully on the image, the output shrank by $k - 1 = 2$ in each direction. The pixels at the true border never get to sit at the centre of a window (the centre cell we defined in Step 3).

WHY care? Two reasons: (1) we lose size every layer, and (2) border information is under-counted. The fix is padding — written $p$ , it is the number of extra rings of zeros we wrap around the image so the window's centre can land on the original corners.

PICTURE. Left: without padding, the corner pixel is only ever counted once. Right: one ring of zeros ( $p = 1$ ) lets the window sit centred on that corner. Zeros contribute $0$ to every vote — they pad the geometry without adding fake signal.

Check on our example: no padding, stride $1$ : $⌊(5 + 0 - 3) /1 ⌋ + 1 = 3$ . ✓ With $p = 1, s = 2$ : $⌊(5 + 2 - 3) /2 ⌋ + 1 = 3$ .

Step 8 — Degenerate cases: read the formula at its limits

WHAT. Push the knobs to their extremes and confirm the picture still makes sense.

WHY. A formula you trust must survive its boundary values, or you have hidden a bug.

Case	Setting	What happens	Picture check
$1 \times 1$ filter	$k = 1$	Window is one pixel; the "sum" has one term; output size $=$ input size. It just rescales each pixel.	No sliding overlap — pure per-pixel weighting.
Filter = whole image	$k = H = W$	Exactly one landing; output is $1 \times 1$ . Convolution collapses to a single dot product (defined in Step 5) between the whole flattened image and the whole flattened filter — a fully-connected neuron!	The window fills the frame; there is nowhere to slide.
All-zero filter	$K \equiv 0$	Every vote is $0$ ; the map is all zeros. The filter has no opinion about anything.	Every product cell reads $0$ .
Stride $\geq$ input	$s \geq H$	Only the first window lands. Massive downsampling.	Second slide falls off the edge.

PICTURE. The two informative extremes side by side: the $1 \times 1$ filter (map same size as input) and the full-image filter (map is a single number, i.e. one dot product).

The one-picture summary

Everything above is one loop: overlay → multiply → sum → slide. Read the figure below left-to-right: an image grid (Step 1) meets the filter overlaid on a window (Steps 2–3); the overlap becomes votes by multiplying (Step 4); the votes are summed into a dot product (Step 5); we slide to the next window (Step 6); and the trail of tallies fills the feature map (the output, sized by Step 7's formula). One filter, reused everywhere, turns an image grid into a feature-map grid.

Recall Feynman retelling — say it like a story

I have a photo made of numbers: bright spots are big numbers, dark spots are small ones. I also carry a tiny stencil — a little square grid, $k$ cells on a side, filled with plus and minus numbers that describe a pattern I'm hunting, like "dark on the left, bright on the right."

I lay my stencil on the top-left corner. For each hole in the stencil I multiply the number under it by the stencil's number: bright-where-I-wanted-bright gives a big positive vote, bright-where-I-wanted-dark gives a big negative vote, and anything dark barely votes at all. I add up all nine votes — that multiply-and-add of two matched lists is a dot product — and write that one number down (at the window's centre cell, one down and one right of the corner for a $3 \times 3$ stencil). It says how much the corner looks like my pattern. (I never flip the stencil; that flip would make it "true" convolution, but since the numbers are learned it changes nothing, so what I really do is cross-correlation.)

Then I shuffle the stencil one step right and do it again, and again, sweeping the whole photo row by row — always the same stencil, so I only had to learn $k^{2}$ numbers. The trail of tallies I leave behind is a brand-new smaller grid that lights up wherever my pattern lives; its height and width each follow the same shrink-and-pad formula, and the " $+ 1$ " in it is just the first free landing I get before moving. If I don't want it to shrink, I wrap the photo in a border of zeros first (zeros vote nothing, so they only fix the geometry). Skip steps while sweeping (stride) and the grid comes out even smaller and faster. That whole ritual — overlay, multiply, sum, slide — is convolution.

Recall

The two $\sum$ signs in the formula do what job? ::: They add up the products over every cell of the $k \times k$ filter window — the "tally the votes" step. What does $k$ stand for? ::: The filter's side length: a square filter has $k$ rows and $k$ columns. Is the operation on this page strict convolution? ::: No — it is cross-correlation (no kernel flip); deep learning calls it convolution because the weights are learned. Why does the output shrink without padding? ::: The window can't hang off the edge, so it starts in $k - 1$ fewer positions per axis. For an odd filter with corner at $(i, j)$ , where is its centre cell? ::: At $(i + \frac{k - 1}{2}, j + \frac{k - 1}{2})$ — for $k = 3$ that is one down and one right of the corner. Why does the output-size formula end in " $+ 1$ "? ::: Landings are one more than the gaps between them (fence-post counting); the $+ 1$ is the first free landing before any slide. A $k \times k$ filter on a $k \times k$ image is equivalent to what? ::: A single dot product — one fully-connected neuron.

See also: 3.4.02-Pooling-layers · 3.4.03-CNN-architectures · 3.3.01-Backpropagation (how the filter weights are learned) · 2.2.05-Receptive-fields · 3.4.10-Depthwise-separable-convolutions · 3.4.06-Batch-normalization · 4.1.03-Transfer-learning