Convolution operation and filters

Overview

The convolution operation is the fundamental building block of Convolutional Neural Networks (CNNs). It's a mathematical operation that slides a small matrix (called a filter or kernel) across an input image to detect local patterns like edges, textures, or shapes. Unlike fully connected layers that treat each pixel independently, convolution preserves spatial relationships between neighboring pixels.

The Mathematics of Convolution

What Is the Discrete2D Convolution?

This is technically cross-correlation, but in deep learning we call it "convolution" (true mathematical convolution flips the kernel, but we learn the weights anyway, so the flip is irrelevant).

Derivation from First Principles

WHY this formula? We want to measure "how much does this local patch of the image match the pattern encoded in the filter?"

Step 1: Local receptive field
The filter looks at a $k \times k$ window of the input. For a 3×3 filter at position $(i,j)$, it sees pixels $I[i:i+3, j:j+3]$.

Step 2: Element-wise multiplication
Each filter weight $K[m,n]$ multiplies the corresponding input pixel $I[i+m, j+n]$. This is a weighted sum where the filter weights determine what pattern we're looking for.

Step 3: Accumulation
We sum all $k^2$ products. A large positive sum means "strong match," near-zero means "pattern not present."

Step 4: Sliding the filter
Repeat for every valid position $(i,j)$ in the input. The stride $s$ controls how many pixels we skip: incrementing by $s$ instead of 1.

where $p$ is padding (zeros added around input borders), $s$ is stride.

WHY padding? Without it, the output shrinks by $k-1$ pixels in each dimension. Padding preserves spatial size. WHY stride $>1$? Downsamples the output, reducing computation (like pooling).

What Does a Filter Actually Detect?

Worked Examples

Example 1: 3×3 Filter on 5×5 Input (Stride 1, No Padding)

Input $I$:

Filter $K$ (vertical edge):

Output size: $H_{out} = (5 - 3)/1 + 1 = 3$, so $3 \times 3$ output.

Compute top-left output element $(0,0)$:

Why this step? We overlay the filter on the top-left 3×3 patch, multiply corresponding elements, and sum. The positive result (3) indicates a weak vertical edge (right side slightly brighter than left).

Full output (computing remaining 8 positions similarly):

Example 2: Effect of Stride and Padding

Same input/filter, stride $s=2$, padding $p=1$.

Paded input (add one row/column of zeros on all sides) → $7\times 7$:

Output size: $H_{out} = (7 - 3)/2 + 1 = 3$.

Top-left computation now starts at the paded position $(0,0)$, but most elements are 0:

Why stride 2? We skip every other position, so the filter moves from $(0,0)$ to $(0,2)$ to $(0,4)$, then $(2,0)$, etc. Output is smaller but faster to compute.

Example 3: Multi-Channel Input (RGB Image)

Real images have 3 channels (R, G, B). The filter must also have 3 channels.

Input: $5 \times 5 \times 3$ (height, width, channels)
Filter: $3 \times 3 \times 3$ (one slice per input channel)

Convolution:

Why this form? We apply the filter separately to each channel, then sum across channels to produce a single output value. The filter learns correlations across color channels (e.g., "red and green high, blue low" = yellow).

Multiple filters: A CNN layer typically has $N$ filters (say 64). Each produces its own feature map, so the output is $H_{out} \times W_{out} \times 64$.

Common Mistakes and Fixes

Active Recall Flashcards

#flashcards/ai-ml

What is the discrete2D convolution formula for position (i,j)? :: $(I * K)[i,j] = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I[i+m, j+n] \cdot K[m,n]$

Feynman Explanation

Inside that magnifying glass, you have a special pattern drawn: "If the left side is dark and the right side is bright, this is a vertical edge!" You multiply each pixel you see by this pattern and add them up. If the sum is big, you write "Found an edge here!" on a new sheet of paper.

Now you slide the magnifying glass one step to the right, and check again. Slide, check, slide, check—all the way across the photo, row by row. The new sheet of paper (the feature map) shows you everywhere the vertical edge pattern appeared.

The amazing part? You only designed the magnifying glass pattern once, but you used it thousands of times across the photo. That's way easier than making a separate magnifying glass for every single pixel!

In a CNN, the computer learns what patterns to look for (edges, colors, shapes) by trying millions of examples. Early layers find simple stuff like edges. Deeper layers combine those edges into complex things like "dog ears" or "bicycle wheels."

Connections

• 3.4.02-Pooling-layers — After convolution, pooling downsamples feature maps
• 3.4.03-CNN-architectures — Stack convolution + pooling to build networks like LeNet, AlexNet
• 3.3.01-Backpropagation — How filters learn: gradients flow back through convolution
• 2.2.05-Receptive-fields — Each output neuron "sees" a local input region
• 3.4.10-Depthwise-separable-convolutions — Efficient variant splitting spatial and channel-wise convolution
• 4.1.03-Transfer-learning — Pretrained filters (ImageNet) transfer low-level features to new tasks
• 3.4.06-Batch-normalization — Normalizes feature maps to stabilize training

Mnemonic

Channels: "Filter Depth Must Match Input Depth" (FD = ID).

Key Takeaways

1. Convolution = local pattern matching: Filters encode templates (edges, textures), output measures match strength at each location.
2. Parameter sharing: Same small filter reused across entire image → far fewer weights than fully connected.
3. Translation invariance: Learned features work anywhere in the image.
4. Dimensions matter: Filter depth = input channels. Output channels = number of filters.
5. Stride & padding control: Stride downsamples, padding preserves size. Formula: $H_{out} = \lfloor (H+2p-k)/s \rfloor + 1$.
6. Learned representations: Backprop tunes filter weights to extract useful features for the task.

Master convolution, and you understand the heart of computer vision models.

Concept Map

Hinglish (regional understanding)