Convolutional neural networks — convolution operation, pooling

1. The Convolution Operation

WHY it works — derive it from scratch

Start with a fully-connected layer: every output unit $y_p = \sum_q W_{pq} x_q + b_p$ . For a $100\times100$ image feeding a $100\times100$ hidden layer, that is $10^8$ weights. Two problems:

No spatial prior — pixel $(0,0)$ and $(99,99)$ get independent weights, so the net cannot "know" that nearby pixels matter more.
No reuse — a feature learned at one location must be re-learned everywhere.

Impose two constraints on $W$ :

Locality: $W_{pq}=0$ unless $q$ is in a small $k\times k$ neighbourhood of $p$ .
Weight sharing: the same $k^2$ weights are used for every output $p$ .

Substituting these constraints into $y_p=\sum_q W_{pq}x_q$ collapses the giant matrix into a single tiny kernel $K$ slid across the image — and out pops the convolution formula above. Convolution is just a constrained fully-connected layer. That is the WHY.

HOW the output size shrinks

Figure — Convolutional neural networks — convolution operation, pooling

Worked example Worked convolution

Input $3\times3$ patch and kernel: $I=\begin{bmatrix}1&2&0\\0&1&3\\2&1&1\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix},\ b=0$ Kernel is $2\times2$ , stride 1, no padding $\Rightarrow W_{out}=(3-2)/1+1=2$ , so a $2\times2$ output.

$S(0,0)=I(0,0)\cdot1 + I(0,1)\cdot0 + I(1,0)\cdot0 + I(1,1)\cdot(-1)=1-1=0$ Why this step? We overlay $K$ on the top-left window and sum element-wise products.
$S(0,1)=I(0,1)\cdot1 + I(1,2)\cdot(-1)=2-3=-1$ — slide right by stride 1.
$S(1,0)=I(1,0)\cdot1 + I(2,1)\cdot(-1)=0-1=-1$ — slide down.
$S(1,1)=I(1,1)\cdot1 + I(2,2)\cdot(-1)=1-1=0$ $S=\begin{bmatrix}0&-1\\-1&0\end{bmatrix}$ This kernel is a diagonal edge detector: it fires where diagonal neighbours differ.

Channels

Real images have $C$ channels (RGB, or stacked physics fields). A filter is then $k\times k\times C$ ; you sum over channels too. With $F$ filters the output has $F$ channels. Parameters $= F(k^2C+1)$ .

2. Pooling

Worked example Max vs average pooling

$S=\begin{bmatrix}1&3&2&4\\5&6&1&2\\0&2&9&1\\1&1&3&8\end{bmatrix},\quad 2\times2,\ \text{stride }2$ Max-pool: top-left window $\{1,3,5,6\}\to6$ ; top-right $\{2,4,1,2\}\to4$ ; bottom-left $\{0,2,1,1\}\to2$ ; bottom-right $\{9,1,3,8\}\to9$ . $P_{max}=\begin{bmatrix}6&4\\2&9\end{bmatrix},\qquad P_{avg}=\begin{bmatrix}3.75&2.25\\1&5.25\end{bmatrix}$ Why this step? Non-overlapping windows because stride = window size. Max keeps the strongest activation (the "did this feature appear?" signal); average smooths.

3. Common Mistakes (Steel-manned)

4. Flashcards

What two constraints turn a fully-connected layer into a convolution?

Locality (weights nonzero only in a small neighbourhood) and weight sharing (same weights at every position).

Output-size formula for a conv/pool layer

W_{out}=\lfloor (W_{in}-k+2p)/s\rfloor + 1

Why does the

+1

appear in the output-size formula?

It counts the kernel's starting position 0 in addition to the strided positions after it.

How many learnable parameters in a conv layer (F filters, k×k, C channels)?

F(k^2C+1)

, the

+1

is the bias per filter; independent of image size.

Does pooling have learnable parameters?

No — it is a fixed max/average function.

Give two reasons to use pooling.

Local translation invariance and dimensionality reduction (also enlarges receptive field).

Why doesn't kernel flipping matter in a CNN?

The kernel weights are learned, so the network simply learns the flipped values.

What property does weight sharing give a CNN?

Translation equivariance — the same feature is detected regardless of position.

Compute output size: input 32, kernel 5, pad 2, stride 1.

(32-5+4)/1+1 = 32

(padding preserves size).

Max-pool vs average-pool intuition?

Max keeps the strongest "feature present?" signal; average smooths/retains overall magnitude.

Recall Feynman: explain to a 12-year-old

Imagine you have a tiny stamp with a pattern on it, and a big sheet of stickers. You press the same stamp all over the sheet and mark wherever the pattern matches — that stamping is convolution, and using one stamp everywhere means you don't need to memorise a different rule for every spot. Then you look at the marked sheet in little $2\times2$ squares and, in each square, keep only the loudest mark — that's max-pooling. Now the sheet is smaller and you still remember where kinds of things are, even if they wobbled a little. Stack many stamps and shrinks, and the computer learns to recognise cracks in a wing or runways from the sky.

Connections

Fully-connected neural networks — conv is a constrained version of this.
Backpropagation — gradients flow through conv (shared-weight sum) and pooling (max routes gradient to the argmax).
Feature maps and receptive fields
Padding and stride
Image classification for aerospace inspection — crack/defect detection.
Translation equivariance vs invariance

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, CNN ka core idea bahut simple hai: ek chhota sa filter (kernel) lo, usse poori image ke upar slide karo, aur har jagah same weights use karo. Isko convolution kehte hain. Fayda kya? Ek toh weights bahut kam ho jaate hain (3x3 filter = sirf 9 numbers, image chahe kitni bhi badi ho), aur doosra, jo edge ya crack top-left mein detect hota hai wahi bottom-right mein bhi detect ho jaata hai — isko translation equivariance bolte hain. Isiliye aerospace mein composite panel ke cracks dhoondhne ya satellite images se runway pehchaanne ke liye CNN full-connected net se kahin better hai.

Output ka size yaad rakhne ke liye ek hi formula: $W_{out}=\lfloor (W_{in}-k+2p)/s\rfloor+1$ . Yahan $k$ kernel size, $p$ padding, $s$ stride hai. Padding image ke chaaro taraf zero laga ke size bacha leta hai, aur stride badhao toh output chhota ho jaata hai. Ye derive karna easy hai — kernel ka left edge jitni positions par baith sakta hai, utne hi outputs.

Pooling ka kaam hai downsampling. $2\times2$ window mein max lo (max-pooling) ya average lo. Isse map chhota ho jaata hai, compute kam, aur thoda-bahut shift hone par bhi feature same rehta hai. Yaad rakho: pooling mein koi learnable parameter nahi hota — ye fixed function hai. Aur convolution vs cross-correlation ke chakkar mein mat pado; network toh weights khud seekh leta hai, isliye kernel flip karne se koi farak nahi padta. Bas formula aur intuition pakad lo, exam mein aur real project dono mein kaam aayega.

Convolutional neural networks — convolution operation, pooling

1. The Convolution Operation

WHAT it is

WHY it works — derive it from scratch

HOW the output size shrinks

Channels

2. Pooling

3. Common Mistakes (Steel-manned)

4. Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Machine Learning (Aerospace Applications)

Connections