CNN architecture design: Foundations — AI-ML deep dive

Before you can read the parent architecture note, you need to own every symbol it throws at you. This page builds each one from nothing. Read top to bottom — each idea is a brick the next one stands on.

1. The image as a grid of numbers

Look at the figure. The height $H$ counts rows, the width $W$ counts columns, and the channel count $C$ counts how many grids are stacked. We write the whole thing as a shape:

$H \times W \times C$

$H$ (height) ::: number of rows of pixels — the picture is $H$ tall.
$W$ (width) ::: number of columns of pixels — the picture is $W$ wide.
$C$ (channels) ::: how many grids are stacked; 1 for gray, 3 for colour, and many more inside the network (each learned filter makes a new grid).

2. The filter (kernel) and the number $k$

The little grid's side length is called $k$ (the kernel size). A $3 \times 3$ filter has $k = 3$ ; a $5 \times 5$ filter has $k = 5$ .

$k$ (kernel size) ::: the side length of the sliding filter grid; a $k \times k$ filter has $k^{2}$ numbers inside it.

3. Stride $s$ , padding $P$ , and pooling — changing the grid size

In the figure, follow the coral filter box: with $s = 2$ it lands on positions $0, 2, 4, \dots$ — every other spot — so half as many outputs come out along each direction.

Look at the pale border in the figure — those are the $P$ zero-pixels. With them, a filter centred on a corner still has neighbours to multiply, instead of hanging off the edge.

The figure shows a $2 \times 2$ max-pool: each coloured $2 \times 2$ block of the input collapses to a single output cell holding its largest value.

$s$ (stride) ::: the jump size of the sliding filter; larger $s$ makes a smaller output but throws away detail.
$P$ (padding) ::: extra pixels added on each side; keeps the grid from shrinking and lets corners be processed.

4. The receptive field — how much a neuron "sees"

Case A — $n$ stacked $3 \times 3$ convs, stride 1, no shrinking. The parent uses:

$R F = 1 + n \cdot (3 - 1) = 2 n + 1$

What it says: start at $1$ (a single pixel sees itself), and each $3 \times 3$ layer adds $2$ to the reach.
Why $(3 - 1) = 2$ : a $3 \times 3$ filter reaches one pixel left and one pixel right of centre, i.e. $2$ extra pixels of span.
What it looks like: in the figure, one output neuron (top) fans out through two layers until it touches a $5 \times 5$ patch of the input ( $n = 2 \Rightarrow 2 \cdot 2 + 1 = 5$ ). ✓

5. Parameters and FLOPs — the two costs

The parent counts weights as $k_{l}^{2} \cdot C_{l - 1} \cdot C_{l}$ . Adding the biases (one per output channel) gives the true total:

$Params_{l} = weights k_{l}^{2} \cdot C_{l - 1} \cdot C_{l} + biases C_{l}$

Read the weight term left to right: each of the $C_{l}$ output channels needs one filter; each filter is $k_{l} \times k_{l}$ ( $= k_{l}^{2}$ numbers) per input channel, and there are $C_{l - 1}$ input channels — multiply them. Then add one bias for each of the $C_{l}$ outputs.

Formula Deriving the FLOPs of a conv layer

First recall the output grid is $H_{o u t} \times W_{o u t}$ (both from the Section 3 formula). Now build the cost up one step at a time:

One output number comes from lining the filter over the input and doing $k^{2} \cdot C_{in}$ multiplies (one per filter weight, across all input channels), then $k^{2} \cdot C_{in} - 1$ adds to total them, plus one more add for the bias. That's $\approx 2 k^{2} C_{in}$ FLOPs (multiplies + adds), or just $k^{2} C_{in}$ multiply-adds if we count a fused multiply-add as one.
One whole output channel repeats that at every of the $H_{o u t} \times W_{o u t}$ positions: $\times H_{o u t} W_{o u t}$ .
All output channels: there are $C_{o u t}$ of them: $\times C_{o u t}$ .

Multiplying the three steps: $FLOPs \approx multiply-adds H_{o u t} \cdot W_{o u t} \cdot C_{in} \cdot C_{o u t} \cdot k^{2} + bias adds H_{o u t} \cdot W_{o u t} \cdot C_{o u t}$ The first term dominates; the bias adds (and any ReLU applied afterward, one comparison per output number) contribute only the small $H_{o u t} W_{o u t} C_{o u t}$ tail, which is why the parent's " $\propto$ " keeps only the leading product.

$H_{o u t}, W_{o u t}$ ::: the height and width of the output grid, from $⌊(H + 2 P - k) / s ⌋ + 1$ (and the same for $W$ ).
$C_{in}, C_{o u t}$ ::: channels going into and coming out of a layer — same as $C_{l - 1}$ and $C_{l}$ .

6. The gradient and the "+1" trick

To reach an early weight, backprop multiplies many small rates together (the parent's long chain of $\frac{\partial a _{i}}{\partial a _{i - 1}}$ ). Multiply many numbers below 1 and the product shrinks toward zero — that is the vanishing gradient problem.

7. Supporting cast (one line each)

Activation (ReLU) ::: a function applied after each layer; ReLU keeps positives, zeros negatives — adds the "nonlinearity" that lets stacked layers learn curved shapes.
Batch normalization ::: rescales each channel to a stable mean/variance during training so gradients behave.
Softmax ::: turns the final list of scores into probabilities that sum to 1.
FC (fully-connected) layer ::: every input connects to every output — used at the very end to make the class decision.
Concatenate (depth) ::: stack feature-grids from parallel branches into one taller stack (Inception uses this).

Prerequisite map

Equipment checklist

I can state what $H \times W \times C$ means for a picture ::: height rows, width columns, $C$ stacked grids (channels).
I know why channels grow inside a network ::: each learned filter produces one new feature-grid, so $C$ counts grids at that layer.
I can say what $k$ is and how many numbers a filter holds ::: $k$ is the filter's side; it holds $k^{2}$ numbers per input channel.
I know why CNN "convolution" is really cross-correlation ::: it slides the kernel without flipping it; learned weights absorb any flip.
I know two ways to shrink the grid and one way to keep it the same ::: stride $s > 1$ or pooling shrink; padding $P$ (e.g. $P = 1$ for $3 \times 3$ ) keeps the size.
I can compute an output size from $H, k, P, s$ and name valid vs same padding ::: $H_{o u t} = ⌊(H + 2 P - k) / s ⌋ + 1$ ; valid $= P = 0$ , same $= P = ⌊ k /2 ⌋$ .
I know when pixels get dropped ::: when $(H + 2 P - k) mod s \neq = 0$ the floor discards leftover edge pixels.
I can define $n$ and compute $R F$ for $n$ stacked stride-1 $3 \times 3$ convs ::: $n$ = number of stacked conv layers; $R F = 2 n + 1$ .
I can adjust RF for a stride or pool using the jump $j$ ::: $R F_{n e w} = R F_{o l d} + (k - 1) j$ , and $j$ multiplies by each stride.
I can state the parameter count of a conv layer with bias, defining $k_{l}$ and $C_{l}$ ::: $k_{l}^{2} \cdot C_{l - 1} \cdot C_{l} + C_{l}$ , where $k_{l}, C_{l}$ are kernel/channels at layer $l$ .
I can derive the FLOPs formula (defining $H_{o u t}, W_{o u t}$ ) and why doubling channels balances pooling ::: cost per output number $k^{2} C_{in}$ , times $H_{o u t} W_{o u t}$ positions, times $C_{o u t}$ channels, plus $H_{o u t} W_{o u t} C_{o u t}$ bias adds.
I know what $C_{0}$ is ::: the channel count of the first stage (typically 64), doubled per stage as $C_{0} \cdot 2^{l}$ .
I know what $\partial L / \partial w$ means and why "+1" stops it vanishing ::: rate of change of error w.r.t. a weight; the skip path guarantees a nonzero gradient route.