Intuition The one core ideaA convolutional neural network (CNN) is a machine that slides small pattern-detectors across an image, stacks the results, and shrinks them step by step until only a short list of "how much does this look like a cat/dog/digit?" scores remains. LeNet and AlexNet are just two recipes for how big those detectors are, how far they slide, and how you shrink between stages.
Everything below is a piece you must own before the parent note's layer tables make sense. We build each symbol from a picture, never from another symbol you haven't seen yet.
Before any math, look at what we are feeding the machine.
An image is a grid of numbers . Each little square (a pixel ) holds a brightness value. For a grayscale image, one number per pixel (0 = black, 255 = white). For a colour image, three numbers per pixel — how much Red, Green, Blue.
The picture on the left shows a 6 × 6
The picture on the right shows the same idea in colour: three stacked grids , one per colour. We call each grid a channel .
Intuition Why "channel" mattersWhen the parent note writes 32 × 32 grayscale it means 1 channel. When it writes 224 × 224 × 3 RGB it means the third number, 3 height × width × channels .
Notation you now own:
H × W × C = 32 × 32 × 1 = 224 × 224 × 3
Definition Filter / kernelA filter (also called a kernel ) is a tiny grid of numbers — for example 5 × 5 one single number , and write that number down. That is one "how strongly does this patch match my pattern?" score.
Look at the figure: the orange 3 × 3
Intuition Why a filter and not a full connection?We could connect every pixel to every output. But an edge in the top-left of a photo looks like an edge in the bottom-right — the same pattern. A small filter that slides re-uses the same 9 (or 25) weights everywhere. This is why CNNs need far fewer parameters than plain networks, and why LeNet has only ~60K weights.
K K = 5 K = 11 A filter always spans all input channels at once. So a 5 × 5 5 × 5 × 3
Number of filters = how many different patterns this layer looks for. LeNet C1 has 6 filters → 6 output channels.
Now the key motion: how the filter moves decides the output size.
Stride S how many pixels the filter jumps each time it moves. Stride 1 = touch every position (top row of figure). Stride 2 = skip every other position (bottom row), so you get fewer outputs and a smaller result.
Padding P before sliding, so the filter can also sit centred on the edge pixels. Padding = 0 means "no border, output shrinks".
Intuition Why we count positionsThe output height/width is simply how many times the window can stop as it slides across. Start at the left edge; each stride of S Output = ⌊ S W − K + 2 P ⌋ + 1
Two pieces of that formula still need defining — do them now.
Definition The floor bracket⌊ ⌋ ⌊ x ⌋ round down to the nearest whole number "; e.g. ⌊ 53.25 ⌋ = 53 partly fits off the edge produces no output — you can't take half a stop.
Worked example Re-deriving LeNet C1 with everything definedInput W = 32 K = 5 P = 0 S = 1 ⌊ 1 32 − 5 + 0 ⌋ + 1 = ⌊ 27 ⌋ + 1 = 28 32 − 5 + 1 28 × 28 × 6
Common mistake The AlexNet "224 vs 227" trapWith W = 224 , K = 11 , P = 0 , S = 4 ⌊ 4 224 − 11 ⌋ + 1 = ⌊ 53.25 ⌋ + 1 = 54 55 227 ⌊ 4 227 − 11 ⌋ + 1 = 55 ⌊ ⌋
More on the sliding operation: Convolutional Layers .
Pooling takes a small window (e.g. 2 × 2 single summary number , then slides on. Average pooling uses the mean of the window; max pooling keeps only the biggest value.
LeNet uses average pooling (top of figure): gentle, cheap for 1998 hardware, spreads gradient to all inputs.
AlexNet uses max pooling (bottom): keeps the strongest response, better at "is the pattern present anywhere here?".
Intuition Why shrink at all?After a conv layer we have a big stack of feature maps. Pooling halves the height and width, so the next layer sees a coarser, cheaper picture and its filters cover more of the original image per step. This is how "early layers see edges, deep layers see whole objects". More at Pooling Layers .
Between layers we pass every number through a simple function that adds the bending a network needs to learn non-straight patterns.
LeNet uses tanh ( x ) ( − 1 , 1 ) ReLU : f ( x ) = max ( 0 , x )
The symbol max ( 0 , x )
Why the switch matters (vanishing gradients, 6× speed) is developed fully in the parent note and in Activation Functions .
You will meet these later in the parent note; own them now.
Definition The learning symbols
⊙ elementwise multiply : multiply two grids cell-by-cell (used in dropout's mask).∼ Bernoulli ( q ) q p dropping a neuron; AlexNet uses p = 0.5 ∑ add up a list of things ; appears in LRN summing squared neighbouring activations.N ( 0 , σ ) bell-curve random draw centred at 0; used in PCA colour augmentation.
Definition Softmax and top-5 error
Softmax turns the final 10 (or 1000) raw scores into probabilities that sum to 1 , so "class 7 = 0.82" reads as 82% confidence.Top-5 error = fraction of test images where the true label is not among the model's 5 highest-probability guesses . AlexNet's 15.3% top-5 error is measured on the ImageNet Dataset .
The training procedure that adjusts all these numbers is Backpropagation ; later architectures that replace LRN and add stability are Batch Normalization , VGGNet and ResNet . Dropout gets its own deep treatment in Dropout .
Image as number grid H x W x C
Filter slides over patches
Stride and padding set output size
Output size floor formula
Stack many conv plus pool stages
Softmax gives class probabilities
Read it top to bottom: pixels become patches, patches become scores, scores get bent and shrunk, stacked, and finally squeezed into class probabilities. LeNet and AlexNet are two ways of choosing the numbers in that pipeline.
Cover the right side and answer; reveal to check.
What does the shape H × W × C Height × Width × Channels — a colour image is 224 × 224 × 3
What is a filter (kernel)? A small grid of weights slid over the image; each stop produces one output number by multiply-and-sum.
What does stride S How many pixels the filter jumps each move; bigger stride → smaller output.
What does padding P Adds a border of zeros so the filter can sit on edge pixels; P = 0
Why is there a ⌊ ⌋ A window that falls partly off the edge gives no output, so we round down the count of valid stops.
Compute output size for input 32, kernel 5, pad 0, stride 1. ⌊( 32 − 5 ) /1 ⌋ + 1 = 28
Difference between max and average pooling? Average keeps the mean of a window (LeNet); max keeps the largest value (AlexNet).
What is max ( 0 , x ) ReLU — outputs x
What does ⊙ Elementwise (cell-by-cell) multiplication.
What does softmax produce? Probabilities across the classes that add up to 1.
What is top-5 error? Fraction of images whose true label is not in the model's 5 highest-probability guesses.
