LeNet and AlexNet: Exercises — AI-ML deep dive

This is a self-testing page. Each problem sits above a collapsible solution. Try it first, then open the box. Problems climb from L1 Recognition (just recall the fact) to L5 Mastery (combine everything). Every symbol is defined the first time it appears.

Parent topic: LeNet and AlexNet. Related building blocks you may want open in another tab: Convolutional Layers, Pooling Layers, Activation Functions, Backpropagation, Dropout, Batch Normalization.

Definition The one formula this whole page leans on

When a square filter (kernel) of side $K$ slides across a square image of side $W$ , jumping $S$ pixels each step, after we glue $P$ pixels of padding onto each side, the output side length is $O = ⌊ \frac{W - K + 2 P}{S} ⌋ + 1$

$W$ = input width in pixels (Width). One idea: how big is the picture.
$K$ = Kernel side. How big is the sliding window.
$P$ = Padding: extra border pixels we bolt on so the window doesn't run out of room.
$S$ = Stride: how far the window hops each move.
$⌊ \cdot ⌋$ = the floor: "round down to the nearest whole number." We use it because you cannot place the window at a fractional pixel — a leftover half-step is simply thrown away.

Read it as a story: $W - K$ is how much room the window has to travel once its left edge starts at pixel 0. Divide by $S$ to count how many hops fit. $+ 1$ because the very first placement (before any hop) also counts.

The figure below turns that story into a picture. A tiny input strip of width $W = 9$ (the cyan cells) is scanned by a $K = 3$ window taking stride $S = 2$ : follow the coloured boxes labelled "hop 0", "hop 1", … — each arrow points up to the input cells that box covers, and each box produces exactly one output value. Count the boxes and you have counted the outputs: $⌊(9 - 3) /2 ⌋ + 1 = 4$ . That count is the formula.

Level 1 — Recognition

Exercise 1.1

LeNet-5 and AlexNet each end with a layer of a specific size that equals the number of classes they predict. State the output-layer size for each network, and say which dataset forces that number.

Recall Solution

LeNet-5: final layer has $10$ neurons, one per digit $0, 1, \dots, 9$ . The dataset is MNIST (handwritten digits). See Image Classification.
AlexNet: final layer has $1000$ neurons, one per class in the ImageNet Dataset (1000 categories). The rule to remember: the last layer's width = number of classes, because each neuron there answers "how much does the picture look like this class?"

Exercise 1.2

Which activation did LeNet use, and which did AlexNet use? Write each function's rule.

Recall Solution

LeNet used $tanh (x)$ (the hyperbolic tangent), a smooth S-shaped curve squashing every input into the range $(- 1, 1)$ .
AlexNet used ReLU (Rectified Linear Unit): $f (x) = max (0, x)$ — "keep positives, flatten negatives to zero." See Activation Functions.

Exercise 1.3

LeNet used average pooling; AlexNet used max pooling. In one sentence each, describe what these two operations do to a $2 \times 2$ block of numbers.

Recall Solution

Average pooling: replace the four numbers with their arithmetic mean (add them, divide by 4).
Max pooling: replace the four numbers with the single largest of the four. See Pooling Layers.

Level 2 — Application

Exercise 2.1

LeNet's first conv layer C1: input $32 \times 32$ , kernel $5 \times 5$ , padding $0$ , stride $1$ . Compute the output side length $O$ . Show each substitution.

Recall Solution

Plug into $O = ⌊ \frac{W - K + 2 P}{S} ⌋ + 1$ with $W = 32, K = 5, P = 0, S = 1$ : $O = ⌊ \frac{32 - 5 + 0}{1} ⌋ + 1 = ⌊ 27 ⌋ + 1 = 28.$ So C1 outputs $28 \times 28$ per filter. With 6 filters the full tensor is $28 \times 28 \times 6$ .

Exercise 2.2

LeNet S2 average-pooling: input $28 \times 28$ , window $2 \times 2$ , stride $2$ , padding $0$ . Compute $O$ .

Recall Solution

Pooling uses the same formula with $K = 2$ (the pool window plays the role of the kernel): $O = ⌊ \frac{28 - 2 + 0}{2} ⌋ + 1 = ⌊ 13 ⌋ + 1 = 14.$ Output $14 \times 14 \times 6$ — exactly what the parent note lists.

Exercise 2.3

AlexNet's real Conv1 uses effective input $227 \times 227$ , kernel $11 \times 11$ , padding $0$ , stride $4$ . Show that the output is $55 \times 55$ , and explain where the floor actually bites.

Recall Solution

$O = ⌊ \frac{227 - 11 + 0}{4} ⌋ + 1 = ⌊ \frac{216}{4} ⌋ + 1 = ⌊ 54 ⌋ + 1 = 55.$ Here $216/4 = 54$ divides evenly, so the floor changes nothing. If you (wrongly) use $W = 224$ : $\frac{213}{4} = 53.25$ , floor to $53$ , giving $54$ — that's where the floor does bite, and why the "224" figure is a paper-side simplification.

The figure contrasts these two inputs side by side: the amber bar is the $W = 224$ case that floors down to $54$ , the cyan bar is the correct $W = 227$ case that lands exactly on $55$ . Notice the amber bar is shorter — that missing pixel is precisely what the floor threw away, and it is why practitioners quietly pad the input to $227$ .

Level 3 — Analysis

Exercise 3.1

A convolutional layer's parameter count (weights only, ignore biases) is $params = (K \times K \times C_{in}) \times C_{out},$ where $C_{in}$ is the number of input channels and $C_{out}$ is the number of filters. Each filter must look at all input channels at once, hence the $C_{in}$ factor. Compute the weight count of AlexNet Conv1 ( $K = 11$ , $C_{in} = 3$ RGB, $C_{out} = 96$ ).

Recall Solution

$(11 \times 11 \times 3) \times 96 = 363 \times 96 = 34848.$ Why $C_{in} = 3$ ? The input is RGB, so every filter is itself a little $11 \times 11 \times 3$ cube; it sees red, green, and blue simultaneously and produces one output map. See Convolutional Layers.

Exercise 3.2

Compare the gradient of $tanh$ and ReLU at a strongly-activated positive input, say $x = 4$ . Explain in one line why this comparison predicts AlexNet trains faster.

Recall Solution

$tanh$ gradient: $\frac{d}{d x} tanh (x) = 1 - tanh^{2} (x)$ . At $x = 4$ , $tanh (4) \approx 0.9993$ , so gradient $\approx 1 - 0.9986 = 0.00134$ — essentially zero (the "vanishing gradient").
ReLU gradient at any $x > 0$ is exactly $1$ . During Backpropagation gradients are multiplied layer-by-layer. A per-layer factor of $0.0013$ shrinks the signal catastrophically over many layers; a factor of $1$ passes it through untouched. That is the mechanical reason for the empirical 6× faster convergence.

Exercise 3.3

Dropout with drop-probability $p$ zeroes each neuron independently. At test time we multiply every activation $a$ by $(1 - p)$ . Show algebraically why $(1 - p)$ is the correct scale, using expectation.

Recall Solution

Let $a$ denote the neuron's output activation — the single number this neuron would pass forward if it were never dropped. Let the random mask $m$ be $1$ with probability $1 - p$ (kept) and $0$ with probability $p$ (dropped). During training the neuron passes $m \cdot a$ . Its expected output is $E [m \cdot a] = a \cdot E [m] = a ((1) (1 - p) + (0) (p)) = (1 - p) a .$ At test time no neuron is dropped, so the raw output is $a$ — a factor $\frac{1}{1 - p}$ too large compared to what the next layer saw during training. Multiplying by $(1 - p)$ restores the training-time expected scale. See Dropout.

Level 4 — Synthesis

Exercise 4.1

Trace AlexNet's spatial size through its first stack: Conv1 ( $227 \to$ ?) then MaxPool1 ( $3 \times 3$ , stride $2$ , padding $0$ ). Give both output side lengths.

Recall Solution

Conv1 (from Ex 2.3): $O_{1} = 55$ . MaxPool1 on $W = 55$ , $K = 3$ , $S = 2$ , $P = 0$ : $O_{2} = ⌊ \frac{55 - 3 + 0}{2} ⌋ + 1 = ⌊ \frac{52}{2} ⌋ + 1 = 26 + 1 = 27.$ So $55 \times 55 \to 27 \times 27$ , matching the parent note's $27 \times 27 \times 96$ .

Exercise 4.2

AlexNet Conv2 keeps spatial size fixed at $27 \times 27$ using padding. Given kernel $5 \times 5$ , stride $1$ , what padding $P$ achieves "same" output ( $O = W = 27$ )? Solve for $P$ .

Recall Solution

With stride $1$ the division is exact, so drop the floor and solve: $27 = (27 - 5 + 2 P) + 1 \Rightarrow 27 = 23 + 2 P \Rightarrow 2 P = 4 \Rightarrow P = 2.$ Padding $2$ — precisely the value in the AlexNet spec. General rule: for stride 1 and an odd kernel side $K$ , "same" padding is $P = \frac{K - 1}{2}$ , which for $K = 5$ gives $2$ . ✓ See VGGNet, which builds entire networks from such same-padded $3 \times 3$ layers.

Exercise 4.3

LeNet has $\sim 60, 000$ parameters; AlexNet has $\sim 60, 000, 000$ . Express AlexNet's parameter count as a multiple of LeNet's, and state which single component of AlexNet dominates that count.

Recall Solution

Ratio $= \frac{60 , 000 , 000}{60 , 000} = 1000$ . AlexNet is 1000× larger. The dominant chunk is the first fully-connected layer FC6: it connects the flattened $6 \times 6 \times 256 = 9216$ features to $4096$ neurons, i.e. $9216 \times 4096 \approx 37.7$ million weights — well over half of all parameters. This is exactly why Dropout was applied to the FC layers: they hold the most parameters and overfit most easily.

Level 5 — Mastery

Exercise 5.1

Build a mini "same-padded" stack and prove size preservation. You feed a $32 \times 32$ input through three consecutive Conv layers, each $3 \times 3$ , stride $1$ , padding $1$ . What is the final side length, and why does depth cost nothing spatially here?

Recall Solution

For one layer: $O = \frac{32 - 3 + 2 ( 1 )}{1} + 1 = \frac{31}{1} + 1 = 32$ . Carefully: $32 - 3 + 2 = 31$ , then $31 + 1 = 32$ . Size unchanged. Because each layer preserves size, three stacked layers give $32 \to 32 \to 32 \to 32$ . Spatial cost is zero; only the receptive field grows (each $3 \times 3$ layer widens what an output pixel "sees" by $2$ , so three layers see a $7 \times 7$ region). This is the core insight behind VGGNet and later ResNet: stack cheap same-padded $3 \times 3$ convs to gain depth without shrinking the map.

Exercise 5.2

Full LeNet spatial trace as a self-check. Starting from $32 \times 32$ , apply C1 ( $5 \times 5$ , s1, p0), S2 ( $2 \times 2$ pool, s2), C3 ( $5 \times 5$ , s1, p0), S4 ( $2 \times 2$ pool, s2), C5 ( $5 \times 5$ , s1, p0). List all five output sizes and confirm C5 collapses to $1 \times 1$ .

Recall Solution

C1: $⌊(32 - 5) /1 ⌋ + 1 = 28$ .
S2: $⌊(28 - 2) /2 ⌋ + 1 = 14$ .
C3: $⌊(14 - 5) /1 ⌋ + 1 = 10$ .
S4: $⌊(10 - 2) /2 ⌋ + 1 = 5$ .
C5: $⌊(5 - 5) /1 ⌋ + 1 = 0 + 1 = 1$ . C5 outputs $1 \times 1 \times 120$ — a $5 \times 5$ kernel on a $5 \times 5$ map has exactly one placement, so it acts like a fully-connected layer. ✓ This is why the parent note calls C5 "effectively FC".

The figure below is your visual answer key: five shrinking cyan squares, each labelled with its side length, marching left to right from $32 \times 32$ down to the single $1 \times 1$ cell. Watch the square shrink at every arrow — conv layers nibble the border away (no padding), pooling halves it, until nothing spatial is left and C5 behaves like a dense layer.

Exercise 5.3

Design decision under a budget. Suppose you must downsample a $224 \times 224$ input to roughly $56 \times 56$ in a single conv layer, using a $7 \times 7$ kernel and padding $3$ . What stride achieves this? Verify.

Recall Solution

We want $O \approx 56$ . Solve $56 = ⌊ \frac{224 - 7 + 2 ( 3 )}{S} ⌋ + 1 = ⌊ \frac{223}{S} ⌋ + 1$ . So $⌊ \frac{223}{S} ⌋ = 55$ . Try $S = 4$ : $⌊ 223/4 ⌋ = ⌊ 55.75 ⌋ = 55$ . ✓ Output $= 56$ . Stride $4$ works. (This is essentially ResNet's stem: a big-kernel, big-stride first layer for cheap aggressive downsampling — see ResNet.)

Recall Rapid self-quiz (cloze)

The output-size formula is ::: $O = ⌊(W - K + 2 P) / S ⌋ + 1$ AlexNet's real Conv1 uses effective input side ::: $227$ "Same" padding for a stride-1 kernel of odd side $K$ is ::: $P = (K - 1) /2$ ReLU's gradient for $x > 0$ is exactly ::: $1$ Dropout test-time scale factor is ::: $(1 - p)$ AlexNet is this many times larger than LeNet in parameters ::: $1000$