3.4.6 · D5Convolutional Neural Networks
Question bank — LeNet and AlexNet
This page is a trap-detector. Every item below targets a spot where students think they understand LeNet and AlexNet but actually hold a subtle misconception. Read the prompt, answer out loud, then reveal. If your reasoning differs from the answer's reasoning (not just the verdict), you found your gap.
True or false — justify
TF1. "AlexNet is deeper than LeNet, so its power comes mostly from having more layers."
False — AlexNet has only 8 layers vs LeNet's 7, barely deeper. The leap came from scale (1000× more parameters, ImageNet's 1.2M images), ReLU, GPUs, and Dropout — not depth. Depth-driven gains arrive later with VGGNet and ResNet.
TF2. "LeNet used average pooling because average pooling is objectively better than max pooling."
False — it was a 1998 pragmatic choice: averaging was cheaper and spread gradients smoothly. AlexNet later switched to max pooling because sharper feature selection beat smoothing on hard natural images.
TF3. "ReLU speeds up training because it makes each forward pass cheaper to compute."
Partly, but that's the minor reason. The real win is that ReLU's gradient is exactly for positive inputs, so it never saturates — unlike whose gradient vanishes toward for large , stalling Backpropagation.
TF4. "Dropout is applied during both training and testing to keep the network stable."
False — dropout only happens at training time. At test time all neurons stay on, and outputs are scaled by to match the expected activation scale seen during training.
TF5. "Local Response Normalization and Batch Normalization do essentially the same job, so LRN is fine to use today."
False — LRN normalizes across nearby channels at one spatial location; Batch Normalization normalizes across the batch per channel. BN is far more effective, so LRN is now abandoned.
TF6. "AlexNet's input is genuinely , exactly as stated in the paper."
Misleading — the code actually feeds (or uses implicit -pixel padding). Only makes come out right; "224" is a paper-side simplification.
TF7. "Because ReLU never saturates, it has no failure mode."
False — the dying ReLU problem: a neuron pushed into the negative region always outputs , so its gradient is and it can never recover. Leaky ReLU exists precisely to fix this.
TF8. "Data augmentation increases the amount of real information the network learns from."
False — augmentation adds no new information; it teaches invariance (to crops, flips, lighting) by showing plausible transformed versions of existing labels, reducing overfitting.
Spot the error
SE1. "The output-size formula for Conv1 gives ."
Error: , not 55. You reach 55 only by using the true input : . Plugging 224 into the formula and claiming 55 hides that mismatch.
SE2. "For LeNet C1 with input 32, kernel 5, stride 1: output ."
Error: you dropped the . The kernel slides to positions, so the output is , not 27. The "" counts the starting position itself.
SE3. "Dropout with keeps half the neurons, so at test time multiply activations by ."
Error: at test time you multiply by , not by . Test-time all neurons fire; scaling down by matches the smaller expected sum seen in training.
SE4. " has a vanishing-gradient problem, so its derivative is always near zero."
Error: near the derivative is close to — perfectly healthy. It vanishes only for large , where . The problem is saturation at the tails, not everywhere.
SE5. "AlexNet's stride-4 Conv1 keeps fine spatial detail because the kernel is large."
Error: a large stride throws away spatial detail — it skips 3 of every 4 positions. The large kernel gives a big receptive field, but stride 4 is deliberately lossy downsampling; AlexNet recovers detail through later depth.
SE6. "LeNet's C5 layer is a convolution, so it produces a 2-D feature map like the earlier conv layers."
Error: with a input and kernels, the output is — a single value per filter. It is a convolution in name but behaves as a fully-connected layer.
SE7. "ReLU's gradient at is because ReLU is increasing there."
Error: by the standard convention the gradient is taken as at (ReLU has a kink at , so it isn't differentiable there; frameworks pick ). Not a smooth transition.
Why questions
WQ1. Why did AlexNet win ImageNet by ~10 points instead of just edging out competitors?
Traditional pipelines used hand-crafted features with limited capacity. AlexNet learned features end-to-end via Backpropagation on 1.2M images with enough capacity to actually use them — a qualitative jump, not an incremental tweak.
WQ2. Why does LRN suppress neighbouring channels rather than acting on a single neuron alone?
It mimics biological lateral inhibition: a strongly firing feature map dampens competing maps at the same location, creating competition so only the most confident features dominate.
WQ3. Why is dropout described as training an ensemble of networks?
Each forward pass randomly zeros a different subset of neurons, so effectively a different sub-network is trained each step. With neurons there are possible masks, and inference averages over them via the scaling.
WQ4. Why did the field wait ~14 years between LeNet and AlexNet if CNNs already worked?
LeNet's ideas were sound but starved of resources: no large labelled dataset, no GPU compute, and no ReLU/dropout tricks to train big nets. AlexNet is what happens when all three arrive together.
WQ5. Why does average pooling distribute gradients to all its inputs while max pooling does not?
Averaging is a linear combination of every input, so each contributes equally to the gradient. Max only passes its output through the single largest input, so during Backpropagation only that one input receives gradient.
WQ6. Why is a -input needed to reproduce AlexNet's stated Conv1 output?
Because the arithmetic only works with . The extra pixels come from implicit padding; the "224" figure alone gives , exposing the discrepancy.
WQ7. Why can PCA color augmentation change pixel colors yet preserve the label?
It shifts colors along the natural directions of variation (principal components of the RGB distribution), scaled by eigenvalues. This mimics realistic lighting/color changes an object survives, so a "dog" stays a dog.
Edge cases
EC1. What happens to a ReLU neuron whose weighted input is exactly ?
It outputs and, by convention, has gradient . If it stays pinned at across all training examples, it is effectively dead and stops learning — the boundary case of the dying-ReLU problem.
EC2. What if you forget to scale by at test time in a dropout network?
Every test activation is inflated by a factor of relative to training, so downstream neurons see systematically larger inputs than they were trained on — degrading accuracy even though nothing "crashes".
EC3. What is the output size of a conv layer when kernel size equals input size (stride 1, no padding)?
— a single spatial value per filter. This is exactly LeNet's C5 and is why it behaves as fully-connected.
EC4. What happens to -based training as network depth grows large?
Vanishing gradients compound multiplicatively: each saturated layer shrinks the gradient toward , so deep nets barely train early layers — a key reason ReLU (and later Batch Normalization and ResNet) became necessary.
EC5. What if the pooling window's stride is smaller than its size (e.g. AlexNet's stride )?
The windows overlap. Overlapping pooling was a deliberate AlexNet choice — it slightly reduced overfitting compared to non-overlapping pooling.
EC6. If dropout probability , what happens to the network during training?
Almost every neuron is zeroed, so essentially no signal passes forward and no gradient flows back — training collapses. The useful regime keeps moderate (AlexNet used in FC layers).
EC7. What if you apply a -image directly to AlexNet's stated pipeline without the extra padding?
Conv1 yields , so every subsequent dimension shifts and the final feature map no longer lines up with FC6's expected input size — the network mismatches unless padding restores .
Recall One-line self-test
Cover the reveals: could you justify every verdict, not just state it? ::: If any answer was "true/false but I'm not sure why", revisit that section — the reasoning is the point, the verdict is not.