LeNet and AlexNet

Overview

LeNet and AlexNet are landmarkconvolutional neural network architectures== that demonstrated CNs could achieve breakthrough performance on image recognition tasks. LeNet (1998) proved the concept on handwritten digits, while AlexNet (2012) sparked the deep learning revolution by winning ImageNet with a massive performance gap.

LeNet-5 Architecture

Layer Sequence:

1. Input: $32 \times 32$ grayscale image
2. C1 (Conv): 6 filters of $5 \times 5$, stride 1 → Output: $28 \times 28 \times 6$
3. S2 (Pool): $2 \times 2$ average pooling, stride 2 → Output: $14 \times 14 \times 6$
4. C3 (Conv): 16 filters of $5 \times 5$, stride 1 → Output: $10 \times 10 \times 16$
5. S4 (Pool): $2 \times 2$ average pooling, stride 2 → Output: $5 \times 5 \times 16$
6. C5 (Conv): 120 filters of $5 \times 5$ → Output: $1 \times 1 \times 120$ (effectively FC)
7. F6 (FC): 84 neurons
8. Output (FC): 10 neurons (digit classes 0-9)

Total Parameters: ~60,000

Deriving Output Dimensions

The spatial dimension after convolution/pooling follows:

$\text{Output size} = \left\lfloor \frac{\text{Input size} - \text{Kernel size} + 2 \times \text{Padding}}{\text{Stride}} \right\rfloor + 1$

Derivation: $\text{Output height/width} = \left\lfloor \frac{32 - 5 + 0}{1} \right\rfloor + 1 = 27 + 1 = 28$

Why this formula?

• The kernel can slide $(32 - 5 + 1) = 28$ positions horizontally
• Each position produces one output value
• With 6 filters, output is $28 \times 28 \times 6$

Why average pooling? LeNet used $2 \times 2$ average pooling (not max):

• Computation: In 1998, computation was expensive; averaging is cheaper than max
• Gradient flow: Averages distribute gradients to all inputs; max only to the largest
• Smoothing: Averaging provides gentle downsampling

AlexNet Architecture

Layer Sequence:

1. Input: $224 \times 224 \times 3$ RGB image
2. Conv1: 96 filters of $11 \times 11$, stride 4, ReLU → $55 \times 55 \times 96$
3. MaxPool1: $3 \times 3$, stride 2 → $27 \times 27 \times 96$
4. Conv2: 256 filters of $5 \times 5$, padding 2, ReLU → $27 \times 27 \times 256$
5. MaxPool2: $3 \times 3$, stride 2 → $13 \times 13 \times 256$
6. Conv3: 384 filters of $3 \times 3$, padding 1, ReLU → $13 \times 13 \times 384$
7. Conv4: 384 filters of $3 \times 3$, padding 1, ReLU → $13 \times 384$
8. Conv5: 256 filters of $3 \times 3$, padding 1, ReLU → $13 \times 13 \times 256$
9. MaxPool3: $3 \times 3$, stride 2 → $6 \times 6 \times 256$
10. FC6: 4096 neurons, ReLU, Dropout(0.5)
11. FC7: 4096 neurons, ReLU, Dropout(0.5)
12. FC8: 1000 neurons (ImageNet classes), Softmax

Total Parameters: ~60 million

Key Innovations in AlexNet

$\text{Output size} = \left\lfloor \frac{224 - 11 + 0}{4} \right\rfloor + 1 = \left\lfloor \frac{213}{4} \right\rfloor + 1 = 53 + 1 = 54$

Wait, why does AlexNet documentation say $55 \times 55$?

• Implementation detail: AlexNet's actual Conv1 uses implicit padding that adds 1 pixel
• With effective input $227 \times 227$: $\left\lfloor \frac{227 - 11}{4} \right\rfloor + 1 = 55$
• The "224" in papers is a simplification; the code uses227

Why such a large stride (4)?

• Aggressive downsampling: Reduces computation immediately
• Large receptive field: $11 \times 11$ kernel with stride 4 covers huge input regions
• Trade-off: Loses spatial detail, but AlexNet compensates with depth

1. ReLU Activation

Why ReLU is faster: Consider gradient flow.

Tanh gradient: $\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x)$ For large $|x|$, $\tanh(x) \to \pm 1$, so gradient $\to 0$ (vanishing gradient).

ReLU gradient: $\frac{d}{dx}\max(0, x) = \begin{cases} 1 & \text{if } x > 0 \\ 0 & \text{if } x \leq 0 \end{cases}$

Key insight: ReLU gradient is either 1 or 0, never near-zero for positive inputs. This allows:

• 6× faster convergence (measured empirically by Krizhevsky)
• No gradient saturation for positive activations
• Simpler computation (just thresholding)

Trade-off: "Dying ReLU" problem when neurons always output 0 (addressed later by Leaky ReLU).

2. Dropout Regularization

Forward pass with dropout: $h_{\text{train}} = \text{mask} \odot a$ where $\text{mask}_i \sim \text{Bernoulli}(1-p)$ (1 with probability $1-p$, 0 with probability $p$).

Test time (inference): $h_{\text{test}} = (1-p) \cdot a$

Why scale by $(1-p)$ at test time?

• Training: On average, $(1-p)$ fraction of neurons are active
• Test: All neurons are active, so outputs are $\frac{1}{1-p}$ times larger
• Solution: Multiply by $(1-p)$ to match expected training activation scale

Why dropout works:

• Ensemble effect: Training different sub-networks (like training $2^n$ networks for $n$ neurons)
• Co-adaptation prevention: Forces neurons to learn robust features independently
• AlexNet used $p = 0.5$ in FC6 and FC7

3. Local Response Normalization (LRN)

$b_{x,y}^i = \frac{a_{x,y}^i}{\left(k + \alpha \sum_{j=\max(0, i-n/2)}^{\min(N-1, i+n/2)} (a_{x,y}^j)^2\right)^\beta}$

where:

• $a_{x,y}^i$ = activity of neuron at position $(x,y)$ in channel $i$
• $n$ = normalization window (AlexNet: $n=5$)
• $k=2, \alpha=10^{-4}, \beta=0.75$ (hyperparameters)

Intuition: Neurons with large activations suppress nearby channels (mimics biological lateral inhibition).

Why it worked (slightly): Created competition between feature maps, improving generalization by ~1%.

Why it's abandoned: Batch Normalization (2015) is far more effective. Modern networks don't use LRN.

4. Data Augmentation

AlexNet pionered aggressive data augmentation:

Technique 1: Random Crops

• Extract random $224 \times 224$ patches from $256 \times 256$ images
• At test time, extract5 crops (corners + center) + their horizontal flips = 10 crops
• Average predictions over 10 crops

Technique 2: PCA Color Augmentation

• PerformCA on RGB values across training set
• Add multiples of principal components to RGB values: $[\text{R}, \text{G}, \text{B}]^T = [\text{R}, \text{G}, \text{B}]^T + [p_1, p_2, p_3][\alpha_1 \lambda_1, \alpha_2 \lambda_2, \alpha_3 \lambda_3]^T$ where $p_i$ = principal components, $\lambda_i$ = eigenvalues, $\alpha_i \sim N(0, 0.1)$

Why this works: Captures natural variations in lighting and color while preserving object identity.

Architectural Comparison

| Aspect | LeNet-5 | AlexNet | |--------|------| | Year | 1998 | 2012 | | Input Size | $32 \times 32$ grayscale | $224 \times 224$ RGB | | Depth | 7 layers | 8 layers | | Parameters | ~60K | ~60M (1000× larger) | | Activation | $\tanh$ / sigmoid | ReLU | | Pooling | Average pooling | Max pooling | | Regularization | Weight decay | Dropout + Data Aug | | Normalization | None | LRN | | Hardware | CPU | 2× GTX 580 GPUs | | Dataset | MNIST (60K images) | ImageNet (1.2M images) | | Classes | 10 | 1000 |

1. Compute: GPUs made training 1000× larger networks feasible
2. Data: ImageNet provided 20× more training data
3. Algorithms: ReLU + Dropout solved vanishing gradients and overfitting
4. Belief: The community didn't believe deep networks would work until AlexNet proved it

Worked Examples

Given:

• 6 filters of size $5 \times 5$
• Input: 1 channel (grayscale)
• Output: 6 feature maps

Calculation: Each filter has $5 \times 5 \times 1 = 25$ weights plus 1 bias. Total per filter: $25 + 1 = 26$ parameters. Total for 6 filters: $6 \times 26 = 156$ parameters.

Why this step? Each filter slides over the entire input, sharing the same 26 parameters everywhere (parameter sharing reduces overfitting and computation).

Given:

• Input: $6 \times 6 \times 256$ (flattened to $6 \times 6 \times 256 = 9216$ neurons)
• Output: 4096 neurons

Calculation: Each of 4096 output neurons connects to all 9216 input neurons: $\text{Weights} = 9216 \times 4096 = 37,748,736$ $\text{Biases} = 4096$ $\text{Total} = 37,748,736 + 4096 = 37,752,832 \approx 37.8M \text{ parameters}$

Why this matters? FC6 alone contains 62% of AlexNet's total parameters. This is why modern architectures (ResNet, VGG) minimize FC layers and use global average pooling instead.

Given:

• Conv1: $11 \times 11$ kernel, stride 4

Calculation: A Conv1 neuron sees an $11 \times 11$ region in the input. The receptive field in the input is directly $11 \times 11$ pixels.

For a Conv2 neuron:

• Conv2 sees $5 \times 5$ in Conv1 feature map
• Each Conv1 pixel corresponds to $11 \times 11$ input region with stride 4
• Receptive field: $11 + (5-1) \times 4 = 11 + 16 = 27$ pixels

General formula for receptive field: $RF_{l} = RF_{l-1} + (k_l - 1) \times \prod_{i=1}^{l-1} s_i$ where $k_l$ = kernel size at layer $l$, $s_i$ = stride at layer $i$.

Why this matters? Deep networks see large input regions. A neuron in Conv5 sees a $195 \times 195$ input region, allowing recognition of large objects.

Common Mistakes

Why it feels right: More parameters = more capacity to learn complex patterns.

The fix:

• Parameters only help if you have enough training data to avoid overfitting
• LeNet on ImageNet would severely overfit (60K params, 1.2M images = underfitting)
• AlexNet on MNIST would severely overfit (60M params, 60K images = wasted capacity)
• Rule: Match model capacity to dataset size. Use regularization (dropout, data aug) to train large models on smaller datasets.

Why it feels right: The network is "learning," so it should adapt to any architecture.

The fix:

• Stride controls information loss. Large strides (AlexNet Conv1 stride 4) agressively downsample; information is permanently lost.
• Padding controls spatial dimension preservation. Without padding, spatial size shrinks every layer; with padding, it's controlled.
• Impact on depth: Without padding, a $224 \times 224$ input with $3 \times 3$ kernels reaches $1 \times 1$ in ~110 layers. With padding, you can go arbitrarily deep (ResNet has 152+ layers).

Example: If you use stride 4 everywhere, your network will lose spatial information too quickly and fail to learn fine-grained features.

Why it feels right: AlexNet won ImageNet with LRN, so it must be important.

The fix:

• LRN provided only ~1% improvement in AlexNet
• Batch Normalization (2015) is far superior: faster training, better generalization,10-30% error reduction
• Modern practice: No one uses LRN anymore. Use Batch Norm instead.
• Lesson: Don't cargo-cult architectural details from old papers. Use modern best practices.

Active Recall Questions

Instead, LeNet (created in 1998) uses a clever trick: it looks at the image in layers. The first layer finds simple things like edges (horizontal lines, vertical lines, curves). The second layer combines those edges to find slightly more complex shapes (like the top of a "5" or the loop in a "6"). By stacking these layers, the computer learns to recognize digits automatically, without anyone programming the rules!

LeNet worked great for simple images (like MNIST digits), but what about recognizing cats, dogs, cars in real photos? That's way harder because photos are bigger, more colorful, and have way more variety.

In 2012, AlexNet solved this by making the network much bigger and deeper. It used:

1. More layers (8 instead of 5) to learn more complex patterns
2. Faster math (ReLU instead of tanh, like switching from slow division to fast addition)
3. Dropout (randomly turning off neurons during training, like practicing a sport with a handicap so you get stronger)
4. Data tricks (flipping images, changing colors slightly to show the computer more examples)

AlexNet was so much better than everything else that it shocked the world and started the "deep learning revolution." Now, similar ideas power face recognition, self-driving cars, and AI assistants!

For AlexNet's 8 layers: "Cats Can Climb Canyon Cliffs For Fun Finally" (Conv, Conv, Conv, Conv, Conv, FC, FC, FC)

Connections

Prerequisites:

• Convolutional Layers — Understanding filters, stride, padding
• Pooling Layers — Max pooling vs average pooling
• Activation Functions — ReLU, tanh, sigmoid
• Backpropagation — How gradients flow through networks

Related Concepts:

• VGGNet — Deeper architecture with uniform $3 \times 3$ filters (2014)
• Batch Normalization — Replaced LRN, normalizes layer inputs (2015)
• Dropout — AlexNet's key regularization technique
• ResNet — Skip connections enable100+ layer networks (2015)
• ImageNet Dataset — The benchmark AlexNet conquered

Applications:

• Transfer Learning — Using pretrained AlexNet/VGG for new tasks
• Object Detection — R-CNN built on AlexNet features
• Image Classification — Foundation for all modern architectures

#flashcards/ai-ml

What is the input size and number of parameters in LeNet-5? :: Input: $32 \times 32$ grayscale image. Parameters: approximately 60,000.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)