LeNet and AlexNet

3.4.6 · AI-ML › Convolutional Neural Networks

Overview

LeNet aur AlexNet landmark convolutional neural network architectures hain jo demonstrate karte hain ki CNNs image recognition tasks pe breakthrough performance achieve kar sakte hain. LeNet (1998) ne handwritten digits pe concept prove kiya, jabki AlexNet (2012) ne massive performance gap ke saath ImageNet jeetkar deep learning revolution start kar diya.

LeNet-5 Architecture

Layer Sequence:

1. Input: $32\times 32$ grayscale image
2. C1 (Conv): 6 filters of $5\times 5$, stride 1 → Output: $28\times 28\times 6$
3. S2 (Pool): $2\times 2$ average pooling, stride 2 → Output: $14\times 14\times 6$
4. C3 (Conv): 16 filters of $5\times 5$, stride 1 → Output: $10\times 10\times 16$
5. S4 (Pool): $2\times 2$ average pooling, stride 2 → Output: $5\times 5\times 16$
6. C5 (Conv): 120 filters of $5\times 5$ → Output: $1\times 1\times 120$ (effectively FC)
7. F6 (FC): 84 neurons
8. Output (FC): 10 neurons (digit classes 0-9)

Total Parameters: ~60,000

Output Dimensions Derive Karna

Convolution/pooling ke baad spatial dimension yeh formula follow karta hai:

$\text{Output size}=\lfloor \frac{\text{Input size}-\text{Kernel size}+2\times \text{Padding}}{\text{Stride}}\rfloor +1$

Derivation: $\text{Output height/width}=\lfloor \frac{32-5+0}{1}\rfloor +1=27+1=28$

Yeh formula kyun?

• Kernel $(32-5+1)=28$ positions horizontally slide kar sakta hai
• Har position ek output value produce karta hai
• 6 filters ke saath, output $28\times 28\times 6$ hoga

Average pooling kyun? LeNet ne $2\times 2$ average pooling use kiya (max nahi):

• Computation: 1998 mein computation expensive tha; averaging, max se sasti thi
• Gradient flow: Averaging saare inputs mein gradients distribute karta hai; max sirf sabse bade mein
• Smoothing: Averaging gentle downsampling provide karta hai

AlexNet Architecture

Layer Sequence:

1. Input: $224\times 224\times 3$ RGB image
2. Conv1: 96 filters of $11\times 11$, stride 4, ReLU → $55\times 55\times 96$
3. MaxPool1: $3\times 3$, stride 2 → $27\times 27\times 96$
4. Conv2: 256 filters of $5\times 5$, padding 2, ReLU → $27\times 27\times 256$
5. MaxPool2: $3\times 3$, stride 2 → $13\times 13\times 256$
6. Conv3: 384 filters of $3\times 3$, padding 1, ReLU → $13\times 13\times 384$
7. Conv4: 384 filters of $3\times 3$, padding 1, ReLU → $13\times 384$
8. Conv5: 256 filters of $3\times 3$, padding 1, ReLU → $13\times 13\times 256$
9. MaxPool3: $3\times 3$, stride 2 → $6\times 6\times 256$
10. FC6: 4096 neurons, ReLU, Dropout(0.5)
11. FC7: 4096 neurons, ReLU, Dropout(0.5)
12. FC8: 1000 neurons (ImageNet classes), Softmax

Total Parameters: ~60 million

AlexNet Ke Key Innovations

$\text{Output size}=\lfloor \frac{224-11+0}{4}\rfloor +1=\lfloor \frac{213}{4}\rfloor +1=53+1=54$

Ruko, AlexNet documentation $55\times 55$ kyun kehta hai?

• Implementation detail: AlexNet ka actual Conv1 implicit padding use karta hai jo 1 pixel add karta hai
• Effective input $227\times 227$ ke saath: $\lfloor \frac{227-11}{4}\rfloor +1=55$
• Papers mein "224" ek simplification hai; code mein 227 use hota hai

Itna bada stride (4) kyun?

• Aggressive downsampling: Computation turant kam karta hai
• Large receptive field: $11\times 11$ kernel with stride 4 input ke bade regions cover karta hai
• Trade-off: Spatial detail kho jaata hai, lekin AlexNet depth se compensate karta hai

1. ReLU Activation

ReLU faster kyun hai: Gradient flow ko consider karo.

Tanh gradient: $\frac{d}{dx}\tanh (x)=1-{\tanh }^2(x)$ Bade $|x|$ ke liye, $\tanh (x)\to \pm 1$, toh gradient $\to 0$ (vanishing gradient).

ReLU gradient: $\frac{d}{dx}\max (0,x)=\{\begin{array}{ll}1& \text{if }x>0\\ 0& \text{if }x\le 0\end{array}$

Key insight: ReLU gradient ya toh 1 ya 0 hota hai, positive inputs ke liye kabhi near-zero nahi. Isse hota hai:

• 6× faster convergence (Krizhevsky ne empirically measure kiya)
• Positive activations ke liye gradient saturation nahi
• Simpler computation (sirf thresholding)

Trade-off: "Dying ReLU" problem jab neurons hamesha 0 output karte hain (baad mein Leaky ReLU se address kiya gaya).

2. Dropout Regularization

Forward pass with dropout: $h_{\text{train}}=\text{mask}\odot a$ jahan ${\text{mask}}_i\sim \text{Bernoulli}(1-p)$ (probability $1-p$ se 1, probability $p$ se 0).

Test time (inference): $h_{\text{test}}=(1-p)\cdot a$

Test time pe $(1-p)$ se scale kyun karte hain?

• Training: Average par, $(1-p)$ fraction neurons active hote hain
• Test: Saare neurons active hote hain, toh outputs $\frac{1}{1-p}$ times bade ho jaate hain
• Solution: Expected training activation scale se match karne ke liye $(1-p)$ se multiply karo

Dropout kyun kaam karta hai:

• Ensemble effect: Alag sub-networks train karna (jaise $n$ neurons ke liye $2^n$ networks train karna)
• Co-adaptation prevention: Neurons ko robust features independently seekhne ke liye force karta hai
• AlexNet ne FC6 aur FC7 mein $p=0.5$ use kiya

3. Local Response Normalization (LRN)

$b_{x,y}^i=\frac{a_{x,y}^i}{{\left(k+\alpha {\sum }_{j=\max (0,i-n/2)}^{\min (N-1,i+n/2)}(a_{x,y}^j{)}^2\right)}^{\beta }}$

jahan:

• $a_{x,y}^i$ = channel $i$ mein position $(x,y)$ par neuron ki activity
• $n$ = normalization window (AlexNet: $n=5$)
• $k=2,\alpha =10^{-4},\beta =0.75$ (hyperparameters)

Intuition: Bade activations wale neurons nearby channels ko suppress karte hain (biological lateral inhibition ko mimic karta hai).

Kyun thoda kaam aaya: Feature maps ke beech competition create kiya, generalization ~1% improve hua.

Ab abandoned kyun hai: Batch Normalization (2015) kaafi zyada effective hai. Modern networks LRN use nahi karte.

4. Data Augmentation

AlexNet ne aggressive data augmentation pioneer kiya:

Technique 1: Random Crops

• $256\times 256$ images se random $224\times 224$ patches extract karo
• Test time pe 5 crops (corners + center) + unke horizontal flips = 10 crops extract karo
• 10 crops pe predictions average karo

Technique 2: PCA Color Augmentation

• Training set ke RGB values pe PCA perform karo
• RGB values mein principal components ke multiples add karo: $[\text{R},\text{G},\text{B}{]}^T=[\text{R},\text{G},\text{B}{]}^T+[p_1,p_2,p_3][{\alpha }_1{\lambda }_1,{\alpha }_2{\lambda }_2,{\alpha }_3{\lambda }_3{]}^T$ jahan $p_i$ = principal components, ${\lambda }_i$ = eigenvalues, ${\alpha }_i\sim N(0,0.1)$

Yeh kyun kaam karta hai: Lighting aur color mein natural variations capture karta hai jabki object identity preserve rehti hai.

Architectural Comparison

| Aspect | LeNet-5 | AlexNet | |--------|------| | Year | 1998 | 2012 | | Input Size | $32\times 32$ grayscale | $224\times 224$ RGB | | Depth | 7 layers | 8 layers | | Parameters | ~60K | ~60M (1000× bada) | | Activation | $\tanh$ / sigmoid | ReLU | | Pooling | Average pooling | Max pooling | | Regularization | Weight decay | Dropout + Data Aug | | Normalization | None | LRN | | Hardware | CPU | 2× GTX 580 GPUs | | Dataset | MNIST (60K images) | ImageNet (1.2M images) | | Classes | 10 | 1000 |

1. Compute: GPUs ne 1000× bade networks train karna feasible bana diya
2. Data: ImageNet ne 20× zyada training data diya
3. Algorithms: ReLU + Dropout ne vanishing gradients aur overfitting solve kiya
4. Belief: Community ko nahi lagta tha ki deep networks kaam karenge jab tak AlexNet ne prove nahi kiya

Worked Examples

Given:

• 6 filters of size $5\times 5$
• Input: 1 channel (grayscale)
• Output: 6 feature maps

Calculation: Har filter mein $5\times 5\times 1=25$ weights plus 1 bias hote hain. Ek filter ke liye total: $25+1=26$ parameters. 6 filters ke liye total: $6\times 26=156$ parameters.

Yeh step kyun? Har filter poore input par slide karta hai, same 26 parameters har jagah share karte hue (parameter sharing overfitting aur computation dono kam karta hai).

Given:

• Input: $6\times 6\times 256$ (flatten hokar $6\times 6\times 256=9216$ neurons)
• Output: 4096 neurons

Calculation: 4096 output neurons mein se har ek saare 9216 input neurons se connect hai: $\text{Weights}=9216\times 4096=37,748,736$ $\text{Biases}=4096$ $\text{Total}=37,748,736+4096=37,752,832\approx 37.8M\text{ parameters}$

Yeh kyun matters karta hai? FC6 akele AlexNet ke 62% total parameters contain karta hai. Isliye modern architectures (ResNet, VGG) FC layers minimize karte hain aur global average pooling use karte hain.

Given:

• Conv1: $11\times 11$ kernel, stride 4

Calculation: Conv1 ka ek neuron input mein $11\times 11$ region dekhta hai. Input mein receptive field directly $11\times 11$ pixels hai.

Conv2 neuron ke liye:

• Conv2, Conv1 feature map mein $5\times 5$ dekhta hai
• Conv1 ka har pixel stride 4 ke saath $11\times 11$ input region se correspond karta hai
• Receptive field: $11+(5-1)\times 4=11+16=27$ pixels

Receptive field ka general formula: $R{F}_l=R{F}_{l-1}+(k_l-1)\times {\prod }_{i=1}^{l-1}{s}_i$ jahan $k_l$ = layer $l$ par kernel size, $s_i$ = layer $i$ par stride.

Yeh kyun matters karta hai? Deep networks input ke bade regions dekhte hain. Conv5 mein ek neuron $195\times 195$ input region dekhta hai, jo bade objects ko recognize karne deta hai.

Common Mistakes

Kyun sahi lagta hai: Zyada parameters = complex patterns seekhne ki zyada capacity.

Fix:

• Parameters tabhi kaam aate hain jab tumhare paas overfitting bachne ke liye enough training data ho
• LeNet on ImageNet severely overfit karega (60K params, 1.2M images = underfitting)
• AlexNet on MNIST severely overfit karega (60M params, 60K images = wasted capacity)
• Rule: Model capacity ko dataset size se match karo. Bade models ko chhote datasets pe train karne ke liye regularization (dropout, data aug) use karo.

Kyun sahi lagta hai: Network "seekh" raha hai, toh use kisi bhi architecture mein adapt ho jaana chahiye.

Fix:

• Stride information loss control karta hai. Bade strides (AlexNet Conv1 stride 4) aggressively downsample karte hain; information permanently kho jaati hai.
• Padding spatial dimension preservation control karta hai. Padding ke bina, spatial size har layer mein shrink hoti hai; padding ke saath, yeh controlled rehti hai.
• Depth par impact: Padding ke bina, $3\times 3$ kernels ke saath $224\times 224$ input ~110 layers mein $1\times 1$ tak pahunch jaata hai. Padding ke saath, aap arbitrarily deep ja sakte ho (ResNet ke 152+ layers hain).

Example: Agar tum har jagah stride 4 use karo, tumhara network spatial information bahut jaldi kho dega aur fine-grained features seekhne mein fail karega.

Kyun sahi lagta hai: AlexNet ne LRN ke saath ImageNet jeeta, toh yeh important hona chahiye.

Fix:

• LRN ne AlexNet mein sirf ~1% improvement diya
• Batch Normalization (2015) kaafi zyada superior hai: faster training, better generalization, 10-30% error reduction
• Modern practice: Ab koi LRN use nahi karta. Uski jagah Batch Norm use karo.
• Lesson: Purane papers ke architectural details ko blindly mat copy karo. Modern best practices use karo.

Active Recall Questions

Iske bajaye, LeNet (1998 mein banaya gaya) ek smart trick use karta hai: yeh image ko layers mein dekhta hai. Pehli layer simple cheezein dhundhti hai jaise edges (horizontal lines, vertical lines, curves). Doosri layer un edges ko combine karti hai thodi complex shapes banane ke liye (jaise "5" ke upar ka hissa ya "6" mein loop). Inhe stack karke, computer automatically digits recognize karna seekhta hai, bina koi rules program kiye!

LeNet simple images (jaise MNIST digits) ke liye bahut achha kaam karta tha, lekin real photos mein cats, dogs, cars recognize karne ka kya? Woh kaafi mushkil hai kyunki photos badi hoti hain, zyada colorful hoti hain, aur kaafi variety hoti hai.

2012 mein, AlexNet ne yeh solve kiya network ko bahut bada aur deeper banake. Usne use kiya:

1. Zyada layers (8 instead of 5) zyada complex patterns seekhne ke liye
2. Faster math (ReLU instead of tanh, jaise slow division se fast addition pe switch karna)
3. Dropout (training ke dauran neurons ko randomly band karna, jaise kisi handicap ke saath koi sport practice karna taaki tum zyada strong ho jao)
4. Data tricks (images flip karna, colors thoda change karna taaki computer ko zyada examples dikhao)

AlexNet itna better tha baaki sab se ki usne duniya ko shock kar diya aur "deep learning revolution" start kar di. Ab, similar ideas face recognition, self-driving cars, aur AI assistants mein kaam aati hain!

AlexNet ke 8 layers ke liye: "Cats Can Climb Canyon Cliffs For Fun Finally" (Conv, Conv, Conv, Conv, Conv, FC, FC, FC)

Connections

Prerequisites:

• Convolutional Layers — Filters, stride, padding ko samajhna
• Pooling Layers — Max pooling vs average pooling
• Activation Functions — ReLU, tanh, sigmoid
• Backpropagation — Networks mein gradients kaise flow karte hain

Related Concepts:

• VGGNet — Uniform $3\times 3$ filters ke saath deeper architecture (2014)
• Batch Normalization — LRN ko replace kiya, layer inputs normalize karta hai (2015)
• Dropout — AlexNet ki key regularization technique
• ResNet — Skip connections 100+ layer networks enable karte hain (2015)
• ImageNet Dataset — Woh benchmark jo AlexNet ne jeet liya

Applications:

• Transfer Learning — Naye tasks ke liye pretrained AlexNet/VGG use karna
• Object Detection — R-CNN ne AlexNet features pe build kiya
• Image Classification — Saare modern architectures ki foundation

#flashcards/ai-ml

LeNet-5 ka input size aur parameters ki sankhya kya hai? :: Input: $32\times 32$ grayscale image. Parameters: approximately 60,000.

Concept Map