CNN architecture design

3.4.5 · AI-ML › Convolutional Neural Networks

Core Design Dimensions

Ye Kyun Matter Karte Hain

• Depth feature hierarchy control karti hai: shallow layers edges/textures seekhti hain, deep layers parts/objects seekhti hain
• Width har level par capacity control karti hai: zyada filters = zyada parallel feature detectors
• Connectivity training problems solve karta hai: skip connections vanishing gradients se ladte hain
• Receptive field context determine karta hai: object recognition ko edge detection se bada field chahiye

Classic Architectures Evolution

1. LeNet-5 (1998)

Structure: Conv → Pool → Conv → Pool → FC → FC → Output

Input(32×32) → Conv(6@5×5) → AvgPool(2×2) → Conv(16@5×5) → AvgPool(2×2) → FC(120) → FC(84) → FC(10)

Design Philosophy:

• Chhota, shallow (sirf 2 conv layers)
• Digit recognition ke liye use hua (MNIST)
• Receptive field: har output neuron input ka 28×28 region dekhta hai

Ye kaam kyun karta hai: Simple patterns (digits) ke liye, tumhe sirf basic edge combinations chahiye. Do conv layers kaafi hain "edges" → "digit parts" → "full digit" tak jaane ke liye.

2. AlexNet (2012)

Structure:

Conv(96@11×11,s=4) → MaxPool → Conv(256@5×5) → MaxPool → 
Conv(384@3×3) → Conv(384@3×3) → Conv(256@3×3) → MaxPool → FC(4096) → FC(4096) → FC(1000)

Key Innovations:

1. ReLU activation (tanh ki jagah): faster training, no vanishing gradient
2. Dropout FC layers mein: regularization
3. Data augmentation: random crops, flips
4. GPU training: 2 GPUs par split

Deeper kyun? ImageNet mein 1000 classes hain complex objects ke saath. Hierarchy chahiye: edges → textures → parts → objects.

3. VGGNet (2014)

Design Principle: Sirf 3×3 convolutions use karo, unhe deep stack karo.

VGG-16 Structure:

• Block 1: Conv(64@3×3) → Conv(64@3×3) → MaxPool
• Block 2: Conv(128@3×3) → Conv(128@3×3) → MaxPool
• Block 3: Conv(256@3×3) → Conv(256@3×3) → Conv(256@3×3) → MaxPool
• Block 4: Conv(512@3×3) × 3 → MaxPool
• Block 5: Conv(512@3×3) × 3 → MaxPool
• FC(4096) → FC(4096) → FC(1000)

Ye kaam kyun karta hai: Har pool ke baad filters double karna computational cost balance karta hai (spatial size aadha hoti hai, channels double hote hain).

$n$ stacked 3×3 convs ke liye general formula: $RF=1+n\cdot (3-1)=2n+1$

Teen 3×3 layers → 7×7 receptive field, lekin 1 ki jagah 3 nonlinearities ke saath.

4. ResNet (2015)

Residual Block:

x → Conv → ReLU → Conv → (+) → ReLU → output
 ↓________________________↑
      (skip connection)

Mathematically: $\mathbf{H}(\mathbf{x})=\mathbf{F}(\mathbf{x})+\mathbf{x}$

Jahaan $\mathbf{F}(\mathbf{x})$ residual mapping hai (layers kya seekhti hain).

Ye kaam kyun karta hai:

1. Gradient flow: $\frac{\partial H}{\partial x}=\frac{\partial F}{\partial x}+1$ — "+1" ensure karta hai ki gradient kabhi vanish na ho
2. Identity shortcut: Agar zaroorat ho, network $F(x)=0$ seekh sakta hai aur sirf input copy kar sakta hai (layers ke through identity seekhne se aasaan)
3. Ensemble interpretation: Input se output tak har path ek alag model jaisa hai; ResNet $2^n$ paths ensemble karta hai

Bottleneck design: 1×1 conv channels reduce karta hai → 3×3 conv process karta hai → 1×1 conv wapas expand karta hai. Computation ~70% reduce karta hai.

5. Inception (GoogLeNet)

Inception Module:

                Input
                      |
        ┌─────────┬───┴────┬────────┐
        |         |          |
     1×1 conv  1×1→3×3  1×1→5×5   3×3 pool→1×1
        |         |          |
        └─────────┴──────────┘
                      |
              Concatenate (depth)

1×1 bade convs se pehle kyun? Dimensionality reduction:

• Bina: 256 channels → 5×5 conv with 256 filters = $256\times 256\times 25=1.6M$ params
• 1×1 ke saath: 256→64→256 = $256\times 64\times 1+64\times 256\times 25=426K$ params (74% reduction)

Concatenate: 28×28×(64+128+64+64) = 28×28×320

Ye step kyun? Har branch alag receptive fields capture karti hai. Network task ke basis par branches weight karna seekhta hai.

Design Principles (Best Practices Derive Karna)

Principle 1: Receptive Field Growth

Object recognition ke liye, final layer ko poora object dekhna chahiye. Do strategies:

1. Pooling: Spatial dimensions aadhe karo → effective RF double hoti hai $R{F}_{new}=2\cdot R{F}_{old}+(k-1)$ size $k$ ke pooling ke liye.

2. Stride: Convolution mein pixels skip karo

   • Stride $s$ RF ko $s$ se multiply karta hai
   • Lekin information lose hoti hai (sparingly use karo)

Pools ke beech 3×3 convs ke saath, 100×100 reach karne ke liye ~15 total conv layers chahiye.

Principle 2: Channel Progression

Standard pattern: Spatial size aadhi hone par channels double karo.

Kyun? Computational balance maintain karo: $\text{FLOPs}\propto H\times W\times C_{in}\times C_{out}\times k^2$

Agar $H,W$ aadhe ho jayein aur $C$ double ho jaye: FLOPs constant rehte hain.

jahaan $C_0$ initial channels hai (typically 64).

Principle 3: Spatial Invariance

Problem: Images mein translation/scale/rotation variance hoti hai. Objects kahin bhi appear ho sakte hain.

Solutions:

1. Data augmentation: Crops, flips, rotations par train karo
2. Pooling: Max-pooling local translation invariance deta hai
3. Global average pooling: FC layers replace karo → full translation invariance

Example: 7×7×512 → Flatten → FC(1000) ki jagah:

7×7×512 → AvgPool(7×7) → 1×1×512 → Conv(1×1, 1000 filters) → 1×1×1000 → Softmax

Isme koi spatial position encoding nahi → kisi bhi input size par kaam karta hai!

Principle 4: Computational Efficiency

$\text{FLOPs}=H^{\prime }\times W^{\prime }\times C_{in}\times C_{out}\times k^2$

Sabse expensive layers: Early wale (bade $H,W$) aur late wale (bade $C$).

Optimization strategies:

1. Depthwise separable convolutions (MobileNet):

   • Depthwise: Har input channel ka apna $k\times k$ filter hota hai → $H^{\prime }\times W^{\prime }\times C\times k^2$
   • Pointwise: Channels mix karne ke liye 1×1 conv → $H^{\prime }\times W^{\prime }\times C\times C$
   • Total: $H^{\prime }\times W^{\prime }\times C\times (k^2+C^{\prime })$ vs standard $H^{\prime }\times W^{\prime }\times C\times C^{\prime }\times k^2$
   • Speedup: $\frac{k^2\times C^{\prime }}{k^2+C^{\prime }}\approx k^2$ (jaise 3×3 ke liye 9×)

2. Bottleneck layers: 1×1 reduce → 3×3 process → 1×1 expand

Common Mistakes & Fixes

Bad (ResNet-50 copy):

• 7×7 conv, stride 2 → 32×32
• MaxPool → 16×16
• 4 stages of residual blocks → 1×1 (bahut zyada aggressive!)
• 2 classes ke liye over-parameterized

Good (adapted):

• 3×3 conv, stride 1 → 64×64 (resolution preserve karo)
• [3×3 conv × 2 + MaxPool] × 3 → 8×8
• AvgPool → 1×1×256
• FC(2) with softmax
• 10× kam parameters, faster train hota hai, similar accuracy

Modern Architecture Patterns

Neural Architecture Search (NAS)

Idea: Algorithms ko architectures design karne do.

Search space: Building blocks (conv, pool, skip) aur constraints (max depth, FLOP budget) define karo.

Search methods:

1. Reinforcement learning: Controller RNN architectures propose karta hai, validation accuracy par trained hota hai
2. Evolution: Architectures mutate/crossover karo, fittest select karo
3. Differentiable NAS: Architecture choices continuous banao, gradient descent use karo

Result: EfficientNet (NAS se discover hua) ResNet-50 accuracy achieve karta hai 10× kam parameters ke saath.

EfficientNet Scaling

Kyun? Higher resolution images ko features extract karne ke liye deeper/wider nets chahiye.

Vision Transformers (2020s)

Convolution se bilkul alag ho kar:

• Image ko 16×16 patches mein divide karo
• Sequence ki tarah treat karo (jaise NLP mein words)
• Transformer self-attention use karo

Tradeoff: CNN performance match karne ke liye huge datasets (ImageNet-21k+) chahiye, lekin better scale karta hai.

Design Checklist

CNN design karte waqt:

1. Input size → Pooling schedule determine karta hai (~7×7 tak aadha karo)
2. Task complexity → Depth determine karta hai (simple: 5-10 layers, complex: 50-100)
3. Compute budget → Limited ho to bottlenecks/depthwise convs use karo
4. Data availability → Zyada data → bina overfitting ke deeper ja sakte ho
5. Interpretability needs → Simpler architectures (VGG-style) visualize karna aasaan hai

Connections

• 3.401-convolution-operation — Sabhi CNN architectures ka building block
• 3.4.02-pooling-layers — Downsampling strategy architecture depth ko affect karti hai
• 3.4.03-activation-functions — ReLU ne deeper networks enable kiye (AlexNet innovation)
• 3.4.04-batch-normalization — Bahut deep nets mein training stabilize karta hai (post-2015 standard)
• 3.5.01-transfer-learning — Pre-trained architectures feature extractors ki tarah
• 4.2.03-vanishing-gradient — Core problem jo ResNet solve karta hai
• 5.1.02-hyperparameter-tuning — Architecture choices hyperparameters hain
• 6.3.01-model-compression — Architectures ko deployment ke liye efficient banana

#flashcards/ai-ml

CNN layer ka receptive field kya hai? :: Input image ka wo spatial region jo ek particular output neuron ko influence karta hai. Depth ke saath convolutions aur pooling ke zariye badhta hai.

Depthwise separable convolution kya hai? :: Standard convolution ko depthwise (har input channel ke liye ek filter) aur pointwise (channels mix karne ke liye 1×1) mein factorize karta hai. k×k kernels ke liye ~k² se computation reduce karta hai.

Concept Map