CNN architecture design

Core Design Dimensions

Why These Matter

• Depth controls feature hierarchy: shallow layers learn edges/textures, deep layers learn parts/objects
• Width controls capacity at each level: more filters = more parallel feature detectors
• Connectivity solves training problems: skip connections combat vanishing gradients
• Receptive field determines context: object recognition needs larger fields than edge detection

Classic Architectures Evolution

1. LeNet-5 (1998)

Structure: Conv → Pool → Conv → Pool → FC → FC → Output

Input(32×32) → Conv(6@5×5) → AvgPool(2×2) → Conv(16@5×5) → AvgPool(2×2) → FC(120) → FC(84) → FC(10)

Design Philosophy:

• Small, shallow (only 2 conv layers)
• Used for digit recognition (MNIST)
• Receptive field: each output neuron sees 28×28 region of input

Why it works: For simple patterns (digits), you only need basic edge combinations. Two conv layers are enough to go from "edges" → "digit parts" → "full digit".

2. AlexNet (2012)

Structure:

Conv(96@11×11,s=4) → MaxPool → Conv(256@5×5) → MaxPool → 
Conv(384@3×3) → Conv(384@3×3) → Conv(256@3×3) → MaxPool → FC(4096) → FC(4096) → FC(1000)

Key Innovations:

1. ReLU activation (instead of tanh): faster training, no vanishing gradient
2. Dropout in FC layers: regularization
3. Data augmentation: random crops, flips
4. GPU training: split across2 GPUs

Why deeper? ImageNet has 1000 classes with complex objects. Need hierarchy: edges → textures → parts → objects.

3. VGGNet (2014)

Design Principle: Use only 3×3 convolutions, stack them deep.

VGG-16 Structure:

• Block 1: Conv(64@3×3) → Conv(64@3×3) → MaxPool
• Block 2: Conv(128@3×3) → Conv(128@3×3) → MaxPool
• Block 3: Conv(256@3×3) → Conv(256@3×3) → Conv(256@3×3) → MaxPool
• Block 4: Conv(512@3×3) × 3 → MaxPool
• Block 5: Conv(512@3×3) × 3 → MaxPool
• FC(4096) → FC(4096) → FC(1000)

Why it works: Doubling filters after each pool balances computational cost (spatial size halves, channels double).

General formula for $n$ stacked 3×3 convs: $RF = 1 + n \cdot (3 - 1) = 2n + 1$

Three 3×3 layers → 7×7 receptive field, but with 3nonlinearities vs 1.

4. ResNet (2015)

Residual Block:

x → Conv → ReLU → Conv → (+) → ReLU → output
 ↓________________________↑
      (skip connection)

Mathematically: $\mathbf{H}(\mathbf{x}) = \mathbf{F}(\mathbf{x}) + \mathbf{x}$

Where $\mathbf{F}(\mathbf{x})$ is the residual mapping (what the layers learn).

Why it works:

1. Gradient flow: $\frac{\partial H}{\partial x} = \frac{\partial F}{\partial x} + 1$ — the "+1" ensures gradient never vanishes
2. Identity shortcut: If needed, the network can learn $F(x) = 0$ and just copy input (easier than learning identity through layers)
3. Ensemble interpretation: Each path from input to output is like a separate model; ResNet ensembles $2^n$ paths

Bottleneck design: 1×1 conv reduces channels → 3×3 conv processes → 1×1 conv expands back. Reduces computation by ~70%.

5. Inception (GoogLeNet)

Inception Module:

                Input
                      |
        ┌─────────┬───┴────┬────────┐
        |         |          |
     1×1 conv  1×1→3×3  1×1→5×5   3×3 pool→1×1
        |         |          |
        └─────────┴──────────┘
                      |
              Concatenate (depth)

Why 1×1 before larger convs? Dimensionality reduction:

• Without: 256 channels → 5×5 conv with 256 filters = $256 \times 256 \times 25 = 1.6M$ params
• With 1×1: 256→64→256 = $256 \times 64 \times 1+ 64 \times 256 \times 25 = 426K$ params (74% reduction)

Concatenate: 28×28×(64+128+64+64) = 28×28×320

Why this step? Each branch captures different receptive fields. The network learns to weight branches based on the task.

Design Principles (Deriving Best Practices)

Principle 1: Receptive Field Growth

For object recognition, final layer needs to see the entire object. Two strategies:

1. Pooling: Halve spatial dimensions → doubles effective RF $RF_{new} = 2 \cdot RF_{old} + (k-1)$ for pooling with size $k$.

2. Stride: Skip pixels in convolution

   • Stride $s$ multiplies RF by $s$
   • But loses information (use sparingly)

With 3×3 convs between pools, need ~15 total conv layers to reach 100×100.

Principle 2: Channel Progression

Standard pattern: Double channels when halving spatial size.

Why? Maintain computational balance: $\text{FLOPs} \propto H \times W \times C_{in} \times C_{out} \times k^2$

If $H,W$ halve and $C$ doubles: FLOPs stay constant.

where $C_0$ is initial channels (typically 64).

Principle 3: Spatial Invariance

Problem: Images have translation/scale/rotation variance. Objects can appear anywhere.

Solutions:

1. Data augmentation: Train on crops, flips, rotations
2. Pooling: Max-pooling provides local translation invariance
3. Global average pooling: Replace FC layers → full translation invariance

Example: Instead of 7×7×512 → Flatten → FC(1000):

7×7×512 → AvgPool(7×7) → 1×1×512 → Conv(1×1, 1000 filters) → 1×1×1000 → Softmax

This has no spatial position encoding → works at any input size!

Principle 4: Computational Efficiency

$\text{FLOPs} = H' \times W' \times C_{in} \times C_{out} \times k^2$

Most expensive layers: Early ones (large $H,W$) and late ones (large $C$).

Optimization strategies:

1. Depthwise separable convolutions (MobileNet):

   • Depthwise: Each input channel gets its own $k \times k$ filter → $H' \times W' \times C \times k^2$
   • Pointwise: 1×1 conv to mix channels → $H' \times W' \times C \times C$
   • Total: $H' \times W' \times C \times (k^2 + C')$ vs standard $H' \times W' \times C \times C' \times k^2$
   • Speedup: $\frac{k^2 \times C'}{k^2 + C'} \approx k^2$ (e.g., 9× for 3×3)

2. Bottleneck layers: 1×1 reduce → 3×3 process → 1×1 expand

Common Mistakes & Fixes

Bad (ResNet-50 copy):

• 7×7 conv, stride 2 → 32×32
• MaxPool → 16×16
• 4 stages of residual blocks → 1×1 (way too aggressive!)
• Over-parameterized for 2 classes

Good (adapted):

• 3×3 conv, stride 1 → 64×64 (preserve resolution)
• [3×3 conv × 2 + MaxPool] × 3 → 8×8
• AvgPool → 1×1×256
• FC(2) with softmax
• 10× fewer parameters, trains faster, similar accuracy

Modern Architecture Patterns

Neural Architecture Search (NAS)

Idea: Let algorithms design architectures.

Search space: Define building blocks (conv, pool, skip) and constraints (max depth, FLOP budget).

Search methods:

1. Reinforcement learning: Controller RNN proposes architectures, trained on validation accuracy
2. Evolution: Mutate/crossover architectures, select fittest
3. Differentiable NAS: Make architecture choices continuous, use gradient descent

Result: EfficientNet (discovered by NAS) achieves ResNet-50 accuracy with 10× fewer parameters.

EfficientNet Scaling

Why? Higher resolution images need deeper/wider nets to extract features.

Vision Transformers (2020s)

Breaking from convolution entirely:

• Divide image into 16×16 patches
• Treat as sequence (like words in NLP)
• Use transformer self-attention

Tradeoff: Needs huge datasets (ImageNet-21k+) to match CNN performance, but scales better.

Design Checklist

When designing a CNN:

1. Input size → Determines pooling schedule (halve to ~7×7)
2. Task complexity → Determines depth (simple: 5-10 layers, complex: 50-100)
3. Compute budget → Use bottlenecks/depthwise convs if limited
4. Data availability → More data → can go deeper without overfitting
5. Interpretability needs → Simpler architectures (VGG-style) easier to visualize

Connections

• 3.401-convolution-operation — Building block of all CNN architectures
• 3.4.02-pooling-layers — Downsampling strategy affects architecture depth
• 3.4.03-activation-functions — ReLU enabled deeper networks (AlexNet innovation)
• 3.4.04-batch-normalization — Stabilizes training in very deep nets (post-2015 standard)
• 3.5.01-transfer-learning — Pre-trained architectures as feature extractors
• 4.2.03-vanishing-gradient — Core problem that ResNet solves
• 5.1.02-hyperparameter-tuning — Architecture choices are hyperparameters
• 6.3.01-model-compression — Making architectures efficient for deployment

#flashcards/ai-ml

What is the receptive field of a CNN layer? :: The spatial region of the input image that influences a particular output neuron. Grows with depth via convolutions and pooling.

What is depthwise separable convolution? :: Factorizes standard convolution into depthwise (one filter per input channel) and pointwise (1×1 to mix channels). Reduces computation by ~k² for k×k kernels.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)