Noesis
3.4.7Convolutional Neural Networks

VGG networks

2,122 words10 min readdifficulty · medium
🇮🇳 Yeh note Hinglish mein padho →

What Problem Does VGG Solve?

Before VGG (2014), AlexNet used large filters (11×11, 5×5). Why is this bad?

  • More parameters → harder to train, more overfitting
  • Less non-linearity → fewer decision boundaries
  • Inefficient receptive fields → waste computation

VGG's insight: replace large filters with stacked small ones.

Core Architecture Principles

Why 3×3 Filters?

Let's derive the receptive field equivalence:

Claim: Two 3×3 convolutions = one 5×5 receptive field.

Derivation:

  1. First 3×3 conv: each output pixel sees a 3×3 input region
  2. Second 3×3 conv: each output pixel sees 3×3 of the previous layer
  3. Trace back: the center pixel of layer 2 sees:
    • Middle pixel of layer 1's 3×3 patch
    • That middle pixel itself saw a 3×3 patch in layer 0
    • Total area: extends 1 pixel beyond in all directions from the layer 1 patch

Formal calculation: RFlayer n=RFlayer n−1+(k−1)×∏i=0n−1si\text{RF}_{\text{layer } n} = \text{RF}_{\text{layer } n-1} + (k-1) \times \prod_{i=0}^{n-1} s_i

For two 3×3 layers with stride 1: RF1=3\text{RF}_1 = 3 RF2=3+(3−1)×1=5\text{RF}_2 = 3 + (3-1) \times 1 = 5

Why this is better:

Why Double Channels After Pooling?

Intuition: Spatial resolution↓ should mean feature richness ↑.

Derivation from information theory:

  • Before pooling: H×W×CH \times W \times C activations → HWCHWC information capacity
  • After 2×2 pooling: H2×W2×C\frac{H}{2} \times \frac{W}{2} \times C → capacity drops to HWC4\frac{HWC}{4}
  • To compensate, double channels: H2×W2×2C\frac{H}{2} \times \frac{W}{2} \times 2C → HWC2\frac{HWC}{2}

This maintains computational balance. Memory decreases, but representational capacity stays reasonable.

Figure — VGG networks

VGG-16 Architecture in Detail

Example Calculation: Layer 3 Parameters

Given: Input 56×56×128, output 56×56×256 (a 3×3 conv)

Conv3-256 parameters: P=(kernel_height×kernel_width×in_channels+1)×out_channelsP =(\text{kernel\_height} \times \text{kernel\_width} \times \text{in\_channels} +1) \times \text{out\_channels}

Why this step? Each of 256 output filters has a full 3×3×1283 \times 3 \times 128 weight tensor (kernel height × width × input channels), plus 1 bias.

P=(3×3×128+1)×256=1153×256=295,168P = (3 \times 3 \times 128 + 1) \times 256 = 1153 \times 256 = 295{,}168

Memory for activations: 56×56×256×4 bytes (float32)≈3.2 MB per image56 \times 56 \times 256 \times 4 \text{ bytes (float32)} \approx 3.2 \text{ MB per image}

Batch of 32 → ~102 MB just for this one layer's activations!

Common Mistakes

Key Formulas

Training Insights

Initialization: VGG-11 trained first, then used as initialization for VGG-16/19. Why? Deep networks hard to train from scratch in2014 (pre-BatchNorm). Shallow→deep bootstrapping stabilized training.

Data augmentation:

  • Random crops (224×224 from 256×256)
  • Horizontal flips
  • RGB color jittering

Why aggressive augmentation? 138M parameters on 1.2M ImageNet images → severe overfitting risk. Augmentation creates ~100× effective dataset size.

Learning rate schedule: Start at 0.01, divide by 10 when validation plateaus. Why not adaptive optimizers? SGD with momentum (0.9) was state-of-art in 2014. Modern Adam would converge faster but might generalize slightly worse.

Recall Explain VGG to a 12-year-old

Imagine you're painting a picture, but instead of using one thick brush, you use many thin brushes—each adds a tiny layer. That's VGG!

Old neural networks used big "brushes" (big filters like11×11). VGG said: "What if we use tiny3×3 brushes many times?"

The magic: Two tiny brushes see the same area as one medium brush, but you have more control! It's like coloring with many light strokes instead of one heavy one—you get smother blending and can fix mistakes easier.

VGG stacks 16-19 layers deep. Each layer looks at a tiny neighborhood, but stacked together, they see the whole image. Early layers spot edges and colors. Middle layers combine them into textures. Deep layers recognize objects like "dog's face" or "car wheel."

The trick: after every few layers, shrink the image (like zoming out) but add more "paint colors" (channels). So you trade detail for understanding—perfect for recognizing "what's in this picture?"

Connections

  • Convolution basics → VGG uses only 3×3
  • AlexNet → VGG's predecessor with large filters
  • ResNet → solved VGG's depth limitations with skip connections
  • Transfer learning → VGG features excel here
  • Batch Normalization → missing in VGG, added in later models
  • Receptive field analysis → why stacked 3×3 works

#flashcards/ai-ml

What are the two key advantages of using two 3×3 convolutions instead of one 5×5? :: (1) 28% fewer parameters (18C² vs 25C² weights), (2) Two ReLU non-linearities instead of one, increasing model expressiveness

Why does VGG double the number of channels after each pooling layer? :: To maintain computational balance—spatial dimensions halve (÷4 area), so doubling channels compensates by maintaining similar information capacity

What is the receptive field of three stacked 3×3 convolutions with stride 1?
7×7 (calculated as 3 + 2 + 2), equivalent to a single 7×7 convolution but with 45% fewer parameters and three ReLU activations
How many parameters does VGG-16 have, and where are most of them?
~138 million total, with most (>100M) in the fully connected layers, not the convolutional layers
What padding strategy does VGG use for its 3×3 convolutions and why?
Same-padding (padding=1 for 3×3 kernel) to preserve spatial dimensions until pooling layers, making the architecture more uniform
Does canonical VGG-16/19 use 1×1 convolutions?
No—canonical VGG-16/19 use ONLY 3×3 convolutions. (Only the experimental configuration C in the paper used 1×1 convs.)
Why did VGG researchers train VGG-11 before training VGG-16/19?
Pre-BatchNorm era made deep networks hard to train from scratch; they used the trained shallow network as initialization for deeper ones (bootstrap training)
What is the computational bottleneck in VGG-16 during training?
Memory for storing activations during backpropagation—a 224×224×3 image generates >100MB of activation maps per batch element

Concept Map

problems

motivates

replace with

deep uniform stacks

two 3x3 equal one 5x5

yields

yields

design rule

design rule

halves resolution

maintains

proves

AlexNet large filters

Too many params
less non-linearity

VGG core insight

Stacked 3x3 filters

VGG-16 and VGG-19

Receptive field equivalence

28 percent fewer params

More ReLU non-linearity

Same padding
preserves size

Max pool 2x2 stride 2

Double channels

Computational balance

Depth matters most

Hinglish (regional understanding)

Intuition Hinglish mein samjho

VGG networks ne ek simple lekin powerful idea introduce kiya: agar apko deep neural network banana hai, toh fancy architecture ki zarurat nahi—bas chote 3×3 filters

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 1,963 wordsD2Visual walkthroughThe derivation in pictures · 2,694 wordsD3Worked examplesEvery scenario covered · 2,128 wordsD4ExercisesGraded, full solutions · 2,129 wordsD5Question bankConcept traps, quick reveals · 2,655 words

Test yourself — Convolutional Neural Networks

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Convolution operation and filtersAI-ML · 3.4.1Batch normalizationAI-ML · 3.2.8