VGG networks: Exercises — AI-ML deep dive

This page is a self-test ladder. Each rung is harder than the last. Cover the answer, try the problem, then open the solution. Every symbol used here is built in the parent VGG networks note — if a formula feels unfamiliar, re-read that note first.

Three tools we lean on repeatedly (all from the parent):

Here $k_{h}, k_{w}$ are the kernel's height and width in pixels; $C_{in}, C_{out}$ are the number of input and output channels (feature maps); $⌊ \cdot ⌋$ means "round down to the nearest whole number" (you cannot have half a pixel of output).

Related notes you may want open: 3.4.01-convolution-operation, 3.4.05-alexnet, 3.6.03-receptive-field, 3.5.02-batch-normalization, 3.4.08-resnet, 3.4.15-transfer-learning.

Level 1 — Recognition

Exercise 1.1

State VGG's five design rules in one line each (no derivations).

Recall Solution

Only 3×3 convolutions everywhere.
Same padding so spatial size is preserved until a pooling layer.
Max-pool 2×2, stride 2 to halve height and width.
Double the channel count after each pooling stage.
Deep uniform stacks (VGG-16 = 16 weight layers, VGG-19 = 19).

Exercise 1.2

The canonical VGG-16 uses which kernel sizes for its convolution layers? Pick all that apply: 1×1, 3×3, 5×5, 7×7, 11×11.

Recall Solution

Only 3×3. The famous VGG-16/19 use no 1×1, 5×5, 7×7, or 11×11 conv layers. (The paper's experimental "configuration C" added 1×1 convs, but that is not the canonical VGG-16.)

Exercise 1.3

VGG-16 has ~138M parameters. In which part of the network do most of them live — the convolution stack or the fully-connected classifier?

Recall Solution

The fully-connected (FC) classifier. The first FC layer alone maps $7 \times 7 \times 512 = 25, 088$ flattened inputs to 4096 outputs, giving $(25, 088 + 1) \times 4096 \approx 102.8$ M parameters — already about three-quarters of the whole network.

Level 2 — Application

Exercise 2.1

Compute the parameter count (weights and biases) of a Conv3-128 layer whose input has 64 channels.

Recall Solution

Use $P = (k_{h} k_{w} C_{in} + 1) C_{out}$ with $k_{h} = k_{w} = 3$ , $C_{in} = 64$ , $C_{out} = 128$ . $P = (3 \times 3 \times 64 + 1) \times 128 = (576 + 1) \times 128 = 577 \times 128 = 73, 856.$ Why this formula: each of the 128 output filters carries a full $3 \times 3 \times 64$ weight tensor (one weight per input pixel per input channel) plus a single bias — hence the " $+ 1$ ".

Exercise 2.2

An input feature map is $56 \times 56$ . Apply a 3×3 conv with same-padding $p = 1$ , stride $s = 1$ . What is the output height?

Recall Solution

$H_{out} = ⌊ \frac{56 + 2 ( 1 ) - 3}{1} ⌋ + 1 = ⌊ \frac{55}{1} ⌋ + 1 = 55 + 1 = 56.$ Same padding with $p = ⌊ k /2 ⌋ = 1$ keeps the spatial size unchanged — the whole point of "same" padding. Only pooling shrinks the map.

Exercise 2.3

Track the spatial size (height only) of a 224×224 input through Block 1 of VGG-16: two Conv3 layers (same padding), then MaxPool2 (2×2, stride 2, no padding).

Recall Solution

Conv 1: $⌊(224 + 2 - 3) /1 ⌋ + 1 = 224$ . Conv 2: $⌊(224 + 2 - 3) /1 ⌋ + 1 = 224$ . MaxPool2: here $k = 2, s = 2, p = 0$ : $⌊(224 + 0 - 2) /2 ⌋ + 1 = ⌊ 222/2 ⌋ + 1 = 111 + 1 = 112$ . So $224 \to 224 \to 224 \to 112$ . The convs preserve, the pool halves. See the figure below.

Level 3 — Analysis

Exercise 3.1

Prove that two stacked 3×3 convolutions (stride 1) have the same receptive field as one 5×5, using the RF recurrence.

Recall Solution

Start with $RF_{1} = k_{1} = 3$ (the first layer sees a 3×3 patch of the input). Apply the recurrence for layer 2, where $k_{2} = 3$ and the only earlier stride is $s_{1} = 1$ : $RF_{2} = RF_{1} + (k_{2} - 1) \prod_{i = 1}^{1} s_{i} = 3 + (3 - 1) \times 1 = 3 + 2 = 5.$ So two 3×3 layers reach a 5×5 window of the original image — identical to one 5×5 conv. The figure shows how the second layer's central output pixel traces back to a 5×5 input region.

Exercise 3.2

Compare parameters (weights only, $C$ channels in and out throughout): one 5×5 conv versus two 3×3 convs. Report the percentage saved.

Recall Solution

One 5×5: $C \times (5 \times 5 \times C) = 25 C^{2}$ . Two 3×3: $C \times (3 \times 3 \times C) + C \times (3 \times 3 \times C) = 9 C^{2} + 9 C^{2} = 18 C^{2}$ . Saving: $\frac{25 - 18}{25} = \frac{7}{25} = 0.28 = 28%$ fewer weights. Bonus: two 3×3 layers pass through two ReLU non-linearities instead of one — more expressive decision boundaries for less cost.

Exercise 3.3

Now extend to three 3×3 convs (stride 1). Find the receptive field, and the percentage of weights saved versus one 7×7 conv ( $C$ → $C$ ).

Recall Solution

Receptive field: build up layer by layer. $RF_{1} = 3$ ; $RF_{2} = 3 + 2 \times 1 = 5$ ; $RF_{3} = 5 + 2 \times 1 = 7$ . So three 3×3 = a 7×7 window. Weights: three 3×3 give $3 \times 9 C^{2} = 27 C^{2}$ ; one 7×7 gives $49 C^{2}$ . Saving: $\frac{49 - 27}{49} = \frac{22}{49} \approx 0.449 = 44.9%$ (the parent note rounds this to "45%"). Plus three ReLUs instead of one.

Level 4 — Synthesis

Exercise 4.1

Design a "mini-VGG" Block 3 that turns a 56×56×128 map into 28×28×256, and compute its total parameters (weights + biases). Use three Conv3-256 layers (same padding) followed by MaxPool2.

Recall Solution

The three conv layers each output 256 channels; only the first has 128 input channels, the next two have 256 input channels.

Conv3-256 (128→256): $(3 \times 3 \times 128 + 1) \times 256 = (1152 + 1) \times 256 = 1153 \times 256 = 295, 168$ .
Conv3-256 (256→256): $(3 \times 3 \times 256 + 1) \times 256 = (2304 + 1) \times 256 = 2305 \times 256 = 590, 080$ .
Conv3-256 (256→256): another $590, 080$ .
MaxPool2: 0 parameters (it just picks the max — no learnable weights). Total $= 295, 168 + 590, 080 + 590, 080 = 1, 475, 328$ parameters. Spatial size: $56 \to 56 \to 56 \to 56$ (convs preserve) then $\to 28$ (pool halves).

Exercise 4.2

Compute the activation memory (float32) of the output of a single Conv3-256 layer on a 56×56 map, for one image and for a batch of 32.

Recall Solution

One activation tensor is $56 \times 56 \times 256$ floats. Each float32 = 4 bytes. $56 \times 56 \times 256 \times 4 = 3, 211, 264 bytes \approx 3.06 MB (using 1 MB = 102 4^{2} bytes) .$ In decimal MB ( $1 0^{6}$ bytes) this is $\approx 3.21$ MB — the value the parent quotes. For a batch of 32: $3, 211, 264 \times 32 = 102, 760, 448$ bytes $\approx 98.0$ MB (binary) or $\approx 102.8$ MB (decimal). Roughly ~100 MB for one layer's activations alone — this is why deep nets are memory-hungry during training (you must store activations for backprop).

Level 5 — Mastery

Exercise 5.1

A team replaces VGG's Block 5 (three Conv3-512 layers, all 512→512) with a single Conv-512 whose receptive field equals the three stacked 3×3s. (a) What kernel size must the single conv have? (b) Compute weights-only parameters for both designs and the percentage the single-conv design adds. (c) Give one non-parameter reason the stacked design is still preferable.

Recall Solution

(a) Three stride-1 3×3 convs reach $RF = 7$ (Exercise 3.3), so the single conv needs a 7×7 kernel. (b) With $C = 512$ :

Stacked (weights only): $3 \times (3 \times 3 \times 512 \times 512) = 3 \times 2, 359, 296 = 7, 077, 888$ .
Single 7×7: $7 \times 7 \times 512 \times 512 = 49 \times 262, 144 = 12, 845, 056$ .
The single conv uses $\frac{12 , 845 , 056 - 7 , 077 , 888}{7 , 077 , 888} = \frac{5 , 767 , 168}{7 , 077 , 888} \approx 0.8148 = 81.5%$ more weights.
(Equivalently, the stacked design saves $\frac{49 - 27}{49} \approx 44.9%$ versus the single 7×7.) (c) The stacked design applies three ReLU non-linearities instead of one, giving a more expressive, more non-linear function for the same-sized receptive field — the extra decision boundaries are exactly VGG's core argument for going deep with small filters.

Exercise 5.2

The full receptive field of a VGG feature-map pixel (not the raw kernel math) is boosted by pooling too. Using the general recurrence, compute the receptive field after: Conv3 → Conv3 → MaxPool2(s=2) → Conv3. Treat max-pool like a $k = 2$ layer with stride 2, and note the strides before each layer.

Recall Solution

Layers in order (kernel $k$ , stride $s$ ): L1 (3,1), L2 (3,1), L3 pool (2,2), L4 (3,1). Track the running product of strides of layers before the current one.

$RF_{1} = 3$ . Strides so far: ${1}$ .
$RF_{2} = 3 + (3 - 1) \times \prod {s_{1}} = 3 + 2 \times 1 = 5$ . Strides: ${1, 1}$ .
$RF_{3} = 5 + (2 - 1) \times \prod {s_{1} s_{2}} = 5 + 1 \times (1 \cdot 1) = 6$ . Strides now include $s_{3} = 2$ : ${1, 1, 2}$ .
$RF_{4} = 6 + (3 - 1) \times \prod {s_{1} s_{2} s_{3}} = 6 + 2 \times (1 \cdot 1 \cdot 2) = 6 + 4 = 10$ . So one pixel after this stack sees a 10×10 region of the input. The pool's stride of 2 magnifies the reach of every later layer — that is why deep VGG pixels "see" almost the whole image.

Recall One-line self-check summary

Conv weights ::: $(k_{h} k_{w} C_{in} + 1) C_{out}$ , bias per output channel Same-pad 3×3 output size ::: unchanged (only pooling halves it) Two/three 3×3 receptive field ::: 5×5 / 7×7 Weights saved: two 3×3 vs 5×5 ::: 28% Weights saved: three 3×3 vs 7×7 ::: ~45% Where 138M params live ::: mostly in the first FC layer (~103M)