Exercises — CNN architecture design

The four knobs (straight from the parent note):

1. Depth — number of layers stacked; controls the feature hierarchy (edges → parts → objects).
2. Width — number of channels $C$ per layer; controls capacity (how many parallel detectors).
3. Connectivity — how layers wire together (plain sequential, skip/residual, or DenseNet-style dense skip connections). Careful: here "dense" means dense skip connections (every layer feeds every later layer, as in DenseNet), not fully-connected "dense layers." This knob solves training problems like vanishing gradients.
4. Receptive field — how much of the input one deep neuron can "see"; controls how much context is available.

• (a) → AlexNet (2012)
• (b) → VGGNet (2014)
• (c) → ResNet (2015)
• (d) → Inception / GoogLeNet (2014)

What "receptive field" (RF) means, from zero. The receptive field of a top neuron is the side length of the patch of input pixels that can possibly influence it. We now derive the growth formula on this page rather than borrowing it.

Why the padding/stride matter here. With stride 1 the window moves one pixel per step, so consecutive receptive fields overlap by the maximum amount — this is what makes the growth a clean $+(k-1)$ per layer. Same padding just keeps every map the same $H\times W$ so we can count reach without the map shrinking; it does not change the RF arithmetic, it only keeps the bookkeeping tidy. (With valid padding the maps would shrink by $(k-1)$ each layer, but each neuron's reach on the input is still $2n+1$ — padding affects map size, not receptive field.)

The picture — walk it row by row. The figure below shows four stacked rows of grid cells drawn like a chalk sketch: the bottom row is the input, and each higher row is the output of one $3\times 3$ conv. A single yellow cell at the very top is one top neuron; blue dashed lines fan down from it forming a "cone of vision," and coloured bands (blue, then pink, then yellow) mark how wide that cone gets at each lower row. Trace downward:

• The yellow top neuron reads a $3\times 3$ window (blue band) in the row just below — width 3.
• Each of those blue neurons itself reads a $3\times 3$ window one row lower (pink band). The leftmost blue neuron reaches 1 further left, the rightmost reaches 1 further right, so the pink span is $3+2=5$ wide.
• Down one more row (yellow band on the input), the span widens again by $+2$ to width 7.

The rule you can read straight off the picture: every extra $3\times 3$ layer adds exactly $(k-1)=2$ to the width — 1 pixel of new reach on each side, because a $3\times 3$ window centered on a pixel extends 1 left and 1 right. The centre pixel is shared with the layer below, so it is not recounted. That is why it is $(k-1)$ per layer, not $k$.

Assemble the formula. Start from a single pixel (width 1), add $(k-1)$ for each of the $n$ layers: $RF=1+n\cdot (k-1)=1+n\cdot (3-1)=2n+1.$ For $n=4$: $RF=2(4)+1=\boxed{9}$, i.e. a $9\times 9$ region. (This same $RF=2n+1$ is quoted in the parent note; above is its self-contained derivation.)

Where the formula comes from: each output channel needs a full $k\times k$ stack across all $C$ input channels, and there are $C$ output channels. So weights $=k^2\cdot C_{\text{in}}\cdot C_{\text{out}}$. Here $C_{\text{in}}=C_{\text{out}}=C$.

• One $5\times 5$: $25C^2$.
• Two $3\times 3$: $(9+9)C^2=18C^2$.

Saving: $\frac{25-18}{25}=\frac{7}{25}=0.28=28\%.$ Bonus insight: the two $3\times 3$ stack also inserts two ReLU nonlinearities instead of one — more expressive and cheaper, which is the whole VGG thesis.

Use weights $=k^2\cdot C_{\text{in}}\cdot C_{\text{out}}$.

• (a) Direct $5\times 5$: $25\cdot 256\cdot 256=1,638,400\approx 1.6\text{M}$.
• (b) $1\times 1$ reduce: $1\cdot 256\cdot 64=16,384$. Then $5\times 5$: $25\cdot 64\cdot 256=409,600$. Total: $16,384+409,600=425,984\approx 426\text{K}$.

Saving: $1-\frac{425,984}{1,638,400}=1-0.26=0.74=74\%.$ Why the $1\times 1$ helps: a $1\times 1$ conv mixes channels without touching spatial patterns — cheap. It shrinks the channel dimension so the expensive $5\times 5$ works on a thinner stack.

Setup. $\mathbf{x}$ is the block input; $\mathbf{F}(\mathbf{x})$ is what the conv layers compute (the "residual"). The output is their elementwise sum, so $\mathbf{H}$, $\mathbf{F}$ and $\mathbf{x}$ are all tensors of the same shape (say $H\times W\times C$ flattened to a vector of length $d$).

What we differentiate and why. During backprop we need how the output changes when the input changes, because that factor multiplies the gradient as it travels backward (chain rule). For vector-valued $\mathbf{H}$ and $\mathbf{x}$ this "factor" is not a single number but a Jacobian matrix $\frac{\partial \mathbf{H}}{\partial \mathbf{x}}$, a $d\times d$ table whose entry $(i,j)$ is $\partial H_i/\partial x_j$. $\frac{\partial \mathbf{H}}{\partial \mathbf{x}}=\frac{\partial \mathbf{F}}{\partial \mathbf{x}}+\frac{\partial \mathbf{x}}{\partial \mathbf{x}}=J_F+I.$ Here $\frac{\partial \mathbf{x}}{\partial \mathbf{x}}=I$ is the identity matrix (1's on the diagonal, 0's elsewhere) — the Jacobian of "copy the input unchanged." It is the matrix version of the scalar "$\partial x/\partial x=1$"; picturing it as a plain "$+1$" is a convenient shorthand, but the true object is the identity matrix $I$.

Plain block (no skip): output is just $\mathbf{F}(\mathbf{x})$, so the local Jacobian is $J_F$ alone. Stack $n$ of these and the backprop factor is a product of matrices $J_{F_n}\cdots J_{F_1}$. If these matrices shrink vectors (all singular values $<1$), the product decays exponentially → vanishing gradient (see 4.2.03-vanishing-gradient).

Residual block: the Jacobian is $J_F+I$. Even if $J_F\to 0$ (the layers learn nothing useful), the block still passes the gradient through as $I\cdot (\text{incoming gradient})=$ the gradient unchanged. The identity term guarantees a "highway" straight back for every channel and every spatial position at once, which is why the trick works for full multi-channel tensors, not just a scalar toy.

Convs alone. Using the $RF=2n+1$ derived in L2.1 (self-contained there), require $2n+1\ge 100$: $2n+1\ge 100\Rightarrow n\ge 49.5\Rightarrow n=50\text{ layers}.$ That's a lot of layers just to widen vision.

Why pooling wins — deriving $R{F}_{\text{new}}=2\cdot R{F}_{\text{old}}+(k-1)$ from scratch. First, a reminder of stride: the stride is how many pixels the window jumps between placements. A $2\times 2$ pool (stride 2) replaces every $2\times 2$ block of pixels with one summary pixel, so one step on the pooled map corresponds to two pixels of movement on the map below — that "jump of 2" is exactly the stride. Now put a $k\times k$ conv on top of the pooled map and ask how wide its input footprint is, measured back on the pre-pool map:

• The conv reads $k$ neighbouring pooled pixels. Neighbouring pooled pixels are $2$ apart on the map below (that's the stride-2 "multiplier"), so their centres span $2\cdot (k-1)$ pixels; but each pooled pixel itself already summarised the old receptive field $R{F}_{\text{old}}$.
• Carrying that reach through: the new receptive field becomes $R{F}_{\text{new}}=2\cdot R{F}_{\text{old}}+(k-1)$. The factor 2 is the stride of the pool (each step downstairs jumps 2 pixels); the $+(k-1)$ is the usual kernel border reach from L2.1.

The punchline: pooling makes RF grow multiplicatively (a factor of 2 every pool) whereas plain convs only add $+2$ each. Multiplication beats addition, so interleaving a handful of pools reaches $100\times 100$ in far fewer layers than 50 — which is exactly why every classic net alternates conv stacks with pooling.

Inception concatenates along the channel (depth) axis. Spatial dims must match so the sheets stack cleanly. $28\times 28\times (64+128+64+64)=28\times 28\times 320.$ Why same spatial size: you cannot stack a $28\times 28$ sheet on a $14\times 14$ sheet — they don't align. Every branch therefore uses padding so each keeps $28\times 28$; only the channel counts add up. See 3.401-convolution-operation for how "same" padding preserves size.

Steps and shapes:

1. Global average pool over the $7\times 7$ plane: average each channel's 49 values → $1\times 1\times 512$.
2. $1\times 1$ conv with 1000 filters (mixes 512 channels into 1000): $1\times 1\times 1000$.
3. Softmax over the 1000 → probabilities.

Why this is nice: no flatten means no dependence on a fixed input size — the average pool collapses whatever spatial size remains, so the same weights accept different image sizes. It also drops the huge FC weight matrix ($7\cdot 7\cdot 512\cdot 1000\approx 25\text{M}$ params) to almost nothing, cutting overfitting (a form of 6.3.01-model-compression).

BN = BatchNorm = batch normalization — a layer that re-centres and re-scales its inputs to roughly zero-mean, unit-variance across the batch (full details in 3.4.04-batch-normalization).

Standard order: $\text{Conv}\to \text{BN}\to \text{ReLU}$.

Why BN before ReLU: BN re-centres and re-scales the conv output to roughly zero-mean, unit-variance before the nonlinearity. If ReLU came first, it would already have thrown away all negatives (clamped them to 0), so BN would be normalising a distribution that is already half-dead — losing information and biasing the mean upward. Normalising first means ReLU sees a well-conditioned, centred input, so a healthy fraction of activations stay positive (alive), and gradients flow through them; this stabilises and speeds up training. (Later variants like "pre-activation ResNet" reorder to $\text{BN}\to \text{ReLU}\to \text{Conv}$, but the classic ResNet block is $\text{Conv}\to \text{BN}\to \text{ReLU}$.)

Apply $k^2\cdot C_{\text{in}}\cdot C_{\text{out}}$ layer by layer ($k=3$, so $k^2=9$):

Total: $1,728+36,864+0+73,728+147,456=\boxed{259,776}$ weights.

Insight: pooling has zero parameters (it just picks maxima), yet the cost per layer keeps rising because channels grow. This is exactly why the channel-doubling rule (L5.2 below) is a budget balancer, not a freebie.

FLOPs model (from parent): $\text{FLOPs}\propto H\cdot W\cdot C_{\text{in}}\cdot C_{\text{out}}\cdot k^2$ — every output position runs one $k^2\times C_{\text{in}}$ multiply-add per output channel.

Before pool: $\text{FLOPs}\propto H\cdot W\cdot C\cdot C\cdot k^2=HW{C}^2{k}^2$.

After pool + double: $H\to H/2$, $W\to W/2$, $C\to 2C$: $\frac{H}{2}\cdot \frac{W}{2}\cdot (2C)\cdot (2C)\cdot k^2=\frac{1}{4}\cdot 4\cdot HW{C}^2{k}^2=HW{C}^2{k}^2.$ The $\frac{1}{4}$ from spatial shrink and the $4$ from channel growth cancel — FLOPs are constant. That's why "double the channels when you halve the resolution" is the standard rule: it keeps each stage's compute balanced.

Decision A — Transfer learning (3.5.01-transfer-learning): with only 2000 images, training a deep net from scratch would overfit badly. Start from a network pretrained on ImageNet (edges/textures/parts are already learned) and fine-tune the top layers. This addresses the data scarcity.

Decision B — Model compression / efficient blocks (6.3.01-model-compression): the phone constraint means you want a small backbone — favour bottleneck ($1\times 1$) blocks (L2.3, 74% fewer params) and a global-average-pool head (L4.2, kills the huge FC). This addresses the memory constraint.

Why this combo: transfer learning fixes the statistics problem (too little data), compression fixes the hardware problem (too little memory). They are orthogonal — one does not solve the other, so you need both. Tune the fine-tuning learning rate and number of unfrozen layers via 5.1.02-hyperparameter-tuning.

Diagnosis — vanishing gradients, not overfitting. The key clue is that training error (not just test error) is higher. Overfitting would make test error worse while training error kept dropping. Higher training error means the deep net cannot even be optimised. With sigmoids, each layer's derivative is $<1$; multiplying 40 of them (chain rule) shrinks the gradient toward the early layers to near zero, so they barely learn (see L3.1 and 4.2.03-vanishing-gradient).

Fix 1 — Residual (skip) connections: the $J_F+I$ Jacobian (L3.1) gives gradients a highway back to early layers.

Fix 2 — BatchNorm (BN) + ReLU (3.4.04-batch-normalization, 3.4.03-activation-functions): BN keeps activation distributions well-scaled layer to layer, and ReLU's derivative is exactly 1 for positive inputs (no shrink), so gradients survive. Together these are precisely why ResNet-50 trains fine at 50 layers while a 40-layer plain sigmoid net degrades.

Weights of a conv layer ::: $k^2\cdot C_{\text{in}}\cdot C_{\text{out}}$ Receptive field of $n$ stacked $3\times 3$ convs ::: $2n+1$ The ResNet backprop Jacobian ::: $J_F+I$ (identity matrix, not a scalar 1) Rule when spatial size halves ::: double the channels (keeps FLOPs constant)