Convolutional Neural Networks

Question 1 (10 marks)

(a) With dilation, the effective kernel size is $K_{eff} = D(K-1)+1$. Sliding it across a padded input of length $H+2P$ with stride $S$: $H_{out} = \left\lfloor \frac{H + 2P - D(K-1) - 1}{S} \right\rfloor + 1$

• Effective kernel size (1)
• Padding term & numerator (1)
• Floor + 1 (1)

(b) $H_{out} = \lfloor (224 + 6 - 1\cdot6 - 1)/2 \rfloor + 1 = \lfloor 223/2 \rfloor + 1 = 111 + 1 = 112$. (2)

(c) Params $= (K\cdot K \cdot C_{in} + 1)\cdot F = (7\cdot7\cdot3 + 1)\cdot 64 = (147+1)\cdot64 = 148\cdot64 = 9472$. (2)

• Correct weight count 7·7·3·64 (1), bias +64 (1)

(d) Output is $112\times112\times64$. Each output element = $7\cdot7\cdot3 = 147$ MACs. $\text{MACs} = 112\cdot112\cdot64\cdot147 = 802816\cdot147 = 118{,}013{,}952 \approx 1.18\times10^8.$

• Output positions $112^2\cdot64$ (1), MACs per output 147 (1), product (1)

Question 2 (10 marks)

(a) Cumulative jump: $j_\ell = j_{\ell-1}\cdot s_\ell$, with $j_0=1$. Receptive field recurrence: $r_\ell = r_{\ell-1} + (k_\ell - 1)\cdot j_{\ell-1}$ with $r_0 = 1$. Reasoning: each new layer's kernel spans $k_\ell$ units, but each unit at the previous layer already covers $j_{\ell-1}$ input pixels, so the extra $(k_\ell-1)$ units add $(k_\ell-1)j_{\ell-1}$. (4)

(b) Stride 1 ⇒ $j=1$ throughout. $r_1 = 1 + 2\cdot1 = 3$; $r_2 = 3 + 2\cdot1 = 5$; $r_3 = 5 + 2\cdot1 = 7$. RF = 7. (2)

(c) Three $3\times3$ convs give the same $7\times7$ receptive field. Parameter count: $3\times(3^2 C^2) = 27C^2$ vs. $7^2 C^2 = 49C^2$ — a ~45% reduction. Additionally, the stack inserts two extra non-linearities, increasing representational (decision-boundary) capacity. (4: RF equivalence 1, param comparison 2, non-linearity point 1)

Question 3 (12 marks)

(a) (8 marks: padding 2, output alloc 2, loops/window 2, correctness 2)

import numpy as np
 
def conv2d(x, w, stride=1, pad=0):
    if pad > 0:
        x = np.pad(x, ((pad, pad), (pad, pad)), mode='constant')
    H, W = x.shape
    kH, kW = w.shape
    oH = (H - kH) // stride + 1
    oW = (W - kW) // stride + 1
    out = np.zeros((oH, oW))
    for i in range(oH):
        for j in range(oW):
            r, c = i * stride, j * stride
            region = x[r:r+kH, c:c+kW]
            out[i, j] = np.sum(region * w)
    return out

(b) Output shape $= \left(\frac{H+2P-kH}{S}+1,\ \frac{W+2P-kW}{S}+1\right)$. (2) Cross-correlation (no flip) is what deep-learning frameworks actually use; a true convolution flips the kernel both horizontally and vertically ($w[::-1,::-1]$). For a learned filter it makes no functional difference — the network just learns the mirrored weights — so omitting the flip is standard and correct in ML. (2)

Question 4 (12 marks)

(a) Block: conv3x3 → BN → ReLU → conv3x3 → BN, then add the input (identity), then final ReLU. Output: $y = \mathcal{F}(x, \{W_i\}) + x.$ (3)

(b) $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\cdot\frac{\partial y}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\left(\frac{\partial \mathcal{F}}{\partial x} + I\right)$. Out loud: because of the identity term $I$, the upstream gradient $\partial\mathcal{L}/\partial y$ flows directly to $x$ even if $\partial\mathcal{F}/\partial x \to 0$. The gradient never fully vanishes as depth grows — the "$+I$" guarantees a highway for gradient flow, enabling very deep networks (100+ layers) to train. (5: chain rule + I term 3, out-loud explanation 2)

(c) (1) Projection shortcut — a $1\times1$ conv with stride on the identity path to match channels/spatial dims (adds parameters). (2) Zero-padding the identity's extra channels (parameter-free but less expressive). Trade-off: projection costs $C_{in}\cdot C_{out}$ params but preserves representational capacity; padding is free but adds no learning. (4)

Question 5 (8 marks)

(a) The $1\times1$ conv is a channel-wise bottleneck: it reduces the depth cheaply before an expensive spatial conv, cutting computation/params.

Direct $5\times5$: params $= 5\cdot5\cdot192\cdot64 = 307{,}200$. Bottleneck: $1\times1$ (192→32): $1\cdot1\cdot192\cdot32 = 6{,}144$; then $5\times5$ (32→64): $5\cdot5\cdot32\cdot64 = 51{,}200$. Total $= 57{,}344$. Saving $= 307{,}200 - 57{,}344 = 249{,}856$ params (≈81% reduction). (6: direct 2, bottleneck two terms 2, saving 2)

(b) DenseNet: dense connectivity — each layer receives feature maps of all preceding layers (concatenation) for feature reuse. EfficientNet: compound scaling — jointly scale depth, width, and resolution by a fixed ratio. (2)

Question 6 (8 marks)

(a) With only 2,000 images: freeze the early/mid convolutional layers (generic edge/texture features transfer well) and fine-tune only the later layers + new classifier head. Use a small learning rate (e.g. $10^{-4}$) so pretrained weights aren't destroyed; optionally a higher LR on the new head. Small dataset ⇒ heavy freezing prevents overfitting. (4)

(b) Suitable: random horizontal flip, random crop/scale, color jitter, rotation (small), random erasing. (any 4 → 2 marks) Inappropriate for MNIST digits: horizontal/vertical flip or large rotation — because they change semantic identity (a flipped/rotated "6" becomes a "9" or invalid glyph), corrupting labels. (2)

[
  {"claim":"Q1b output height = 112","code":"K,P,S,D=7,3,2,1; H=224; result=(( (H+2*P-D*(K-1)-1)//S)+1)==112"},
  {"claim":"Q1c params = 9472","code":"result=((7*7*3+1)*64)==9472"},
  {"claim":"Q1d MACs = 118013952","code":"result=(112*112*64*147)==118013952"},
  {"claim":"Q2b receptive field of three 3x3 = 7","code":"r=1\nfor _ in range(3): r=r+(3-1)*1\nresult=r==7"},
  {"claim":"Q5 bottleneck saving = 249856","code":"direct=5*5*192*64; bott=1*1*192*32+5*5*32*64; result=(direct-bott)==249856"}
]