Convolutional Neural Networks

Question 1

(a) Formula (per spatial dim), effective kernel $k_{\text{eff}}=d(k-1)+1$: $O=\left\lfloor \frac{I + 2P - k_{\text{eff}}}{s}\right\rfloor + 1.$ (1 mark for correct formula incl. dilation.)

• L1: $k=7,P=3,s=2,d=1\Rightarrow k_{\text{eff}}=7$: $\lfloor(224+6-7)/2\rfloor+1=\lfloor223/2\rfloor+1=111+1=112$. → 112×112×64. (1.5)
• L2: $k=3,P=1,s=2$: $\lfloor(112+2-3)/2\rfloor+1=\lfloor111/2\rfloor+1=55+1=56$. → 56×56×64. (1.5)
• L3: $k=3,d=2\Rightarrow k_{\text{eff}}=2(2)+1=5$; $P=2,s=1$: $\lfloor(56+4-5)/1\rfloor+1=55+1=56$. → 56×56×128. (1)
• L4: $k=3,P=1,s=1,d=1\Rightarrow k_{\text{eff}}=3$: $\lfloor(56+2-3)/1\rfloor+1=55+1=56$. → 56×56×128. (1)

(b) Derivation (recurrence unrolling, 4 marks). Initialize $r_0=1$, $j_0=1$. A layer with effective kernel $k_{\text{eff}}$ and stride $s$: each output unit sees $k_{\text{eff}}$ consecutive units of the previous map, and adjacent previous units are spaced by the previous jump $j_{\ell-1}$ in input pixels. Extra span added $=(k_{\text{eff}}-1)j_{\ell-1}$, giving $r_\ell=r_{\ell-1}+(k_{\text{eff},\ell}-1)j_{\ell-1}, \qquad j_\ell=j_{\ell-1}s_\ell.$

Computation (3 marks). Jumps: $j_0=1$; $j_1=1\cdot2=2$; $j_2=2\cdot2=4$; $j_3=4\cdot1=4$; $j_4=4\cdot1=4$. RF ($k_{\text{eff}}$: L1=7, L2=3, L3=5, L4=3):

• $r_1=1+(7-1)\cdot1=7$
• $r_2=7+(3-1)\cdot2=7+4=11$
• $r_3=11+(5-1)\cdot4=11+16=27$
• $r_4=27+(3-1)\cdot4=27+8=\mathbf{35}$ input pixels. (final 1 mark)

(c) L3 output grid $56\times56=3136$ positions; out ch $=128$; kernel elems $=3\times3=9$; in ch $=64$. $\text{MACs}=3136\times128\times9\times64 = 231{,}211{,}008 \approx 2.31\times10^8.$ (2 setup, 2 result)

(d) Two stacked non-dilated $3\times3$ convs have $k_{\text{eff}}=3$ each → RF grows $+2+2=4$ (contiguously sampled), vs one dilated conv $k_{\text{eff}}=5$ → RF grows $+4$ but with gaps (sparse/gridding sampling). Parameters: two $3\times3$ layers cost $2\times(9C_{in}C_{out})$ ≈ double one dilated layer, but give denser coverage and extra nonlinearity. (3: RF density + param comment)

Question 2

(a) (5 marks derivation, 3 explanation.) By induction on the recurrence $\mathbf{x}_{i+1}=\mathbf{x}_i+\mathcal{F}_i(\mathbf{x}_i)$: summing telescopically from $i=\ell$ to $L-1$, $\mathbf{x}_L=\mathbf{x}_\ell+\sum_{i=\ell}^{L-1}\mathcal{F}_i(\mathbf{x}_i).$ Differentiate via chain rule: $\frac{\partial\mathcal{E}}{\partial\mathbf{x}_\ell}=\frac{\partial\mathcal{E}}{\partial\mathbf{x}_L}\frac{\partial\mathbf{x}_L}{\partial\mathbf{x}_\ell}=\frac{\partial\mathcal{E}}{\partial\mathbf{x}_L}\Big(1+\tfrac{\partial}{\partial\mathbf{x}_\ell}\textstyle\sum_i\mathcal{F}_i\Big).$ Why: the constant "1" provides an identity (shortcut) path so $\partial\mathcal{E}/\partial\mathbf{x}_L$ propagates undiminished to any earlier layer even if the $\mathcal{F}$-Jacobian product is near zero; the gradient cannot fully vanish because the bracket is $1+(\text{something})$, never a product of many small factors.

(b) (6 marks.) Bottleneck weights (kernel²·Cin·Cout):

• $1\times1$: $1\cdot C\cdot(C/4)=C^2/4$
• $3\times3$: $9\cdot(C/4)\cdot(C/4)=9C^2/16$
• $1\times1$: $1\cdot(C/4)\cdot C=C^2/4$

Total $=\dfrac{C^2}{4}+\dfrac{9C^2}{16}+\dfrac{C^2}{4}=\dfrac{4C^2+9C^2+4C^2}{16}=\dfrac{17C^2}{16}$.

Basic block (two $3\times3$, C→C): $2\times9C^2=18C^2$. For $C=256$: bottleneck $=17\cdot65536/16=69{,}632\cdot... =17\times4096=\mathbf{69{,}632}$; basic $=18\times65536=\mathbf{1{,}179{,}648}$. Bottleneck ≈ 17× cheaper. (3 derivation, 3 numbers)

(c) (8 marks: correct convs 3, BN/ReLU order 2, shortcut logic 2, final add+ReLU 1.)

import torch.nn as nn
 
class Bottleneck(nn.Module):
    expansion = 4
    def __init__(self, in_ch, mid_ch, stride=1):
        super().__init__()
        out_ch = mid_ch * self.expansion
        self.conv1 = nn.Conv2d(in_ch, mid_ch, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(mid_ch)
        self.conv2 = nn.Conv2d(mid_ch, mid_ch, 3, stride=stride,
                               padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(mid_ch)
        self.conv3 = nn.Conv2d(mid_ch, out_ch, 1, bias=False)
        self.bn3   = nn.BatchNorm2d(out_ch)
        self.relu  = nn.ReLU(inplace=True)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_ch != out_ch:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_ch))
    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x)      # add BEFORE final activation
        return self.relu(out)

Question 3

(a) (5 marks.) Intersection corners: $x=[\max(10,30),\min(60,90)]=[30,60]$, width $=30$; $y=[\max(20,40),\min(80,100)]=[40,80]$, height $=40$. Intersection $=30\times40=1200$. Areas: $A=(60-10)(80-20)=50\times60=3000$; $B=(90-30)(100-40)=60\times60=3600$. Union $=3000+3600-1200=5400$. $\text{IoU}=\frac{1200}{5400}=\frac{2}{9}\approx\mathbf{0.222}.$

(b) (6 marks, 2 per axis + choice.) (i) YOLO: no separate proposal stage—dense grid cells directly regress boxes+class. R-CNN: a Region Proposal Network / selective search generates candidate ROIs, then a classifier refines them. (ii) YOLO is faster (single forward pass, real-time) but historically lower localization accuracy on small/overlapping objects; two-stage R-CNN is slower but more accurate. (iii) YOLO uses multiple anchors/bounding-box predictors per cell; R-CNN handles many objects via many proposals + per-ROI classification (NMS to dedup). Choice: YOLO (or SSD) for the 30 FPS drone — single-shot meets the real-time/compute budget of embedded hardware.

(c) (4 marks.) Skip connections copy high-resolution feature maps from the contracting path and concatenate them onto the upsampled decoder features, restoring fine spatial detail lost during pooling and giving the decoder both semantic (deep) and localization (shallow) information. Concatenation preserves both feature sets as separate channels for the next conv to combine, whereas ResNet addition fuses them into one tensor (requires matching channel count, learns a residual). Elastic deformation matters because biomedical datasets are small and cells/tissue vary in shape; realistic deformations teach invariance and expand effective training data.

(d) (3 marks.) $\text{Dice}=\dfrac{2|P\cap G|}{|P|+|G|}$ (soft: $\frac{2\sum p_i g_i}{\sum p_i+\sum g_i}$). Preferred because it directly measures foreground overlap and is insensitive to the vast correct-background majority, so it does not get inflated by class imbalance the way pixel accuracy does.

[
 {"claim":"L4 output spatial size is 56","code":"def O(I,P,k,s,d):\n keff=d*(k-1)+1\n return (I+2*P-keff)//s+1\nL1=O(224,3,7,2,1); L2=O(L1,1,3,2,1); L3=O(L2,2,3,1,2); L4=O(L3,1,3,1,1)\nresult = (L1==112 and L