Exercises — Stride, padding, and dilation

Dilation $d=1$ means "insert $(d-1)=0$ gaps" — i.e. no gaps at all, an ordinary dense kernel. $k_{\text{eff}}=k+(k-1)(d-1)=3+(2)(0)=3.$ Answer: $k_{\text{eff}}=3$. A dilation of 1 is just a normal convolution.

Stride. Stride $s$ decides how many positions we place the filter — bigger $s$ ⇒ fewer positions ⇒ smaller output. Dilation $d$ decides how wide a single placement reaches, leaving the count of positions (roughly) unchanged. Answer: stride.

Set $W_{\text{out}}=W_{\text{in}}$ in the master formula with $s=1$, $d=1$ (so $k_{\text{eff}}=k$): $W_{\text{in}}=W_{\text{in}}+2p-k+1\Rightarrow 2p=k-1\Rightarrow p=\frac{k-1}{2}.$ Answer: $p=\frac{k-1}{2}$. For $k=3$ that is $p=1$; for $k=5$ it is $p=2$.

$k_{\text{eff}}=3$ (since $d=1$). Reach $=W_{\text{in}}+2p-k_{\text{eff}}=7+0-3=4$ (how far the filter's left edge can travel before its right edge leaves the image). $W_{\text{out}}=\lfloor \frac{4}{2}\rfloor +1=2+1=3.$ Filter sits at positions $0,2,4$ — all inside $[0,6]$. Answer: $3$.

$k_{\text{eff}}=3$. Padded size $=32+2=34$. Reach $=34-3=31$. $W_{\text{out}}=\lfloor \frac{31}{1}\rfloor +1=31+1=32.$ This is same padding: $p=(k-1)/2=1$ preserves the $32\times 32$ shape. Answer: $32\times 32$.

First the effective kernel: $k_{\text{eff}}=3+(3-1)(2-1)=3+2=5.$ Reach $=28+0-5=23$. $W_{\text{out}}=\lfloor \frac{23}{1}\rfloor +1=24.$ Answer: $24\times 24$. The dilated kernel behaves size-wise like a plain $5\times 5$, even though it only holds $9$ weights.

Reach $=3+0-5=-2$, which is negative. $W_{\text{out}}=\lfloor \frac{-2}{1}\rfloor +1=-2+1=-1.$ A negative "size" is the formula's way of shouting the kernel does not fit — a $5\times 5$ window cannot sit anywhere inside a $3\times 3$ image without padding. Answer: no valid output (formula gives $-1$). In practice you must add padding or shrink $k$.

Padded size $=64+2=66$, $k_{\text{eff}}=3$. Try $s=2$: $W_{\text{out}}=\lfloor \frac{66-3}{2}\rfloor +1=\lfloor \frac{63}{2}\rfloor +1=31+1=32.✓$ Answer: $s=2$ gives exactly $32\times 32$. This is the standard "strided conv downsample" — an alternative to a pooling layer that learns its own downsampling.

Path A parameters: $11^2=121$. Path B effective size: $k_{\text{eff}}=3+(3-1)(5-1)=3+8=11$ ✓ — same reach. Path B parameters: $3^2=9$. Answer: A needs $121$ weights, B needs $9$ — a $\approx 13\times$ saving. Dilation buys reach cheaply (see Receptive Field Analysis), at the cost of sparse sampling: B never looks at the pixels sitting in its gaps.

For $k=4$: $p=(4-1)/2=1.5$ — you cannot add half a pixel of border. The geometric reason: same-padding needs $k-1$ extra pixels split evenly across the two sides. When $k$ is odd, $k-1$ is even and splits cleanly ($p$ each side). When $k$ is even, $k-1$ is odd and cannot split symmetrically. Fix in practice: pad asymmetrically — e.g. $1$ pixel on the left, $2$ on the right — which is why odd kernels ($3,5,7$) dominate CNN design.

Step 1 — effective kernel: $k_{\text{eff}}=3+(3-1)(2-1)=5$. Step 2 — padded input: $32+2(1)=34$. Step 3 — reach: $34-5=29$. Step 4 — count positions: $W_{\text{out}}=\lfloor \frac{29}{2}\rfloor +1=14+1=15.$ Answer: $15\times 15$. Note the floor mattered here: $29/2=14.5$, and the half-step is a filter that would hang off the edge, so we drop it.

Layer 1 (same padding): $\lfloor (56+2-3)/1\rfloor +1=55+1=56$. Still $56\times 56$. Layer 2: $\lfloor (56+2-3)/2\rfloor +1=\lfloor 55/2\rfloor +1=27+1=28$. Answer: $28\times 28$. A same-conv followed by a strided conv — the classic "process then downsample" block seen when trading depth against width.

$k_{\text{eff}}=3+(3-1)(3-1)=3+4=7$. Same-padding condition with $s=1$: $W_{\text{in}}=W_{\text{in}}+2p-k_{\text{eff}}+1\Rightarrow 2p=k_{\text{eff}}-1=6\Rightarrow p=3.$ Answer: $p=3$. General rule: for same padding, pad by $p=(k_{\text{eff}}-1)/2$, using the effective kernel, not the raw one.

$k_{\text{eff}}=3$, padded $=102$, reach $=99$. We need: $50=\lfloor \frac{99}{s}\rfloor +1\Rightarrow \lfloor \frac{99}{s}\rfloor =49.$ Solve $99/s\approx 49$: try $s=2$ ⇒ $\lfloor 99/2\rfloor =49$ ✓. Answer: $s=2$. Because $99/2=49.5$ and the floor knocks it to $49$, then $+1$ gives exactly $50$. The half-pixel of reach is discarded — the last would-be filter overhangs the edge.

With $p=0,s=1$: $W_{\text{out}}=W_{\text{in}}-k_{\text{eff}}+1=W_{\text{in}}-[k+(k-1)(d-1)]+1.$ The bracket grows linearly in $d$ with slope $(k-1)>0$ (for $k\ge 2$). So $W_{\text{out}}$ decreases without bound as $d$ increases. It hits $\le 0$ once $k+(k-1)(d-1)\ge W_{\text{in}}+1\Rightarrow d\ge 1+\frac{W_{\text{in}}+1-k}{k-1}.$ Concrete check: for $k=3,W_{\text{in}}=20$ the threshold is $d\ge 1+(20+1-3)/2=1+9=10$. At $d=10$: $k_{\text{eff}}=3+2\cdot 9=21$, so $W_{\text{out}}=20-21+1=0\le 0$ ✓. Conclusion: the effective kernel outgrows the image, so no filter fits — you must add padding or cap $d$. This is why dilated stacks in practice use increasing then resetting dilation schedules rather than unbounded growth.

Part 1: $k_{\text{eff}}=3+(3-1)(4-1)=3+6=9$. Same padding: $p=(k_{\text{eff}}-1)/2=(9-1)/2=4$. Check: $\lfloor (20+8-9)/1\rfloor +1=19+1=20$ ✓. Part 2: same $p=4$, input $13$: $\lfloor (13+8-9)/1\rfloor +1=12+1=13$. Answer: $p=4$; and with that padding a $13$-input gives $13$ out. Same padding is shift-invariant in input size — once $p$ is set for a given $(k_{\text{eff}},s=1)$, it preserves any input size.

Effective kernel formula ::: $k_{\text{eff}}=k+(k-1)(d-1)$ Output size formula ::: $W_{\text{out}}=\lfloor (W_{\text{in}}+2p-k_{\text{eff}})/s\rfloor +1$ Same padding (general) ::: $p=(k_{\text{eff}}-1)/2$ Negative $W_{\text{out}}$ means ::: kernel does not fit; add padding Floor applies to ::: the division only, and rounds down