Stride, padding, and dilation

Why These Matter

Standard convolution slides a kernel pixel-by-pixel. But:

1. Stride > 1: Downsamples faster than pooling, saves computation
2. Padding: Prevents spatial dimensions from shrinking too fast (border pixels get used fairly)
3. Dilation: Captures long-range dependencies without deep stacks or large kernels

Together they give you architectural flexibility: control receptive field, output size, and parameter count independently.

1. Stride: The Step Size

Output Size Formula Derivation

Setup: Input size $W_{in}$, kernel size $k$, stride $s$, padding $p$.

WHY this formula? Count how many valid filter positions fit:

• After padding, effective input size: $W_{in} + 2p$
• First filter position: pixel 0
• Each stride moves $s$ pixels
• Last valid position: filter must fit entirely, so rightmost filter edge≤ $W_{in} + 2p -1$
• Rightmost filter starts at: $(W_{in} + 2p) - k$

Number of positions from0 to $(W_{in} + 2p - k)$ with step $s$:

$W_{out} = \left\lfloor \frac{W_{in} + 2p - k}{s} \right\rfloor + 1$

WHAT each term means:

• Numerator $(W_{in} + 2p - k)$: "slack space" after placing first filter
• Divide by $s$: how many strides fit in that slack
• $+1$: count the starting position itself
• Floor: can only take integer steps

2. Padding: Border Strategy

Types of Padding

Valid Padding ($p = 0$): No padding, output shrinks $W_{out} = W_{in} - k + 1\quad \text{(when } s=1\text{)}$

Same Padding: Choose $p$ so $W_{out} = W_{in}$ (when $s=1$)

Derivation: Set $W_{out} = W_{in}$ in the formula: $W_{in} = \frac{W_{in} + 2p - k}{1} + 1$ $W_{in} - 1 = W_{in} + 2p - k$ $2p = k -1$ $p = \frac{k-1}{2}$

For odd $k$ (e.g., 3, 5, 7), this gives integer $p$. For even $k$, pad asymetrically.

Why Padding Matters

1. Information preservation: Edge pixels participate in multiple convolutions
2. Depth control: Can build deep networks without spatial collapse
3. Symmetry: Border pixels not unfairly underepresented

3. Dilation: Expanding Receptive Field

Derivation of Effective Kernel Size

WHAT is dilation? Insert $(d-1)$ zeros between each pair of adjacent kernel weights.

Original kernel: $k$ elements in a row
Gaps inserted: $(k-1)$ gaps (between $k$ elements)
Pixels per gap: $(d-1)$ pixels

Total span:

• Kernel elements: $k$ pixels
• Gap space: $(k-1)(d-1)$ pixels
• Total: $k + (k-1)(d-1)$

HOW it works: For a $3 \times 3$ kernel with $d=2$:

Original:        Dilated (d=2):
X X X            X . X . X
X X X    -->     . . . .
X X X            X . X . X
                . . . . .
                X . X . X

Effective size: $3 + (3-1)(2-1) = 3 + 2 = 5 \times 5$ receptive field

Combined Formula

Design Patterns

Atrous Spatial Pyramid Pooling (ASPP): Use multiple parallel dilations $(d=1,6,12,18)$ with $k=3$ to capture multi-scale context efficiently.

Connections

• Convolutional Layer Basics - these are hyperparameters of the basic conv operation
• Receptive Field Analysis - stride and dilation directly expand receptive field
• Pooling Layers - stride provides alternative to pooling for downsampling
• Semantic Segmentation Architectures - dilated convolutions critical for dense prediction
• Network Depth vs Width - dilation allows depth without excessive parameters

#flashcards/ai-ml

What is stride in convolution? :: The number of pixels the filter moves between applications. Stride $s$ controls the step size and downsampling factor.

Concept Map

Hinglish (regional understanding)