ResNet and skip connections

Overview

Residual Networks (ResNet) revolutionized deep learning by solving the degradation problem—the counterintuitive observation that deeper networks can perform worse than shallower ones, even on training data. The key innovation is skip connections (or residual connections), which allow gradients to flow directly through the network.

Core Problem: The Degradation Paradox

Intuition Why Can't We Just Stack More Layers?

You'd think: more layers → more capacity → better performance. But experimentally, plain 56-layer networks performed worse than 20-layer networks even on training data. This isn't overfitting (which would only hurt test accuracy)—the deeper network literally cannot learn the training set as well.

Why? Two compounding issues:

Vanishing gradients: In backprop, gradients multiply through layers. With sigmoid/tanh, gradients shrink exponentially: $(\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial a_L} \cdot \prod_{i=2}^{L} \frac{\partial a_i}{\partial a_{i-1}} \cdot \frac{\partial a_1}{\partial w_1})$ . If each $\frac{\partial a_i}{\partial a_{i-1}} < 1$ , early layers get negligible updates.
Optimization difficulty: Even with ReLU (which helps gradients), very deep networks create complex error surfaces with many saddle points and local minima. Optimization gets stuck.

The Residual Learning Solution

Intuition: Learning the Delta, Not the Function

Mathematical Formulation

Formula Residual Block

A residual block transforms input $x$ as: $y = F(x, \{W_i\}) + x$

where:

$F(x, \{W_i\})$ is the residual mapping learned by the stacked layers (typically 2-3 conv layers with ReLU)
$x$ is the skip connection or identity shortcut
$y$ is the output Derivation from first principles:

Standard layer stack: Normally, we'd have $y = F(x, \{W_i\})$ where $F$ might be: $F(x) = W_2 \cdot \text{ReLU}(W_1 \cdot x + b_1) + b_2$
Add identity shortcut: We modify this to: $y = W_2 \cdot \text{ReLU}(W_1 \cdot x + b_1) + b_2 + x$
Dimension matching: If $x$ and $F(x)$ have different dimensions, use a projection: $y = F(x, \{W_i\}) + W_s \cdot x$ where $W_s$ is a $1 \times 1$ convolution (or linear projection) that matches dimensions.

Why this step? Spatial dimensions change with stride, channels change by design. The projection ensures $x$ and $F(x)$ can be element-wise added.

Gradient Flow Analysis

Formula Backward Pass Through Residual Connections

The skip connection creates a gradient highway. During backpropagation: $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial}{\partial x}[F(x) + x]$

$= \frac{\partial \mathcal{L}}{\partial y} \cdot \left[\frac{\partial F(x)}{\partial x} + 1\right]$ Derivation:

Start with chain rule: $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial y}{\partial x}$
Substitute $y = F(x) + x$ : $\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + \frac{\partial x}{\partial x} = \frac{\partial F(x)}{\partial x} + I$ (where $I$ is the identity matrix)
Therefore: $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} + \frac{\partial \mathcal{L}}{\partial y} \cdot \frac{\partial F(x)}{\partial x}$ Critical observation: The gradient has a direct path (the " $+1$ " term) that doesn't pass through any weight layers. Even if $\frac{\partial F}{\partial x} \rightarrow 0$ , gradients still flow backward through the shortcut. This solves vanishing gradients.

Why this matters: In a 100-layer ResNet, the gradient can flow directly from layer 100 to layer 1 without being multiplied through 99 nonlinearities.

ResNet Architecture Details

Common Configurations

Model	Layers	Blocks	Parameters	Top-5 Error
ResNet-18	18	Basic	11.7M	10.76%
ResNet-34	34	Basic	21.8M	10.12%
ResNet-50	50	Bottleneck	25.6M	7.13%
ResNet-101	101	Bottleneck	44.6M	6.44%
ResNet-152	152	Bottleneck	60.2M	6.16%

Worked Examples

Worked example Example 1: Basic Residual Block Forward Pass

Setup: Input $x$ is $64 \times 56 \times 56$ (64 channels, 56×56 spatial). Block has two $3 \times 3$ convs, both with 64 filters.

Forward pass:

First conv: $x \rightarrow \text{Conv}_{3×3, 64}(x) \rightarrow \text{BN} \rightarrow \text{ReLU}$

Output: $64 \times 56 \times 56$
Why this step? Extract features through $3 \times 3$ spatial filters, normalize activations, apply nonlinearity

Second conv: $\rightarrow \text{Conv}_{3×3, 64} \rightarrow \text{BN}$

Output: $F(x)$ of shape $64 \times 56 \times 56$
Why this step? Second transformation layer; no ReLU yet (applied after addition)

Add skip connection: $y = F(x) + x$

Both $F(x)$ and $x$ are $64 \times 56 \times 56$ , so direct addition works
Why this step? Combine learned residual with original input

Final activation: $y \rightarrow \text{ReLU}(y)$

Why this step? Apply nonlinearity after combining residual and identity

Result: Output is $64 \times 56 \times 56$ , same as input. The block learned adjustments to the input rather than a complete transformation.

Worked example Example 2: Projection Shortcut (Dimension Mismatch)

Setup: Input $x$ is $64 \times 56 \times 56$ . We want to downsample spatially (stride 2) and increase channels to 128.

Problem: If $F(x)$ outputs $128 \times 28 \times 28$ , we can't directly add $x$ which is $64 \times 56 \times 56$ . Solution: Use projection shortcut $W_s \cdot x$ Forward pass:

Residual path $F(x)$ :

Conv $1 \times 1$ , 64 filters: $64 \times 56 \times 56 \rightarrow 64 \times 56 \times 56$
Why this step? Bottleneck: reduce computation before expensive $3 \times 3$ conv
Conv $3 \times 3$ , 64 filters, stride 2: $64 \times 56 \times 56 \rightarrow 64 \times 28 \times 28$
Why this step? Spatial downsampling with stride 2 halves dimensions
Conv $1 \times 1$ , 128 filters: $64 \times 28 \times 28 \rightarrow 128 \times 28 \times 28$
Why this step? Expand to target channel count

Shortcut path $W_s \cdot x$ :

Conv $1 \times 1$ , 128 filters, stride 2: $64 \times 56 \times 56 \rightarrow 128 \times 28 \times 28$
Why this step? Match both spatial dimensions (via stride) and channels (via 128 filters) to $F(x)$

Add: $y = F(x) + W_s \cdot x$

Both are $128 \times 28 \times 28$ now Key insight: The projection $W_s$ is learned but simple (just $1 \times 1$ conv). Most computation happens in $F(x)$ .

Worked example Example 3: Gradient Flow Comparison

Setup: Compare gradient flow in a 20-layer plain network vs. 20-layer ResNet. Plain network: $\frac{\partial \mathcal{L}}{\partial x_1} = \frac{\partial \mathcal{L}}{\partial x_{20}} \prod_{i=2}^{20} \frac{\partial x_i}{\partial x_{i-1}}$

If each $\frac{\partial x_i}{\partial x_{i-1}} = 0.8$ (typical for ReLU networks): $\frac{\partial \mathcal{L}}{\partial x_1} \approx \frac{\partial \mathcal{L}}{\partial x_{20}} \cdot (0.8)^{19} \approx \frac{\partial \mathcal{L}}{\partial x_{20}} \cdot 0.0144$

Why this is bad: The gradient is 1.44% of its original magnitude. Layer 1 learns ~70× slower than layer 20.

ResNet with 10 residual blocks (2 layers each): Each block contributes: $\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I$

Simplified gradient path (considering shortcuts): $\frac{\partial \mathcal{L}}{\partial x_1} = \frac{\partial \mathcal{L}}{\partial x_{20}} \cdot \left[1 + \sum_{i=1}^{10} \frac{\partial F_i}{\partial x_1}\right]$

Even if all $\frac{\partial F_i}{\partial x_1} = 0$ , we still have: $\frac{\partial \mathcal{L}}{\partial x_1} = \frac{\partial \mathcal{L}}{\partial x_{20}} \cdot 1$

Why this matters: The gradient maintains its magnitude. Layer 1 gets the same signal strength as layer 20, enabling effective learning in very deep networks.

Common Mistakes and Misconceptions

Common mistake Mistake 1: "Skip connections just help gradients flow backward"

Why this feels right: The gradient flow explanation is the most commonly cited benefit, and it's true.

What's incomplete: Skip connections also help with forward propagation:

Ensemble effect: ResNets can be viewed as an ensemble of exponentially many shallow networks (each subset of residual blocks forms a valid path)
Feature reuse: Early features (edges, textures) are directly accessible to later layers without re-learning
Implicit deep supervision: Each layer effectively has a shortcut to the loss, creating an implicit form of deeply-supervised learning

The fix: Skip connections solve both forward and backward flow problems. They're not just a gradient trick—they fundamentally change how information propagates through the network.

Common mistake Mistake 2: "The identity mapping is always better than learning nothing"

Why this feels right: If a layer learns $F(x) = 0$ , then $y = x$ (identity), which seems like the "default safe option."

What's wrong: This reasoning confuses initialization with optimization. The claim isn't that identity is optimal, but that:

Identity is easier to learn than any specific transformation when starting from random weights
Residuals are easier to optimize because the function space includes the identity as a special case (when $F = 0$ )
The network can selectively use or ignore blocks during training

Consider: If layer 50 already has perfect features, layer 51 can learn $F = 0$ (do nothing) more easily than a plain network could learn the exact identity mapping $H(x) = x$ throug

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, yahaan ka core problem samajhna sabse zaroori hai. Logically hum sochte hain ki jitni zyada layers add karenge, network utna hi powerful banega—more capacity, better results. Lekin experiment mein ulta hua: 56-layer wala plain network 20-layer wale se training data pe bhi kharab perform karta tha. Yeh overfitting nahi hai (woh sirf test accuracy hurt karta), balki yeh ek optimization failure hai jise degradation problem kehte hain. Iske do reasons hain—vanishing gradients (jahan backprop mein gradients har layer se multiply hote-hote itne chhote ho jaate hain ki shuru ki layers seekhna hi band kar deti hain) aur deep networks ka complex error surface jahan optimization phas jaata hai.

ResNet ka jugaad genius hai: layers ko poora mapping $H(x)$ seekhne ke bajaye, sirf residual $F(x) = H(x) - x$ seekhne do, aur phir input $x$ ko wapas add kar do—yaani $H(x) = F(x) + x$ . Yeh skip connection hai. Iska fayda yeh hai ki agar optimal mapping identity ke kareeb hai (matlab input ko zyada badalna nahi hai), toh network ko bas $F(x)$ ko zero ke paas push karna hai, jo bahut aasan hai. Analogy yeh samjho—agar tum apni manzil ke 99% paas ho, toh baaki 1% adjustment describe karna easy hai bajaye poora 100% raasta dobara banane ke.

Ab why-it-matters: jab backprop hota hai, skip connection ek "gradient highway" bana deta hai. Gradient ka formula ban jaata hai $\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y} \cdot [\frac{\partial F(x)}{\partial x} + 1]$ —dekho woh +1 term! Iski wajah se gradient chahe kitni bhi deep layers ho, kabhi puri tarah vanish nahi hota, kyunki hamesha kam se kam ek direct path hai jahan se gradient bina shrink hue flow karta hai. Isi ek chhote se idea ki wajah se aaj hum 100+, yahaan tak ki 1000+ layer wale networks train kar paate hain, aur yeh modern deep learning ki foundation ban gaya. Isliye ResNet ko revolutionary kaha jaata hai bhai.