ResNet and skip connections — Hinglish notes

3.4.9 · AI-ML › Convolutional Neural Networks

Overview

Residual Networks (ResNet) ne deep learning mein revolution la diya tha, kyunki inhone degradation problem ko solve kiya—yeh ek aisa counterintuitive observation tha ki deeper networks shallow ones se bhi worse perform kar sakti hain, training data par bhi. Iska key innovation hai skip connections (ya residual connections), jo gradients ko directly network ke through flow karne deta hai.

Core Problem: The Degradation Paradox

Intuition Hum Bas Zyada Layers Stack Kyun Nahi Kar Sakte?

Tumhara sochna hoga: zyada layers → zyada capacity → better performance. Lekin experimentally, plain 56-layer networks ne 20-layer networks se bhi worse perform kiya training data par bhi. Yeh overfitting nahi hai (jo sirf test accuracy ko hurt karta)—deeper network literally training set utna achha nahi seekh sakta.

Kyun? Do compounding issues hain:

Vanishing gradients: Backprop mein, gradients layers se multiply hote hain. Sigmoid/tanh ke saath, gradients exponentially shrink hote hain: $(\frac{\partial L}{\partial w _{1}} = \frac{\partial L}{\partial a _{L}} \cdot \prod_{i = 2}^{L} \frac{\partial a _{i}}{\partial a _{i - 1}} \cdot \frac{\partial a _{1}}{\partial w _{1}})$ . Agar har $\frac{\partial a _{i}}{\partial a _{i - 1}} < 1$ ho, toh early layers ko negligible updates milte hain.
Optimization difficulty: ReLU ke saath bhi (jo gradients mein help karta hai), bahut deep networks complex error surfaces create karte hain jisme kaafi saddle points aur local minima hote hain. Optimization stuck ho jaata hai.

The Residual Learning Solution

Intuition: Function Nahi, Delta Seekhna

Mathematical Formulation

Formula Residual Block

Ek residual block input $x$ ko is tarah transform karta hai: $y = F (x, {W_{i}}) + x$

jahan:

$F (x, {W_{i}})$ residual mapping hai jo stacked layers seekhti hain (typically 2-3 conv layers with ReLU)
$x$ skip connection ya identity shortcut hai
$y$ output hai First principles se derivation:

Standard layer stack: Normally, hamare paas $y = F (x, {W_{i}})$ hota, jahan $F$ kuch aisa ho sakta hai: $F (x) = W_{2} \cdot ReLU (W_{1} \cdot x + b_{1}) + b_{2}$
Identity shortcut add karo: Hum ise modify karte hain: $y = W_{2} \cdot ReLU (W_{1} \cdot x + b_{1}) + b_{2} + x$
Dimension matching: Agar $x$ aur $F (x)$ ke dimensions alag hain, toh projection use karo: $y = F (x, {W_{i}}) + W_{s} \cdot x$ jahan $W_{s}$ ek $1 \times 1$ convolution (ya linear projection) hai jo dimensions match karta hai.

Yeh step kyun? Stride ke saath spatial dimensions change hoti hain, channels by design change hote hain. Projection ensure karta hai ki $x$ aur $F (x)$ ko element-wise add kiya ja sake.

Gradient Flow Analysis

Formula Residual Connections ke Through Backward Pass

Skip connection ek gradient highway banata hai. Backpropagation ke dauran: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial}{\partial x} [F (x) + x]$

$= \frac{\partial L}{\partial y} \cdot [\frac{\partial F ( x )}{\partial x} + 1]$ Derivation:

Chain rule se shuru karo: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}$
$y = F (x) + x$ substitute karo: $\frac{\partial y}{\partial x} = \frac{\partial F ( x )}{\partial x} + \frac{\partial x}{\partial x} = \frac{\partial F ( x )}{\partial x} + I$ (jahan $I$ identity matrix hai)
Isliye: $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} + \frac{\partial L}{\partial y} \cdot \frac{\partial F ( x )}{\partial x}$ Critical observation: Gradient ka ek direct path hai (woh " $+ 1$ " term) jo kisi bhi weight layer se nahi guzarta. Chahe $\frac{\partial F}{\partial x} \to 0$ bhi ho jaye, gradients shortcut ke through backward flow karte rehte hain. Yahi vanishing gradients ko solve karta hai.

Yeh kyun matter karta hai: Ek 100-layer ResNet mein, gradient directly layer 100 se layer 1 tak flow kar sakta hai bina 99 nonlinearities se multiply hue.

ResNet Architecture Details

Common Configurations

Model	Layers	Blocks	Parameters	Top-5 Error
ResNet-18	18	Basic	11.7M	10.76%
ResNet-34	34	Basic	21.8M	10.12%
ResNet-50	50	Bottleneck	25.6M	7.13%
ResNet-101	101	Bottleneck	44.6M	6.44%
ResNet-152	152	Bottleneck	60.2M	6.16%

Worked Examples

Worked example Example 1: Basic Residual Block Forward Pass

Setup: Input $x$ hai $64 \times 56 \times 56$ (64 channels, 56×56 spatial). Block mein do $3 \times 3$ convs hain, dono 64 filters ke saath.

Forward pass:

Pehla conv: $x \to Conv_{3 \times 3, 64} (x) \to BN \to ReLU$

Output: $64 \times 56 \times 56$
Yeh step kyun? $3 \times 3$ spatial filters se features extract karo, activations normalize karo, nonlinearity apply karo

Doosra conv: $\to Conv_{3 \times 3, 64} \to BN$

Output: $F (x)$ of shape $64 \times 56 \times 56$
Yeh step kyun? Doosra transformation layer; abhi ReLU nahi (addition ke baad apply hoga)

Skip connection add karo: $y = F (x) + x$

Dono $F (x)$ aur $x$ hain $64 \times 56 \times 56$ , toh direct addition kaam karta hai
Yeh step kyun? Seekhe gaye residual ko original input ke saath combine karo

Final activation: $y \to ReLU (y)$

Yeh step kyun? Residual aur identity ko combine karne ke baad nonlinearity apply karo

Result: Output hai $64 \times 56 \times 56$ , input ke same. Block ne complete transformation ki bajay input mein adjustments seekhe.

Worked example Example 2: Projection Shortcut (Dimension Mismatch)

Setup: Input $x$ hai $64 \times 56 \times 56$ . Hum spatially downsample karna chahte hain (stride 2) aur channels 128 tak badhana chahte hain.

Problem: Agar $F (x)$ output karta hai $128 \times 28 \times 28$ , toh hum directly $x$ add nahi kar sakte jo ki $64 \times 56 \times 56$ hai. Solution: Projection shortcut $W_{s} \cdot x$ use karo Forward pass:

Residual path $F (x)$ :

Conv $1 \times 1$ , 64 filters: $64 \times 56 \times 56 \to 64 \times 56 \times 56$
Yeh step kyun? Bottleneck: expensive $3 \times 3$ conv se pehle computation reduce karo
Conv $3 \times 3$ , 64 filters, stride 2: $64 \times 56 \times 56 \to 64 \times 28 \times 28$
Yeh step kyun? Stride 2 ke saath spatial downsampling dimensions ko half kar deta hai
Conv $1 \times 1$ , 128 filters: $64 \times 28 \times 28 \to 128 \times 28 \times 28$
Yeh step kyun? Target channel count tak expand karo

Shortcut path $W_{s} \cdot x$ :

Conv $1 \times 1$ , 128 filters, stride 2: $64 \times 56 \times 56 \to 128 \times 28 \times 28$
Yeh step kyun? $F (x)$ se dono spatial dimensions (stride ke zariye) aur channels (128 filters ke zariye) match karo

Add karo: $y = F (x) + W_{s} \cdot x$

Dono ab $128 \times 28 \times 28$ hain Key insight: Projection $W_{s}$ seekha jaata hai lekin simple hai (sirf $1 \times 1$ conv). Zyaadatar computation $F (x)$ mein hota hai.

Worked example Example 3: Gradient Flow Comparison

Setup: Ek 20-layer plain network aur 20-layer ResNet mein gradient flow compare karo. Plain network: $\frac{\partial L}{\partial x _{1}} = \frac{\partial L}{\partial x _{20}} \prod_{i = 2}^{20} \frac{\partial x _{i}}{\partial x _{i - 1}}$

Agar har $\frac{\partial x _{i}}{\partial x _{i - 1}} = 0.8$ ho (ReLU networks ke liye typical): $\frac{\partial L}{\partial x _{1}} \approx \frac{\partial L}{\partial x _{20}} \cdot (0.8)^{19} \approx \frac{\partial L}{\partial x _{20}} \cdot 0.0144$

Yeh bura kyun hai: Gradient apni original magnitude ka sirf 1.44% reh gaya. Layer 1, layer 20 se ~70× slower seekhta hai.

ResNet with 10 residual blocks (2 layers each): Har block contribute karta hai: $\frac{\partial y}{\partial x} = \frac{\partial F}{\partial x} + I$

Simplified gradient path (shortcuts consider karte hue): $\frac{\partial L}{\partial x _{1}} = \frac{\partial L}{\partial x _{20}} \cdot [1 + \sum_{i = 1}^{10} \frac{\partial F _{i}}{\partial x _{1}}]$

Chahe sab $\frac{\partial F _{i}}{\partial x _{1}} = 0$ bhi ho, phir bhi hamare paas hai: $\frac{\partial L}{\partial x _{1}} = \frac{\partial L}{\partial x _{20}} \cdot 1$

Yeh kyun matter karta hai: Gradient apni magnitude maintain karta hai. Layer 1 ko layer 20 ke jaisi hi signal strength milti hai, jo bahut deep networks mein effective learning enable karta hai.

Common Mistakes and Misconceptions

Common mistake Mistake 1: "Skip connections sirf gradients ko backward flow karne mein help karte hain"

Yeh sahi kyun lagta hai: Gradient flow explanation sabse zyada commonly cited benefit hai, aur yeh sach bhi hai.

Kya incomplete hai: Skip connections forward propagation mein bhi help karte hain:

Ensemble effect: ResNets ko exponentially many shallow networks ka ensemble mana ja sakta hai (residual blocks ka har subset ek valid path banata hai)
Feature reuse: Early features (edges, textures) baad ki layers ko re-learn kiye bina directly accessible hoti hain
Implicit deep supervision: Har layer effectively loss ka shortcut rakhti hai, jisse ek implicit form of deeply-supervised learning create hoti hai

Fix: Skip connections dono forward aur backward flow problems solve karte hain. Yeh sirf ek gradient trick nahi hai—yeh fundamentally change kar deta hai ki information network ke through kaise propagate karti hai.

Common mistake Mistake 2: "Identity mapping hamesha kuch nahi seekhne se better hai"

Yeh sahi kyun lagta hai: Agar koi layer $F (x) = 0$ seekhti hai, toh $y = x$ (identity), jo "default safe option" lagta hai.

Kya galat hai: Yeh reasoning initialization ko optimization ke saath confuse kar rahi hai. Claim yeh nahi hai ki identity optimal hai, balki yeh hai ki:

Identity kisi bhi specific transformation se seekhna easier hai jab random weights se shuruat karte hain
Residuals optimize karne mein easier hain kyunki function space mein identity ek special case hai (jab $F = 0$ ho)
Network training ke dauran blocks ko selectively use ya ignore kar sakta hai

Socho: Agar layer 50 ke paas already perfect features hain, toh layer 51 ke liye $F = 0$ (kuch mat karo) seekhna kaafi easier hai, rather than ek plain network ke liye exact identity mapping $H (x) = x$ ko through seekhna.