Vanishing and exploding gradients — Hinglish notes

3.1.11 · AI-ML › Neural Network Fundamentals

Kya ho raha hai actually

KYA multiply hota hai? Ek plain feed-forward net mein, har layer compute karta hai: $a^{(l)} = σ (z^{(l)}), z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)} .$ Backprop error $δ^{(l)} = \partial L / \partial z^{(l)}$ ko chain rule use karke backward push karta hai.

Derivation from first principles

Maano "layer $l$ pe error" $δ^{(l)} = \frac{\partial L}{\partial z ^{(l)}}$ hai.

Step 1 — consecutive layers ko link karo. Kyunki $z^{(l + 1)} = W^{(l + 1)} σ (z^{(l)}) + b^{(l + 1)}$ hai, chain rule deta hai: $δ^{(l)} = (W^{(l + 1)})^{⊤} δ^{(l + 1)} ⊙ σ^{'} (z^{(l)}) .$ Yeh step kyun? Layer $l$ ka error loss tak sirf layer $l + 1$ ke through pahunchta hai, toh hum $l + 1$ ka error lete hain, use weights $(W^{(l + 1)})^{⊤}$ ke through pull back karte hain, aur scale karte hain is baat se ki is neuron ka output apni input ke saath kitna change hota hai, yaani $σ^{'} (z^{(l)})$ .

Step 2 — $L$ layers ke across unroll karo. Step 1 ko last layer se layer 1 tak baar baar apply karne par: $δ^{(1)} = (\prod_{l = 2}^{L} (W^{(l)})^{⊤} diag (σ^{'} (z^{(l - 1)}))) δ^{(L)} .$ Yeh step kyun? Har layer ek aur multiplicative factor insert karti hai. Layer $L$ se layer $1$ tak jaane mein $L - 1$ links cross hote hain, toh product mein $L - 1$ factors hote hain.

Step 3 — magnitude ko bound karo (key insight). Norms lo. Agar har factor satisfy karta hai $(W^{(l)})^{⊤} diag (σ^{'}) \leq γ$ , toh: $δ^{(1)} \leq γ^{L - 1} δ^{(L)} .$ Yeh step kyun? Ek product ka norm $\leq$ norms ka product hota hai, aur $L - 1$ factors hain. Ab behaviour clearly samajh aata hai:

$γ < 1 \Rightarrow γ^{L - 1} \to 0$ → vanish.
$γ > 1 \Rightarrow γ^{L - 1} \to \infty$ → explode.
$γ \approx 1 \Rightarrow$ signal preserved. Yahi poora design goal hai.

Activation choice kyun matter karti hai

$σ_{sigmoid}^{'} (z) = σ (z) (1 - σ (z)) \leq \frac{1}{4}, tanh^{'} (z) = 1 - tanh^{2} (z) \leq 1.$

Sigmoid kyun vanish karta hai: iska derivative kabhi $0.25$ se zyada nahi hota. Toh perfectly scaled weights ke saath bhi, $γ \leq 0.25 \cdot ∥ W ∥$ ; kaafi layers mein $0.25$ ke factors compound hokar near zero ho jaate hain. ReLU ka derivative positive inputs ke liye $1$ hota hai, toh yeh signal ko shrink nahi karta (iska factor exactly $1$ hai), yahi reason hai ki deep nets mein yeh dominate karta hai.

Figure — Vanishing and exploding gradients

Fixes (80/20 jo zyaadatar cases solve karta hai)

Weight initialization — $W$ ko scale karo taaki variance preserve ho.
- Xavier/Glorot (tanh/sigmoid ke liye): $Var (W) = \frac{2}{n _{in} + n _{o u t}}$ .
- He (ReLU ke liye): $Var (W) = \frac{2}{n _{in}}$ (extra $\times 2$ kyunki ReLU aadhe inputs ko zero kar deta hai). Kyun: shuruaat mein $∥ W ∥ \cdot max σ^{'} \approx 1$ rakhta hai.
Non-saturating activations — ReLU / LeakyReLU (slope $1$ ).
Batch/Layer Normalization — har layer mein $z^{(l)}$ ko re-center/scale karta hai taaki $σ^{'}$ healthy range mein rahe.
Residual (skip) connections — $a^{(l)} = a^{(l - 1)} + f (a^{(l - 1)})$ Jacobian ko ek $I + (\dots)$ term deta hai, toh gradient ke paas hamesha ek " $1$ " path hota hai (factor $\approx 1$ ).
Gradient clipping (exploding ke liye) — $g \leftarrow g \cdot \frac{threshold}{∥ g ∥}$ rescale karo agar $∥ g ∥ >$ threshold ho.
LSTM/GRU gates (RNNs ke liye) — additive cell state near- $1$ recurrence deta hai.

Worked examples

Common mistakes

Active recall

Backprop mein kaunsa operation gradients ko vanish ya explode karta hai?

Jacobians ki repeated multiplication (weights × activation derivatives) layers ke across — bahut saare factors ka ek product.

L

-layer net ke gradient mein kitne multiplicative factors hote hain?

L - 1

, ek per layer-to-layer link, toh bound

γ^{L - 1}

jaisa scale karta hai.

Stability kaunsi ek quantity se determine hoti hai?

Per-layer gain

γ \approx ∥ W ∥ \cdot max σ^{'}

;

γ \approx 1

chahiye.

Sigmoid vanishing gradients kyun promote karta hai?

Iska derivative

σ (1 - σ) \leq 0.25

hai, toh factors hamesha signal ko shrink karte hain.

Sigmoid derivative ki maximum value kya hai aur kahan?

0.25

z = 0

par (jahan

σ = 0.5

hoti hai).

He vs Xavier init — kab aur kyun factor of 2?

He ReLU ke liye (

Var = 2/ n_{in}

);

\times 2

ReLU ke ~aadhe activations zero karne ko compensate karta hai. Xavier tanh/sigmoid ke liye (

2/ (n_{in} + n_{o u t})

Kaunsi fix specifically exploding gradients ko target karti hai?

Gradient clipping (gradient ko rescale karo agar uska norm ek threshold se zyada ho).

Residual connections kyun help karte hain?

Layer Jacobian

I + (\dots)

ban jaata hai, gradient ke liye ek guaranteed gain-

\approx 1

identity path deta hai.

γ = 0.25

L = 10

ke liye shrink factor estimate karo.

0.2 5^{9} \approx 3.8 \times 1 0^{- 6}

(

L - 1 = 9

factors use karke); order

1 0^{- 6}

either way.

Kya loss high ho sakta hai jabki gradients vanish ho rahe hon?

Haan — early layers frozen rehti hain jabki later layers still seekhti hain.

Recall Feynman: 12-saal ke bacche ko samjhao

Socho tum ek message 20 doston ki line mein whisper kar rahe ho. Agar har dost thoda dheere whisper kare, toh end mein koi awaaz nahi — yahi vanishing gradient hai (pehle doston ko correction sunti hi nahi). Agar har dost thoda zyada unchhi awaaz mein bole, toh end mein sab chilla rahe hain — yahi exploding hai. Hum ise theek karte hain sabko message same volume par repeat karna sikhaakar (gain $\approx 1$ ), achhe starting rules se (He/Xavier) aur shortcuts se (skip connections) taaki message poori line mein survive kare.

Connections

Backpropagation — woh chain-rule product jo problem create karta hai.
Activation Functions — sigmoid/tanh/ReLU aur unke derivatives.
Weight Initialization — Xavier & He as $γ$ -control.
Batch Normalization — pre-activations ko well-scaled rakhta hai.
Residual Networks (ResNet) — gradient flow ke liye identity paths.
LSTM and GRU — additive cell state recurrent version fix karta hai.
Gradient Descent — yeh gradients weights update karne ke liye kaise use hote hain.