Vanishing gradients in RNNs — Hinglish notes

3.5.3 · AI-ML › Sequence Models

Overview

Vanishing gradient problem tab hota hai jab gradients exponentially small ho jaate hain jaise-jaise wo recurrent neural networks mein time ke through backward propagate hote hain, jisse long-term dependencies seekhna impossible ho jaata hai. Yahi woh fundamental obstacle hai jisne RNNs ko real-world sequential tasks pe effective banne se roka — jab tak LSTM aur GRU architectures develop nahi ho gayi.

The Mathematical Root Cause

Gradient descent se train karne ke liye, humein $\frac{\partial L}{\partial W _{hh}}$ chahiye jahan $L$ hamara loss hai. Chain rule ke zariye, gradients ko saare time steps ke through backward flow karna padta hai.

Derivation from First Principles

Chaliye derive karte hain ki gradients kyun vanish hote hain. Ek simple RNN consider karo jisme loss $L$ time step $T$ par hai.

Step 1: Chain rule through time

Time $t$ par parameters update karne ke liye, humein chahiye: $\frac{\partial L}{\partial h _{t}} = \frac{\partial L}{\partial h _{T}} \cdot \frac{\partial h _{T}}{\partial h _{t}}$

Yeh step kyun? Time $T$ par loss, $h_{t}$ par depend karta hai sirf hidden states ki chain $h_{t} \to h_{t + 1} \to \dots \to h_{T}$ ke through. Humein is poore path ke across gradients accumulate karne honge.

Step 2: Chain ko expand karo

$\frac{\partial h _{T}}{\partial h _{t}} = \frac{\partial h _{T}}{\partial h _{T - 1}} \cdot \frac{\partial h _{T - 1}}{\partial h _{T - 2}} \dots \frac{\partial h _{t + 1}}{\partial h _{t}}$

$= \prod_{k = t + 1}^{T} \frac{\partial h _{k}}{\partial h _{k - 1}}$

Yeh step kyun? Har hidden state directly sirf pichle wale par depend karta hai, isliye hum saari local derivatives ko chain kar lete hain.

Step 3: Local derivative compute karo

$h_{k} = tanh (W_{hh} h_{k - 1} + W_{x h} x_{k} + b_{h})$ se:

$\frac{\partial h _{k}}{\partial h _{k - 1}} = diag (tanh^{'} (z_{k})) \cdot W_{hh}$

jahan $z_{k} = W_{hh} h_{k - 1} + W_{x h} x_{k} + b_{h}$ aur $tanh^{'} (z) = 1 - tanh^{2} (z)$ .

Yeh step kyun? Ek linear transformation par apply hone wale $tanh (\cdot)$ ke composition ke liye chain rule humein activation ki derivative times weight matrix deta hai.

Step 4: Product ko bound karo

Kyunki $tanh^{'} (z) \leq 1$ sabhi $z$ ke liye, har diagonal entry zyada se zyada 1 hai. Isliye:

$\frac{\partial h _{k}}{\partial h _{k - 1}} \leq ∥ W_{hh} ∥$

Yeh step kyun? Tanh ki derivative bounded hai (har diagonal entry $\leq 1$ ), aur matrix norms satisfy karte hain $∥ A B ∥ \leq ∥ A ∥ \cdot ∥ B ∥$ . Toh tanh factor norm ko sirf shrink kar sakta hai, $∥ W_{hh} ∥$ ko effective per-step multiplier ke roop mein chhodta hai.

Step 5: Exponential decay

$(T - t)$ terms ke product ke liye:

$\frac{\partial h _{T}}{\partial h _{t}} \leq ∥ W_{hh} ∥^{T - t}$

Agar $∥ W_{hh} ∥ < 1$ (chahe default initialization se ho ya weight decay se spectral norm 1 se neeche aa jaaye), toh:

$\frac{\partial h _{T}}{\partial h _{t}} \leq γ^{T - t}$

jahan $γ < 1$ . Yeh time gap $(T - t)$ mein exponentially decay karta hai.

Ek zaroori distinction: Yeh do mechanisms alag-alag factors par kaam karte hain. Spectral norm $∥ W_{hh} ∥$ weight matrix ki property hai aur isi ka $< 1$ hona zaroori hai is bound ko decay produce karne ke liye. Tanh saturation $∥ W_{hh} ∥$ ko nahi badalta — balki yeh $diag (tanh^{'} (z_{k}))$ factor ko $0$ ki taraf shrink karta hai, jo $∥ W_{hh} ∥$ term ke upar multiply hota hai. Dono ek hi (shrinking) direction mein push karte hain, lekin yeh alag contributions hain aur inhe conflate nahi karna chahiye.

Yeh step kyun? Ek number jo 1 se kam hai use khud se kai baar multiply karo toh exponential decay milta hai. Sirf 10 time steps ke baad $γ = 0.9$ ke saath, gradient $0. 9^{10} \approx 0.35$ ho jaata hai. 50 steps ke baad: $0. 9^{50} \approx 0.005$ . Gradient effectively vanish ho jaata hai.

Setup: Humare paas length 50 ki ek sequence hai, aur hum dekhna chahte hain ki $t = 50$ par ek gradient, $t = 0$ par parameter updates ko kaise affect karta hai.

Computation: $\frac{\partial h _{50}}{\partial h _{0}} = 0. 9^{50} = 5.15 \times 1 0^{- 3}$

Interpretation: Output par magnitude 1.0 ka ek gradient shuruaat tak pahunchte-pahunchte 0.005 ho jaata hai. Agar hamara learning rate 0.01 hai, toh parameter update hai $0.01 \times 0.005 = 0.00005$ — essentially zero. Network 50 steps span karne waali dependencies nahi seekh sakta.

Typical RNN ke saath: Upar se $tanh$ saturation add karo. Agar inputs bade hain, toh $tanh^{'} (z) \approx 0.1$ per step. Ab combined per-step multiplier (tanh factor $\times$ weight factor) hai: $Effective multiplier per step = 0.1 \times 0.9 = 0.09$ $0.0 9^{50} \approx 2 \times 1 0^{- 53}$

Yeh float32 ke liye machine epsilon se bhi chhota hai. Gradient numerically vanish ho chuka hai. Dhyan do ki $0.1$ tanh ki derivative se aaya aur $0.9$ weight norm se — alag factors jo multiply karte hain.

$C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ \tilde{C}_{t}$

Key difference: Cell state ke through gradient path yeh hai: $\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}$

Yeh additive hai, kai layers ke through multiplicative nahi. Forget gate $f_{t}$ learned hai aur 1 ke close ho sakta hai, ek uninterrupted gradient highway create karta hai.

Yeh step kyun? Chhote numbers se baar-baar multiply karne ke exponential decay ki jagah, hume ek learned gate milta hai jo control karta hai ki purani information (aur uska gradient) kitna preserve karna hai. Jab $f_{t} \approx 1$ hota hai, gradients kaafi time steps ke across nearly unchanged flow karte hain.

Yeh Kab aur Kyun Hota Hai

Teeno usually practice mein saath hote hain, lekin product ke alag-alag factors par kaam karte hain.

Common Mistakes aur Misconceptions

Kyun galat hai: ReLU shrinking tanh-derivative factor ko remove karta hai (local derivative ya toh $0$ ya $1$ ho jaati hai), lekin yeh explosion guarantee nahi karta. Gradients explode honge ya nahi yeh poori tarah spectral norm of $W_{hh}$ par depend karta hai: agar $∥ W_{hh} ∥ > 1$ , toh product $\prod diag (ReLU^{'} (z_{k})) W_{hh}$ ab grow kar sakta hai kyunki tanh damping chali gayi hai; agar $∥ W_{hh} ∥ < 1$ , toh gradients phir bhi vanish ho sakte hain. Toh ReLU ek safety cushion (activation damping) ko remove karta hai bina underlying credit-assignment problem ko fix kiye — yeh inherently explosion cause nahi karta, bas weight spectral norm ko sole controlling factor bana deta hai.

Fix: ReLU RNNs ke saath aapko typically gradient clipping aur $W_{hh}$ ki careful (jaise identity/orthogonal) initialization chahiye, aur phir bhi architecture credit assignment problem solve nahi karta. Carefully gated additions wale LSTMs/GRUs proper solution hain.

Kyun galat hai: Gradients alag-alag time steps par alag rates se decay karte hain. Early steps mein vanished gradients ho sakte hain ( $1 0^{- 50}$ ) jabki recent steps mein reasonable gradients hote hain ( $1 0^{- 2}$ ). Ek single learning rate dono fix nahi kar sakta: early steps ke liye itna bada hoga toh recent steps ke liye exploding updates ho jaayenge.

Fix: Architectural solutions (LSTM/GRU) ya adaptive optimizers (Adam, RMSprop) jo per-parameter learning rates maintain karte hain. Aur bhi better: dono saath mein.

Kyun galat hai: BatchNorm batch dimension ke across normalize karta hai, lekin vanishing gradients RNNs mein time dimension ke across hote hain. Alag sequences ko normalize karna ek single sequence ke andar temporal gradient flow fix nahi karta. Layer normalization (har time step par features ke across) thoda zyada help karta hai, lekin phir bhi LSTM gates ki tarah gradient highways create nahi karta.

Fix: Layer normalization + LSTM/GRU architecture use karo. Gates essential hain.

Practical Implications

1. Maximum learnable dependency length Practice mein, vanilla RNNs sirf 5-10 time steps span karne wali dependencies seekh sakte hain. Language ke liye (20-50 token context chahiye) ya time series ke liye (saikdon steps par seasonal patterns), yeh completely fail ho jaate hain.

2. Training symptoms

Loss bahut jaldi plateau kar jaata hai
Parameters pehle kuch epochs ke baad mushkil se change hote hain
Recent outputs seekhte hain, lekin early sequence context ignore ho jaata hai
Memory require karne wale tasks par validation performance bahut kharab

3. Jab aap modern architectures mein bhi yeh dekhte hain LSTMs bhi vanishing gradients experience kar sakte hain agar:

Sequences extremely long hain (1000+ steps)
Forget gates poor initialization ki wajah se 0 par saturate ho jaate hain
Bahut deep RNN stacks hain (kaafi layers)

Solution Landscape

Approach	Kaise Help Karta Hai	Limitations
LSTM/GRU	Additive cell state updates gradient highways create karte hain	Zyada parameters, slower training
Gradient clipping	Explosion prevent karta hai lekin vanishing nahi	Naye gradient paths create nahi karta
Layer normalization	Saturation reduce karta hai	Multiplicative decay fix nahi karta
Truncated BPTT	Backprop depth limit karta hai	Truncation window se lambi dependencies nahi seekh sakta
Attention mechanisms	Sequential processing ko completely bypass karta hai	High memory cost, temporal inductive bias kho jaata hai
Skip connections	ResNets ki tarah gradient highways add karte hain	Recurrent structure ke liye carefully design karna padta hai

Recall Aise Samjhao Jaise Main Baara Saal Ka Hoon

Imagine karo tum 50 logon ki line mein telephone ka game khel rahe ho. Pehla insaan ek message whisper karta hai, aur har insaan aage pass karta hai. Jab tak yeh end tak pahunchta hai, message completely garbled ho jaata hai.

Ab imagine karo tumhe pata karna hai ki message kisne kharab kiya. Tum end se shuru karke backward jaate ho, har insaan se poochte ho "tumne ise kitna change kiya?" Lekin har insaan sirf thoda yaad rakhta hai jo unhone kiya (jaise "maine ise 10% change kiya"). Jab tum 0.9 × 0.9 × 0.9 ko pachaas baar multiply karte ho, toh itna chhota number milta hai ki practically zero hai.

Yahi vanishing gradients hai: "feedback signal" ki kya galat hua itna tiny ho jaata hai jab tak sequence ki shuruaat tak pahunchta hai ki network early parts ke baare mein kuch nahi seekh sakta.

LSTMs aise hain jaise har insaan ko ek notepad dena jahan wo original message likh sake aur WOHI aage pass kare, sirf apna modified version nahi. Ab tumhare paas do paths hain: telephone game (jo phir bhi garbled ho jaata hai) aur notepad (jo accurate rehta hai). Network seekhta hai important long-term information ke liye notepad use karna.

Connections

Backpropagation Through Time - Woh training algorithm jahan yeh hota hai
LSTM Architecture - Vanishing gradients ka primary solution
GRU Architecture - LSTM ka ek simplified alternative
Exploding Gradients - Opposite problem jab $∥ W_{hh} ∥ > 1$
Gradient Clipping - Exploding address karta hai lekin vanishing nahi
Attention Mechanisms - Sequential processing ko completely bypass karne ka tarika
Residual Connections - Feedforward networks mein similar additive gradient highways
Activation Functions - Tanh saturation ek key contributor hai

#flashcards/ai-ml

RNNs mein vanishing gradient problem kya hai? :: Jab gradients time ke through backpropagate hote waqt exponentially small ho jaate hain, jisse long-term dependencies seekhna impossible ho jaata hai kyunki early time steps ke liye parameter updates zero ke paas aa jaate hain.

Vanishing gradients ka mathematical cause kya hai?

Gradients Jacobians ke product

\prod_{k = t + 1}^{T} \frac{\partial h _{k}}{\partial h _{k - 1}}

ke through flow karte hain. Jab

∥ \frac{\partial h _{k}}{\partial h _{k - 1}} ∥ < 1

hota hai (tanh saturation aur/ya spectral norm

∥ W_{hh} ∥ < 1

ki wajah se), toh product exponentially

γ^{T - t}

ke roop mein decay karta hai jahan

γ < 1

Vanilla RNNs long-term dependencies kyun nahi seekh sakte?

T time steps span karne ke liye, gradients

γ^{T}

ke roop mein decay karte hain.

γ = 0.9

jaisi typical values ke saath, 50 steps ke baad gradients ~0.005 ho jaate hain, jisse parameter updates negligibly small ho jaate hain. Network bahut pehle ho chuki events se kuch nahi seekh sakta.

LSTM vanishing gradient problem ko kaise solve karta hai?

LSTMs ek cell state use karte hain additive updates ke saath:

C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ \tilde{C}_{t}

. Cell ke through gradient path hai

\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}

, jo additive hai multiplicative nahi. Jab forget gates 1 ke paas hote hain, gradients kaafi time steps ke across nearly unchanged flow karte hain — ek "gradient highway" create karte hain.

RNNs mein gradient decay formula kya hai?

\frac{\partial h _{T}}{\partial h _{t}} \leq ∥ W_{hh} ∥^{T - t} \cdot \prod_{k = t + 1}^{T} ∥ diag (tanh^{'} (z_{k})) ∥

. Kyunki

tanh^{'} (z) \leq 1

aur typically

∥ W_{hh} ∥ < 1

, yeh product gap

(T - t)

mein exponentially decay karta hai.

Tanh saturation vanishing gradients ko kyun aur kharab karta hai?

Jab

∣ z ∣

bada hota hai,

tanh (z) \to \pm 1

aur

tanh^{'} (z) = 1 - tanh^{2} (z) \to 0

. Har saturated time step gradient product ke activation-derivative factor mein ek near-zero multiplier contribute karta hai (weight norm se alag), exponential decay ko accelerate karta hai.

Sirf ReLU activation use karna vanishing gradients solve kyun nahi karega?

ReLU shrinking tanh-derivative factor ko remove karta hai (local derivative 0 ya 1 ban jaati hai) lekin inherently explosion cause nahi karta. Gradients explode honge ya vanish yeh abhi bhi

W_{hh}

ke spectral norm par depend karta hai:

> 1

explode kar sakta hai,

< 1

abhi bhi vanish ho sakta hai. ReLU activation damping cushion ko credit assignment fix kiye bina remove karta hai.

Vanishing gradients ke teen main causes kya hain?

(1) Long sequences — zyada time steps matlab zyada multiplications matlab zyada decay; (2) Tanh saturation — activation-derivative factor ko zero ki taraf shrink karta hai; (3) Small spectral norm — agar

∥ W_{hh} ∥ < 1

, toh weight factor har step gradients shrink karta hai. Yeh product mein alag-alag factors par kaam karte hain.

Ek bada learning rate vanishing gradients compensate kyun nahi kar sakta?

Gradients alag time steps ke liye alag rates se vanish hote hain. Early steps mein

1 0^{- 50}

gradients ho sakte hain jabki recent steps mein

1 0^{- 2}

. Ek single learning rate dono fix nahi kar sakta: early steps ke liye itna bada matlab recent steps ke liye exploding updates, numerical instability create karta hai.

Vanilla RNNs maximum kitni dependency length seekh sakte hain? :: Practice mein approximately 5-10 time steps. Iske aage, exponential gradient decay parameter updates ko itna small kar deta hai ki effective learning nahi ho sakti, chahe theoretically RNNs arbitrarily long dependencies model kar sakti hain.

Batch normalization RNN vanishing gradients kyun solve nahi karta?

BatchNorm batch dimension ke across normalize karta hai, lekin vanishing gradients RNNs mein time dimension ke across hote hain. Alag sequences ko normalize karna ek sequence ke andar temporal gradient flow fix nahi karta. Layer normalization (har time step par features ke across) thoda help karta hai lekin LSTM gates ki tarah gradient highways create nahi karta.

RNN aur LSTM gradient flow mein key structural difference kya hai?

RNNs multiplicative hidden state updates use karte hain:

h_{t} = f (W_{hh} h_{t - 1} + \dots)

, deta hai

\frac{\partial h _{T}}{\partial h _{t}} = \prod

(products decay karte hain). LSTMs additive cell state updates use karte hain:

C_{t} = f_{t} ⊙ C_{t - 1} + \dots

, deta hai

\frac{\partial C _{T}}{\partial C _{t}} = \prod f_{k}

jahan

f_{k}

1 ke paas ho sakta hai (gradients preserve karta hai).

Training ke dौran vanishing gradients diagnose kaise karte hain?

Symptoms include: (1) Loss bahut jaldi plateau kar jaata hai, (2) Early layers ke liye gradient norms late layers se orders of magnitude chhote hain, (3) Parameters initial epochs ke baad mushkil se change hote hain, (4) Model recent context par well perform karta hai lekin earlier sequence information ignore karta hai, (5) Long-term memory require karne wale tasks par validation performance kharab hai.