Recurrent neural networks — hidden state, BPTT

5.6.12 · Coding › Machine Learning (Aerospace Applications)

1. Hidden state kya hai? (WHAT)

tanh kyun? Yeh $(- 1, 1)$ mein squash karta hai, state ko bounded rakhta hai taaki recirculate hote waqt blow up na ho, aur 0 par centered hai (sigmoid ke unlike), jo gradients ko healthier rakhta hai.

Weights share kyun? Sequence kisi bhi length ki ho sakti hai. Agar har step ke apne weights hote toh (a) variable length handle nahi ho sakti, aur (b) "same physical rule har instant apply hoti hai" yeh generalize nahi ho sakta. Sharing mein time-invariance bake hoti hai.

2. Network ko unroll karna (HOW sochna hai)

Ek RNN ek loop hai, lekin gradients compute karne ke liye hum ise ek deep feed-forward net mein unroll karte hain — ek layer har time step ke liye, saari layers ek saath tied hoti hain.

Figure — Recurrent neural networks — hidden state, BPTT

3. BPTT ko first principles se derive karna

Hum chahte hain $\frac{\partial L}{\partial W _{hh}}$ ek total loss $L = \sum_{t = 1}^{T} L_{t}$ ke liye, jahan $L_{t}$ , $y_{t}$ ko target se compare karta hai.

Step 0 — pre-activation. Define karo $a_{t} = W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h}$ , toh $h_{t} = tanh (a_{t})$ . Kyun? Pre-activation ko naam dene se hum chain rule cleanly apply kar sakte hain.

Step 1 — gradient hidden state mein wapas flow karna. $h_{t}$ loss ko do tareekon se affect karta hai: seedha $y_{t}$ ke through, aur indirectly next step $h_{t + 1}$ ke through. Toh define karo $δ_{t} \equiv \frac{\partial L}{\partial h _{t}}$ : $δ_{t} = via output y_{t} \frac{\partial L _{t}}{\partial h _{t}} + via future \frac{\partial h _{t + 1}}{\partial h _{t}}^{⊤} δ_{t + 1}$ Sum kyun? Multivariate chain rule: ek variable jo kai paths feed karta hai woh sum of path gradients contribute karta hai. Yahi BPTT ka dil hai — gradient time mein peeche pass hota hai.

Step 2 — recurrent Jacobian. Kyunki $h_{t + 1} = tanh (W_{hh} h_{t} + \dots)$ , $\frac{\partial h _{t + 1}}{\partial h _{t}} = diag (1 - h_{t + 1}^{2}) W_{hh}$ $tanh^{'} (a) = 1 - tanh^{2} (a)$ use karke. Kyun? $tanh$ ke through phir linear map $W_{hh}$ ke through chain rule.

Step 3 — shared weights par accumulate karna. Kyunki $W_{hh}$ har step par reuse hoti hai, iska gradient saare steps ke contributions ka sum hai: $\frac{\partial L}{\partial W _{hh}} = t = 1 \sum T (1 - h_{t}^{2}) ⊙ δ_{t} h_{t - 1}^{⊤}$ Phir sum kyun? Same weight-sharing logic: ek parameter, kai uses ⇒ saare "jobs" add up karo.

4. RNNs train karna kyun mushkil hai: vanishing/exploding gradients

$δ_{t}$ ko $δ_{T}$ tak expand karo: door-future losses ke gradients Jacobians ke product se multiply hote hain: $\frac{\partial h _{T}}{\partial h _{t}} = \prod_{k = t + 1}^{T} diag (1 - h_{k}^{2}) W_{hh}$

5. Worked example — haath se tiny scalar RNN

Sab kuch scalar ho: $w_{hh} = 0.5, w_{x h} = 1, b_{h} = 0$ , inputs $x_{1} = 1, x_{2} = 1$ , $h_{0} = 0$ .

Forward.

$a_{1} = 0.5 \cdot 0 + 1 \cdot 1 = 1 \Rightarrow h_{1} = tanh (1) = 0.7616$ . Kyun? Update rule mein plug karo.
$a_{2} = 0.5 \cdot 0.7616 + 1 = 1.3808 \Rightarrow h_{2} = tanh (1.3808) = 0.8811$ .

Loss. Maano $L = \frac{1}{2} (h_{2} - 1)^{2}$ (final step par target 1). $\frac{\partial L}{\partial h _{2}} = h_{2} - 1 = - 0.1189$ .

Backward (BPTT).

$δ_{2} = - 0.1189$ .
$\frac{\partial h _{2}}{\partial h _{1}} = (1 - h_{2}^{2}) w_{hh} = (1 - 0.7763) \cdot 0.5 = 0.1118$ . Kyun? $tanh^{'}$ times $w_{hh}$ .
$δ_{1} = 0 + 0.1118 \cdot (- 0.1189) = - 0.0133$ . Pehla term 0 kyun? Step 1 par koi output loss nahi.

$w_{hh}$ ke w.r.t. gradient (steps par sum, $h_{t - 1}$ "input" ke taur par): $\frac{\partial L}{\partial w _{hh}} = (1 - h_{1}^{2}) δ_{1} h_{0} + (1 - h_{2}^{2}) δ_{2} h_{1}$ $= (0.4200) (- 0.0133) (0) + (0.2237) (- 0.1189) (0.7616) = - 0.02025$ Do terms kyun? Weight $w_{hh}$ step 1 par use hua ( $h_{0}$ multiply karke) aur step 2 par ( $h_{1}$ multiply karke). Dono sum karo.

Task: detect karo "airspeed 3 consecutive samples ke liye badh rahi hai." 3 ki fixed window par ek feed-forward net ko slot-1, slot-2, slot-3 ke liye alag weights seekhne padte hain. Ek RNN ek comparison rule seekhta hai aur use stream mein chalte waqt apply karta hai — toh yeh bina retraining ke 5-sample window ke liye bhi kaam karta hai. Yeh step kyun important hai: yeh dikhata hai ki weight sharing = ek prior ki "physics har instant same hai."

Common mistake Common errors ko steel-man karna

"BPTT backprop se alag algorithm hai." Kyun sahi lagta hai: iska apna scary naam hai. Fix: yeh ordinary backprop hi hai — sirf unrolled graph par apply kiya gaya. Ek hi twist hai ki same weight kai baar appear hota hai, toh iska gradient ek sum hai.

"Bas $h_{t} = W_{hh} h_{t - 1} + \dots$ (tanh nahi) — simple!" Kyun sahi lagta hai: linear analyze karna aasaan hai. Fix: linear recurrence mein zero nonlinearity hai — pura network ek linear map mein collapse ho jaata hai, aur gradients pure $λ^{T}$ hain, gate karne ka koi tarika nahi aur explode/vanish guaranteed.

"Truncated BPTT koi information nahi khoता." Kyun sahi lagta hai: aap phir bhi recent steps ke through backprop karte ho. Fix: $k$ steps par truncate karna long-range gradients zero set karta hai — aap $k$ se lambi dependencies nahi seekh sakte. Yeh compute trade-off hai, free nahi.

" $\partial h_{t + 1} / \partial h_{t} = W_{hh}$ ." Kyun sahi lagta hai: $W_{hh}$ recurrent matrix hai. Fix: aap $tanh$ derivative bhool gaye — yeh $diag (1 - h_{t + 1}^{2}) W_{hh}$ hai.

Recall Feynman: 12-saal ke bachche ko explain karo

Socho tum ek story ek word at a time padh rahe ho aur apne dimaag mein ek chhota sticky note rakh rahe ho jisme likha hai "abhi tak kya hua." Har naya word, sticky note update karo. Woh sticky note hidden state hai. Jab ending galat ho, tum poori story mein peeche jaate ho yeh pata lagane ke liye ki kaun se pehle words ne tumhe gadbad kiya — aur tum same reading rules poore time use karte ho, toh saari corrections add up karo. Woh story-mein-peeche-blame karna BPTT hai. Mushkil: bahut lambi story ke baad tumhara "blame signal" kuch nahi ho jaata (vanishing gradient), toh net shuruaat bhool jaata hai.

Flashcards

Hidden state

h_{t}

kya represent karta hai?

Time

t

tak dekhe gaye saare inputs ka ek running summary/memory, jo har step par

h_{t - 1}

aur

x_{t}

se update hota hai.

RNN hidden-state update likho.

h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})

RNN weights time ke across share kyun hoti hain?

Variable-length sequences handle karne ke liye aur time-invariance encode karne ke liye (har step same rule); ek param, kai uses.

Ek sentence mein BPTT kya hai?

Ordinary backprop jo RNN par time ke over unrolled apply hota hai, jahan shared weights ke gradients saare steps par summed hote hain.

h_{t}

ko do sources se gradient kyun milta hai?

Yeh loss ko seedha output

y_{t}

ke via aur indirectly next state

h_{t + 1}

ke via affect karta hai; multivariate chain rule paths ko sum karta hai.

Recurrent Jacobian

\partial h_{t + 1} / \partial h_{t}

kya hai?

diag (1 - h_{t + 1}^{2}) W_{hh}

RNNs mein gradients kyun vanish/explode hote hain?

Backprop kai similar Jacobians multiply karta hai;

λ^{T - t}

jaisa behave karta hai, toh

λ < 1

vanish hota hai aur

λ > 1

explode hota hai.

Exploding/vanishing gradients ke do standard fixes?

Gradient clipping (explode) aur gated cells LSTM/GRU near-identity memory path ke saath (vanish).

Plain linear recurrence ki jagah tanh kyun use karo?

Linear net ko ek linear map mein collapse karta hai aur memory gate nahi kar sakta; tanh state ko bound karta hai aur nonlinearity add karta hai.

Truncated BPTT kya sacrifice karta hai?

Truncation window se lambe gradients (aur is tarah learnable dependencies) zero set ho jaate hain.

Connections

Feed-forward Neural Networks — RNN unrolled = deep tied-weight feed-forward net.
Backpropagation — BPTT unrolled graph par backprop hai.
LSTM and GRU — vanishing gradients ke liye gated fixes.
Vanishing and Exploding Gradients
Time Series Forecasting — flight-data / sensor prediction.
Kalman Filter — classical recursive state estimator; learned hidden state ka conceptual cousin.
Sequence Modeling in Flight Data