Visual walkthrough — Recurrent neural networks — hidden state, BPTT

5.6.12 · D2 · Coding › Machine Learning (Aerospace Applications) › Recurrent neural networks — hidden state, BPTT

Hum sab kuch scalar rakhenge (single numbers, matrices nahi) taaki kuch bhi linear algebra ke peeche na chhupe. Ek baar tum ise scalars mein dekh lo, matrix version wohi kahaani hai bas $\odot$ aur transposes ke saath.

---

Step 1 — Memory ki chain (WHAT hum differentiate kar rahe hain)

WHAT. Ek recurrent net ek sequence ko ek item at a time padhta hai. Step $t$ pe yeh ek memory number $h_t$ rakhta hai (hidden state). Yeh us memory ko naye input $x_t$ aur purani memory $h_{t-1}$ se update karta hai, hamesha same teen numbers $w_{hh},w_{xh},b_h$ se:

$$\underset{\text{new memory}}{\underbrace{h_t}}=\tanh (\underset{\text{memory→memory}}{\underbrace{w_{hh}}}\underset{\text{old memory}}{\underbrace{h_{t-1}}}+\underset{\text{input→memory}}{\underbrace{w_{xh}}}\underset{\text{new input}}{\underbrace{x_t}}+\underset{\text{offset}}{\underbrace{b_h}})$$

WHY. Hum chahte hain ki loss $L$ neeche jaaye, isliye hume jaanna hai ki har weight error ke liye kitna zimmedaar hai. Yeh zimmedaari ek derivative hai $\partial L/\partial w$. Us derivative ko chase karne se pehle hume woh raasta dekhna hoga jis par signal travel karta hai — woh raasta neeche ki chain hai.

PICTURE. Time left se right chalta hai. Har box ek memory $h_t$ hai; har black arrow same weight $w_{hh}$ ka ek use hai; har red arrow ek output $y_t$ drop karta hai jo target se compare hota hai.

---

Step 2 — Pre-activation ko naam dena (WHY hum $a_t$ introduce karte hain)

WHAT. $\tanh$ ek plain linear combination wrap karta hai. Us andar wale part ko apna naam do:

$$a_t=\underset{\text{linear part}}{\underbrace{w_{hh}{h}_{t-1}+w_{xh}{x}_t+b_h}},h_t=\tanh (a_t)$$

WHY. Chain rule ek baar mein ek simple function pe kaam karta hai. Agar hum linear part aur squash ko ek saath lump karein toh hume ek monster differentiate karna padega. $a_t$ pe split karne se do easy links milte hain:

$$\text{weights}⟶a_t\stackrel{\tanh }{\to }{h}_t$$

WHY tanh at all? Yeh kisi bhi number ko $(-1,1)$ mein squash karta hai, isliye memory hamesha ke liye recirculate ho sakti hai bina blow up kiye. Yeh origin se bhi pass hota hai, isliye zero input zero-ish memory deta hai (ek fair starting point). Alternatives ko picture mein compare karo.

PICTURE. Teen curves. Sirf $\tanh$ (red) bounded bhi hai aur zero pe centered bhi — yahi property hume ek looping memory ke liye chahiye.

Recall Kyun tanh drop nahi karte aur linear nahi rehte?

Question: kya toot jaata hai agar $h_t=a_t$ ho (no squash)? ::: Poora unrolled net ek bade linear map mein collapse ho jaata hai ($h_T$ ek constant times $h_0$ plus input terms ban jaata hai), aur gradient pure $w_{hh}^T$ ban jaata hai — guaranteed vanish ya explode hoga kyunki control karne ke liye koi nonlinearity nahi. Dekho Vanishing and Exploding Gradients.

---

Step 3 — Single-link derivative: $\tanh$ ka twin (WHAT ${\tanh }^{\prime }$ hai)

WHAT. Blame ko squash ke through move karne ke liye hume $\tanh$ ki slope chahiye. Ek khoobsoorat fact:

$$\frac{d}{da}\tanh (a)=1-{\tanh }^2(a)=1-h^2$$

Toh slope us output se likhi jaati hai jo hum pehle se compute kar chuke hain — koi naya evaluation nahi chahiye.

WHY. Backward pass ke dauran hum $h_t$ pe khade hain aur $a_t$ tak step back karna chahte hain. Chain rule kehta hai is slope se multiply karo. Kyunki yeh $1-h_t^2$ ke barabar hai, aur $h_t\in (-1,1)$, yeh slope hamesha $(0,1]$ mein rehti hai: $\tanh$ ke through har backward step signal ko sirf shrink kar sakta hai, grow nahi. Yeh thought Step 6 ke liye pakad ke rakho.

PICTURE. Red curve $1-h^2$ hai. Yeh middle pe sabse tall (=1) hai aur edges pe 0 tak squash ($h=\pm 1$). Ek memory jo saturate ho gayi hai ($\pm 1$ pe push ho gayi) almost koi gradient pass nahi karti — woh "deaf" ho gayi hai.

---

Step 4 — Hidden state pe blame aana (BPTT ka dil)

WHAT. Memory $h_t$ pe blame define karo ${\delta }_t\equiv \frac{\partial L}{\partial h_t}$ ke roop mein: total loss kitna badalta hai agar hum $h_t$ ko nudge karein. Kyunki $h_t$ do roads mein fork hua tha (Step 1), uska blame har road se wapas aane wale blame ka sum hai:

$${\delta }_t=\underset{\text{road 1: apna output }{y}_t}{\underbrace{\frac{\partial L_t}{\partial h_t}}}+\underset{\text{road 2: future se blame}}{\underbrace{\frac{\partial h_{t+1}}{\partial h_t}{\delta }_{t+1}}}$$

Term by term:

• $\frac{\partial L_t}{\partial h_t}$ — is step ne apne output ke through jo local error kiya.
• ${\delta }_{t+1}$ — next memory ke liye already computed blame (hum right-to-left kaam karte hain, isliye yeh ready hai).
• $\frac{\partial h_{t+1}}{\partial h_t}$ — bridge jo future blame ko ek step wapas le jaata hai. Steps 2–3 se yeh $(1-h_{t+1}^2)w_{hh}$ hai: $\tanh$ ke twin se pass karo, phir weight se.

WHY sum? Yeh multivariable chain rule hai: ek quantity jo kai paths se answer ko influence karti hai woh sum contribute karti hai un paths ka. $h_t$ ke exactly do paths hain, isliye exactly do terms. Yahi "back-propagation through time" ka idea hai: step $t+1$ ka blame step $t$ ko diya jaata hai, phir step $t$ apna total step $t-1$ ko deta hai, aur aise backward chalta hai.

PICTURE. Red arrows right to left flow karte hain. $h_t$ pe do red arrows merge hote hain: ek chhota apne output se, ek lamba $h_{t+1}$ se aata hua. Unka sum ${\delta }_t$ hai.

---

Step 5 — Shared weight ko blame karna: saare kamon ka sum

WHAT. Ab payoff. Weight $w_{hh}$ har step pe use hua tha (Step 1 ki picture ka har black arrow). Uska total blame us har step pe jo usne kiya uska sum hai. Step $t$ pe uska kaam tha $h_{t-1}$ ko $a_t$ ke andar multiply karna, isliye uska local contribution hai $(1-h_t^2){\delta }_t$ ($h_t$ pe blame, twin ke through $a_t$ tak push kiya) times woh number jise usne multiply kiya, $h_{t-1}$:

$$\boxed{\frac{\partial L}{\partial w_{hh}}=\sum _{t=1}^T\underset{\text{twin: }{h}_t\to a_t}{\underbrace{(1-h_t^2)}}\underset{\text{blame at }{h}_t}{\underbrace{{\delta }_t}}\underset{\text{woh jo }{w}_{hh}\text{ ne multiply kiya}}{\underbrace{h_{t-1}}}}$$

WHY phir sum? Same weight-sharing logic Step 4 ke sum jaisi, lekin ab paths ki jagah parameter uses ke upar. Ek knob, kai kaam $\Rightarrow$ har kaam ka blame add karo. Yeh single boxed line BPTT derivation ka poora point hai.

PICTURE. Center mein $w_{hh}$ label wala ek knob; wires har time step tak pahunchte hain; knob pe total turn har wire se wapas aane wale pulls ka sum hai.

---

Step 6 — Edge case: bahut zyada steps peeche = ek product jo mar jaata hai ya explode karta hai

WHAT. Final loss se blame early memory tak kaise pahunchta hai? Bridges ko chain karo. Step $t$ se step $T$ tak jaane mein har hop pe ek bridge multiply hota hai:

$$\frac{\partial h_T}{\partial h_t}=\prod _{k=t+1}^T\underset{\in (0,1]}{\underbrace{(1-h_k^2)}}\underset{\text{fixed}}{\underbrace{w_{hh}}}$$

Har factor roughly $w_{hh}$ hai jo $\le 1$ number se scale hua hai. Ek factor ki typical size $\lambda$ rakho. Toh $n=T-t$ steps ka gap blame ko roughly ${\lambda }^n$ se multiply karta hai.

WHY yeh matter karta hai — $\log \lambda$ ke dono signs cover karo:

• $\lambda <1$ (e.g. chhota $w_{hh}$, ya saturated memories): ${\lambda }^n\to 0$. Door se blame vanish ho jaata hai — net cannot learn long-range dependencies. Yeh common case hai, aur isliye LSTM and GRU exist karte hain (woh near-identity memory path add karte hain jahan $\lambda \approx 1$).
• $\lambda >1$ (bada $w_{hh}$, unsaturated): ${\lambda }^n\to \infty$. Blame explode ho jaata hai — training diverge karti hai. Gradient clipping se fix hota hai.
• $\lambda =1$ (knife-edge): blame preserve hota hai — exactly woh regime jise gated cells engineer karne ki koshish karte hain.

PICTURE. ${\lambda }^n$ ki teen red curves time-gap $n$ ke against: ek zero tak decay karti, ek 1 pe flat, ek upar shoot karti. Sirf flat wali memory ko alive rakhti hai.

---

Step 7 — Real numbers pe sanity check (degenerate + normal case)

WHAT. Tiny scalar RNN end-to-end chalao: $w_{hh}=0.5,w_{xh}=1,b_h=0$, inputs $x_1=x_2=1$, start $h_0=0$, target $1$ last step pe with $L=\frac{1}{2}(h_2-1{)}^2$.

Forward:

• $a_1=0.5\cdot 0+1\cdot 1=1\Rightarrow h_1=\tanh (1)=0.7616$
• $a_2=0.5\cdot 0.7616+1=1.3808\Rightarrow h_2=\tanh (1.3808)=0.8811$

Backward (right to left):

• ${\delta }_2=\partial L/\partial h_2=h_2-1=-0.1189$.
• bridge $\frac{\partial h_2}{\partial h_1}=(1-h_2^2)w_{hh}=(1-0.7763)(0.5)=0.1118$.
• ${\delta }_1=\underset{\text{step 1 pe koi output loss nahi}}{\underbrace{0}}+0.1118\cdot (-0.1189)=-0.01329$.

Degenerate first term. Kyunki $h_0=0$ hai, step-1 ka contribution $\partial L/\partial w_{hh}$ mein zero se multiply hoke vanish ho jaata hai — yeh ek clean illustration hai ki weight sirf wahan blame earn karta hai jahan usne actually nonzero memory multiply ki:

$$\frac{\partial L}{\partial w_{hh}}=\underset{=0(\text{since }{h}_0=0)}{\underbrace{(1-h_1^2){\delta }_1{h}_0}}+\underset{\text{sirf yeh live term hai}}{\underbrace{(1-h_2^2){\delta }_2{h}_1}}$$ $$=0+(0.2237)(-0.1189)(0.7616)=-0.02025$$

WHY. Yahan har symbol Steps 1–5 mein define kiya gaya tha. Yeh number Step 5 ka boxed formula action mein hai, aur yeh "sum over steps" structure confirm karta hai jisme ek term zero initial state se switch off hai.

PICTURE. Sum ki do candles side by side: ek grey zero candle (step 1, $h_0=0$ se kill hua) aur ek red live candle (step 2). Unki heights sum karke $-0.02025$ deti hain.

---

Ek-picture summary

Sab kuch ek canvas pe: top pe forward $\tanh$ chain, bottom pe red blame backward flow karta hua, aur shared weight har step se pulls ka sum collect karta hua.

Recall Feynman: poora walkthrough plain words mein

Tum ek sequence ko ek item at a time padhte ho aur ek single memory number rakhte ho jise tum same rule se har step refresh karte ho — woh rule $\tanh$ se squash karta hai taaki memory hamesha ke liye loop ho sake bina explode kiye. Jab ending galat ho, tum poori sequence ke through backward chalte ho har step se pehle wale ko blame hand karte hue. Har hand-off pe tum blame ko "$\tanh$ ke twin" se pass karte ho ($1-h^2$, hamesha 0 aur 1 ke beech) aur phir weight se. Kyunki wahi weight har step pe use hua tha, tum un sab se uska blame add up karte ho — "same weight, sum the blame." Catch yeh hai: bahut steps wapas jaane se un between-0-and-1 factors ke kai saare multiply hote hain, isliye door ka blame usually fade ho ke kuch nahi rehta (vanishing gradient) — yahi woh leak hai jo LSTM and GRU plug karta hai. Yeh bhi dekho Time Series Forecasting aur Sequence Modeling in Flight Data jahan yeh memory apna kaam karta hai.