Recurrent neural networks — hidden state, BPTT: Question bank

5.6.12 · D5 · Coding › Machine Learning (Aerospace Applications) › Recurrent neural networks — hidden state, BPTT

Shuru karne se pehle, ek one-screen refresher symbols ka taaki neeche kuch bhi anjaan na lage:

$x_{t}$ = time step $t$ par input (jaise ek airspeed reading).
$h_{t}$ = hidden state, ek vector "sticky note" jo $t$ tak sab kuch dekha uska summary rakhta hai.
$W_{hh}, W_{x h}, W_{h y}$ = teen weight matrices, har step par same (weight sharing).
$a_{t} = W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h}$ = pre-activation, $tanh$ ke squash karne se pehle.
$L = \sum_{t} L_{t}$ = total loss; $δ_{t} = \partial L / \partial h_{t}$ = "step $t$ par pahunchne wala blame."
$⊙$ = element-wise multiply; $diag (v)$ = ek matrix jisme vector $v$ diagonal par ho.
BPTT = backprop jo network ko unroll karke ek layer per time step ki tarah run hota hai.

True or false — justify

Har item ek claim hai. True/false decide karo, phir ek sentence mein justify karo.

"BPTT ek fundamentally alag algorithm hai ordinary backpropagation se."

False. Ye ordinary backprop hi hai, bas unrolled graph par run hota hai; ek hi twist hai ki shared weight kaafi baar appear hoti hai, toh uska gradient steps ke upar ek sum ban jaata hai.

"Update ko linear banana,

h_{t} = W_{hh} h_{t - 1} + W_{x h} x_{t}

, model ko simplify karega bina kisi real harm ke."

False. Linear maps ka ek stack ek hi linear map mein collapse ho jaata hai, toh net saari expressive power kho deta hai aur uske gradients pure

λ^{T}

ban jaate hain — guarantee hai ki vanish ya explode hoga, koi gating ka rasta nahi.

"Truncated BPTT koi information lose nahi karta jab tak recent steps rakhe jaayein."

False. Backward pass ko

k

steps par cut karna har us gradient ko exactly zero force kar deta hai jo

k

se purana ho, toh net koi bhi dependency

k

se lambi nahi seekh sakta — ye ek compute trade-off hai, free lunch nahi.

"Kyunki same

W_{hh}

reuse hoti hai, RNN ke paas ek equivalent unrolled feed-forward net se kam parameters hote hain."

True. Weight sharing ka matlab hai ek matrix har step ke liye kaam karta hai, toh parameter count sequence length se independent hai, unlike feed-forward net jo har slot ke liye fresh weights maangta.

\partial h_{t + 1} / \partial h_{t} = W_{hh}

False. Tum

tanh

derivative bhool gaye; sahi Jacobian hai

diag (1 - h_{t + 1}^{2}) W_{hh}

, kyunki state

tanh

se guzarti hai aage feed hone se pehle.

"Weight sharing ek trained RNN ko training se lambi sequence handle karne deta hai bina retraining ke."

True. Ek rule step by step apply hota hai length ki parwah kiye bina, toh jo net "airspeed 3 in a row rising" seekh chuka hai woh already 5-sample stream pe generalise kar leta hai.

"Agar

L

mein loss term sirf final step

T

par ho, toh

δ_{t} = 0

sabhi

t < T

ke liye."

False. Sirf direct term

\partial L_{t} / \partial h_{t}

zero hai

t < T

ke liye; indirect term

W_{hh}^{⊤} [(1 - h_{t + 1}^{2}) ⊙ δ_{t + 1}]

abhi bhi final-step blame ko backward through time le jaata hai.

"Vanishing-gradient problem ka matlab hai ki forward hidden states bhi zero tak shrink ho jaate hain."

False. Ye backward gradient signal hai jo decay karta hai; forward states

tanh

ki wajah se

(- 1, 1)

mein bounded rehte hain aur bilkul non-trivial ho sakte hain.

"Gradient clipping vanishing-gradient problem ko cure karta hai."

False. Clipping sirf exploding gradients ko tame karta hai unka size cap karke; vanishing gradients already bahut chhote hain, toh clipping kuch nahi karta — uske liye LSTM and GRU jaise gated cells chahiye.

Spot the error

Har line mein ek hidden galti ke saath claim hai. Correction reveal karo.

"Koi likhta hai

\frac{\partial L}{\partial W _{hh}} = (1 - h_{T}^{2}) ⊙ δ_{T} h_{T - 1}^{⊤}

sirf last step use karke."

Error: sum missing hai. Kyunki

W_{hh}

har step par use hoti hai, uska gradient hai

\sum_{t = 1}^{T} [(1 - h_{t}^{2}) ⊙ δ_{t}] h_{t - 1}^{⊤}

— "same weight, sum the blame."

"Ek student claim karta hai ki sigmoid aur

tanh

update mein interchangeable hain kyunki dono squash karte hain."

Error: centering.

tanh

0

par centered hai (output

(- 1, 1)

mein) jo gradients ko healthier rakhta hai, jabki sigmoid

0.5

par centered hai aur activations ko positive bias deta hai, gradient flow ko hurt karta hai.

δ_{t}

nikaalte waqt woh likhte hain

δ_{t} = \partial L_{t} / \partial h_{t} \times W_{hh}^{⊤} [(1 - h_{t + 1}^{2}) ⊙ δ_{t + 1}]

(ek product)."

Error: ye product nahi, sum hai. Multivariate chain rule direct path aur future path ko add karta hai:

δ_{t} = \partial L_{t} / \partial h_{t} + W_{hh}^{⊤} [(1 - h_{t + 1}^{2}) ⊙ δ_{t + 1}]

"Koi backward pass mein

tanh^{'} (a) = 1 - a^{2}

use karta hai."

Error: argument confusion. Sahi hai

tanh^{'} (a) = 1 - tanh^{2} (a) = 1 - h^{2}

, yaani one minus output squared, na ki one minus pre-activation squared.

"Ek note kehta hai ki recurrent Jacobian ki sabse badi singular value stable training ke liye exactly 1 honi chahiye."

Error: exactly 1 hona zaroori nahi. Tum chahte ho ki effective factor (

W_{hh}

ki singular value times

tanh^{'}

) 1 ke paas ho; bahut neeche 1 se vanish hoga, bahut upar 1 se explode hoga, lekin 1 ke aaspaas ek healthy band theek hai.

"Koi

W_{h y}

(output weights) ko bhi future hidden states se summed contributions milne wali treat karta hai."

Error: $W_{h y}$ sirf $h_{t}$ ko locally read karta hai. Uska gradient step

t

par sirf us step ke output par depend karta hai, though ye

t

ke upar summed zaroor hota hai kyunki ye bhi time across shared hai.

Why questions

Reason ke saath jawab do, sirf fact nahi.

"Shared weight ka gradient sum kyun hona chahiye, average ya ek step se kyun nahi?"

Kyunki weight ka har use parameter se loss tak ek independent path hai, aur chain rule saare path contributions add karta hai; average karna true gradient ko silently

1/ T

se scale kar dega.

"Unrolling ek RNN ko aise kyun bana deta hai jis par ordinary backprop kaam kar sake?"

Unrolling loop ko ek finite chain

h_{0} \to h_{1} \to \dots \to h_{T}

of tied layers se replace kar deta hai, jo bas ek (deep) feed-forward graph hai, aur backprop kisi bhi aise acyclic graph par defined hai.

"Bahut saare similar Jacobians ka product kyun vanishing/exploding gradients cause karta hai?"

T - t

near-identical matrices multiply karna

λ^{T - t}

ki tarah behave karta hai (jahan

λ

dominant factor ho), aur koi bhi

λ \neq = 1

badi power tak raise hone par

0

\infty

ki taraf race karta hai.

"LSTM/GRU cells plain

tanh

RNN se zyada kyun help karte hain jahan woh struggle karta hai?"

Ye ek gated, near-identity memory path introduce karte hain toh backward Jacobian us path pe

1

ke close rehta hai, long-range gradient product ko collapse hone se bachata hai — dekho LSTM and GRU aur Vanishing and Exploding Gradients.

"Hidden state ko store ki jagah summary kyun kehte hain?"

Ye ek fixed-size vector hai jo arbitrarily lambi history ko compress karta hai, toh necessarily detail discard karta hai aur sirf wahi rakhta hai jo past updates ne relevant samjha — ye sab kuch losslessly store nahi kar sakta.

"Weight sharing flight data ke liye ek useful prior ki tarah kyun kaam karta hai?"

Ye time-invariance bake in karta hai — "same physics har instant apply hoti hai" — toh model attitude ya airspeed jaise streams ke liye ek rule seekhta hai position-specific patterns memorise karne ki jagah; compare Sequence Modeling in Flight Data.

Edge cases

Har rule ko uski boundary tak push karo.

h_{1}

kya hoga jab

h_{0} = 0

aur input zero hai,

x_{1} = 0

, aur

b_{h} = 0

a_{1} = 0

toh

h_{1} = tanh (0) = 0

; state origin par hi rehti hai, yaani "abhi koi information nahi" update ka ek genuine fixed point hai.

"Length-1 sequence (

T = 1

) ke liye BPTT kya reduce ho jaata hai?"

Ye ordinary single-step backprop mein reduce ho jaata hai bina kisi backward-in-time term ke, kyunki koi future step hi nahi hai jisse

δ

propagate ho sake.

"Agar

W_{hh} = 0

ho, toh steps ke across memory ka kya hoga?"

State apna past bilkul ignore kar deti hai (

h_{t} = tanh (W_{x h} x_{t} + b_{h})

), toh RNN ek per-step feed-forward net mein degenerate ho jaata hai bina kisi memory ke.

"Jab ek hidden unit saturate hoti hai,

h_{t} \to \pm 1

— usse guzarne wale gradient ka kya hota hai?"

Factor

1 - h_{t}^{2} \to 0

, toh us unit ka backward gradient zero ki taraf throttle ho jaata hai — saturation ek local cause hai vanishing gradients ka kisi bhi lambe product se pehle bhi.

"Agar target sirf final step par exist kare, toh kya early inputs abhi bhi train ho sakte hain?"

Haan, provided sequence itni chhoti ho ki summed backward gradient vanish na hua ho; final-step blame

x_{1}

tak recurrent path se pahunchta hai, though lambi sequences ke liye kamzori se.

"Step 1 par

\partial L / \partial W_{hh}

ka kya hoga agar

h_{0} = 0

ho?"

Us step ka contribution

(1 - h_{1}^{2}) ⊙ δ_{1} h_{0}^{⊤}

zero hai kyunki

h_{0} = 0

ise multiply karke out kar deta hai — pehla step simply recurrent weight ke gradient mein kuch add nahi karta.