Recurrent neural networks — hidden state, BPTT

1. What is the hidden state? (WHAT)

WHY tanh? It squashes to $(-1,1)$ , keeping the state bounded so it can't blow up as it recirculates, and it is centered at 0 (unlike sigmoid), which keeps gradients healthier.

WHY share weights? A sequence can be any length. If each step had its own weights you (a) couldn't handle variable length, and (b) couldn't generalise "the same physical rule applies at every instant." Sharing bakes in time-invariance.

2. Unrolling the network (HOW to think about it)

An RNN is a loop, but to compute gradients we unroll it into a deep feed-forward net — one layer per time step, all layers tied together.

Figure — Recurrent neural networks — hidden state, BPTT

3. Deriving BPTT from first principles

We want $\dfrac{\partial L}{\partial W_{hh}}$ for a total loss $L=\sum_{t=1}^{T} L_t$ , where $L_t$ compares $y_t$ to the target.

Step 0 — pre-activation. Define $a_t = W_{hh}h_{t-1} + W_{xh}x_t + b_h$ , so $h_t=\tanh(a_t)$ . Why? Naming the pre-activation lets us apply the chain rule cleanly.

Step 1 — gradient flowing back into a hidden state. $h_t$ affects the loss in two ways: directly through $y_t$ , and indirectly through the next step $h_{t+1}$ . So define $\delta_t \equiv \dfrac{\partial L}{\partial h_t}$ : $\delta_t = \underbrace{\frac{\partial L_t}{\partial h_t}}_{\text{via output } y_t} + \underbrace{\frac{\partial h_{t+1}}{\partial h_t}^{\!\top}\delta_{t+1}}_{\text{via future}}$ Why the sum? Multivariate chain rule: a variable that feeds several paths contributes the sum of the path gradients. This is the heart of BPTT — gradient is passed backward in time.

Step 2 — the recurrent Jacobian. Since $h_{t+1}=\tanh(W_{hh}h_t + \dots)$ , $\frac{\partial h_{t+1}}{\partial h_t} = \operatorname{diag}\!\big(1-h_{t+1}^2\big)\,W_{hh}$ using $\tanh'(a)=1-\tanh^2(a)$ . Why? Chain rule through $\tanh$ then through the linear map $W_{hh}$ .

Step 3 — accumulate over shared weights. Because $W_{hh}$ is reused at every step, its gradient is the sum of contributions from all steps: $\boxed{\;\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \big(1-h_t^2\big)\odot\delta_t \;\; h_{t-1}^{\top}\;}$ Why sum again? Same weight-sharing logic: one parameter, many uses ⇒ add up all its "jobs."

4. Why RNNs are hard to train: vanishing/exploding gradients

Expand $\delta_t$ back to $\delta_T$ : gradients from far-future losses get multiplied by a product of Jacobians: $\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \operatorname{diag}(1-h_k^2)\,W_{hh}$

5. Worked example — tiny scalar RNN by hand

Let everything be scalar: $w_{hh}=0.5,\; w_{xh}=1,\; b_h=0$ , inputs $x_1=1, x_2=1$ , $h_0=0$ .

Forward.

$a_1 = 0.5\cdot0 + 1\cdot1 = 1 \Rightarrow h_1=\tanh(1)=0.7616$ . Why? Plug into update rule.
$a_2 = 0.5\cdot0.7616 + 1 = 1.3808 \Rightarrow h_2=\tanh(1.3808)=0.8811$ .

Loss. Suppose $L=\tfrac12(h_2 - 1)^2$ (target 1 at final step). $\dfrac{\partial L}{\partial h_2}=h_2-1=-0.1189$ .

Backward (BPTT).

$\delta_2 = -0.1189$ .
$\dfrac{\partial h_2}{\partial h_1}=(1-h_2^2)\,w_{hh}=(1-0.7763)\cdot0.5=0.1118$ . Why? $\tanh'$ times $w_{hh}$ .
$\delta_1 = 0 + 0.1118\cdot(-0.1189) = -0.0133$ . Why 0 first term? No output loss at step 1.

Gradient w.r.t. $w_{hh}$ (sum over steps, $h_{t-1}$ as the "input"): $\frac{\partial L}{\partial w_{hh}} = (1-h_1^2)\delta_1 h_0 + (1-h_2^2)\delta_2 h_1$ $= (0.4200)(-0.0133)(0) + (0.2237)(-0.1189)(0.7616) = -0.02025$ Why the two terms? Weight $w_{hh}$ was used at step 1 (multiplying $h_0$ ) and step 2 (multiplying $h_1$ ). Sum both.

Task: detect "airspeed increasing for 3 consecutive samples." A feed-forward net over a fixed window of 3 must learn separate weights for slot-1, slot-2, slot-3. An RNN learns one comparison rule and applies it as it walks the stream — so it also works for a 5-sample window with no retraining. Why this step matters: it shows weight sharing = a prior that "the physics is the same at every instant."

Common mistake Steel-manning the common errors

"BPTT is a different algorithm from backprop." Why it feels right: it has its own scary name. Fix: it is ordinary backprop — just applied to the unrolled graph. The only twist is that the same weight appears many times, so its gradient is a sum.

"Just make $h_t = W_{hh}h_{t-1}+\dots$ (no tanh) — simpler!" Why it feels right: linear is easy to analyse. Fix: a linear recurrence has zero nonlinearity — the whole network collapses to one linear map, and gradients are pure $\lambda^{T}$ , guaranteeing explode/vanish with no way to gate.

"Truncated BPTT loses no information." Why it feels right: you still backprop through recent steps. Fix: truncating at $k$ steps sets long-range gradients to zero — you can't learn dependencies longer than $k$ . It's a compute trade-off, not free.

" $\partial h_{t+1}/\partial h_t = W_{hh}$ ." Why it feels right: $W_{hh}$ is the recurrent matrix. Fix: you forgot the $\tanh$ derivative — it's $\operatorname{diag}(1-h_{t+1}^2)W_{hh}$ .

Recall Feynman: explain it to a 12-year-old

Imagine reading a story one word at a time and keeping a tiny sticky note in your head that says "what's happened so far." Each new word, you update the sticky note. That sticky note is the hidden state. When you get the ending wrong, you go backwards through the whole story figuring out which earlier words nudged you off — and you use the same reading rules the whole time, so you add up all the corrections. That backward-through-the-story blaming is BPTT. Trouble: after a very long story your "blame signal" fades to nothing (vanishing gradient), so the net forgets the beginning.

Flashcards

What does the hidden state

h_t

represent?

A running summary/memory of all inputs seen up to time

t

, updated each step from

h_{t-1}

and

x_t

Write the RNN hidden-state update.

h_t=\tanh(W_{hh}h_{t-1}+W_{xh}x_t+b_h)

Why are RNN weights shared across time?

To handle variable-length sequences and encode time-invariance (same rule every step); one param, many uses.

What is BPTT in one sentence?

Ordinary backprop applied to the RNN unrolled over time, where shared weights get their gradients summed over all steps.

Why does

h_t

receive gradient from two sources?

It affects the loss directly via output

y_t

and indirectly via the next state

h_{t+1}

; multivariate chain rule sums the paths.

What is the recurrent Jacobian

\partial h_{t+1}/\partial h_t

\operatorname{diag}(1-h_{t+1}^2)\,W_{hh}

Why do gradients vanish/explode in RNNs?

Backprop multiplies many similar Jacobians; behaves like

\lambda^{T-t}

, so

\lambda<1

vanishes and

\lambda>1

explodes.

Two standard fixes for exploding/vanishing gradients?

Gradient clipping (explode) and gated cells LSTM/GRU with near-identity memory path (vanish).

Why use tanh instead of a plain linear recurrence?

Linear collapses the net to one linear map and can't gate memory; tanh bounds the state and adds nonlinearity.

What does truncated BPTT sacrifice?

Gradients (and thus learnable dependencies) longer than the truncation window are set to zero.

Connections

Feed-forward Neural Networks — RNN unrolled = deep tied-weight feed-forward net.
Backpropagation — BPTT is backprop on the unrolled graph.
LSTM and GRU — gated fixes for vanishing gradients.
Vanishing and Exploding Gradients
Time Series Forecasting — flight-data / sensor prediction.
Kalman Filter — classical recursive state estimator; conceptual cousin of learned hidden state.
Sequence Modeling in Flight Data

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek normal neural network har input ko alag-alag dekhta hai — usko yaad nahi rehta ki pehle kya aaya tha. Lekin sequence data (jaise flight ka airspeed, attitude time ke saath badalta hai) mein history matter karti hai. Isiliye RNN ek chhota sa hidden state $h_t$ rakhta hai — samjho ek "sticky note" jo abhi tak ki poori memory ka summary hai. Har time step par woh sticky note update hota hai: naya input $x_t$ aur purana $h_{t-1}$ milाके naya $h_t = \tanh(W_{hh}h_{t-1}+W_{xh}x_t+b)$ . Aur important baat — wahi weights har step par use hote hain (weight sharing), isliye sequence chhoti ho ya lambi, same rule chalega.

Ab training. Jab output galat aata hai, humein weights update karne ke liye gradient chahiye. Yahi hai BPTT (Backpropagation Through Time). Ismein kuch naya nahi hai — ye wahi purana backprop hai, bas RNN ko time ke along "unroll" karke ek lambe deep network ki tarah treat karte hain. Twist sirf itna: kyunki same weight bar-bar use hua, uska gradient sabhi steps ka sum hota hai. Mantra yaad rakho: "Same weight, sum the blame."

Ek dikkat: agar sequence bahut lambi ho, to gradient bar-bar $W_{hh}$ aur $\tanh'$ se multiply hota hai, aur ye product $\lambda^{T}$ ki tarah behave karta hai. Agar $\lambda<1$ to gradient vanish ho jaata hai (net shuruaat bhool jaata hai), agar $\lambda>1$ to explode ho jaata hai (training phat jaati hai). Iske solution: gradient clipping, aur LSTM/GRU jaise gated cells jo memory ka ek seedha (identity jaisa) rasta banate hain.

Aerospace mein ye kyun useful hai? Kyunki flight data pura temporal hota hai. RNN se aap next state predict kar sakte ho, ya anomaly detect kar sakte ho — sirf abhi ka snapshot nahi, balki poori history dekh kar. Yehi RNN ki asli taakat hai.

Recurrent neural networks — hidden state, BPTT

1. What is the hidden state? (WHAT)

2. Unrolling the network (HOW to think about it)

3. Deriving BPTT from first principles

4. Why RNNs are hard to train: vanishing/exploding gradients

5. Worked example — tiny scalar RNN by hand

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Machine Learning (Aerospace Applications)

Connections