Backpropagation through time

---

The Problem: Gradients in Recurrent Networks

A standard RNN has the form: $h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$ $y_t = W_{hy} h_t + b_y$

We want to minimize a loss over a sequence: $L = \sum_{t=1}^{T} L_t$

where $L_t$ is the loss at time step $t$ (e.g., cross-entropy for classification).

Challenge: To update $W_{hh}$, we need $\frac{\partial L}{\partial W_{hh}}$. But $W_{hh}$ is used at every time step, so changes to it affect all future hidden states. The gradient must account for all paths through time.

---

Derivation from First Principles

Step 1: Unroll the Computation Graph

Write the forward pass explicitly for $T$ time steps:

$$\begin{align} h_1 &= \tanh(W_{hh} h_0 + W_{xh} x_1 + b_h) \\ h_2 &= \tanh(W_{hh} h_1 + W_{xh} x_2 + b_h) \\ &\vdots \\ h_T &= \tanh(W_{hh} h_{T-1} + W_{xh} x_T + b_h) \end{align}$$

Each $h_t$ depends on $h_{t-1}$, which depends on $h_{t-2}$, and so on. This creates a chain of dependencies.

Step 2: Apply the Chain Rule Across Time

To find $\frac{\partial L}{\partial W_{hh}}$, sum contributions from all time steps: $\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial W_{hh}}$

Why this step? $W_{hh}$ appears in the computation of $h_1, h_2, \dots, h_T$, so the total gradient is the sum of partial effects at each time.

Now, $L_t$ depends on $y_t$, which depends on $h_t$. But $h_t$ depends on $h_{t-1}, h_{t-2}, \dots, h_1$, all of which involve $W_{hh}$:

$\frac{\partial L_t}{\partial W_{hh}} = \frac{\partial L_t}{\partial h_t} \cdot \frac{\partial h_t}{\partial W_{hh}}$

Here's the key: $\frac{\partial h_t}{\partial W_{hh}}$ must account for the direct effect (at time $t$) and indirect effects (through earlier hidden states):

$\frac{\partial h_t}{\partial W_{hh}} = \frac{\partial h_t}{\partial h_{t-1}} \cdot \frac{\partial h_{t-1}}{\partial W_{hh}} + \frac{\partial h_t}{\partial W_{hh}}\bigg|_{\text{direct}}$

Why this step? By the multivariate chain rule, the gradient flows through $h_{t-1}$ (indirect path) and also directly affects $h_t$ (direct path).

Step 3: Recursive Gradient Flow

Define $\delta_t = \frac{\partial L}{\partial h_t}$ (the gradient w.r.t. the post-activation hidden state). Then: $\delta_t = \frac{\partial L_t}{\partial h_t} + \frac{\partial L}{\partial h_{t+1}} \cdot \frac{\partial h_{t+1}}{\partial h_t} = \frac{\partial L_t}{\partial h_t} + \delta_{t+1} \cdot \frac{\partial h_{t+1}}{\partial h_t}$

Why this step? Gradient at $h_t$ has two sources: the immediate loss $L_t$ and the future loss propagated back from $h_{t+1}$.

For the RNN update rule, let $a_{t+1} = W_{hh} h_t + W_{xh} x_{t+1} + b_h$ be the pre-activation, so $h_{t+1} = \tanh(a_{t+1})$: $\frac{\partial h_{t+1}}{\partial h_t} = \text{diag}(1 - \tanh^2(a_{t+1})) \cdot W_{hh} = \text{diag}(1 - h_{t+1}^2) \cdot W_{hh}$

Why this step? Derivative of $\tanh$ is $1 - \tanh^2$, and $h_{t+1}$ is computed as $\tanh(\text{linear combination involving } h_t)$. The Jacobian is diagonal (elementwise nonlinearity) times $W_{hh}$.

Thus: $\delta_t = \frac{\partial L_t}{\partial h_t} + W_{hh}^T \cdot \text{diag}(1 - h_{t+1}^2) \cdot \delta_{t+1}$

Step 4: Gradient w.r.t. the Weights (don't forget the $\tanh$ factor!)

The weight $W_{hh}$ enters the pre-activation $a_t = W_{hh} h_{t-1} + \dots$, and $h_t = \tanh(a_t)$. So the local gradient w.r.t. $W_{hh}$ at step $t$ must include the $\tanh$ derivative: $\frac{\partial L_t}{\partial W_{hh}} = \left(\delta_t \odot (1 - h_t^2)\right) h_{t-1}^T$

Why this step? $\delta_t = \partial L/\partial h_t$ is post-activation. To reach $W_{hh}$ we pass through the $\tanh$: $\partial h_t / \partial a_t = \text{diag}(1 - h_t^2)$, and $\partial a_t / \partial W_{hh} = h_{t-1}^T$. Skipping the $(1-h_t^2)$ term is a common bug.

Defining the pre-activation gradient $\hat{\delta}_t = \delta_t \odot (1 - h_t^2)$, the accumulated gradient is:

The BPTT Algorithm:

1. Forward pass: Compute $h_1, \dots, h_T$ and losses $L_1, \dots, L_T$.
2. Backward pass: Initialize $\delta_T$, then for $t = T-1$ down to $1$, compute $\delta_t$ recursively.
3. Accumulate gradients: $\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \big(\delta_t \odot (1 - h_t^2)\big)\, h_{t-1}^T = \sum_{t=1}^{T} \hat{\delta}_t\, h_{t-1}^T$

---

---

Visualizing the Unrolled Network

The diagram shows:

• Forward pass (left to right): hidden states $h_t$ computed sequentially.
• Backward pass (right to left): gradients $\delta_t$ flow backward through time.
• Shared weights $W_{hh}$: gradient contributions from all time steps are summed.

---

---

Worked Example 1: Simple 2-Step RNN

Setup: 1-dimensional RNN, $T=2$, $W_{hh} = 0.5$, $W_{xh} = 1.0$, $\tanh$ activation, $h_0 = 0$.

Input: $x_1 = 1.0, x_2 = 0.5$. Target outputs: $\hat{y}_1 = 0.5, \hat{y}_2 = 0.3$. Loss: $L_t = \frac{1}{2}(y_t - \hat{y}_t)^2$ with $y_t = h_t$ (identity output for simplicity).

Forward pass: $h_1 = \tanh(0.5 \cdot 0 + 1.0 \cdot 1.0) = \tanh(1.0) \approx 0.7616$ $y_1 = h_1 = 0.7616, \quad L_1 = \frac{1}{2}(0.7616 - 0.5)^2 = 0.0342$

$h_2 = \tanh(0.5 \cdot 0.7616 + 1.0 \cdot 0.5) = \tanh(0.8808) \approx 0.7068$ $y_2 = h_2 = 0.7068, \quad L_2 = \frac{1}{2}(0.7068 - 0.3)^2 = 0.0827$

Why these steps? We apply the RNN equations directly. Each hidden state depends on the previous one and the current input.

Backward pass (compute post-activation $\delta_t$):

At $t=2$: $\delta_2 = \frac{\partial L_2}{\partial h_2} = h_2 - 0.3 = 0.4068$

Why this step? Derivative of squared error w.r.t. $h_2 = y_2$.

At $t=1$: $\frac{\partial L_1}{\partial h_1} = h_1 - 0.5 = 0.2616$

$\frac{\partial h_2}{\partial h_1} = (1 - h_2^2) \cdot W_{hh} = (1 - 0.7068^2) \cdot 0.5 = 0.5006 \cdot 0.5 = 0.2503$

Why this step? Derivative of $\tanh$ at the pre-activation is $1 - h_2^2$, and the linear part contributes $W_{hh}$.

$\delta_1 = 0.2616 + 0.4068 \cdot 0.2503 = 0.3634$

Gradient — include the $\tanh$ factor $(1 - h_t^2)$!

Pre-activation gradients: $\hat{\delta}_2 = \delta_2 \cdot (1 - h_2^2) = 0.4068 \cdot (1 - 0.7068^2) = 0.4068 \cdot 0.5006 = 0.2036$ $\hat{\delta}_1 = \delta_1 \cdot (1 - h_1^2) = 0.3634 \cdot (1 - 0.7616^2) = 0.3634 \cdot 0.4200 = 0.1526$

$\frac{\partial L}{\partial W_{hh}} = \hat{\delta}_1 \cdot h_0 + \hat{\delta}_2 \cdot h_1 = 0.1526 \cdot 0 + 0.2036 \cdot 0.7616 \approx 0.1551$

Why this step? Each time step contributes $\hat{\delta}_t \cdot h_{t-1}$ (pre-activation gradient times the incoming hidden state). The $t=2$ term is $0.4068 \cdot (1 - 0.7068^2) \cdot 0.7616 \approx 0.155$ — not $0.310$. Forgetting the $(1 - h_t^2)$ factor doubles the answer here; that is the classic error.

---

Worked Example 2: Gradient Vanishing in Long Sequences

Setup: $T = 10$, $W_{hh} = 0.9$, $x_t = 1.0$ for all $t$, $h_0 = 0$.

Forward pass (computed carefully): $h_1 = \tanh(0.9 \cdot 0 + 1.0) = \tanh(1.0) \approx 0.7616$ $h_2 = \tanh(0.9 \cdot 0.7616 + 1.0) = \tanh(1.6854) \approx 0.9331$ $h_3 = \tanh(0.9 \cdot 0.9331 + 1.0) = \tanh(1.8398) \approx 0.9506$ $h_4 = \tanh(0.9 \cdot 0.9506 + 1.0) = \tanh(1.8555) \approx 0.9522$ $h_5 \approx \tanh(0.9 \cdot 0.9522 + 1.0) = \tanh(1.8570) \approx 0.9524, \dots$

Why these steps? Hidden states saturate toward a fixed point near $0.952$ because $\tanh$ squashes large values.

Backward pass: Suppose $\delta_{10} = 1.0$. The recurrence factor is: $\frac{\partial h_{t+1}}{\partial h_t} = (1 - h_{t+1}^2)\cdot W_{hh}$

Using $h \approx 0.952$ in the saturated region, $1 - h^2 \approx 1 - 0.906 = 0.094$, so each factor is $\approx 0.094 \cdot 0.9 \approx 0.0846$.

At $t=9$: $\delta_9 = 1.0 \cdot 0.0846 \approx 0.0846$ At $t=8$: $\delta_8 = 0.0846 \cdot 0.0846 \approx 0.00716$

Why this step? Each backward step multiplies by $(1 - h_{t+1}^2)\cdot W_{hh} \approx 0.085 < 1$. Since $h_t$ saturates near $0.952$, $(1 - h_t^2)$ is tiny, causing exponential decay.

After 9 steps: $\delta_1 \approx 0.0846^{9} \approx 2.4\times 10^{-10}$. Vanishing gradient: information from time 10 barely reaches time 1.

Why this matters: Long-range dependencies cannot be learned. This motivates LSTMs and GRUs.

---

---

---

---

Truncated BPTT: Practical Compromise

For long sequences, storing all hidden states and computing exact gradients is memory-intensive. Truncated BPTT splits the sequence into chunks of length $k$:

1. Forward pass for $k$ steps.
2. Backward pass for those $k$ steps only.
3. Detach the gradient and continue forward for the next $k$ steps.

Trade-off: Gradients cannot propagate beyond $k$ steps, limiting the model's ability to learn long-range dependencies. But it reduces memory from $O(T)$ to $O(k)$.

Formula: $\frac{\partial L}{\partial W_{hh}} \approx \sum_{t=t_0}^{t_0 + k} \hat{\delta}_t\, h_{t-1}^T$

where $t_0$ is the start of the current chunk.

---

---

Recall Explain to a 12-Year-Old

Imagine you're learning to play a song on the piano. The song has 100 notes, and you want to get better.

Normal learning (feedforward): You play one note, your teacher corrects you, you move to the next note. Each note is independent. Recurrent learning (RNN): Each note you play depends on the previous notes. If you mess up note 50, it affects notes 51, 52, ..., 100. To learn, you need to figure out how each note contributed to the final mistake.

Backpropagation through time: After playing all 100 notes, you "rewind" the song and ask: "How did note 1 affect the final sound?" You trace backward: note 100 depended on 99, which depended on 98, .., back to note 1. You adjust how you played note 1 based on the entire chain of effects.

The problem: the further back you go, the fuzzier your memory of what happened (vanishing gradients). That's why RNNs struggle with long songs (sequences). LSTMs are like taking better notes so you remember earlier parts clearly.

---

---

Connections

• 3.5.01-Recurrent-Neural-Networks-RNNs: BPTT is the training algorithm for RNNs.
• 3.5.03-Vanishing-and-Exploding-Gradients: BPTT exposes the vanishing/exploding gradient problem in deep sequences.
• 3.5.04-Long-Short-Term-Memory-LSTM: LSTMs mitigate vanishing gradients by gating gradient flow.
• 3.5.05-Gated-Recurrent-Unit-GRU: GRUs simplify LSTMs while preserving long-term gradient flow.
• 2.2.03-Backpropagation-Algorithm: BPTT is backprop applied to the unrolled computational graph.
• 3.5.07-Bidirectional-RNNs: BPTT extends to bidirectional RNNs by backpropping through both directions.

---

Summary

Backpropagation through time

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RNN mein ek hi weight matrix $W_{hh}$ har time step pe baar baar use hota hai — yahi core cheez hai. Ab problem yeh hai ki jab hum gradient calculate karte hain training ke liye, tab yeh shared weight future ke saare hidden states ko affect karta hai. Toh idea yeh hai ki hum RNN ko "unroll" karte hain — matlab time ke har step ko ek alag layer ki tarah imagine karte hain, jaise ek bahut deep feedforward network ho jismein saari layers same weights share karti hain. Isiliye iska naam hai "Backpropagation Through Time" — kyunki gradient time steps ke through peeche flow karta hai. Simple words mein: temporal recurrence ko spatial depth ki tarah treat kar rahe hain.

Ab why-it-matters wala part. Chain rule apply karte waqt, kisi bhi time $t$ pe gradient ke do sources hote hain — ek immediate loss $L_t$ jo abhi ke output se aata hai, aur doosra future se aane wala gradient jo $h_{t+1}$ ke through back-propagate hota hai. Isko hum recursively likhte hain as $\delta_t$, aur har step pe $W_{hh}^T$ aur $\tanh$ ka derivative $(1 - h^2)$ multiply hota hai. Yeh jo repeated multiplication hai, wahi asli kahani hai — kyunki agar yeh factors baar-baar chhote hote gaye toh gradient vanish ho jaata hai (long-term memory nahi bachti), aur agar bade ho gaye toh explode ho jaata hai. Isi vanishing/exploding gradient problem ki wajah se aage chalke LSTM aur GRU jaise architectures invent huye.

Toh bhai, BPTT samajhna isliye zaroori hai kyunki yeh foundation hai — jab tak tumhe yeh clear nahi hoga ki gradient time ke through kaise flow karta hai aur kyun weak ya explode hota hai, tab tak tum yeh appreciate nahi kar paoge ki modern sequence models (LSTM, attention, transformers) yeh saari problems kaise solve karte hain. Ek chhoti si tip: notation mein confuse mat hona — $W_{hh}$ hamesha hidden-to-hidden hai, aur yahi wala matrix time ke across shared hota hai, isko baaki $W_{xh}$ aur $W_{hy}$ se alag rakhna.