Visual walkthrough — Backpropagation through time

Prerequisites we lean on: 3.5.01-Recurrent-Neural-Networks-RNNs (what an RNN is) and 2.2.03-Backpropagation-Algorithm (the chain rule as gradient flow). The payoff of this picture — why the backward arrows shrink or blow up — is 3.5.03-Vanishing-and-Exploding-Gradients, repaired later by 3.5.04-Long-Short-Term-Memory-LSTM and 3.5.05-Gated-Recurrent-Unit-GRU.

---

Step 1 — One cell, drawn as a machine

WHAT. An RNN is a single little machine that you feed one input at a time. At step $t$ it takes the input $x_t$ and its own memory from last step $h_{t-1}$, mixes them, squashes the mix, and produces a new memory $h_t$. Then a second, simpler stage reads that memory out into a prediction $y_t$.

WHY a picture first. Before any calculus, you must see the two arrows going in ($x_t$ and $h_{t-1}$), the memory $h_t$ produced, and the read-out arrow that turns $h_t$ into the prediction $y_t$. Every gradient later is just these same arrows, walked backward.

PICTURE.

Term by term, the memory update and the read-out are:

$h_t=\tanh (\underset{\text{memory mixer}}{\underbrace{W_{hh}}}\underset{\text{old memory}}{\underbrace{h_{t-1}}}+\underset{\text{input mixer}}{\underbrace{W_{xh}}}\underset{\text{new input}}{\underbrace{x_t}}+\underset{\text{bias}}{\underbrace{b_h}})$

$y_t=\underset{\text{read-out mixer}}{\underbrace{W_{hy}}}\underset{\text{new memory}}{\underbrace{h_t}}+\underset{\text{read-out bias}}{\underbrace{b_y}}$

• $y_t$ — the network's prediction at step $t$ (what it says), obtained by reading the memory $h_t$ out through $W_{hy}$.

---

Step 2 — Repeat the machine: time becomes depth

WHAT. To process a sequence of length $T$, we don't build $T$ machines. We use the same machine $T$ times, feeding its own memory back as next-step memory, and reading out a prediction $y_t$ at each step. If we "unroll" this loop and lay the copies side by side, it looks exactly like a deep feedforward network — one layer per time step.

WHY. Backprop is a recipe for feedforward graphs. By unrolling, we turn the loop into a graph backprop already understands. The one twist: every copy shares the same $W_{hh}$ (and the same $W_{xh}$, $W_{hy}$, $b_h$, $b_y$).

PICTURE.

$h_0\stackrel{x_1}{\to }{h}_1\stackrel{x_2}{\to }{h}_2\stackrel{x_3}{\to }\cdots \stackrel{x_T}{\to }{h}_T$

Read the yellow arrows as memory flowing forward. Each blue box is one call of the same cell from Step 1, and each drops a prediction $y_t$ downward. We compare every prediction $y_t$ to the target (the correct answer we wanted), which we write ${\hat{y}}_t$. The total loss adds up a small loss at each step:

$L={\sum }_{t=1}^T{L}_t{L}_t=\frac{1}{2}(y_t-{\hat{y}}_t{)}^2\text{(one common choice)}$

• $y_t$ — the prediction the network made (from Step 1's read-out).
• ${\hat{y}}_t$ — the target: the correct value we wanted at step $t$. The hat means "the true label", not "estimate" here — read $y_t-{\hat{y}}_t$ as prediction minus truth, i.e. the error.
• $L_t$ — the mistake made at step $t$ (half the squared error).
• The sum $\sum$ — total mistake over the whole sequence.

---

Step 3 — Read-out first: how $h_t$ reaches the loss

WHAT. Before blame can flow through time, it has to enter at each step from the loss. The loss $L_t$ sees the prediction $y_t$; the prediction sees the memory $h_t$ through $W_{hy}$. So the "local" blame on $h_t$ is a two-link chain: loss → prediction → memory.

WHY. The recursion in the next step uses a term $\frac{\partial L_t}{\partial h_t}$. It is not magic — it is exactly this read-out chain, and we compute it now so nothing is assumed later.

PICTURE.

$\frac{\partial L_t}{\partial h_t}=\underset{=y_t-{\hat{y}}_t}{\underbrace{\frac{\partial L_t}{\partial y_t}}}\cdot \underset{=W_{hy}}{\underbrace{\frac{\partial y_t}{\partial h_t}}}=W_{hy}^T(y_t-{\hat{y}}_t)$

• $\frac{\partial L_t}{\partial y_t}=y_t-{\hat{y}}_t$ — differentiate $\frac{1}{2}(y_t-{\hat{y}}_t{)}^2$: the prediction's error.
• $\frac{\partial y_t}{\partial h_t}=W_{hy}$ — since $y_t=W_{hy}{h}_t+b_y$, a nudge in $h_t$ moves $y_t$ by $W_{hy}$.

While the read-out weights are in view, their own gradients drop out of the very same chain:

$\frac{\partial L}{\partial W_{hy}}={\sum }_{t=1}^T(y_t-{\hat{y}}_t)h_t^T,\frac{\partial L}{\partial b_y}={\sum }_{t=1}^T(y_t-{\hat{y}}_t)$

• $W_{hy}$ multiplied $h_t$, so its gradient carries an $h_t^T$; $b_y$ was just added, so its gradient is the bare error.

---

Step 4 — Send back one error signal: define ${\delta }_t$

WHAT. Backprop asks, at each memory $h_t$: "if you nudged $h_t$ up a little, how much would the total loss $L$ change?" We name that number

${\delta }_t=\frac{\partial L}{\partial h_t}.$

WHY a name. ${\delta }_t$ is the "blame" landing on memory $h_t$. Instead of recomputing it from scratch each time, we will show it can be built from the next step's blame — that recursion is the heart of BPTT.

PICTURE.

The pink arrows are ${\delta }_t$ flowing right to left — opposite the yellow forward arrows.

• ${\delta }_t$ — total blame on memory at time $t$.
• $\partial$ — "tiny change in". $\frac{\partial L}{\partial h_t}$ reads: tiny change in $L$ per tiny change in $h_t$.

Blame on $h_t$ has exactly two sources (look at the two pink arrows meeting at $h_t$ in the figure):

${\delta }_t=\underset{\text{local: through the read-out}}{\underbrace{\frac{\partial L_t}{\partial h_t}}}+\underset{\text{from the future}}{\underbrace{{\delta }_{t+1}\cdot \frac{\partial h_{t+1}}{\partial h_t}}}$

• First term: the mistake $h_t$ causes right now through its own prediction $y_t$ — exactly the read-out chain from Step 3, $\frac{\partial L_t}{\partial h_t}=W_{hy}^T(y_t-{\hat{y}}_t)$.
• Second term: $h_t$ also fed $h_{t+1}$, so it is partly to blame for every future mistake. That future blame is ${\delta }_{t+1}$, passed back through the link $\frac{\partial h_{t+1}}{\partial h_t}$.

---

Step 5 — What is the link $\frac{\partial h_{t+1}}{\partial h_t}$?

WHAT. We need the number that tells us how a nudge in $h_t$ ripples into $h_{t+1}$. Recall $h_{t+1}=\tanh (a_{t+1})$ where $a_{t+1}=W_{hh}{h}_t+W_{xh}{x}_{t+1}+b_h$ is the pre-activation (the raw sum before squashing).

WHY the chain of two pieces. A nudge in $h_t$ first changes the raw sum $a_{t+1}$ (through $W_{hh}$), then $\tanh$ passes that change on with its own slope $1-{\tanh }^2=1-h_{t+1}^2$. Two gates in a row, so we multiply.

PICTURE.

$\frac{\partial h_{t+1}}{\partial h_t}=\underset{\text{slope of }\tanh }{\underbrace{(1-h_{t+1}^2)}}\cdot \underset{\text{memory mixer}}{\underbrace{W_{hh}}}$

• $(1-h_{t+1}^2)$ — how steep the S-curve is at this point. Look at the figure: near the flat ends of $\tanh$ this is almost $0$ (a nudge barely moves the output); near the steep middle it is close to $1$.
• $W_{hh}$ — the linear stretch inside.

Putting Steps 4 and 5 together, the backward recursion (in matrix form, $\text{diag}$ = put the vector on a diagonal, ${}^T$ = transpose) is:

$\boxed{{\delta }_t=W_{hy}^T(y_t-{\hat{y}}_t)+W_{hh}^T\text{diag}(1-h_{t+1}^2){\delta }_{t+1}}$

---

Step 6 — Cross the squash to reach the weight: ${\hat{\delta }}_t$

WHAT. ${\delta }_t$ is blame on the post-activation memory $h_t$. But $W_{hh}$ (and $W_{xh}$ and $b_h$) live before the squash, inside $a_t$. To send blame to them we must first push ${\delta }_t$ back through the $\tanh$, gaining one more slope factor.

WHY it's easy to forget. The picture makes it obvious: there is a $\tanh$ box sitting between $h_t$ and $a_t$. You cannot walk to $W_{hh}$ without passing that box, and passing it costs a $(1-h_t^2)$.

PICTURE.

Define the pre-activation gradient:

${\hat{\delta }}_t={\delta }_t\odot (1-h_t^2)$

• $\odot$ — elementwise multiply (multiply matching entries; here, in 1-D, ordinary multiply).
• $(1-h_t^2)$ — the same $\tanh$ slope, now for step $t$'s own memory.

---

Step 7 — Collect blame onto the shared weights (the sums!)

WHAT. Now we harvest. At each step, the pre-activation $a_t$ was built as $W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h$, so a nudge in $W_{hh}$ moves $a_t$ in proportion to $h_{t-1}$, a nudge in $W_{xh}$ in proportion to $x_t$, and a nudge in $b_h$ moves it directly. The blame each earns from step $t$ is ${\hat{\delta }}_t$ times its multiplier. Because all these parameters are shared, add these over all steps.

WHY the sum. Back in Step 2 we saw the parameters appear in every box. Each box's opinion is one term; the total gradient is their sum.

PICTURE.

$\boxed{\frac{\partial L}{\partial W_{hh}}=\sum _{t=1}^T{\hat{\delta }}_t{h}_{t-1}^T=\sum _{t=1}^T({\delta }_t\odot (1-h_t^2))h_{t-1}^T}$

The input weight and the memory bias follow the same logic — same ${\hat{\delta }}_t$, different multiplier ($x_t$ for $W_{xh}$, and $1$ for $b_h$):

$\frac{\partial L}{\partial W_{xh}}={\sum }_{t=1}^T{\hat{\delta }}_t{x}_t^T,\frac{\partial L}{\partial b_h}={\sum }_{t=1}^T{\hat{\delta }}_t$

Together with the read-out gradients from Step 3 ($\partial L/\partial W_{hy}$ and $\partial L/\partial b_y$), every parameter of the RNN now has its gradient. The recipe is complete.

---

Step 8 — The edge cases you must not trip on

WHAT. Three corners that break naive code.

PICTURE.

1. The last step $t=T$. There is no future, so ${\delta }_{t+1}$ does not exist. The recursion starts clean: ${\delta }_T=W_{hy}^T(y_T-{\hat{y}}_T)$ only. (Yellow box, right edge.)
2. The first step $t=1$. Its incoming memory is $h_0$, the initial state. In the weight sum the $t=1$ term is ${\hat{\delta }}_1{h}_0^T$. If $h_0=0$ (a common default), that whole term vanishes — not a bug, just zero times anything. (Note $b_h$'s $t=1$ term does not vanish: its multiplier is $1$, not $h_0$.)
3. Saturated $\tanh$. If a memory sits near $\pm 1$, then $1-h^2\approx 0$, so both ${\hat{\delta }}_t$ and the backward hop shrink toward $0$. Blame stops flowing — the network "forgets". (Pink arrow fading to nothing.)

Recall Quick self-checks

Where does the sum over $t$ come from? ::: $W_{hh}$ is shared across all time steps, so each step contributes one gradient term and we add them. What is the local term $\partial L_t/\partial h_t$, written out? ::: The read-out chain $W_{hy}^T(y_t-{\hat{y}}_t)$ — loss to prediction $y_t$ (error $y_t-{\hat{y}}_t$) times prediction to memory ($W_{hy}$). Why must you multiply by $(1-h_t^2)$ before touching $W_{hh}$? ::: Because $W_{hh}$ sits before the $\tanh$; you cross the squash to reach it, and crossing costs its slope $1-h_t^2$. What starts the backward recursion, and why? ::: ${\delta }_T=W_{hy}^T(y_T-{\hat{y}}_T)$, because step $T$ has no future step feeding blame back.

---

The one-picture summary

Yellow = forward memory flow, and each cell reads out a prediction $y_t$ compared to target ${\hat{y}}_t$. Pink = backward blame flow. Blame enters at each step through the read-out ($W_{hy}^T(y_t-{\hat{y}}_t)$), rides back through time multiplying by $(1-h^2)W_{hh}$ per hop, crosses each squash (gaining $1-h^2$) and drops a term into the shared bucket for every parameter: ${\hat{\delta }}_t{h}_{t-1}^T$ for $W_{hh}$, ${\hat{\delta }}_t{x}_t^T$ for $W_{xh}$, ${\hat{\delta }}_t$ for $b_h$. Empty the buckets = your gradients.

Recall Feynman retelling (say it out loud)

Picture one little machine that eats an input and its own last memory, mixes them, squashes with a tanh, and hands out a fresh memory — then a second little step reads that memory out into a prediction $y_t$. To handle a whole sentence, we run the same machine over and over and lay the runs in a row — now it's just a deep network where every layer is the same machine, and each layer makes a prediction we compare to the truth ${\hat{y}}_t$.

To train it, we walk backward. At each memory we ask, "how much are you to blame for the total mistake?" That blame, $\delta$, comes from two places: the mistake you made right here — which arrives through the read-out, error $y_t-{\hat{y}}_t$ times the read-out weight — plus all the future mistakes you caused by feeding the next memory, passed back through a link that is the tanh's steepness times the memory-mixing weight.

To finally fix each shared weight, at every step I push the blame through the tanh (that's the tiny $1-h^2$ factor people forget), multiply by whatever that weight multiplied on the way in (the old memory for $W_{hh}$, the input for $W_{xh}$, nothing for the bias), and toss the result in that weight's shared bucket. Because the weights were reused everywhere, I add every step's contribution. Those totals — plus the read-out weight and its bias — are the gradients. And notice: every backward step multiplies by that same $1-h^2$ times $W_{hh}$, so chain a lot of them and the signal either withers away or blows up, which is the famous headache the LSTM and GRU were invented to cure.

Next: watch that repeated multiplication run wild in 3.5.03-Vanishing-and-Exploding-Gradients, or read this whole page in Hinglish.