Exercises — Backpropagation through time

The shared weight is $==W_{hh}==$. In fact $W_{xh}$ and $b_h$ are shared across steps too, but $W_{hh}$ is the one that links step to step (it multiplies the previous hidden state). Because the same $W_{hh}$ is used at $t=1,2,\dots ,T$ (recall $T$ is the sequence length), a tiny change in it ripples into every hidden state, so its total gradient is a sum of contributions from all time steps: $\frac{\partial L}{\partial W_{hh}}={\sum }_{t=1}^T\frac{\partial L_t}{\partial W_{hh}}.$ Ordinary layer-by-layer backprop assumes each weight lives in one place; BPTT is exactly the bookkeeping for a weight that lives in many places at once.

• (a) → (ii): ${\delta }_t=\frac{\partial L}{\partial h_t}$ is the gradient at the output side of the $\tanh$.
• (b) → (iii): ${\hat{\delta }}_t={\delta }_t\odot (1-h_t^2)$ is ${\delta }_t$ pushed back through the $\tanh$ to its input side.
• (c) → (i): $1-h_t^2$ (our shorthand $s_t$) is the derivative of $\tanh$ written using its own output $h_t=\tanh (a_t)$ (where $a_t$ is the pre-activation defined above), since $\frac{d}{dz}\tanh z=1-{\tanh }^{2}z$.

• $\text{diag}(1-h_{t+1}^2)$ comes from the $\tanh$ nonlinearity — its slope at step $t+1$. "diag" means we place these slopes on the diagonal of a matrix because $\tanh$ acts element-by-element.
• $W_{hh}$ comes from the linear part, the pre-activation $a_{t+1}=W_{hh}{h}_t+W_{xh}{x}_{t+1}+b_h$ (see the symbol list); differentiating this linear term with respect to $h_t$ gives $W_{hh}$.

WHAT/WHY: just run the forward recurrence; each $h$ needs the previous $h$ plus this step's input. Recall $a_t=W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h$ is the pre-activation. $a_1=0.4\cdot 0+1.0\cdot 1.0=1.0,h_1=\tanh (1.0)=0.7616$ $a_2=0.4\cdot 0.7616+1.0\cdot 1.0=1.3046,h_2=\tanh (1.3046)=0.8632$

At the last step there is no future to receive gradient from, so ${\delta }_2$ is just the local loss slope ${\ell }_2$: ${\delta }_2={\ell }_2=\frac{\partial L_2}{\partial h_2}=h_2-{\hat{y}}_2=0.8632-0.9=-\mathrm{0.0368.}$

WHY this formula: $h_1$ affects the loss two ways — directly through $L_1$, and indirectly by feeding $h_2$. The second term is exactly $\frac{\partial h_2}{\partial h_1}{\delta }_2$. ${\ell }_1=\frac{\partial L_1}{\partial h_1}=h_1-{\hat{y}}_1=0.7616-0.6=0.1616$ $1-h_2^2=1-0.8632^2=0.2549$ ${\delta }_1=0.1616+0.2549\cdot 0.4\cdot (-0.0368)=0.1616-0.00375=0.1578$

${\hat{\delta }}_2={\delta }_2(1-h_2^2)=-0.0368\cdot 0.2549=-0.009380$ ${\hat{\delta }}_1={\delta }_1(1-h_1^2)=0.1578\cdot (1-0.7616^2)=0.1578\cdot 0.4200=0.06628$ $\frac{\partial L}{\partial W_{hh}}=0.06628\cdot 0+(-0.009380)\cdot 0.7616=-0.007144$ Since $h_0=0$, only the $t=2$ term survives.

Each backward step multiplies by $(1-h^2)W_{hh}=0.5\cdot 0.9=0.45$. After 10 steps: $0.45^{10}=3.405\times 10^{-4}.$ The factor is far below 1, so the signal from 10 steps ago is crushed — this is vanishing gradient. See 3.5.03-Vanishing-and-Exploding-Gradients.

Per-step factor $=0.9\cdot 3.0=2.7$. After 8 steps: $2.7^8=2.824\times 10^3.$ The gradient is amplified thousands of times — exploding gradient.

When $\tanh$ saturates, $h_t\to \pm 1$, so its slope $s_t=1-h_t^2\to 0$. Every backward hop then multiplies by a near-zero number, so gradients from far away die even faster than $W_{hh}$ alone would predict. An LSTM sidesteps this with an additive cell state whose gradient path multiplies by values near 1 (the forget gate) instead of a shrinking $\tanh$ slope, keeping long-range signal alive.

Per-step factor $=(1-h^2)W_{hh}=0.5\cdot (-0.9)=-0.45$. Its magnitude ($0.45$) still vanishes, but the sign alternates: after $k$ steps the factor is $(-0.45{)}^k$, which is positive for even $k$ and negative for odd $k$. $(-0.45{)}^1=-0.45,(-0.45{)}^2=+0.2025,(-0.45{)}^3=-0.091,\dots$ Why it matters: when the total gradient $\partial L/\partial W_{hh}={\sum }_t{\hat{\delta }}_t{h}_{t-1}$ collects contributions of alternating sign, some terms cancel each other. Two sequences with almost identical magnitudes can end up with very different — even near-zero — total gradients purely because of these sign flips. A magnitude-only analysis ($|W_{hh}|$) would completely miss this cancellation. (For $|W_{hh}|>1$ the same alternating sign instead amplifies with flipping direction, making updates oscillate.)

1. Forward pass — compute and store every $h_1,\dots ,h_T$ and every loss $L_1,\dots ,L_T$. Why store them? The backward pass needs each $h_t$ twice: once for the slope $s_t=1-h_t^2$ and once as the incoming state $h_{t-1}$ in the weight sum. We trade memory (holding all $T$ hidden states) for the ability to compute exact gradients — without them we'd have to recompute the forward pass repeatedly, which is far more expensive. This memory-for-compute trade is exactly why long sequences later force truncation (Exercise 5.1).

2. Backward pass — set ${\delta }_T={\ell }_T=\partial L_T/\partial h_T$ (no future term, since $h_T$ has no successor), then for $t=T-1$ down to $1$ apply the recursion below. Why backward and why this shape? Gradient at $h_t$ has two sources — the local loss ${\ell }_t$ and everything downstream, which is already summarised in ${\delta }_{t+1}$. Working right→left lets us reuse ${\delta }_{t+1}$ instead of re-deriving the whole future each time. The one-step Jacobian is $\frac{\partial h_{t+1}}{\partial h_t}=\text{diag}(s_{t+1})W_{hh},s_{t+1}=1-h_{t+1}^2.$ To send the gradient vector ${\delta }_{t+1}$ backward through this map we multiply by its transpose (a gradient is pulled back by the transpose of the forward Jacobian — that is what "backprop" means). Transposing $\text{diag}(s_{t+1})W_{hh}$ gives $W_{hh}^T\text{diag}(s_{t+1})$ (the diagonal is its own transpose, and the order of the two factors flips). Hence, written consistently as a pull-back: ${\delta }_t={\ell }_t+W_{hh}^T\text{diag}(s_{t+1}){\delta }_{t+1}.$ In the 1-D exercises every matrix is a single number, so transpose does nothing and this reads ${\delta }_t={\ell }_t+s_{t+1}{W}_{hh}{\delta }_{t+1}$ — exactly what we used in Exercise 2.3.

3. Accumulate gradients — form the pre-activation gradient ${\hat{\delta }}_t={\delta }_t\odot s_t$ and sum over all steps, for every shared parameter: $\frac{\partial L}{\partial W_{hh}}={\sum }_{t=1}^T{\hat{\delta }}_t{h}_{t-1}^T,\frac{\partial L}{\partial W_{xh}}={\sum }_{t=1}^T{\hat{\delta }}_t{x}_t^T,\frac{\partial L}{\partial b_h}={\sum }_{t=1}^T{\hat{\delta }}_t.$ The bias gradient is just ${\sum }_t{\hat{\delta }}_t$ because $b_h$ enters the pre-activation with a constant multiplier of $1$ ($\partial a_t/\partial b_h=1$), so no extra $h_{t-1}$ or $x_t$ factor appears.

Forward. $h_1=\tanh (0.5)=0.4621,h_2=\tanh (0.5\cdot 0.4621+0.5)=\tanh (0.7311)=0.6241$ $h_3=\tanh (0.5\cdot 0.6241+0.5)=\tanh (0.8121)=0.6710$ Local loss slopes ${\ell }_t=\partial L_t/\partial h_t=h_t-0=h_t$ (targets are 0): ${\ell }_1=0.4621,{\ell }_2=0.6241,{\ell }_3=0.6710$. $\tanh$ slopes $s_t=1-h_t^2$: $s_1=0.7865,s_2=0.6105,s_3=0.5498$. Backward (using ${\delta }_t={\ell }_t+s_{t+1}{W}_{hh}{\delta }_{t+1}$): ${\delta }_3={\ell }_3=0.6710$ ${\delta }_2={\ell }_2+s_3{W}_{hh}{\delta }_3=0.6241+0.5498\cdot 0.5\cdot 0.6710=0.6241+0.1845=0.8086$ ${\delta }_1={\ell }_1+s_2{W}_{hh}{\delta }_2=0.4621+0.6105\cdot 0.5\cdot 0.8086=0.4621+0.2468=0.7089$ Pre-activation ${\hat{\delta }}_t={\delta }_t{s}_t$: ${\hat{\delta }}_1=0.7089\cdot 0.7865=0.5575,{\hat{\delta }}_2=0.8086\cdot 0.6105=0.4937,{\hat{\delta }}_3=0.6710\cdot 0.5498=0.3689$ Accumulate the weight gradient ${\sum }_{t=1}^3{\hat{\delta }}_t{h}_{t-1}$ (with $h_0=0$, so the $t=1$ term drops): $\frac{\partial L}{\partial W_{hh}}={\hat{\delta }}_1\cdot h_0+{\hat{\delta }}_2\cdot h_1+{\hat{\delta }}_3\cdot h_2=0.5575\cdot 0+0.4937\cdot 0.4621+0.3689\cdot 0.6241$ $=0+0.2281+0.2302=\boxed{0.4583}$

With truncation, ${\delta }_t={\ell }_t=h_t$, so ${\hat{\delta }}_t=h_t{s}_t$: ${\hat{\delta }}_1=0.4621\cdot 0.7865=0.3635,{\hat{\delta }}_2=0.6241\cdot 0.6105=0.3810,{\hat{\delta }}_3=0.6710\cdot 0.5498=0.3689$ $\frac{\partial L}{\partial W_{hh}}={\hat{\delta }}_2{h}_1+{\hat{\delta }}_3{h}_2=0.3810\cdot 0.4621+0.3689\cdot 0.6241=0.1761+0.2302=0.4063$ Truncated $0.4063$ vs full $0.4583$: the truncated gradient is smaller because it drops the long-range terms that push ${\delta }_2,{\delta }_1$ up. Truncation trades some accuracy (missing long dependencies) for cheaper, more stable computation — the standard practical choice for long sequences, and the direct payoff of the memory-vs-compute trade noted in Exercise 4.1.

The symptom is vanishing gradients (the per-step product $(1-h_t^2)W_{hh}<1$ compounds to $\sim 10^{-9}$).

• 3.5.04-Long-Short-Term-Memory-LSTM — introduces an additive cell state with gated, near-identity gradient flow, so the multiplicative shrinkage of $\tanh$ slopes is bypassed.
• 3.5.05-Gated-Recurrent-Unit-GRU — a lighter gated design achieving the same protected gradient highway with fewer parameters. Both replace the fragile repeated-multiplication path of plain BPTT with a controllable one. (Gradient clipping from 3.5.03-Vanishing-and-Exploding-Gradients fixes exploding, not vanishing.)