Worked examples — Recurrent neural networks — hidden state, BPTT

The scenario matrix

Every BPTT calculation is built from the same three moving parts:

1. the hidden state value $h_t$ (near $0$? near $\pm 1$? exactly $0$?),
2. the recurrent weight $w_{hh}$ (magnitude $<1$, $=1$, $>1$, or negative?),
3. the loss (one output at the end, or an output at every step?).

The table below lists every combination that behaves differently. The last column names the example that covers it.

We keep everything scalar (one number, not vectors) so you can check every digit by hand. For scalars the fearsome matrix transpose $W_{hh}^{\top }$ is just the number $w_{hh}$, and $\odot$ is ordinary multiplication.

The picture below plots $\tanh$ (blue) against its backward multiplier $1-{\tanh }^2$ (yellow) — read the legend inside the figure for which colour is which, and the labelled axes for the scale. Read it as your master key for every example: wherever blue is flat (saturated), yellow is near $0$ → blame dies; wherever blue is steep (near the origin), yellow is near $1$ → blame passes freely. We will point back to this shape in Ex 3.

Example 1 — Cell A, B, C: the healthy baseline

Forecast: Guess the sign first. $h_2=0.88$ is below the target $1$, so the loss wants $h_2$ bigger. Since $h_0=0$, the step-1 contribution is dead. Do you expect the final gradient positive or negative?

1. Forward, step 1. $a_1=0.5\cdot 0+1\cdot 1=1$, so $h_1=\tanh (1)=0.7616$. Why this step? The recipe's forward line — we cannot blame anything until we know the actual states.
2. Forward, step 2. $a_2=0.5\cdot 0.7616+1=1.3808$, so $h_2=\tanh (1.3808)=0.8811$. Why? Same rule, now the previous state $h_1$ feeds in through $w_{hh}$ — this is the memory in action.
3. Output blame. $\frac{\partial L}{\partial h_2}=h_2-1=-0.1189$. Why negative? $h_2<1$ means we undershot; the derivative of $\frac{1}{2}(h_2-1{)}^2$ is $(h_2-1)$, which is negative here. This is Cell C — a negative blame signal.
4. Set ${\delta }_2$. Since the only loss is at step 2, ${\delta }_2=\frac{\partial L}{\partial h_2}=-0.1189$. Why no future term? There is no step 3, so nothing flows back from the future.
5. Carry blame to step 1. ${\delta }_1=\underset{\text{no loss at }t=1}{\underbrace{0}}+w_{hh}(1-h_2^2){\delta }_2=0.5\cdot (1-0.8811^2)\cdot (-0.1189)$. Compute $1-h_2^2=1-0.7763=0.2237$, so ${\delta }_1=0.5\cdot 0.2237\cdot (-0.1189)=-0.01330$. Why? Backward blame line: step 1's state only matters through the future, filtered by Tanh's twin.
6. Sum the weight gradient over both steps. $\frac{\partial L}{\partial w_{hh}}=(1-h_1^2){\delta }_1{h}_0+(1-h_2^2){\delta }_2{h}_1.$ Term 1 $=(0.4200)(-0.01330)(0)=0$ — killed by $h_0=0$ (Cell B). Term 2 $=(0.2237)(-0.1189)(0.7616)=-0.02025$. Why two terms? "Same weight, sum the blame" — $w_{hh}$ did a job at step 1 (times $h_0$) and step 2 (times $h_1$).

Answer: $\frac{\partial L}{\partial w_{hh}}=-0.02025$.

Example 2 — Cell B, D: same net, target below → positive gradient

Forecast: $h_2=0.8811$ is now above target $0$. We want it smaller. Sign of the gradient?

1. Forward is unchanged. $h_1=0.7616,h_2=0.8811$. Why? The forward pass does not depend on the target.
2. Output blame flips sign. $\frac{\partial L}{\partial h_2}=h_2-0=+0.8811$. Cell D — positive blame. Why? Derivative of $\frac{1}{2}{h}_2^2$ is $h_2$, which is positive.
3. ${\delta }_2=0.8811$, only-end loss.
4. Weight gradient, second term only (first dies on $h_0=0$): $\frac{\partial L}{\partial w_{hh}}=(1-h_2^2){\delta }_2{h}_1=(0.2237)(0.8811)(0.7616)=\mathrm{0.15012.}$

Answer: $\frac{\partial L}{\partial w_{hh}}=+0.15012$.

Example 3 — Cell E: saturated $\tanh$, gradient vanishes

Forecast: Both $a$ values are large, so $h_1,h_2\approx 1$. What happens to Tanh's twin $(1-h^2)$?

1. Forward. $a_1=0.9\cdot 0+3=3$, $h_1=\tanh 3=0.99505$. $a_2=0.9\cdot 0.99505+3=3.89555$, $h_2=\tanh 3.89555=0.99919$. Why? Straight recipe — but notice how close to $1$ we already are. This is the flat-top region of the master-key figure (s01).

2. Tanh's twin is tiny. $1-h_2^2=1-0.99838=0.001622$. That's the geometric flatness of the figure below. Why does this matter? Every backward step multiplies by this factor. Cell E starts here.

3. Output blame. $\frac{\partial L}{\partial h_2}=h_2-1=-0.000810$. Why so small? We are already almost exactly at target $1$.

4. Carry to step 1. ${\delta }_1=0.9\cdot (1-h_2^2)\cdot {\delta }_2=0.9\cdot 0.001622\cdot (-0.000810)=-1.182\times 10^{-6}$. Why so tiny? Both the small error and the tiny Tanh's twin multiply together — the gradient has vanished before reaching step 1.

Answer: ${\delta }_1\approx -1.18\times 10^{-6}$ — effectively zero.

Example 4 — Cell F: $|w_{hh}|>1$, gradient explodes

Forecast: Each backward hop multiplies by $w_{hh}(1-h^2)\approx 2\cdot 1=2$. Travelling from step 5 to step 1 is 4 hops (5→4→3→2→1). Over those 4 hops?

1. Count the hops first. Going from state 5 to state 1 you cross the arrows $5\to 4$, $4\to 3$, $3\to 2$, $2\to 1$ — that is $5-1=4$ backward hops. Why care? The number of hops is the exponent below; "5 states" but "4 hops between the ends."
2. Per-step multiplier. $\frac{\partial h_{t+1}}{\partial h_t}=(1-h_{t+1}^2)w_{hh}\approx 1\cdot 2=2$. Why $1$? With $h\approx 0$, Tanh's twin $1-0^2=1$ — no shrinkage from the nonlinearity here.
3. Chain the 4 hops (step 5 → step 1). $\frac{\partial h_5}{\partial h_1}={\prod }_{k=2}^{5}2=2^4=16$. Why a product, and why does it become a power? The chain rule says a two-hop derivative is the product of one-hop derivatives: $\frac{\partial h_5}{\partial h_1}=\frac{\partial h_5}{\partial h_4}\frac{\partial h_4}{\partial h_3}\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial h_1}$. When every hop equals the same number $\lambda$, multiplying $n$ copies of $\lambda$ is ${\lambda }^n$ by definition of a power. That is the whole reason powers appear.
4. General law. $\frac{\partial h_T}{\partial h_t}\approx {\lambda }^{T-t}$ with $\lambda =|w_{hh}|\cdot \max (1-h^2)=2$; here $T-t=5-1=4$. Cell F — explosion because $\lambda >1$.

Answer: the blame is amplified $16\times$ over 4 hops; over 20 hops it would be $2^{20}\approx 1.05\times 10^6$.

Example 5 — Cell G: a loss at every step (multi-output)

Forecast: Now ${\delta }_1$ has a direct term (its own loss $L_1$) plus the future term. Bigger or smaller than Ex 1?

1. Forward unchanged. $h_1=0.7616,h_2=0.8811$.
2. ${\delta }_2$. Only $L_2$ touches $h_2$: ${\delta }_2=\frac{\partial L_2}{\partial h_2}=h_2-1=-0.1189$. Why? $h_2$ has no future.
3. ${\delta }_1$ now has TWO parts. ${\delta }_1=\underset{\partial L_1/\partial h_1}{\underbrace{(h_1-1)}}+\underset{\text{via }{h}_2}{\underbrace{w_{hh}(1-h_2^2){\delta }_2}}.$ Direct part $=0.7616-1=-0.2384$. Future part $=0.5\cdot 0.2237\cdot (-0.1189)=-0.01330$. So ${\delta }_1=-0.2384-0.01330=-0.25170$. Cell G — the direct term at every step. Why add? Multivariate chain rule: $h_1$ influences the loss through both $L_1$ directly and $L_2$ via $h_2$. Sum the paths.
4. Weight gradient, both steps: $\frac{\partial L}{\partial w_{hh}}=(1-h_1^2){\delta }_1{h}_0+(1-h_2^2){\delta }_2{h}_1.$ First term $=0$ (still $h_0=0$). Second $=(0.2237)(-0.1189)(0.7616)=-0.02025$.

Answer: $\frac{\partial L}{\partial w_{hh}}=-0.02025$ (same value as Ex 1, because $h_0=0$ still kills the term that ${\delta }_1$ lives in).

Example 6 — Cell H: a 4-step chain, watch ${\lambda }^{T-t}$ numerically

Forecast: Per-step multiplier is $w_{hh}(1-h^2)=0.5\cdot 0.64=0.32<1$. So the blame should shrink geometrically.

1. Where does ${\delta }_4=1$ come from? The final blame is ${\delta }_4=\frac{\partial L}{\partial h_4}=h_4-t_{\text{gt}}$. We deliberately picked the target $t_{\text{gt}}=h_4-1$ so this equals exactly $1$ — a clean round number to watch decay. Why allowed? The target is just a number we compare against; choosing it fixes the size of the starting blame.
2. Per-step factor. $r=w_{hh}(1-h_{t+1}^2)=0.5\cdot 0.64=0.32$. Why one number? All states equal, so every hop multiplies by the same $r$.
3. ${\delta }_3=r{\delta }_4=0.32\cdot 1=0.32$.
4. ${\delta }_2=r{\delta }_3=0.32\cdot 0.32=0.1024$.
5. ${\delta }_1=r{\delta }_2=0.32\cdot 0.1024=0.032768$. Why the pattern? ${\delta }_{4-n}=r^n{\delta }_4=0.32^n$ — the promised ${\lambda }^{T-t}$ decay, made of real digits.

Answer: ${\delta }_3=0.32,{\delta }_2=0.1024,{\delta }_1=0.032768$.

Example 7 — Cell I: real flight-data word problem (full backward pass)

Forecast: Steady positive inputs + memory retention → the state should ramp up toward $1$. Does it get there in 3 seconds?

1. Second 1. $a_1=0.8\cdot 0+0.5=0.5$, $h_1=\tanh 0.5=0.46212$. Why? Turn the physics into the recipe — $x_1$ is the fresh evidence, $h_0=0$ is "no history yet."
2. Second 2. $a_2=0.8\cdot 0.46212+0.5=0.86970$, $h_2=\tanh 0.86970=0.70067$. Why $w_{hh}=0.8$? The net remembers $80\%$ of last second's belief and adds new evidence.
3. Second 3. $a_3=0.8\cdot 0.70067+0.5=1.06053$, $h_3=\tanh 1.06053=0.78741$. Reads: memory has ramped $0.46\to 0.70\to 0.79$ — climbing but not yet at $1$.
4. Output blame at the final step. ${\delta }_3=\frac{\partial L}{\partial h_3}=h_3-1=-0.21259$. Why $h_3-1$? The loss is $\frac{1}{2}(h_3-1{)}^2$; by the power-and-chain rule $\frac{d}{d{h}_3}\frac{1}{2}(h_3-1{)}^2=(h_3-1)\cdot 1=h_3-1$. Negative because we undershot the target $1$.
5. Carry blame to step 2. No output loss at step 2, so the direct term is $0$: ${\delta }_2=0+w_{hh}(1-h_3^2){\delta }_3=0.8(1-0.62002)(-0.21259)=0.8\cdot 0.37998\cdot (-0.21259)=-\mathrm{0.064624.}$ Why? Step 2's memory only reaches the loss through step 3, filtered by Tanh's twin $1-h_3^2$.
6. Carry blame to step 1. Again no output loss at step 1: ${\delta }_1=0+w_{hh}(1-h_2^2){\delta }_2=0.8(1-0.49094)(-0.064624)=0.8\cdot 0.50906\cdot (-0.064624)=-\mathrm{0.026318.}$ Why? Same backward hop, one step earlier — the blame is shrinking as it travels back (each hop's factor $0.8\cdot \text{twin}<1$ here).
7. Sum the weight gradient over all three steps (weight $w_{hh}$ was used at each step, multiplying the previous state): $\frac{\partial L}{\partial w_{hh}}=(1-h_1^2){\delta }_1{h}_0+(1-h_2^2){\delta }_2{h}_1+(1-h_3^2){\delta }_3{h}_2.$
   • Term 1 $=(\cdot )(\cdot )(h_0=0)=0$ — killed by the zero start.
   • Term 2 $=(0.50906)(-0.064624)(0.46212)=-0.015202$.
   • Term 3 $=(0.37998)(-0.21259)(0.70067)=-0.056598$.
   • Sum $=-0.071800$. Why sum three terms? "Same weight, sum the blame" — one parameter, one job per step, gradients add.

Answer: $h_1=0.4621,h_2=0.7007,h_3=0.7874$; $\frac{\partial L}{\partial w_{hh}}=-0.07180$.

Example 8 — Cell J: exam twist, remove $\tanh$ (linear RNN)

Forecast: No squashing means the state can grow without a ceiling and Tanh's twin becomes... what?

1. Tanh's twin dies. With $h_t=a_t$, the derivative $\frac{\partial h_{t+1}}{\partial h_t}=w_{hh}$ — no $(1-h^2)$ factor at all. Why? $\frac{d}{da}a=1$, so the nonlinearity's damping is gone. This is Cell J.
2. Forward (pure linear recurrence). $h_1=0.5\cdot 0+1=1$. $h_2=0.5\cdot 1+1=1.5$. $h_3=0.5\cdot 1.5+1=1.75$. Why no ceiling? Nothing squashes it; the state just accumulates.
3. Jacobian over 2 hops. $\frac{\partial h_3}{\partial h_1}=w_{hh}^2=0.5^2=0.25$. Why exactly ${\lambda }^{T-t}$? With $\tanh$ gone the whole network is a single linear map, and its Jacobian is pure $w_{hh}^{T-t}$ — vanishing if $|w_{hh}|<1$, exploding if $>1$, with no way to gate in between.

Answer: $h_3=1.75$, $\frac{\partial h_3}{\partial h_1}=0.25$. What breaks: the net collapses to one linear map (stacking layers buys nothing) and its gradients are locked into pure ${\lambda }^T$ behaviour — exactly the backprop pathology the parent's mistake box warns about.

Example 9 — Cell K: the borderline $|w_{hh}|=1$

Forecast: The naive fear is "$|w_{hh}|=1$ means $\lambda =1$, so blame is preserved." But Tanh's twin is still $<1$. Does the blame stay put, or slowly fade?

1. Per-hop factor. $r=w_{hh}(1-h^2)=1\cdot 0.91=0.91$. Why not exactly 1? Even with $w_{hh}=1$, the $\tanh$ nonlinearity contributes its twin $0.91<1$. So $\lambda =0.91$, just under the knife-edge.
2. March back 4 hops (step 5 → step 1).
   • ${\delta }_4=0.91\cdot 1=0.91$
   • ${\delta }_3=0.91\cdot 0.91=0.8281$
   • ${\delta }_2=0.91^3=0.753571$
   • ${\delta }_1=0.91^4=0.68574961$ Why powers again? Same-number-per-hop ⇒ ${\delta }_{5-n}=0.91^n$ (the power rule from Ex 4).
3. Interpretation. Blame kept $\approx 69\%$ of its strength over 4 steps — neither vanished (Ex 6: $3\%$) nor exploded (Ex 4: $16\times$). This is the slow, graceful borderline. Only a strictly linear ($\tanh$-free) unit with $|w_{hh}|=1$ preserves blame exactly.

Answer: ${\delta }_4=0.91,{\delta }_3=0.8281,{\delta }_2=0.75357,{\delta }_1=0.68575$.

Example 10 — Cell L: negative $w_{hh}$, alternating-sign blame

Forecast: Each hop now multiplies by a negative number $w_{hh}(1-h^2)=-0.8\cdot 0.75=-0.6$. What does multiplying by a negative do to the sign, step after step?

1. Per-hop factor. $r=w_{hh}(1-h^2)=(-0.8)(0.75)=-0.6$. Why negative? Tanh's twin $0.75$ is always positive, but $w_{hh}$ is negative — the product inherits the sign of $w_{hh}$. This is Cell L.
2. March back.
   • ${\delta }_3=-0.6\cdot 1=-0.6$
   • ${\delta }_2=-0.6\cdot (-0.6)=+0.36$
   • ${\delta }_1=-0.6\cdot 0.36=-0.216$ Why the flip-flop? Multiplying by a negative number flips the sign every hop, so blame alternates $+,-,+,-$ as it travels back in time.
3. Magnitude still governed by $|\lambda |$. $|r|=0.6<1$, so despite the sign flips the size still shrinks like $0.6^n$ — it still vanishes, just with oscillating sign.

Answer: ${\delta }_3=-0.6,{\delta }_2=+0.36,{\delta }_1=-0.216$ (alternating sign, shrinking magnitude).

Recall