Worked examples — Backpropagation through time

Before anything else, three symbols you must own (all from the parent — re-anchored here so line one is readable):

---

The scenario matrix

Every worked example below is tagged with the cell it covers. The columns are the distinct behaviours BPTT can produce.

Cell	Scenario	What is being stress-tested	Example
A	Small weight $|W_{hh}|<1$	Gradient shrinks backward → foreshadows vanishing	Ex 1
B	Large weight $|W_{hh}|>1$	Gradient grows backward → foreshadows exploding	Ex 2
C	Negative $W_{hh}$	Sign flips each step; direction of the update	Ex 3
D	Zero input / degenerate ($W_{hh}=0$)	Time-links cut; only the local term survives	Ex 4
E	Saturated $\tanh$ ($h_t\to \pm 1$)	$(1-h_t^2)\to 0$ kills the gradient	Ex 5
F	Negative target → negative $\delta$	Loss pushing the state down	Ex 6
G	Real-world word problem	Reading a story into $x_t,{\hat{y}}_t$	Ex 7
H	Exam twist: 3 steps, share the chain	Full recursion, no shortcuts	Ex 8

The reusable rules (all from the parent, restated so this page stands alone):

The blueprint above is your map: each example is a walk right-to-left along the bottom arrows, and the figure marks where the $(1-h_t^2)$ gate sits.

---

Cell A — the gradient that shrinks

1. Forward. $h_1=\tanh (1.0)=0.7616$, $h_2=\tanh (0.5\cdot 0.7616+0.5)=\tanh (0.8808)=0.7068$. Why: apply the forward line twice; each memory feeds the next.
2. Backward, last step. ${\delta }_2=h_2-0.3=0.4068$. Why: the last step has no future, so ${\delta }_2$ is purely the local loss slope.
3. Backward, step 1. ${\delta }_1=(h_1-0.5)+0.5(1-h_2^2){\delta }_2=0.2616+0.5(0.5006)(0.4068)=0.3634$. Why: two sources — local loss $0.2616$ plus the future loss squeezed by the $\tanh$ gate and the weight.
4. Weight gradient. ${\hat{\delta }}_2=0.4068(1-0.7068^2)=0.2036$, ${\hat{\delta }}_1=0.3634(0.4200)=0.1526$. $\partial L/\partial W_{hh}={\hat{\delta }}_1{h}_0+{\hat{\delta }}_2{h}_1=0+0.2036\cdot 0.7616=0.1551$. Why: multiply each pre-activation gradient by the memory that entered that step.

Verify: $h_0=0$ deletes step 1's contribution, so only step 2 counts. $0.1551\ne 0.310$ — the wrong (no-$\tanh$) answer would be $0.4068\cdot 0.7616=0.310$, exactly double. ✓ (matches parent).

---

Cell B — the gradient that grows

1. Forward. $h_1=\tanh (1.5\cdot 0+1.0)=\tanh (1.0)=0.7616$. $h_2=\tanh (1.5\cdot 0.7616+0.5)=\tanh (1.6424)=0.9277$. Why: bigger $W_{hh}$ pushes the pre-activation higher, so $h_2$ is closer to $1$.
2. ${\delta }_2$. ${\delta }_2=0.9277-0.3=0.6277$.
3. ${\delta }_1$. $(1-h_2^2)=1-0.8607=0.1393$. ${\delta }_1=(0.7616-0.5)+1.5(0.1393)(0.6277)=0.2616+0.1312=0.3928$. Why: even though $h_2$ is starting to saturate (small gate $0.1393$), the large weight $1.5$ partly compensates.
4. Weight gradient. ${\hat{\delta }}_2=0.6277(0.1393)=0.0874$. $\partial L/\partial W_{hh}=0+{\hat{\delta }}_2{h}_1=0.0874\cdot 0.7616=0.0666$.

Verify: the un-gated future multiplier $W_{hh}=1.5>1$ would explode the chain, but the $\tanh$ gate $0.1393$ tames it here — this is exactly why LSTMs replace the gate. ✓

---

Cell C — negative weight flips the sign

1. Forward. $h_1=\tanh (1.0)=0.7616$. $h_2=\tanh (-0.5\cdot 0.7616+0.5)=\tanh (0.1192)=0.1186$. Why: the negative recurrent term subtracts the past memory, so $h_2$ lands near zero.
2. ${\delta }_2$. ${\delta }_2=0.1186-0.3=-0.1814$ (negative — the state is below target).
3. ${\delta }_1$. $(1-h_2^2)=0.9859$. ${\delta }_1=0.2616+(-0.5)(0.9859)(-0.1814)=0.2616+0.0894=0.3510$. Why: two negatives multiply to a positive, so the future term adds here — sign bookkeeping matters.
4. Weight gradient. ${\hat{\delta }}_2=-0.1814(0.9859)=-0.1788$. $\partial L/\partial W_{hh}=0+{\hat{\delta }}_2{h}_1=-0.1788\cdot 0.7616=-0.1362$.

Verify: gradient is negative — gradient descent will increase $W_{hh}$ (make it less negative), pulling $h_2$ up toward $0.3$. Direction makes physical sense. ✓

---

Cell D — degenerate: cut the time link

1. Forward. $h_1=\tanh (0+1.0)=0.7616$. $h_2=\tanh (0+0.5)=\tanh (0.5)=0.4621$. Why: $W_{hh}=0$ removes $h_{t-1}$; each step sees only its own input.
2. ${\delta }_2=0.4621-0.3=0.1621$.
3. ${\delta }_1=(0.7616-0.5)+0\cdot (\cdots )=0.2616$. Why: the future-loss channel is physically severed; memory of step 2's error cannot reach step 1.
4. Weight gradient. ${\hat{\delta }}_2=0.1621(1-0.4621^2)=0.1621(0.7865)=0.1275$. $\partial L/\partial W_{hh}=0+0.1275\cdot 0.7616=0.0971$.

Verify: the gradient w.r.t. $W_{hh}$ is non-zero even though $W_{hh}=0$ — that is correct! The gradient measures the effect of turning the link on, i.e. $h_{t-1}$ still multiplies into the weight slot ($h_1=0.7616$). A zero weight is not a zero derivative. ✓

---

Cell E — saturation kills the gradient

1. Forward. $h_1=\tanh (3.0)=0.9951$. $h_2=\tanh (1.0\cdot 0.9951+3.0)=\tanh (3.9951)=0.99933$. Why: large pre-activations push $\tanh$ onto its flat tails.
2. ${\delta }_2=0.99933-0.9=0.09933$.
3. Gate at step 2: $(1-h_2^2)=1-0.99866=0.00134$. ${\delta }_1=(0.9951-0.9)+1.0(0.00134)(0.09933)=0.0951+0.000133=0.09523$. Why: the future term is crushed to $\sim 10^{-4}$ because the gate is nearly zero.
4. Weight gradient. ${\hat{\delta }}_2=0.09933(0.00134)=0.000133$. $\partial L/\partial W_{hh}=0+0.000133\cdot 0.9951=0.000132$.

Verify: gradient $\approx 1.3\times 10^{-4}$, essentially zero despite a real loss — saturation ⇒ dead gradient. This is why LSTMs and GRUs add un-squashed carry paths. ✓

---

Cell F — negative target, negative gradient

1. Forward (same weights/inputs as Ex 1): $h_1=0.7616$, $h_2=0.7068$.
2. ${\delta }_2=0.7068-(-0.2)=0.9068$.
3. ${\delta }_1=(0.7616-(-0.4))+0.5(1-0.7068^2)(0.9068)=1.1616+0.5(0.5006)(0.9068)=1.1616+0.2270=1.3886$. Why: the mismatch is now huge ($1.16$ locally), and the future term adds more.
4. Weight gradient. ${\hat{\delta }}_2=0.9068(0.5006)=0.4539$. $\partial L/\partial W_{hh}=0+0.4539\cdot 0.7616=0.3457$.

Verify: far larger than Ex 1's $0.1551$ (bigger error ⇒ bigger gradient), and positive, so descent lowers $W_{hh}$-driven output toward the negative targets. Consistent. ✓

---

Cell G — a real-world word problem

1. Read the story into numbers. Inputs $x_1=0.8,x_2=-0.6$; targets $0.5,0.0$. Why: the word problem is just Ex-1 machinery with new $x,\hat{y}$.
2. Forward. $h_1=\tanh (0.8)=0.6640$. $h_2=\tanh (0.5\cdot 0.6640-0.6)=\tanh (-0.2680)=-0.2618$. Why: the cold input drags the memory below zero — the "mood" cooled.
3. Backward. ${\delta }_2=-0.2618-0.0=-0.2618$. $(1-h_2^2)=0.9315$. ${\delta }_1=(0.6640-0.5)+0.5(0.9315)(-0.2618)=0.1640-0.1220=0.0420$.
4. Weight gradient. ${\hat{\delta }}_2=-0.2618(0.9315)=-0.2439$. ${\hat{\delta }}_1=0.0420(1-0.6640^2)=0.0420(0.5591)=0.0235$. $\partial L/\partial W_{hh}={\hat{\delta }}_1{h}_0+{\hat{\delta }}_2{h}_1=0+(-0.2439)(0.6640)=-0.1620$.

Verify: negative gradient ⇒ descent raises $W_{hh}$, letting yesterday's warmth carry more strongly so $h_2$ climbs toward the neutral target instead of overshooting negative. Story-consistent. ✓

---

Cell H — exam twist: full 3-step chain

1. Forward. $h_1=\tanh (1.0)=0.7616$. $h_2=\tanh (0.5\cdot 0.7616+0.5)=\tanh (0.8808)=0.7068$. $h_3=\tanh (0.5\cdot 0.7068+0.2)=\tanh (0.5534)=0.5030$.
2. ${\delta }_3$. ${\delta }_3=0.5030-0.1=0.4030$. Why: newest step, no future.
3. ${\delta }_2$. $(1-h_3^2)=0.7470$. ${\delta }_2=(0.7068-0.3)+0.5(0.7470)(0.4030)=0.4068+0.1505=0.5573$.
4. ${\delta }_1$. $(1-h_2^2)=0.5006$. ${\delta }_1=(0.7616-0.5)+0.5(0.5006)(0.5573)=0.2616+0.1395=0.4011$. Why: ${\delta }_2$ already carries step-3's loss, so passing it back gives ${\delta }_1$ its share of all three losses.
5. Weight gradient. ${\hat{\delta }}_1=0.4011(0.4200)=0.1685$, ${\hat{\delta }}_2=0.5573(0.5006)=0.2790$, ${\hat{\delta }}_3=0.4030(0.7470)=0.3010$. $\partial L/\partial W_{hh}={\hat{\delta }}_1{h}_0+{\hat{\delta }}_2{h}_1+{\hat{\delta }}_3{h}_2=0+0.2790\cdot 0.7616+0.3010\cdot 0.7068=0.2125+0.2127=0.4253$.

Verify: with $h_0=0$ only the last two terms survive; both positive and comparable, giving $\approx 0.425$. Each backward hop shrank the future term ($0.15,0.14$) — a live preview of why very long sequences lose signal. ✓

---

Recall Self-test: predict before you compute

If $W_{hh}=0$, is $\partial L/\partial W_{hh}$ necessarily zero? ::: No — the incoming memory $h_{t-1}$ still multiplies the weight slot; a zero weight has a non-zero derivative (Ex 4 gave $0.097$). Which factor turns a live loss into a nearly-dead gradient? ::: The saturation gate $(1-h_t^2)\to 0$ when $h_t\to \pm 1$ (Ex 5). When does the future term make ${\delta }_1$ smaller than its local part? ::: When $W_{hh}$ or ${\delta }_{t+1}$ is negative, so the future contribution subtracts (Ex 3, Ex 7). Why must the weight gradient use ${\hat{\delta }}_t$ and not ${\delta }_t$? ::: Because $W_{hh}$ feeds the pre-activation; you must pass back through $\tanh$, whose slope is $(1-h_t^2)$ — skipping it doubled the answer in Ex 1.

Related: 3.5.01-Recurrent-Neural-Networks-RNNs · 2.2.03-Backpropagation-Algorithm · 3.5.03-Vanishing-and-Exploding-Gradients · 3.5.07-Bidirectional-RNNs.