Vanishing gradients in RNNs: Worked examples — AI-ML deep dive

This page is the "no scenario left behind" companion to Vanishing gradients in RNNs. Before we compute anything, let us agree on what all the symbols mean, because every example below reuses them.

Definition The cast of symbols (read this first)

$h_{t}$ — the hidden state at time step $t$ . Think of it as the network's memory: a list of numbers it carries forward as it reads a sequence one item at a time.
$x_{t}$ — the input fed in at step $t$ : the actual item being read right now (e.g. the current word's vector). It arrives fresh each step and does not depend on the memory.
$W_{hh}$ — the recurrent weight matrix. It is the fixed rule that says "given last step's memory, here is my raw next memory". The same matrix is reused at every step — that reuse is the whole story.
$W_{x h}$ — the input weight matrix: the fixed rule that converts the current input $x_{t}$ into its contribution to the raw next memory.
$b_{h}$ — the bias vector: a constant offset added at every step, independent of memory or input.
$z_{k}$ — the pre-activation at step $k$ : the raw number $z_{k} = W_{hh} h_{k - 1} + W_{x h} x_{k} + b_{h}$ that goes into the tanh before it is squashed (it stacks the memory contribution, the input contribution, and the bias). It is what the slope $tanh^{'}$ is measured at.
$T$ — the final time index: the last step of the sequence, where the loss is read off. It is simply the largest value $t$ takes, so a length- $50$ sequence running from $t = 0$ has $T = 50$ .
$γ$ — the per-step factor: the number each hidden step multiplies the gradient by. For a matrix $W_{hh}$ its non-negative size is the spectral norm $∥ W_{hh} ∥$ (the largest stretch factor). But for a scalar RNN, or when we look at a single dominant direction, the true per-step multiplier is an eigenvalue (the spectral radius carries the sign), which can be negative — we treat that case explicitly in Example 4. Norms are never negative; eigenvalues can be.
$tanh^{'} (z) = 1 - tanh^{2} (z)$ — the slope of the tanh squashing function at input $z$ . It lives in $(0, 1]$ : it is $1$ when $z = 0$ (steepest) and shrinks toward $0$ as $∣ z ∣$ grows (flat, "saturated").
$n = T - t$ — the time gap: how many steps separate the early memory $h_{t}$ from the late memory $h_{T}$ . Every example measures decay against this single number.
$\frac{\partial h _{T}}{\partial h _{t}}$ — the gradient factor that tells us how a nudge in the early memory $h_{t}$ ripples out to the late memory $h_{T}$ . If this is tiny, the network cannot learn long-range links.

Everything on this page is really about ONE quantity: this gradient factor across a time gap of length $n = T - t$ .

The next figure plots exactly this: the gradient factor $(base)^{n}$ against the gap $n$ for the three defining bases $0.9$ , $1.0$ , $1.1$ . Follow the blue curve diving toward zero (Example 1), the flat green line at $1$ (Example 3), and the orange curve rocketing up (Example 2) — the three examples share one picture.

The scenario matrix

Every question this topic can throw at you is one cell of this table. The examples that follow are each tagged with the cell they cover, so by the end no cell is left unworked.

#	Case class	Concrete trigger	Expected behaviour
A	Shrinking weight, no activation	$γ < 1$ , linear	Exponential vanish
B	Growing weight	$γ > 1$	Exponential explode
C	Boundary / degenerate	$γ = 1$ exactly	Neither — gradient preserved
D	Negative / sign-flip factor	eigenvalue $< 0$	Vanishing magnitude + oscillating sign
E	Saturated / degenerate step	$tanh^{'} \to 0$ on one step	Near-zero factor that step
F	Saturation stacked on weight	$tanh^{'} \to 0$ every step	Even faster vanish
G	Limiting behaviour	$n \to \infty$	Factor $\to 0$ or $\infty$
H	Real-world word problem	subject–verb agreement across a clause	Can it learn the link?
I	Exam twist: LSTM gate	forget gate $f_{t}$ near $1$	Vanish defeated
J	Exam twist: mixed rates	per-step $γ_{i}$ varies with $i$	One learning rate can't fix

Example 1 — Cell A: shrinking weight, pure linear

Forecast: guess before reading — will the influence be closer to $1$ , to $0.5$ , or to $0.005$ ?

Write the per-step derivative. From $h_{t} = 0.9 h_{t - 1} + x_{t}$ , differentiate w.r.t. $h_{t - 1}$ : $\frac{\partial h _{t}}{\partial h _{t - 1}} = 0.9$ . Why this step? $x_{t}$ does not depend on $h_{t - 1}$ , so it drops; the coefficient $0.9$ is the whole local derivative.
Chain across the gap. $\frac{\partial h _{50}}{\partial h _{0}} = \prod_{k = 1}^{n} 0.9 = 0. 9^{n} = 0. 9^{50}$ . Why this step? Each step multiplies the same factor, so a gap of $n$ steps means the factor to the $n$ th power (note-passing line of $n$ people).
Evaluate. $0. 9^{50} \approx 5.15 \times 1 0^{- 3}$ . Why this step? This is the actual size of the "note" that survives across the gap $n = 50$ .
Turn into a parameter update. update $\approx lr \times grad = 0.01 \times 5.15 \times 1 0^{- 3} \approx 5.15 \times 1 0^{- 5}$ . Why this step? A gradient this small produces a change too tiny to move the weights — the dependency is effectively unlearnable.

Verify: $0. 9^{50}$ : take logs, $50 ln 0.9 = 50 (- 0.10536) = - 5.268$ , and $e^{- 5.268} \approx 0.00515$ . ✓ Consistent with step 3.

Example 2 — Cell B: growing weight (the opposite failure)

Forecast: vanish or blow up?

Write the per-step derivative $= 1.1$ . Why this step? Exactly as in Example 1, step 1: differentiating $h_{t} = 1.1 h_{t - 1} + x_{t}$ w.r.t. $h_{t - 1}$ drops the input term $x_{t}$ and leaves the coefficient $1.1$ as the local derivative.
Chain: $\partial h_{50} / \partial h_{0} = 1. 1^{n} = 1. 1^{50}$ . Why this step? Reuse of the same factor across the gap $n$ again.
Evaluate: $1. 1^{50} \approx 117.4$ . Why this step? Now each pass brightens the note, so it grows without bound — this is the Exploding Gradients regime, cured by Gradient Clipping.

Verify: $50 ln 1.1 = 50 (0.09531) = 4.766$ , $e^{4.766} \approx 117.4$ . ✓ Opposite sign of exponent-effect from Example 1 — confirming vanish and explode are two ends of one dial (the blue vs orange curves in the figure above).

Example 3 — Cell C: the exact boundary $γ = 1$

Forecast: what does $1$ raised to the $50$ th power do?

Per-step factor $= 1$ .
Chain: $1^{n} = 1^{50} = 1$ exactly. Why this step? Multiplying by $1$ changes nothing, no matter how large the gap $n$ .
Interpret: this is the ideal — an uninterrupted "gradient highway" (the flat green line in the opening figure). It is exactly what the LSTM forget gate and Residual Connections deliberately engineer by keeping the effective factor near $1$ .

Verify: $1. 0^{50} = 1$ . ✓ The boundary is neither vanish nor explode — this is why $γ = 1$ (orthogonal initialization) is the target for plain RNNs.

Example 4 — Cell D: negative factor (sign-flipping oscillation)

Forecast: does the magnitude vanish like Example 1, and what does the sign do?

Per-step derivative $= - 0.9$ . Why this step? Same differentiation as before; the coefficient is simply negative now. A norm can never be negative, so this sign lives in the eigenvalue (spectral radius with sign), not the spectral norm — a real matrix can have negative eigenvalues, and a scalar factor can be $< 0$ .
Chain across the gap: $\partial h_{50} / \partial h_{0} = (- 0.9)^{n} = (- 0.9)^{50}$ . Why this step? A negative base raised to a power keeps flipping sign: odd $n$ gives a negative factor, even $n$ gives positive.
Split magnitude and sign. $(- 0.9)^{50} = (+ 1) \cdot 0. 9^{50} \approx + 5.15 \times 1 0^{- 3}$ ; but $(- 0.9)^{49} \approx - 5.72 \times 1 0^{- 3}$ . Why this step? The magnitude $∣ - 0.9 ∣^{n} = 0. 9^{n}$ still vanishes exactly like Example 1 — sign flipping does not rescue it. The oscillating sign just means the gradient's direction alternates every step, which can slow or destabilise learning even before it vanishes.

Verify: $(- 0.9)^{50} = 0. 9^{50} \approx 5.15 \times 1 0^{- 3}$ (positive); $(- 0.9)^{49} \approx - 5.72 \times 1 0^{- 3}$ (negative). ✓ Magnitude identical to Example 1, sign alternates with parity of $n$ .

Example 5 — Cell E: a single saturated / degenerate step

Forecast: does the big weight $2$ rescue the flat tanh, or does the flat tanh win?

Local derivative $= tanh^{'} (z_{k}) \cdot W_{hh}$ (the standard BPTT form). Why this step? Both factors are present: the tanh slope and the weight both multiply. The weight magnitude does not disappear.
Evaluate the slope: $tanh (10) \approx 0.99999999588$ , so $tanh^{'} (10) = 1 - tanh^{2} (10) \approx 8.24 \times 1 0^{- 9}$ . Why this step? Deep in saturation the tanh curve is flat, so its slope is nearly $0$ — see the Activation Functions shape in the next figure.
Multiply in the weight: factor $= 2 \times 8.24 \times 1 0^{- 9} \approx 1.65 \times 1 0^{- 8}$ . Why this step? The weight $2$ does affect the answer — it doubles the small factor. But because the tanh slope is so tiny, even a large weight leaves the factor near zero: the saturated step dominates, though it does not literally cancel $W_{hh}$ .

Verify: $1 - tanh (10)^{2} \approx 8.24 \times 1 0^{- 9}$ ; times $2 \approx 1.65 \times 1 0^{- 8}$ . ✓ Notice the weight $2$ visibly scales the result — it is not irrelevant, just outmatched.

The figure above shows tanh (blue) flattening out and its slope $tanh^{'}$ (red) collapsing to zero as $∣ z ∣$ grows. The red dot marks $z = 4$ where the slope has already nearly vanished — at $z = 10$ it is off the bottom of the plot.

Example 6 — Cell F: saturation stacked on a shrinking weight

Forecast: guess the order of magnitude — $1 0^{- 5}$ ? $1 0^{- 20}$ ? $1 0^{- 50}$ ?

Combine per-step factors: effective multiplier $= 0.1 \times 0.9 = 0.09$ . Why this step? The tanh slope and the weight norm are separate factors that multiply — the parent note stresses not to conflate them.
Chain $n = 50$ steps: $0.0 9^{n} = 0.0 9^{50}$ .
Evaluate: $0.0 9^{50} \approx 5.15 \times 1 0^{- 53}$ . Why this step? This is below float32's smallest normal number ( $\approx 1.2 \times 1 0^{- 38}$ ) — the gradient is numerically zero, not just small.

Verify: $50 lo g_{10} (0.09) = 50 (- 1.0458) = - 52.29$ , so $0.0 9^{50} \approx 1 0^{- 52.29} \approx 5.1 \times 1 0^{- 53}$ . ✓ Two separate factors ( $0.1$ and $0.9$ ) confirmed multiplying.

Example 7 — Cell G: limiting behaviour as the gap grows

Forecast: roughly $10$ steps, $50$ steps, or $100$ steps?

Set up the inequality: we want $0.9 5^{n} < 0.01$ . Why this step? We are asking "how long a note-passing line before the note is <1% bright?"
Take logs: $n ln 0.95 < ln 0.01 \Rightarrow n > \frac{ln 0.01}{ln 0.95}$ (inequality flips because $ln 0.95 < 0$ ). Why this step? Logs turn the exponent into a solvable linear inequality; dividing by a negative flips the sign.
Evaluate: $\frac{- 4.6052}{- 0.051293} \approx 89.8$ , so $n = 90$ . Why this step? Even a gentle $γ = 0.95$ still kills long-range signal — just at a longer horizon. As $n \to \infty$ , $0.9 5^{n} \to 0$ always.

Verify: $0.9 5^{90} \approx 0.0099 < 0.01$ while $0.9 5^{89} \approx 0.0104 > 0.01$ , so the first integer crossing is $n = 90$ . ✓

Example 8 — Cell H: the real-world word problem

Forecast: more than half, or almost nothing?

Count the gap: $n = 11$ tokens between "cat" and "was".
Compute survival: $0. 8^{n} = 0. 8^{11}$ . Why this step? Each intervening word re-multiplies the carried memory by $γ$ ; the "cat" signal must survive $n = 11$ such multiplications on its way forward, and the correcting gradient must survive the same $11$ on the way back.
Evaluate: $0. 8^{11} \approx 0.0859$ , i.e. about $8.6%$ . Why this step? Under $9%$ of the original "cat" memory component (and equally of the returning gradient) still reaches across the clause — separately, the parameter update is this gradient times the learning rate, but the point here is that the raw signal itself is already down to $8.6%$ . This is exactly the case Attention Mechanisms and LSTMs were built to rescue.

Verify: $0. 8^{11} = 0. 8^{10} \cdot 0.8 = 0.1074 \times 0.8 = 0.0859$ . ✓ ( $8.59%$ .)

Example 9 — Cell I: the LSTM forget-gate escape (exam twist)

Forecast: will $0.9 9^{50}$ behave like the vanished $0. 9^{50}$ , or survive?

Chain the gate: $\frac{\partial C _{50}}{\partial C _{0}} = \prod_{k = 1}^{n} f_{k} = 0.9 9^{n} = 0.9 9^{50}$ . Why this step? The additive cell update makes the local derivative just $f_{t}$ — no weight matrix, no tanh slope stacked on top.
Evaluate: $0.9 9^{50} \approx 0.605$ . Why this step? Over $60%$ of the gradient survives $n = 50$ steps — versus $0.5%$ for the plain RNN. A gate near $1$ builds the highway.
Contrast: plain RNN $0. 9^{50} = 0.005$ vs LSTM $0.9 9^{50} = 0.605$ — a factor of $\sim 117 \times$ more signal. Why this step? Same gap, same "exponential" form, but a base close to $1$ delays vanishing enormously. GRUs achieve the same with a single update gate.

Verify: $0.9 9^{50}$ : $50 ln 0.99 = 50 (- 0.01005) = - 0.5025$ , $e^{- 0.5025} \approx 0.605$ . Ratio $0.605/0.00515 \approx 117$ . ✓

The figure below overlays both survival curves against the gap $n$ : the red plain-RNN curve crashes to $0.005$ by $n = 50$ while the green LSTM curve holds at $0.605$ — the highway made visible.

Example 10 — Cell J: mixed / varying per-step rates (exam twist)

Forecast: can a single learning rate be "just right" for both the near and far parameters?

Product with varying factors. When $γ$ changes per step the chain is a product of different numbers: $\frac{\partial h _{T}}{\partial h _{t}} = \prod_{i = t + 1}^{T} γ_{i}$ , not a single base to a power. Why this step? Example 1 assumed every $γ_{i}$ equal; in reality each step's slope and weight differ, so we must multiply the actual sequence of factors. This is the honest general form behind all earlier examples.
Recent gradient (gap $n = 2$ ): the last two factors are $0.99$ each, so $γ_{i}$ -product $= 0.9 9^{2} \approx 0.98$ . Why this step? A recent parameter only has to survive $2$ mild multiplications, so its gradient is near $1$ — perfectly healthy.
Early gradient (gap $n = 50$ ): the full product is $0.9 9^{2} \times 0. 9^{48}$ . Why this step? The first parameter's signal must run the whole gauntlet: the $2$ mild steps and the $48$ shrinking steps stack up.
Evaluate the early product: $0.9 9^{2} \times 0. 9^{48} \approx 0.9803 \times 6.36 \times 1 0^{- 3} \approx 6.24 \times 1 0^{- 3}$ . Why this step? The $0. 9^{48}$ term dominates — the early gradient has still essentially vanished despite the friendly recent steps.
Pick lr to boost the early step to update $= 0.1$ : $lr = 0.1/6.24 \times 1 0^{- 3} \approx 16.0$ . Why this step? To move the far-away parameters at all, we need a huge learning rate.
Apply that lr to the recent step: update $= 16.0 \times 0.98 \approx 15.7$ . Why this step? The recent parameters get slammed with an update of size $\sim 16$ — wildly unstable. One learning rate cannot serve a gap- $2$ gradient of $0.98$ and a gap- $50$ gradient of $0.006$ at once. This is the parent note's "Mistake 2"; the cure is adaptive optimizers (Adam, RMSprop) that keep per-parameter rates, plus gated architecture.

Verify: $0.9 9^{2} \times 0. 9^{48} \approx 6.24 \times 1 0^{- 3}$ ; $0.1/6.24 \times 1 0^{- 3} \approx 16.0$ ; $16.0 \times 0.9 9^{2} \approx 15.7$ . ✓ Recent gradient $0.9 9^{2} \approx 0.980$ vs early $\approx 0.006$ — a $\sim 160 \times$ spread that no single lr can bridge.

Recall Self-test — reveal after guessing

Per-step factor $0.9$ , gap $n = 50$ , gradient factor? ::: $0. 9^{50} \approx 5.15 \times 1 0^{- 3}$ Per-step factor $1.1$ , gap $n = 50$ ? ::: $1. 1^{50} \approx 117.4$ (explodes) Per-step factor exactly $1$ , any gap $n$ ? ::: $1$ — preserved, the ideal highway Per-step eigenvalue $- 0.9$ , gap $n = 49$ ? ::: $\approx - 5.72 \times 1 0^{- 3}$ : same tiny magnitude as $+ 0.9$ , but negative (odd gap) A single step with $tanh^{'} (10)$ and weight $2$ ? ::: $\approx 1.65 \times 1 0^{- 8}$ — near-zero, weight only scales it LSTM forget gate $0.99$ over gap $n = 50$ ? ::: $0.9 9^{50} \approx 0.605$ , over $60%$ survives Varying factors $0.9 9^{2} \times 0. 9^{48}$ ? ::: $\approx 6.24 \times 1 0^{- 3}$ — the shrinking steps still dominate