Vanishing gradients in RNNs: Exercises — AI-ML deep dive

This page is a self-test. Read each problem, try it on paper, THEN open the collapsible solution. The exercises climb from recognising the vanishing-gradient pattern to designing fixes.

Before anything, let us define every symbol used on this page, in plain words, so you never meet a piece of notation cold:

Definition The symbols you will use here

$h_{t}$ — the hidden state: the RNN's memory, a list of numbers (a vector) it carries at time step $t$ .
$x_{t}$ (and $x_{k}$ ) — the input fed in at step $t$ (or $k$ ): the word/number arriving right now.
$W_{hh}$ — the recurrent weight matrix: the fixed grid of numbers that mixes the old memory into the new memory each step.
$W_{x h}$ — the input weight matrix: the grid that mixes the current input into the memory.
$b_{h}$ — the bias: a constant vector added in.
$z_{k}$ — the pre-activation at step $k$ : the raw number(s) before squashing, $z_{k} = W_{hh} h_{k - 1} + W_{x h} x_{k} + b_{h}$ .
$tanh^{'} (z) = 1 - tanh^{2} (z)$ — the slope of the tanh squashing curve at $z$ (how steep tanh is there).
$∥ \cdot ∥$ — the norm (size) of a matrix or vector; for a matrix, $∥ W_{hh} ∥$ is the spectral norm = the largest factor by which $W_{hh}$ can stretch any vector.
$γ$ — a single number we use as shorthand for the per-step multiplier of the gradient (the factor the gradient gets multiplied by each time step). " $γ < 1$ " means "each step shrinks the gradient."
$f_{t}$ — the LSTM forget gate at step $t$ : a learned number between 0 and 1 that decides how much old memory to keep.

Figure s01 below plots the tanh curve and its slope — Exercise 1.2 refers to it directly.

Level 1 — Recognition

Recall Solution 1.1

When every step multiplies the gradient by the same number $γ$ , doing it $(T - t)$ times means raising $γ$ to that power — that is just what "multiply by $γ$ , $20$ times over" means. $0. 8^{20} \approx 0.0115$ The gradient shrinks to about $1.15%$ of its size — this is vanishing (each step multiplies by a number below 1, so 20 of them push toward zero).

Recall Solution 1.2

Look at figure s01 (embedded near the top of this page, titled "s01 Activation factor: tanh and its slope"). The slope is largest where the tanh curve is steepest — at $z = 0$ , where $tanh (0) = 0$ so $tanh^{'} (0) = 1 - 0 = 1$ . That is the maximum, value $= 1$ . As $∣ z ∣ \to \infty$ , $tanh (z) \to \pm 1$ , so $tanh^{2} (z) \to 1$ and $tanh^{'} (z) \to 1 - 1 = 0$ . The tanh saturates (goes flat) and its slope dies to 0. This is why large pre-activations are dangerous: the activation factor collapses.

Recall Solution 1.3

(a) $0.5 < 1 \Rightarrow$ vanishing (weight factor shrinks each step). (b) $2.0 > 1 \Rightarrow$ exploding (weight factor grows each step). See Exploding Gradients. (c) Saturation makes $tanh^{'} \to 0 \Rightarrow$ vanishing (activation factor shrinks). (d) Weight factor $= 1$ and slope $= 1 \Rightarrow$ product stays near 1 $\Rightarrow$ neither (the borderline "healthy" case).

Level 2 — Application

Recall Solution 2.1

With a linear recurrence the local derivative $\partial h_{k} / \partial h_{k - 1}$ is exactly $0.95$ every step (the coefficient in front of $h_{k - 1}$ ), so chaining 40 of them gives $0.9 5^{40}$ . $\frac{\partial h _{40}}{\partial h _{0}} = 0.9 5^{40} \approx 0.1285$ The update scale is $0.01 \times 0.1285 \approx 1.28 \times 1 0^{- 3}$ — small but not dead yet. $γ = 0.95$ decays slowly; the problem bites harder for longer sequences or smaller $γ$ .

Recall Solution 2.2

Why group first? Each single step contributes the product $tanh^{'} (z_{k}) \cdot W_{hh} = 0.2 \times 0.95$ . So one step's multiplier is $0.2 \times 0.95 = 0.19$ . Because multiplication is associative and every step has the same $0.19$ , the whole $(T - t)$ -fold product is just $0.19$ raised to the power of the gap — we are allowed to collapse the two same-valued factors into one number before exponentiating, since $(ab)^{n} = a^{n} b^{n}$ and here $a, b$ repeat identically every step. $(0.2 \times 0.95)^{40} = 0.1 9^{40} \approx 3.9 \times 1 0^{- 30}$ Numerically vanished. Note $0.2$ came from tanh's slope and $0.95$ from the weight norm — separate factors that just happen to multiply, exactly as the L1 trap warned.

Recall Solution 2.3

LSTM cell path (see LSTM Architecture): $0.9 9^{100} \approx 0.366$ — about $37%$ survives. This is the gradient highway: the additive cell update means the multiplier is the learned gate, kept near 1. Plain RNN: $0. 9^{100} \approx 2.66 \times 1 0^{- 5}$ — essentially gone. The LSTM preserves roughly 13,000× more gradient over 100 steps.

Level 3 — Analysis

Recall Solution 3.1

Dividing two powers of the same base subtracts exponents, so the unknown $a$ (a possible constant) cancels: $\frac{γ ^{20}}{γ ^{10}} = γ^{10} = \frac{0.0115}{0.107} \approx 0.1075$ . So $γ = 0.107 5^{1/10} \approx 0.800$ . Check: $0. 8^{10} \approx 0.107$ ✓ and $0. 8^{20} \approx 0.0115$ ✓. The measured base is $γ \approx 0.8$ .

Recall Solution 3.2

We need $1 0^{- 2} \leq ∥ W_{hh} ∥^{30} \leq 1 0^{2}$ . Why the 30th root is legal: the function $u \mapsto u^{1/30}$ is strictly increasing for $u > 0$ , so applying it to all three sides of the inequality preserves the order (it never flips $\leq$ ). That is why we may root through: $1 0^{- 2/30} \leq ∥ W_{hh} ∥ \leq 1 0^{2/30}$ $0.858 \leq ∥ W_{hh} ∥ \leq 1.079$ So the largest spectral norm that still avoids explosion at 30 steps is $\approx 1.079$ , and the smallest that avoids vanishing is $\approx 0.858$ . The healthy window is the narrow band $[0.858, 1.079]$ around 1. Push above $1.079$ and you enter Exploding Gradients; drop below $0.858$ and you vanish. Real saturation ( $tanh^{'} < 1$ ) shifts the whole band upward (you need a bigger $∥ W_{hh} ∥$ just to break even), squeezing plain RNNs from both sides — which is why they are stuck.

Recall Solution 3.3

With $ReLU^{'} = 1$ the activation factor is exactly 1, so the whole product is $∥ W_{hh} ∥^{25}$ : $1. 3^{25} \approx 705.6$ The gradient exploded by a factor of ~700. Removing tanh's damping did not fix credit assignment — it just handed control entirely to $∥ W_{hh} ∥$ , and $1.3 > 1$ blows up. See Activation Functions and Gradient Clipping.

Recall Solution 3.4

Only the magnitude of an eigenvalue drives vanishing/exploding, but the sign and phase shape the path:

Negative eigenvalue (e.g. $- 0.9$ ): each step also flips the sign, so the gradient's sign oscillates $+, -, +, -$ while its magnitude still decays like $0. 9^{n}$ . The envelope vanishes, but non-monotonically.
Complex eigenvalues (they come in conjugate pairs for real $W_{hh}$ ): the gradient rotates in the plane spanned by that pair. The norm follows $∣ λ ∣^{n}$ (so still vanishes if $∣ λ ∣ < 1$ , explodes if $∣ λ ∣ > 1$ ), but the components spiral — the norm can rise and fall along the way rather than dropping smoothly. Takeaway: the $∥ W_{hh} ∥^{T - t}$ curve is an envelope; the true per-component gradient can wobble or spiral inside it. This is why measured gradient norms are often bumpy, not clean exponentials.

Level 4 — Synthesis

Recall Solution 4.1

We need $1 0^{- 2} \leq ∥ W_{hh} ∥^{50} \leq 1 0^{2}$ . Why we may root through: $u \mapsto u^{1/50}$ is strictly increasing on $u > 0$ , so it preserves the direction of both $\leq$ signs. Take 50th roots of all sides: $1 0^{- 2/50} \leq ∥ W_{hh} ∥ \leq 1 0^{2/50}$ $0.912 \leq ∥ W_{hh} ∥ \leq 1.096$ An extremely narrow band around 1. Real training can't hold the spectral norm in a $\pm 9%$ window for 50 steps — hence the need for gated highways where the effective multiplier ( $f_{t} \approx 1$ ) is learned rather than luck.

Recall Solution 4.2

$0.9 8^{200} \approx 0.0176$ About $1.8%$ survives over 200 steps — usable for learning. The forget gate (LSTM Architecture) is doing the heavy lifting: it keeps the per-step multiplier near 1 additively. Gradient Clipping never activates on the vanishing side (clipping only caps large norms) — it is insurance against the occasional exploding spike, not a cure for decay. Two different jobs, two different tools.

Recall Solution 4.3

Differentiate: $\frac{\partial h _{t}}{\partial h _{t - 1}} = I + diag (tanh^{'} (z_{t})) W_{hh}$ , where $I$ is the identity matrix (leaves any vector unchanged). The identity term is the key: even if the tanh-weight part shrinks to 0, the derivative floors at $I$ , so the product $\prod (I + \dots)$ never collapses to zero. This is exactly the LSTM's additive cell path ( $\partial C_{t} / \partial C_{t - 1} = f_{t}$ with $f_{t} \approx 1$ ) and the idea behind Residual Connections and Attention Mechanisms — give gradients an un-multiplied shortcut back through time.

Level 5 — Mastery

Recall Solution 5.1

(a) Plain RNN: $0.8 5^{60} \approx 5.8 \times 1 0^{- 5}$ . LSTM: $0.99 5^{60} \approx 0.741$ . (b) Threshold is $1 0^{- 3}$ . Plain RNN gives $5.8 \times 1 0^{- 5} < 1 0^{- 3}$ ✗ — it cannot learn the cat→was agreement; the singular subject signal has vanished. LSTM gives $0.741 > 1 0^{- 3}$ ✓ — the highway keeps ~ $74%$ of the signal, so the agreement is learnable. This is the parent note's motivating example made quantitative.

Recall Solution 5.2

(a) $\frac{S ( 30 )}{S ( 10 )} = r^{20} = \frac{0.216}{0.60} = 0.36$ , so $r = 0.3 6^{1/20} \approx 0.9500$ . Then $a = S (10) / r^{10} = 0.60/0.9 5^{10} = 0.60/0.5987 \approx 1.002$ . (b) Solve $a r^{g} < 1 0^{- 2}$ . Taking logs is legal because $ln$ is strictly increasing (preserves $<$ ), and dividing by $ln r$ flips the inequality since $ln r < 0$ : $g > \frac{ln ( 9.98 \times 1 0 ^{- 3} )}{ln 0.95} \approx \frac{- 4.610}{- 0.0513} \approx 89.9$ . So beyond a gap of about 90 steps the gradient drops below $1 0^{- 2}$ .

Recall Solution 5.3

What axes does BatchNorm act on? In an RNN the hidden state at step $t$ for one example is a vector $h_{t}$ . During training we process a batch of $N$ sequences at once, so at each time step we hold a tensor of shape (batch $N$ , features $H$ ). BatchNorm computes, for each feature channel, the mean and variance across the $N$ examples and rescales that channel to zero-mean/unit-variance. Its averaging axis is the batch axis ( $N$ different sequences), one normalisation per time step.

Steel-man (why it seems plausible): in feedforward nets this keeps each pre-activation $z$ near 0, so $tanh^{'} (z)$ stays near its max of 1 — it genuinely fights the activation-factor shrinkage.

Refutation — the dimension mismatch: the vanishing product $\prod_{k} diag (tanh^{'} (z_{k})) W_{hh}$ accumulates along the time axis ( $k = t + 1, \dots, T$ ) within a single sequence. BatchNorm never touches that axis — it only averages across sequences at a fixed time step. So it operates on the batch dimension while the decay lives on the time dimension: normalising one sequence-vs-another does nothing to the chain of multiplications running down a single sequence's timeline, and it does nothing to $∥ W_{hh} ∥$ (a weight, not an activation). It can slightly reduce saturation, but the temporal weight-factor decay is fully intact. The real fix is architectural — gates (LSTM Architecture, GRU Architecture), residual shortcuts (Residual Connections), or Attention Mechanisms.