Vanishing gradients in RNNs: Visual walkthrough

This page rebuilds the central result of the parent note from absolute zero. We will draw every idea before we write a single symbol. By the end you will see why a signal that has to travel back through 50 time steps arrives as a whisper — and why that one fact broke plain RNNs.

Prerequisite ideas we lean on: Backpropagation Through Time, Activation Functions. The cures we point to at the end: LSTM Architecture, GRU Architecture, Gradient Clipping, Residual Connections, Attention Mechanisms.

Step 1 — What is a hidden state, and why a chain?

WHAT. An RNN reads a sequence one item at a time (word 1, word 2, …). After reading item $t$ it holds a memory vector called the hidden state, written $h_{t}$ . Each new memory is built from the previous memory plus the new input.

WHY. This is the only way a network with fixed size can summarise a sequence of any length: it keeps overwriting one memory slot. That overwriting is exactly what creates a long chain — and the chain is where trouble lives.

PICTURE. Look at the beads on a string below. Each bead $h_{t}$ is fed forward into the next bead by an arrow. The input $x_{t}$ (the peach box) is stirred in at every bead.

Step 2 — What is a gradient, and which one do we want?

WHAT. Training means: the network makes a mistake at the end, measured by a number $L$ (the loss). We want to know how much each early memory was to blame. That "how much to blame" is the gradient $\frac{\partial L}{\partial h _{t}}$ — literally "if I nudge $h_{t}$ a tiny bit, how much does the loss change?"

WHY. We can only fix what we can blame. If the blame signal for $h_{t}$ arrives as zero, the early part of the sequence is never corrected — the network stays deaf to long-range clues like the word cat deciding was vs were.

PICTURE. The forward beads (Step 1) now get a backward red arrow layered on top: blame flows from the loss $L$ at the far right back toward $h_{t}$ on the left.

Step 3 — Why the trip unfolds into a product

WHAT. A nudge at $h_{t}$ can only reach $h_{T}$ by passing through every bead in between: $h_{t} \to h_{t + 1} \to \dots \to h_{T}$ . The chain rule of calculus says: to compose steps, you multiply their local sensitivities.

WHY. Each bead only listens to the one just before it. There is no shortcut edge. So the total sensitivity is the product of every one-step sensitivity along the road.

PICTURE. Each link in the string is stamped with a little factor $\frac{\partial h _{k}}{\partial h _{k - 1}}$ . The whole trip is all those stamps multiplied together.

Step 4 — What one link actually equals

WHAT. Open up a single link. Because $h_{k} = tanh (z_{k})$ with $z_{k} = W_{hh} h_{k - 1} + W_{x h} x_{k} + b_{h}$ , differentiating the composition gives two pieces multiplied.

WHY. The chain rule again, but within one step: first the $tanh$ squasher contributes its slope, then the linear mixer $W_{hh}$ contributes itself. Two dials on every link.

PICTURE. Each link is a two-stage gearbox: dial ① is the $tanh$ slope (a value between $0$ and $1$ ), dial ② is the memory-mixer $W_{hh}$ .

Step 5 — Multiply the links: the exponential

WHAT. Stack $T - t$ identical-ish gearboxes in a row and multiply. Using the size ("norm") of a product being at most the product of sizes, we get a bound.

WHY. We cannot compute the exact matrix product for every sequence, but we can bound its magnitude — and the bound already tells the whole doom story.

PICTURE. A ladder of factors. Since dial ① $\leq 1$ , the surviving per-step multiplier is essentially $∥ W_{hh} ∥$ (the "stretch factor" of the mixer). Multiply it against itself once per rung.

Step 6 — Feel the numbers (the worked example)

WHAT. Put in real values. Take a scalar toy memory $h_{t} = 0.9 h_{t - 1} + x_{t}$ , so $γ = 0.9$ , over a gap of $50$ steps.

WHY. Abstract "exponential decay" only bites once you watch $0.9$ nose-dive. Then add $tanh$ saturation on top and watch it hit machine zero.

PICTURE. A curve of $γ^{n}$ vs the gap $n$ , on a log axis, with the two markers $n = 10$ and $n = 50$ dropped in.

Step 7 — The degenerate & edge cases (nothing left unshown)

WHAT. We now sweep every regime so no reader meets a surprise. The figure below draws the four decay-curve regimes (vanish, ridge, explode, zero-gap); the table then adds two sub-cases of vanishing (deep vs mild) and the zero-input check for completeness.

WHY. The Contract: cover all signs and limits.

PICTURE. A four-panel map of the four core regimes, each panel a tiny gradient-over-gap sketch.

Formula Every regime on the ridge

Case	$γ = ∥ W_{hh} ∥$	$tanh^{'}$	Trip $γ^{T - t}$	Meaning
Deep vanishing	$< 1$	saturated $\approx 0$	$\to 0$ fast	learns nothing long-range
Mild vanishing	$< 1$	active $\approx 1$	$\to 0$ slowly	short memory only
Knife-edge	$= 1$	$\approx 1$	$\approx$ const	ideal, unstable
Exploding	$> 1$	$\approx 1$	$\to \infty$	see Exploding Gradients, Gradient Clipping
Zero gap	$T = t$	—	empty product $= 1$	no decay: you are the loss step
Zero input	$x_{k} = 0$	slope from $b_{h}, h$ only	same law	decay unchanged, driven by $W_{hh}$

The four panels in the figure show the four qualitatively different curve shapes (vanish, ridge, explode, zero-gap); "deep vs mild vanishing" are two speeds of the same vanish shape, and "zero input" leaves the shape unchanged — so they share panels rather than needing their own.

Step 8 — The cure in one glance: turn $\times$ into $+$

WHAT. Every fix replaces the multiplicative chain with an additive highway. In an LSTM Architecture the cell state updates as $C_{t} = f_{t} ⊙ C_{t - 1} + i_{t} ⊙ \tilde{C}_{t}$ , whose backward link is just $\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}$ .

WHY. If the forget gate $f_{t} \approx 1$ , the trip becomes $1 \times 1 \times \dots \times 1 = 1$ — a gradient highway with no toll. GRU Architecture, Residual Connections, and Attention Mechanisms all exploit the same "add, don't repeatedly multiply" trick.

PICTURE. Two tracks side by side: the RNN's shrinking multiply-chain vs the LSTM's flat add-highway.

The one-picture summary

This final figure compresses everything: blame enters at the loss (right), passes through $T - t$ gearboxes each with dials $tanh^{'} \leq 1$ and $W_{hh}$ , gets multiplied into $γ^{T - t}$ , and arrives at the early step as a whisper — unless we swap the chain for an additive highway.

Recall Feynman retelling — say it back in plain words

An RNN keeps one memory bead per step, each fed from the last. To learn from a mistake at the end, we send a "blame signal" backward through every bead. Because each bead only touches its neighbour, the total blame is a product of one-step sensitivities — one factor per step apart. Each factor is a $tanh$ slope (never bigger than 1) times the memory-mixer $W_{hh}$ . If $W_{hh}$ 's stretch factor $γ$ is below 1, multiplying it $T - t$ times gives $γ^{T - t}$ , which crashes toward zero as the gap grows. Saturated $tanh$ piles on more shrinkage. So blame for distant beads arrives as essentially nothing, and the early steps never learn. Fixes (LSTM, GRU, residuals, attention) all break the multiply-chain into an add-highway where the per-step factor is $\approx 1$ , so blame travels far intact.

Self-test:

Which factor in a single backward link can only shrink the gradient, never grow it?

The

tanh

derivative

diag (tanh^{'} (z_{k}))

, since

tanh^{'} \leq 1

everywhere.

What is

\frac{\partial h _{T}}{\partial h _{t}}

when

T = t

, and why?

It is

1

— an empty product — because there are no links to traverse; a state is perfectly sensitive to itself.

Which single quantity must be

< 1

for the Step 5 bound to guarantee decay?

The spectral norm

∥ W_{hh} ∥ = γ

Why does the LSTM cell state dodge the problem?

Its backward link is the additive gate

\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}

; with

f_{t} \approx 1

the trip stays near

1

instead of shrinking.