Limitations of RNNs motivating transformers: Visual walkthrough

This is the picture-story of the single most important reason RNNs struggle: the learning signal that should reach the start of a long sentence gets multiplied by itself hundreds of times and shrinks to zero. We will build this from absolutely nothing — no calculus assumed, every symbol earned before use.

Step 1 — What is a hidden state, and what is a "chain"?

WHAT. An RNN reads a sequence one item at a time. At each moment $t$ it holds a memory vector called the hidden state $h_{t}$ . Think of $h_{t}$ as "everything the network remembers so far, packed into a list of numbers."

WHY. Before we can talk about a signal flowing back through the network, we must see that the states form a single-file chain — each one built only from the one before it. That chain is the road every signal must travel.

PICTURE. Below, each box is a hidden state. The red arrow is the one rule that governs everything: $h_{t}$ is made from $h_{t - 1}$ .

Key term: the memory at time $t$ is called the hidden state.

Step 2 — What is a gradient, and why does it "flow backward"?

WHAT. After the network guesses, we measure how wrong it was with a single number $L$ , the loss. To improve, we ask of every number in the network: "if I nudge you a little, does the loss go up or down, and how fast?" That sensitivity is the gradient.

WHY. We choose the gradient (and not, say, just trial-and-error) because it is the one tool that tells us the direction of steepest improvement for millions of parameters at once. Learning = follow the gradient downhill.

PICTURE. The loss lives at the end of the chain (step $T$ ). But the memory we want to fix — the one that read the word "not" — lives near the start. So the sensitivity has to travel backward along the same chain, box to box.

This backward travel is Backpropagation Through Time. The question of this whole page is: does the gradient survive the trip from $h_{T}$ back to $h_{1}$ ?

Step 3 — The single link: how much signal survives one step back?

WHAT. To go back one box — from $h_{t}$ to $h_{t - 1}$ — we multiply by one local factor, $\frac{\partial h _{t}}{\partial h _{t - 1}}$ . Let us open it up.

WHY. If we understand one link of the chain, we understand all of them, because every link has the exact same form (same $W_{hh}$ ). One link is the atom of the whole problem.

PICTURE. The factor has two pieces multiplied together — the squasher's slope and the mixing grid.

Why does $tanh^{'}$ show up at all? Because $h_{t}$ was squashed through $tanh$ ; the chain rule says the sensitivity through a squash is scaled by the squash's steepness. That steepness is never bigger than $1$ — the crucial fact of Step 4.

Step 4 — Why the squasher's slope is a leak (never bigger than 1)

WHAT. The function $tanh (z)$ takes any number and gently bends it into the range $(- 1, 1)$ . Its slope $tanh^{'} (z) = 1 - tanh^{2} (z)$ is at most $1$ , and shrinks toward $0$ as inputs grow large.

WHY. We need to know the biggest possible value of one link's shrink factor. The $tanh^{'}$ piece guarantees each step can only hold or lose signal — never amplify it through the squash. It is a valve that leaks.

PICTURE. The red curve is the slope $tanh^{'}$ . Its peak is exactly $1$ (at $z = 0$ ); everywhere else it is smaller and drops to $0$ in the tails.

Step 5 — Collapse the whole chain: one factor raised to a power

WHAT. Going all the way from the loss back to $h_{1}$ means walking every link. The chain rule turns "walk the road" into "multiply all the link factors together."

WHY. This is the moment the disaster becomes visible. Because every link has the same size (same $W_{hh}$ , same average slope $γ$ ), multiplying them $T - 1$ times is the same as raising one number to a power — and powers of a number below $1$ crash toward zero.

PICTURE. The gradient at the start equals the gradient at the end times a product of identical shrinks — a tower of the same factor.

Step 6 — Watch it die: the three regimes of $ρ$

WHAT. Everything now hinges on whether $ρ$ is below, above, or exactly $1$ . All three cases must be covered — the reader should never meet a scenario we skipped.

WHY. A power behaves wildly differently on each side of $1$ . Only $ρ = 1$ preserves signal; the vanishing case ( $ρ < 1$ ) is the one that dooms long-range learning.

PICTURE. Three curves of $ρ^{T - 1}$ against chain length $T$ : one crashing (red, $ρ < 1$ ), one flat ( $ρ = 1$ ), one exploding ( $ρ > 1$ ).

Recall Why do the gates in

LSTMs and GRUs only delay this? Because the forget gate $f_{t} \in (0, 1)$ still multiplies the cell-state gradient: $\frac{\partial c _{t}}{\partial c _{t - 1}} = f_{t}$ . A product of numbers below $1$ still decays — just more slowly (range ~100 instead of ~10). The power structure never disappears; it is only softened. ::: The exponential-decay picture returns for long enough sequences.

Step 7 — The escape: replace the chain with one hop

WHAT. The fix is not a better $W_{hh}$ . It is to delete the chain. Self-attention lets position $T$ read position $1$ directly, so the gradient path between them is one link, not $T - 1$ .

WHY. If there is only one link, there is no power $ρ^{T - 1}$ to collapse. $ρ^{0}$ … there simply is no tower. Distance stops mattering. (The price is $O (T^{2})$ work per layer — see Computational Complexity of Sequence Models — but it is parallel and gradient-safe.)

PICTURE. Left: the RNN's long fragile chain. Right: the transformer's direct red edge from far to near — path length $1$ .

The one-picture summary

One vector road ( $h_{1} \to \dots \to h_{T}$ ); a gradient sent home from the end; each step multiplies it by the same leaky factor $ρ$ ; after $T - 1$ steps that becomes $ρ^{T - 1}$ , which vanishes when $ρ < 1$ . Self-attention swaps the road for a single hop, and the problem evaporates.

Recall Feynman retelling — say it like you'd tell a friend

"An RNN remembers by passing a note down a line: each person rewrites the note using the same rulebook. When the network is wrong, we want to send blame backward down that line to the person who first made a mistake. But every hand-off keeps only a fraction — say 95% — of the blame. Pass it back through 300 people and 95% × 95% × … × 95% is basically zero, so the first person never hears they messed up and never fixes themselves. The math name is $ρ^{T - 1}$ : one shrink-factor raised to the length of the line. If the factor is under 1 the blame vanishes; if it's over 1 it explodes; only exactly 1 is stable, and you can't sit on a knife-edge. LSTM gates make the note fade more slowly but the fading still wins for long lines. Transformers throw out the line entirely: person 300 can whisper straight to person 1, one hop, no fading — and every person can talk at once, so it's fast too."

Retell all seven steps aloud from this. If you can, you own the page. ::: Chain → gradient must travel it → one link = $tanh^{'} \cdot W_{hh}$ → $tanh^{'} \leq 1$ leaks → links multiply into $ρ^{T - 1}$ → below 1 it dies → attention makes path length 1.