Worked examples — Limitations of RNNs motivating transformers

This is the hands-on companion to the parent note. There we argued why RNNs struggle; here we compute the failure in every scenario it can appear. Nothing new is assumed — every symbol is rebuilt as it enters.

Before we start, three tiny reminders (each is a picture you should hold in your head):

One more symbol before the formula: $n$ = the number of boxes (hand-offs) the gradient must cross to get from where it starts to where it lands. Picture counting the red arrows between two boxes; $n$ is how many arrows you pass through.

The one formula that powers almost everything below:

The figure below makes this concrete: the outlined bars are the original strength $1$ at each box, the filled blue bars are what actually survives after each red "$\times r$" hand-off, and the yellow numbers are $g_n=0.95^n$ for $n=0,\dots ,5$. Watch the blue bar shrink one hand-off at a time — that shrinking is the vanishing gradient.

---

The scenario matrix

Every question this topic can throw at you lives in exactly one of these cells. The worked examples afterward are labelled by cell, and together they hit all of them.

Cell	Regime / input	What decides it	Example
A	$r<1$ — vanishing	many boxes, leaky bucket	Ex 1
B	$r>1$ — exploding	many boxes, overflowing bucket	Ex 2
C	$r=1$ — the knife-edge	perfect preservation	Ex 3
D0	Degenerate: $r=0$	dead weights, one-step wipeout	Ex 4
D1	Degenerate: $T=1$	only one box, no chain at all	Ex 5
E	Limiting: $T\to \infty$	what happens to $r^T$ as $T$ grows without bound	Ex 6
F	Parallel-time cost	sequential $O(T)$ vs parallel $O(1)$	Ex 7
G	Real-world word problem	translate a story into $r$, $n$, $T$	Ex 8
H	LSTM partial-fix	forget gate $f_t$ replaces $r$	Ex 9
I	Exam twist: bottleneck bits	information capacity vs demand	Ex 10

Prerequisites you can revisit if a step feels shaky: Vanishing and Exploding Gradients, Backpropagation Through Time, Computational Complexity of Sequence Models, LSTMs and GRUs, Encoder-Decoder Architecture.

---

Example 1 — Cell A: the vanishing case

1. Identify the numbers. $r=0.95$, $n=100$, $g_0=1$. Why this step? Every gradient problem reduces to plugging into $g_n=g_0{r}^n$. First we must know which number is which — a wrong assignment poisons everything.

2. Apply the power formula. $g_{100}=1\cdot (0.95{)}^{100}.$ Why this step? Each of the 100 boxes multiplies the signal by $0.95$; stacking 100 of them is the power $0.95^{100}$ (see the leaking-bucket figure).

3. Evaluate. $(0.95{)}^{100}\approx 0.00592$. Why this step? We need an actual number to feel the size. A quick sanity anchor: $0.95^{10}\approx 0.60$, and $100$ steps is ten such blocks, so $\approx 0.60^{10}$, which is tiny — consistent.

Answer: $\approx 0.0059$. The signal is about 170× weaker — practically gone. The RNN cannot learn that the far-away word mattered.

Recall Verify (sanity check)

Since $r<1$ and $n$ is large, $r^n$ must be a small positive number below $1$. $0.0059$ is small, positive, below $1$. ✓ Units: gradients are dimensionless ratios, so a pure number is correct.

---

Example 2 — Cell B: the exploding case

1. Assign. $r=1.1$, $n=80$, $g_0=1$. Why this step? Same machinery — only $r$ crossed the threshold $1$, which flips leak into overflow.

2. Apply. $g_{80}=(1.1{)}^{80}$. Why this step? 80 amplifications compound multiplicatively, exactly the same power structure, just with $r>1$.

3. Evaluate. $(1.1{)}^{80}\approx 2.05\times 10^3$… let's be precise: $(1.1{)}^{80}\approx 2048$. Why this step? We need magnitude. Anchor: $1.1^{10}\approx 2.59$, eight blocks give $\approx 2.59^8$, which is in the thousands — consistent.

Answer: $\approx 2048$. The gradient blows up ~2000×; training diverges (loss becomes NaN). This is the exploding twin of vanishing — same formula, $r>1$.

Recall Verify

$r>1$, large $n$ ⇒ $r^n\gg 1$. $2048\gg 1$. ✓ Note $2048=2^{11}$; and indeed $1.1^{80}$ rounds to $2048$ — checked in VERIFY.

---

Example 3 — Cell C: the knife-edge $r=1$

1. Assign. $r=1$, $n=500$.
2. Apply. $g_{500}=1\cdot 1^{500}$. Why this step? $1$ raised to any power is $1$ — the bucket neither leaks nor overflows.
3. Answer: exactly $1$.

Meaning: $r=1$ is the ideal the whole field chases. LSTMs try to sit near it with their cell-state "highway"; transformers sidestep the chain entirely so there is no $r$ to tune. See Vanishing and Exploding Gradients.

Recall Verify

$1^{500}=1$. ✓ This is the boundary between Cells A ($r<1\to 0$) and B ($r>1\to \infty$).

---

Example 4 — Cell D0: degenerate dead weights, $r=0$

1. One box. $g_1=g_0\cdot r^1=1\cdot 0=0$. Why this step? Multiplying by $0$ once annihilates the signal — the very first hand-off empties the bucket completely.

2. Many boxes. $g_n=1\cdot 0^n=0$ for every $n\ge 1$. Why this step? $0^n=0$ for any positive $n$; there is nothing left to keep multiplying, so extra boxes cannot make it "more zero."

3. Contrast with $T=1$. Note the gradient within box 1 itself ($n=0$ hops, $r^0=1$) is untouched; the wipeout only strikes the moment you try to reach an earlier box.

Answer: $r=0$ is the most extreme vanishing — total in a single step, not gradual. This is why practitioners initialize $W_{hh}$ away from zero: dead recurrent weights sever the past instantly. It is the left endpoint of the whole $r$-axis (Cell A is $0<r<1$; this is the boundary $r=0$).

Recall Verify

$0^1=0$ and $0^5=0$, while $0^0=1$ (the $n=0$ / same-box case). ✓

---

Example 5 — Cell D1: degenerate single-step, $T=1$

1. Sequential steps. With $T=1$ there is exactly one hidden state $h_1$ — nothing before it. Depth $=T=1$. Why this step? The bottleneck we complain about is the chain; with one link there is no chain.

2. Gradient hops. The gradient crosses $n=T-1=0$ boxes. Why this step? To reach $h_1$ from $h_1$ you cross zero boxes.

3. Apply. $g_0=g_0\cdot r^0=g_0\cdot 1=g_0$. Why this step? $r^0=1$ by the meaning of "multiply by nothing." The signal is untouched.

Answer: no vanishing, no exploding, perfect parallelizability (there is nothing to parallelize). RNNs are only bad on long sequences — the degenerate case exposes that the disease is length, not the recurrence itself. (Contrast with Cell D0: there the weights were dead; here the chain is trivially short.)

Recall Verify

$r^0=1$ for any $r$ (including $r=0$). So $T=1$ is immune. ✓

---

Example 6 — Cell E: the limit $T\to \infty$

1. Set up the limit. We study $\underset{T\to \infty }{\lim }(0.95{)}^{T-1}$. Why this step? "Limiting behaviour" asks what value we approach, not any single $T$.

2. Reason about the sign of $\ln r$. Since $0<r<1$, $\ln r<0$. Then $r^{T-1}=e^{(T-1)\ln r}$, and $(T-1)\ln r\to -\infty$. Why this step? Writing a power as $e^{(\text{exponent})\ln r}$ turns "repeated multiply" into "add up $\ln r$", where the sign of $\ln r$ decides the fate. This is exactly why the exponential tool appears — it converts multiplicative decay into additive slope.

3. Conclude. $e^{-\infty }=0$.

Answer: $\underset{T\to \infty }{\lim }{r}^{T-1}=0$ for every $r<1$. No matter how gentle the leak, an infinitely long sentence loses all early-word gradient. Contrast the three regimes on the figure below.

Recall Verify

$0.95^{10000}$ is astronomically small (underflows to $0$ in floating point). ✓ For $r=1.1$ the same limit is $+\infty$; for $r=1$ it stays $1$ — all three plotted.

---

Example 7 — Cell F: sequential vs parallel wall-clock

1. RNN time. Sequential steps $=T=512$, each $2$ ms. $t_{\text{RNN}}=512\times 2\text{ ms}=1024\text{ ms}=1.024\text{ s}.$ Why this step? $h_t$ needs $h_{t-1}$, so the 512 updates cannot overlap — they add up. This is the $O(T)$ depth from Computational Complexity of Sequence Models.

2. Transformer time. All 512 positions computed in one parallel wave, so the depth is $O(1)$ — say one $2$ ms wave (ignoring the parallel work each does). $t_{\text{TF}}\approx 2\text{ ms}.$ Why this step? Self-attention lets position 512 read position 1 directly, so there is no forced ordering. See Self-Attention Mechanism.

3. Speedup. $\frac{1024}{2}=512\times$. Why this step? Ratio of wall-clock times gives the practical win.

Answer: ~512× faster forward pass in this idealized model — matching "$O(T)$ vs $O(1)$ depth."

Recall Verify

$512\times 2=1024$ ms; $1024/2=512$. ✓ Units: ms / ms = dimensionless speedup factor.

---

Example 8 — Cell G: real-world word problem

1. Count the hops from loss to "not". From position $300$ back to position $5$ is $n=300-5=295$ boxes. Why this step? The gradient must travel from where the loss is measured to where the important parameter lives. Distance = box count.

2. Apply. $g=(0.95{)}^{295}$. Why this step? Same leaking bucket, now 295 hand-offs deep.

3. Evaluate. $(0.95{)}^{295}\approx 3.1\times 10^{-7}$. Why this step? We need to see whether it is learnable. Anything below ~$10^{-4}$ is effectively drowned by numerical noise.

Answer: $\approx 3\times 10^{-7}$ — the model essentially never learns that "not" flips the sentiment. Exactly the failure the parent note narrated, now a number.

Recall Verify

$295$ hops, $r<1$ ⇒ deeply tiny positive number. $3.1\times 10^{-7}$ ✓ (checked below).

---

Example 9 — Cell H: the LSTM partial fix

1. Vanilla baseline. From Ex 8: $0.95^{295}\approx 3.1\times 10^{-7}$.
2. LSTM version. Replace $r$ by $f_t=0.99$: $g_{\text{LSTM}}=(0.99{)}^{295}.$ Why this step? The additive cell-state path makes the effective per-box multiplier the forget gate, not $W_{hh}$. Bigger multiplier ⇒ slower leak.
3. Evaluate. $(0.99{)}^{295}\approx 0.0522$. Why this step? We compare magnitudes: $0.052$ vs $3\times 10^{-7}$.

Answer: the LSTM keeps $\approx 5.2\%$ of the signal versus $0.00003\%$ — about 170,000× stronger, yet still not $1$. This is precisely the parent's point: LSTMs mitigate, they do not eliminate. Push $n$ to thousands and even $0.99^n\to 0$.

Recall Verify

Ratio $0.99^{295}/0.95^{295}=(0.99/0.95{)}^{295}\approx 1.7\times 10^5$. ✓

---

Example 10 — Cell I: exam twist, information bottleneck

1. Bits available (honest upper bound). The vector has $d=512$ slots, each a $32$-bit float, so the raw container holds at most $B_{\text{avail}}=512\times 32=16,384\text{ bits}.$ Why this step? A float's storage is exactly its bit-width; $512$ of them is $512\times 32$. This is a generous ceiling — real usable capacity is lower because floats waste bits on sign/exponent and because the network cannot exploit every bit pattern independently. We use the ceiling so our conclusion is conservative.

2. Bits required. Each token picks one of $V$ options: ${\log }_{2}V$ bits; $T$ tokens ⇒ $T{\log }_{2}V$. $B_{\text{req}}=1000\times {\log }_{2}50000\approx 1000\times 15.61=15,610\text{ bits}.$ Why this step? Information to reconstruct the exact sequence is bits-per-token times number of tokens.

3. Compare. $B_{\text{req}}=15,610$ vs the ceiling $B_{\text{avail}}=16,384$. They are already the same order of magnitude — and the usable capacity is far below the ceiling, so in practice demand exceeds supply. Why this step? If even the generous ceiling barely clears the demand, the realistic (much smaller) capacity cannot store the sentence losslessly — the encoder must discard information. This is the compression bottleneck of Encoder-Decoder Architecture.

Answer: required $\approx 15,610$ bits sits right against the $16,384$-bit storage ceiling, and effective capacity is well below that ceiling — so long inputs are compressed lossily and degrade. Attention fixes this by letting the decoder look back at all hidden states instead of one squeezed vector.

Recall Verify

$512\times 32=16384$; $1000{\log }_{2}50000\approx 15610$; the two are within a factor of $\approx 1.05$, confirming demand and the storage ceiling are the same order. ✓ Units: bits vs bits, dimensionless comparison.

---

Recall