Limitations of RNNs motivating transformers: Question bank

This page is a self-test for the ideas in Limitations of RNNs motivating transformers. Each line is a trap that catches most learners. Read the left side, say your answer OUT LOUD, then reveal the right side. If your reason doesn't match the reasoning given, you found a gap — go back and re-read.

Every symbol below appears in the parent note; if any feels unfamiliar, here is the one-line reminder before we start:

Definition Symbols used on this page

$x_{1}, \dots, x_{T}$ ::: the input tokens (words), fed in order; $T$ is how many there are.
$h_{t}$ ::: the "hidden state" — a vector holding what the RNN remembers after reading token $t$ .
$W_{hh}$ ::: the matrix that mixes the old memory $h_{t - 1}$ into the new memory.
$tanh$ ::: the squashing function that keeps $h_{t}$ between $- 1$ and $1$ .
$tanh^{'}$ ::: the slope of $tanh$ ; it equals $1 - tanh^{2}$ , always between $0$ and $1$ .
$γ$ ::: a single "average" value of $tanh^{'}$ over the steps — a shorthand for the typical per-step shrink factor from the squashing.
$σ_{m a x} (W_{hh})$ ::: the largest stretch factor of the matrix $W_{hh}$ (its maximum singular value) — the most any vector can be lengthened when multiplied by $W_{hh}$ .
$L$ ::: the "loss", one number saying how wrong the model's final answer is.
$\frac{\partial L}{\partial h _{1}}$ ::: how much the loss changes if we nudge the first memory — the "learning signal" for early words.

True or false — justify

RNNs cannot be parallelized at all, ever.

False. The time axis cannot be parallelized (each

h_{t}

needs

h_{t - 1}

), but many sequences in a batch can run side by side. The bottleneck is depth-in-time, not total work.

The vanishing gradient problem is a bug that better code could fix.

False. It follows from repeatedly multiplying by numbers

< 1

(

tanh^{'} \leq 1

times a small

W_{hh}

); the exponential decay

(something < 1)^{T}

is baked into the recursive structure itself.

LSTMs completely eliminate vanishing gradients.

False. They mitigate it via the additive cell-state highway, but the forget gate

f_{t} < 1

still multiplies the gradient each step, so decay returns for very long sequences (~1000+ tokens).

σ_{m a x} (W_{hh}) \cdot γ > 1

, gradients vanish faster.

False. When the per-step factor (largest stretch of

W_{hh}

times the typical

tanh^{'}

shrink) exceeds 1, gradients explode (blow up) instead of vanishing — the opposite failure, but still caused by the same repeated multiplication.

A transformer does less total computation than an RNN.

False. Self-attention makes each position do

O (T)

work (it looks at all others), so total work can be more; the win is that this work happens in parallel, not in sequence.

The compression bottleneck only matters for very long inputs.

False. It bites whenever the input's information (in bits) exceeds what one fixed vector

h_{T} \in R^{d}

can hold; even moderate sentences lose fine detail because

d lo g_{2} d

bits is a hard ceiling.

Making the hidden state bigger (

d

larger) fully solves the compression bottleneck.

False. It raises the ceiling but capacity grows only like

O (d lo g_{2} d)

, while required bits grow like

T lo g_{2} V

; long enough inputs still overflow, and big

d

costs memory and slows training.

In BPTT the gradient to word 1 passes through one matrix multiply.

False. It passes through the whole chain

\frac{\partial h _{T}}{\partial h _{T - 1}} \dots \frac{\partial h _{2}}{\partial h _{1}}

— that's

T - 1

multiplications, exactly what causes the decay. Self-attention is what gives a single-hop path.

Spot the error

h_{3}

can be computed as soon as we have

x_{3}

Error:

h_{3} = tanh (W_{hh} h_{2} + \dots)

needs

h_{2}

, which needs

h_{1}

. You need every earlier hidden state first, not just the current input token.

"Because

tanh^{'} (z) \in (0, 1]

, gradients always shrink by exactly the same factor each step."

Error: the factor

tanh^{'} (z)

varies with

z

and combines with

σ_{m a x} (W_{hh})

; we use an average

γ

only as an estimate. The decay is approximate, not a fixed constant.

"The cell-state update

c_{t} = f_{t} ⊙ c_{t - 1} + i_{t} ⊙ \tilde{c}_{t}

has no gradient decay."

Error:

\frac{\partial c _{t}}{\partial c _{t - 1}} = f_{t}

, and

f_{t}

is between 0 and 1, so gradients still shrink whenever the network chooses to forget — which it usually must.

"A sequence of

T

tokens from vocabulary

V

needs

T \cdot V

bits to store."

Error: each token needs

lo g_{2} V

bits (which of

V

words), so the sequence needs

T lo g_{2} V

bits, far less than

T \cdot V

"For a batch of 32 sequences each 512 tokens long, the RNN takes 512/32 ≈ 16 sequential timesteps."

Error: here 512 is the sequence length (tokens per sentence) and 32 is the batch size. Batching parallelizes over sequences, not time, so it still takes the full 512 timesteps; the 32 sequences just march through those steps together.

"Transformers are faster because self-attention is cheaper per position."

Error: per position it's actually more expensive (

O (T)

O (1)

). The speed comes from doing all positions at once, so wall-clock time drops even though work rises.

Why questions

Why can't we "skip ahead" to compute a late hidden state directly?

Because each

h_{t}

is a non-linear function of

h_{t - 1}

; there's no closed-form shortcut that jumps past the intermediate states, so they must be built one at a time.

Why do modern GPUs sit mostly idle when running an RNN?

A GPU has thousands of cores, but the time-recurrence lets only one timestep run at a time, so the remaining cores have nothing to do until that step finishes.

Why does the "not ... terrible" example fail for a vanilla RNN?

The gradient linking the far-apart words must travel ~295 recursive steps; multiplied by a factor <1 each step it becomes ~

1 0^{- 7}

, far too weak to teach the model that early "not" flips the meaning.

Why does self-attention avoid this exponential decay?

Position 1000 attends to position 1 through a single weighted connection, so the gradient between them passes through one hop, not 999 multiplications — no exponential shrinking. See Self-Attention Mechanism.

Why is the encoder's final vector called a "bottleneck"?

All of the input's information has to be squeezed through one fixed-size vector

h_{T}

; like a narrow pipe, it limits how much can pass, forcing detail to be dropped. See Encoder-Decoder Architecture.

Why does making

f_{t}

close to 1 keep long-term memory but risk instability?

With

f_{t} \approx 1

the cell state barely forgets, so gradients survive — but old information never fades, so the cell can accumulate and blow up (exploding activations). See LSTMs and GRUs.

Why does the vanishing gradient problem count as a training failure, not a prediction failure?

The forward pass still runs fine; it's the backward learning signal that dies, so the model can never learn long-range rules even though it could in principle use them.

Edge cases

What happens to gradients when the per-step factor

σ_{m a x} (W_{hh}) \cdot γ

equals exactly 1?

They neither vanish nor explode — the ideal knife-edge — but it's practically impossible to stay there for many steps, so real training drifts to vanishing or exploding. See Vanishing and Exploding Gradients.

What if the sequence length

T = 1

There is no recurrence to unroll, so parallelization, vanishing gradients, and the recurrence bottleneck all disappear; the RNN degenerates to a single feed-forward step.

What if two sequences in a batch have very different lengths?

The whole batch still marches for as many timesteps as the longest sequence (shorter ones are padded), so the slow-per-timestep problem is set by the longest input, not the average.

Does a bidirectional RNN remove the sequential bottleneck?

No. It adds a second pass reading right-to-left, but each direction is still a strict left-to-right (or right-to-left) chain — it doubles the passes, it doesn't parallelize time. See Backpropagation Through Time.

If we shorten inputs enough (say

T < 10

), do RNN limitations vanish?

Largely yes for gradients — 10 steps of a factor like 0.95 loses only ~40% signal — but you also lose the ability to model any genuinely long-range structure, so it's a workaround, not a fix. See Computational Complexity of Sequence Models.

What about attention added on top of an RNN (pre-transformer models)?

It relieves the compression bottleneck (the decoder can look back at all

h_{t}

), but the encoder still processes tokens sequentially, so the parallelization problem remains — which is why transformers dropped recurrence entirely. See Positional Encoding.

Recall Quick self-check

Name the three distinct RNN limitations. ::: (1) Sequential processing — can't parallelize over time; (2) vanishing/exploding gradients over long chains; (3) fixed-size compression bottleneck in the encoder vector. Which single transformer idea attacks all three at once? ::: Self-attention: it computes all positions in parallel, gives one-hop gradient paths, and lets the decoder read every input position directly.