Foundations — Limitations of RNNs motivating transformers

Before you can understand why Recurrent Neural Networks (RNNs) struggle, you need to be fluent in the symbols the parent note throws at you. This page builds each one from nothing — plain words first, a picture second, and a reason it matters third. Nothing here assumes you have seen the notation before.

1. A sequence and its tokens: $x_1,x_2,\dots ,x_T$

Picture it. Imagine beads threaded on a wire, left to right. Each bead is a token; its position on the wire is its subscript.

Why the topic needs it. Every complaint about RNNs is really a complaint about how they walk along this wire. If $T$ is small (a 3-word phrase) nothing goes wrong; the trouble only appears when $T$ is large. So we must name the position and the length before we can say "the memory of $x_1$ fades by the time we reach $x_{1000}$".

2. A vector and the space ${\mathbb{R}}^d$

Each token isn't fed to the network as a word — it's turned into a list of numbers called a vector.

Picture it. A vector in ${\mathbb{R}}^3$ is an arrow in 3-D space. We can't draw ${\mathbb{R}}^{512}$, but the idea is the same: a point whose location is given by $d$ coordinates. Think of it as $d$ dials, each turned to some value.

Why the topic needs it. The parent note says the RNN's memory is a vector $h_T\in {\mathbb{R}}^d$. The whole "compression bottleneck" argument compares how many numbers we have ($d$ of them) to how much information we must store. You cannot follow that argument until "${\mathbb{R}}^d$ = a box holding exactly $d$ numbers" is second nature.

3. The hidden state $h_t$ — the memory

Picture it. A single sticky note the network keeps rewriting. Read a word → cross out the old note → write a new summary. The sticky note is always the same size ($d$ numbers), no matter how long the sentence.

Why the topic needs it. "Information loss" and "fixed compression bottleneck" both come from the fact that this one sticky note has a fixed size and gets overwritten every step. Name the sticky note $h_t$ and the arguments become concrete.

4. The recurrence $h_t=f(h_{t-1},x_t)$

This is the single most important line in the whole parent note. Let us earn every piece.

Picture it. A machine with two input chutes — old memory drops in one chute, the new word drops in the other — and one output chute producing the new memory. Then you feed that output back into the old-memory chute for the next word. That feed-back loop is what "recurrent" means.

Why this notation and not a plain formula? Writing it as $h_t=f(h_{t-1},x_t)$ hides the internal details of $f$ so we can focus on the dependency: to get $h_t$ you must already have $h_{t-1}$. That dependency — one step chained to the last — is exactly the "sequential bottleneck". The notation was chosen to make the chain visible.

5. The concrete update: $\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$

The parent note replaces the abstract $f$ with a specific formula. Let's decode each symbol.

Picture it. Each matrix is a mixing board: sliders that decide how much each old number contributes to each new number. $W_{hh}$ mixes memory-with-itself; $W_{xh}$ stirs in the new word; $b_h$ nudges the result.

Why $\tanh$ and not just leave the numbers? Without squashing, repeatedly multiplying by $W_{hh}$ every step would let the memory numbers blow up to infinity. $\tanh$ keeps every memory value politely between $-1$ and $+1$. But — and this is the seed of the vanishing-gradient problem — its slope is almost always less than $1$ (steepest, slope $1$, only at $z=0$; nearly flat at the edges). That slope-below-one is what shrinks gradients later.

6. "Big-O" time: $O(T)$ and $O(1)$

The parent note keeps writing $O(T)$ and $O(1)$. These are just size labels.

Picture it. $O(T)$ is a line of people passing a bucket one to the next — a chain of $T$ hand-offs. $O(1)$ is everyone dunloading their own bucket at the same moment.

Why the topic needs it. "Cannot parallelize" is precisely the claim that the RNN forward pass is $O(T)$ sequential steps, while a transformer is $O(1)$ sequential depth. The notation is the shorthand that lets us compare them in one glyph.

7. Gradients and the derivative $\frac{\partial \mathcal{L}}{\partial h_t}$

Picture it. Imagine the loss as the height of a hill and $h_t$ as a knob you can turn. The gradient is the steepness you feel when you wiggle the knob — which way, and how strongly, is downhill.

Why the topic needs it. Learning happens by sending these steepness-signals backward through the chain (this is Backpropagation Through Time). Because the chain multiplies many small $\tanh$ slopes together, the signal reaching early tokens becomes microscopic — the vanishing gradient. You need the derivative symbol before that story makes sense. See Vanishing and Exploding Gradients and Backpropagation Through Time for the full mechanics.

8. Information in bits, and ${\log }_2$

The compression argument needs one more tool.

Why the topic needs it. The bottleneck argument compares the bits needed to store a sentence ($T{\log }_{2}V$) against the bits a vector holds. You can't weigh those two without knowing what a bit and ${\log }_2$ mean.

How these foundations feed the topic

Each foundation on the left is a prerequisite for one of the three limitations in the parent topic. Related deep vault notes: LSTMs and GRUs, Self-Attention Mechanism, Positional Encoding, Encoder-Decoder Architecture, Computational Complexity of Sequence Models.

Equipment checklist

Cover the right side and answer each before moving on.