Recurrent Neural Networks (RNN) architecture: Visual walkthrough

This page builds the RNN update equation from nothing. No prior notation is assumed. If you have never seen a vector, a matrix, or the word "state", start here at line one and we will earn every symbol as we go. The parent note is Recurrent Neural Networks (RNN) architecture — this page is its slow-motion, picture-by-picture replay.

Step 1 — What is a "sequence", and why memory is the whole problem

WHAT. A sequence is just a line of boxes, read left to right, one box at a time. Each box holds one thing: a word, a character, a number. We write the box at position $t$ as $x^{(t)}$ .

$x^{(t)}$ — the input at step $t$ . The little " $(t)$ " up top is not a power. It is a label meaning "the one at time $t$ ". So $x^{(1)}$ is the first thing, $x^{(2)}$ the second, and so on up to $x^{(T)}$ , where $T$ is how many boxes there are.

WHY. A plain neural network sees one box and forgets it instantly before the next box arrives. That is fine for a photo, useless for a sentence — you cannot understand "the mat" if you have already forgotten "the cat sat on". So we must carry something forward.

PICTURE. Below: three input boxes in a row. The dashed amber arrow is the thing we still need to invent — a wire that carries a memory from one box to the next.

Figure — Recurrent Neural Networks (RNN) architecture

Step 2 — Invent the memory vector $h^{(t)}$

WHAT. We invent a second box that lives between the input boxes and carries information forward. Call it the hidden state, written $h^{(t)}$ .

$h^{(t)} \in R^{d_{h}}$ — the symbol $R^{d_{h}}$ just means "a list of $d_{h}$ real numbers". If $d_{h} = 128$ , then $h^{(t)}$ is 128 numbers stacked in a column. $d_{h}$ is a knob we choose (bigger = more room to remember).

WHY a vector and not a single number? One number can only remember one thing (say, "was the last word positive?"). A list of 128 numbers can remember 128 different features at once — tense, subject, sentiment, open brackets… We want richness, so we use a vector.

PICTURE. The memory box $h^{(t)}$ drawn as a stack of little cells; the amber wire from Step 1 now plugs into it and flows rightward.

Step 3 — The memory must depend on two things

WHAT. The new memory $h^{(t)}$ must be built from exactly two ingredients:

the old memory $h^{(t - 1)}$ (what we knew a moment ago), and
the new input $x^{(t)}$ (the word we just read).

So we want a rule of the shape $h^{(t)} = (something that blends) h^{(t - 1)} and x^{(t)} .$

WHY these two and nothing else? If we left out $h^{(t - 1)}$ , there is no memory — we are back to the forgetful network. If we left out $x^{(t)}$ , the machine ignores the actual sentence. Both must go in.

PICTURE. Two arrows feeding a single "blender" node: one arrow labelled $h^{(t - 1)}$ coming from the left, one labelled $x^{(t)}$ coming from below.

Step 4 — How do we "blend" two vectors? Weighted sums (matrices)

WHAT. The simplest fair way to blend two lists of numbers is: scale each one and add them. Scaling a whole vector is done by a matrix.

We introduce two mixing desks and a nudge:

$z^{(t)} = mixes old memory W_{hh} h^{(t - 1)} + mixes new input W_{x h} x^{(t)} + constant nudge b_{h}$

Term by term, right where each lives:

$W_{hh} h^{(t - 1)}$ — takes the old memory and re-mixes it. $W_{hh}$ is the recurrent weight matrix ( $h \to h$ ), size $d_{h} \times d_{h}$ .
$W_{x h} x^{(t)}$ — takes the new input and re-mixes it into memory-shaped numbers. $W_{x h}$ has size $d_{h} \times d_{x}$ (input has $d_{x}$ numbers, memory has $d_{h}$ ).
$b_{h}$ — the bias, a fixed vector added every time. It lets the machine have a default lean even when everything else is zero.
$z^{(t)}$ — the raw blended result, before we tame it in Step 5.

WHY a plain weighted sum? Because it is the most general linear mix, and (crucially) it is easy to differentiate later for training. Any fancier behaviour we need will come from the non-linearity in the next step, not from a complicated blend here.

PICTURE. The blender opened up: two mixing desks $W_{hh}$ and $W_{x h}$ , their outputs summed with $b_{h}$ into $z^{(t)}$ .

Step 5 — Tame the blend with $tanh$

WHAT. The raw sum $z^{(t)}$ can be any size — a thousand, a million. We squash it through the function $tanh$ , applied to every number separately: $h^{(t)} = tanh (z^{(t)}) = tanh (W_{hh} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h}) .$

WHY $tanh$ and not just leave it linear, or use another squash?

Why squash at all? Without a squash, stacking many time steps is just one giant matrix multiply — a purely linear machine that cannot learn curvy patterns. The squash is what gives the network its power.
Why $tanh$ specifically? Its output is centred at zero (ranges $- 1$ to $+ 1$ ), so memories can be positive or negative — useful for "yes/no" style features. A sigmoid (range $0$ to $1$ ) can never say "strongly negative". $tanh$ also keeps values bounded, so memory cannot blow up to infinity as we loop.

PICTURE. The S-curve of $tanh$ with the flat ceilings at $\pm 1$ marked, and an example input arrow being squashed down into range.

That boxed line is the RNN. Everything the parent note states follows from these five moves.

Step 6 — Reading an answer out: the output $y^{(t)}$

WHAT. The memory $h^{(t)}$ is internal. To get a usable answer we project it once more: $y^{(t)} = W_{h y} h^{(t)} + b_{y} .$

$W_{h y}$ — the hidden-to-output desk, size $d_{y} \times d_{h}$ . It reshapes the $d_{h}$ -long memory into the $d_{y}$ numbers the task needs.
$b_{y}$ — output bias.
$y^{(t)} \in R^{d_{y}}$ — the raw output (later passed through softmax for probabilities, or sigmoid for yes/no).

WHY a separate desk instead of using $h^{(t)}$ directly? The memory dimension $d_{h}$ (say 128) rarely equals the answer dimension $d_{y}$ (say 26 letters, or 1 sentiment score). $W_{h y}$ is the translator between "how much I remember" and "what I must report". Separating them lets the memory stay rich while the report stays the right shape.

PICTURE. The memory box feeding a small desk $W_{h y}$ that emits a short output vector $y^{(t)}$ .

Step 7 — Unfold: the same machine reused every step

WHAT. There is only one set of desks $W_{hh}, W_{x h}, W_{h y}$ . We do not build a new machine per word — we reuse the same machine, feeding its own output-memory back into itself. Drawing that reuse spread out across time is called unfolding.

WHY reuse (weight sharing)?

Fixed size, any length. One machine handles a 3-word tweet or a 3000-word essay.
A rule that does not change with position. "How words affect memory" should be the same at word 2 as at word 200.
It mirrors how Backpropagation Through Time (BPTT) later sends gradients back along the shared wires.

PICTURE. Three copies of the identical cell in a row, the amber memory wire threading $h^{(1)} \to h^{(2)} \to h^{(3)}$ , every copy stamped "same weights".

Step 8 — The degenerate cases (never leave a gap)

Every edge the reader could hit, shown explicitly.

The one-picture summary

One figure compressing all eight steps: inputs flow up, the two desks blend old memory with new input, $tanh$ squashes, memory flows right (reused weights), outputs drop down — with the $h^{(0)} = 0$ start marked.

Recall Feynman retelling — say it back in plain words

We wanted a tiny machine that reads a line of boxes and remembers. So we gave it a scratch-pad of numbers, the memory $h$ . At each box it does the same three things forever: (1) mix the old memory (desk $W_{hh}$ ) with the new box (desk $W_{x h}$ ) and add a fixed nudge $b_{h}$ ; (2) squash the mix with $tanh$ so it stays between $- 1$ and $+ 1$ — that squash is what makes it smart, and the zero-centred range lets it hold "positive" and "negative" facts; (3) if we need an answer, translate the memory with a third desk $W_{h y}$ . The memory then slides to the next box and the same desks run again. It starts blank ( $h^{(0)} = 0$ ), and after a sentence we wipe it clean. The one flaw: reusing $W_{hh}$ across a long chain makes signals fade or explode — which is the doorway to LSTMs, GRUs and attention.

Recall Quick self-check

Why is the $(t)$ in $x^{(t)}$ not a power? ::: It is a label meaning "at time step $t$ ", not exponentiation. What is $h^{(0)}$ set to and why? ::: The zero vector $0$ — the scratch-pad is blank before any input arrives. Which two ingredients build $h^{(t)}$ ? ::: The previous memory $h^{(t - 1)}$ and the current input $x^{(t)}$ . Why $tanh$ rather than sigmoid? ::: $tanh$ is zero-centred (range $- 1$ to $+ 1$ ), so memory can hold negative facts, and it stays bounded. Why the same weights every step? ::: Fixed parameters handle any length, and the sequence's rules shouldn't change with position.