Foundations — Recurrent Neural Networks (RNN) architecture

Before you can read a single line of the parent note, you must be fluent with the pieces it silently assumes: what a vector is, what a weight matrix does to a vector, what the superscript $(t)$ means, why we bother with $\tanh$, and what a gradient is. This page builds each one from absolutely nothing, in an order where each idea leans only on the ones before it.

1. A number, a list of numbers, a vector

Start with the smallest thing: a single number, like $3$ or $-0.7$. We call a lone number a scalar — "scalar" just means "one number, no direction." Ordinary numbers like these (including fractions and decimals, positive or negative) are called real numbers, and mathematicians give the whole collection of them a name: $\mathbb{R}$. So $\mathbb{R}$ is just shorthand for "all the ordinary numbers on the number line."

Now stack several numbers into an ordered list:

$x=\left[\begin{array}{c}0.2\\ -1.0\\ 0.5\end{array}\right]$

That ordered list is a vector. "Ordered" matters: the first slot and the second slot mean different things and you may not swap them.

Why does the RNN need this? Because every single thing an RNN handles — the input word $x^{(t)}$, the memory $h^{(t)}$, the output $y^{(t)}$ — is a vector. A word like "movie" is turned into a list of 300 numbers (that's what Word Embeddings do). The memory is a list of, say, 128 numbers. The whole topic is about vectors flowing and changing.

So when the parent note writes $x^{(t)}\in {\mathbb{R}}^{d_x}$, translate it to: "the input at time $t$ is a list of $d_x$ ordinary numbers." Nothing scarier than that.

2. The superscript $(t)$ — a clock, not a power

This trips up everyone, so read carefully. When you see $x^{(t)}$, the $(t)$ in round brackets is not an exponent. It is a time stamp.

So the sequence $x^{(1)},x^{(2)},x^{(3)},\dots ,x^{(T)}$ is: the piece we read at tick 1, the piece at tick 2, and so on, up to the last tick $T$. Here $T$ (capital) is just the length of the sequence — how many pieces there are in total.

Why the topic needs it: an RNN's whole purpose is to process things in order over time, so it needs a notation that says "this thing, at this tick." That is exactly $(t)$.

3. A weight matrix — a machine that reshapes a vector

Next assumed piece: the capital-$W$ symbols like $W_{xh}$, $W_{hh}$, $W_{hy}$. Each is a matrix — a rectangular grid of numbers.

Here is the only fact about matrices you truly need:

Concretely, each output number is a weighted sum of all input numbers. For example:

$W=\left[\begin{array}{cc}2& 0\\ 1& 3\end{array}\right],x=\left[\begin{array}{c}5\\ 1\end{array}\right]\Rightarrow Wx=\left[\begin{array}{c}2\cdot 5+0\cdot 1\\ 1\cdot 5+3\cdot 1\end{array}\right]=\left[\begin{array}{c}10\\ 8\end{array}\right]$

Why the topic needs three different $W$'s, and what the subscripts mean:

The dimensions in the parent, e.g. $W_{xh}\in {\mathbb{R}}^{d_h\times d_x}$, now read plainly: "a machine that eats a $d_x$-slot input and outputs a $d_h$-slot vector" — exactly the shape needed to feed the memory.

4. Adding vectors and the bias $b$

Inside $W_{hh}{h}^{(t-1)}+W_{xh}{x}^{(t)}+b_h$ there are plus signs between vectors. Vector addition is the easy one: line them up and add slot by slot.

$\left[\begin{array}{c}1\\ 4\end{array}\right]+\left[\begin{array}{c}2\\ -1\end{array}\right]=\left[\begin{array}{c}3\\ 3\end{array}\right]$

The last term $b_h$ is the bias — just a fixed vector that gets added on every time. Why do we need it? Because without a bias, when the input is all zeros the result is forced to be zero; the bias lets the network shift its output to any baseline it wants. It is the "+ c" of the linear world. (There is a second bias, $b_y$, for the output step — same idea, we meet it in §7.)

So the whole expression $W_{hh}{h}^{(t-1)}+W_{xh}{x}^{(t)}+b_h$ means: reshape the old memory, reshape the new input, add both together, then nudge by a fixed offset. We give this pre-squash result its own name, $z^{(t)}$ (a "pre-activation"):

$z^{(t)}=W_{hh}{h}^{(t-1)}+W_{xh}{x}^{(t)}+b_h$

5. Why $\tanh$? The squasher

The parent wraps that $z^{(t)}$ in $\tanh$: $h^{(t)}=\tanh (z^{(t)})$. Before asking why $\tanh$, know what it is.

Why the topic needs it: without $\tanh$, the memory update would be purely linear and the RNN could not model anything interesting. The squash also keeps the memory numbers bounded in $(-1,1)$ so they don't blow up as they cycle around the loop tick after tick.

6. The recurrence — the loop that makes it "recurrent"

Now assemble the two pieces the parent's core equation is made of:

$h^{(t)}=\tanh (W_{hh}{h}^{(t-1)}+W_{xh}{x}^{(t)}+b_h)$

Read it left to right as a story: "The new memory $h^{(t)}$ is built from (a reshaped copy of the old memory $h^{(t-1)}$) plus (a reshaped copy of the new input $x^{(t)}$), all squashed by $\tanh$."

The single crucial thing: the old memory $h^{(t-1)}$ appears on the right, and the new memory $h^{(t)}$ comes out on the left — then next tick, that becomes the "old" one. The output feeds back into the input. This feed-back loop is the entire meaning of the word recurrent.

The starting point $h^{(0)}$: at the very first tick there is no "previous memory," so we set $h^{(0)}=0$ (a vector of all zeros — the boldface $0$ means "the zero vector, every slot 0," matching our bold-is-a-vector rule from the top of the page). This is the "blank mind before reading anything" case, and you must always handle it explicitly.

7. From memory to output: $y^{(t)}$, softmax, and sigmoid

The memory $h^{(t)}$ is internal bookkeeping. To actually say something, the RNN turns it into an output vector $y^{(t)}$ with one more reshape-plus-bias:

$y^{(t)}=W_{hy}{h}^{(t)}+b_y$

Here $W_{hy}$ is the hidden-to-output matrix from §3 and $b_y$ is the output bias (the second bias promised in §4). The raw numbers in $y^{(t)}$ are called scores or "logits" — they can be any size, positive or negative, and don't yet mean anything friendly. We convert them depending on the task.

Why the topic needs both: softmax answers "which of many?", sigmoid answers "how likely is this one thing?". Both convert raw scores $y^{(t)}$ into honest probabilities.

8. Loss, gradient, and how the RNN learns — from zero

The training section of the parent talks about a total loss $\mathcal{L}$ and its gradient $\frac{\partial \mathcal{L}}{\partial W_{hh}}$. Here is the from-zero version, and we start by clearing up the two loss symbols.

Now the piece the parent leans on but never builds from zero: the gradient, and why it points where it does.

How these foundations feed the topic

This whole map feeds the parent: RNN architecture. Downstream, the same foundations power Sequence-to-Sequence Models, the Attention Mechanism, and Transformers.

Equipment checklist