Foundations — Gated Recurrent Units (GRU)

Before you can read a single GRU equation, you need to recognise the pieces it is built from. This page defines every symbol and every tool the parent note uses — from what a vector even is, up to the mixing formula — with a picture attached to each. Read top to bottom; each block earns the next.

1. What a sequence is, and why we index by $t$

A GRU reads a sequence: an ordered list of things, one after another. In text, that list is the words of a sentence, left to right.

We label positions with a counter called the timestep, written $t$. The first item is at $t=1$, the next at $t=2$, and so on. "$t$" is just "which step are we on right now".

Look at the figure: reading the sentence is like walking a row of stepping stones. On each stone we do the same computation, but with new input. That "same computation, new input, memory carried across" pattern is exactly what a basic RNN does, and GRU is a smarter version of that stone.

2. Vectors — the shape of everything here

A word cannot be fed to math as letters. We turn it into a vector: a fixed-length list of numbers.

The symbol $\mathbb{R}$ just means "the real numbers" (any decimal, positive or negative). ${\mathbb{R}}^{d_h}$ means "a point in $d_h$-dimensional space". You never have to picture 200 dimensions — picture an arrow of numbers whose length is $d_h$.

3. The hidden state $h_t$ — the memory carried forward

This is the star symbol of the whole topic.

Because GRU (unlike LSTM, which also carries a separate cell state $c_t$) keeps only $h_t$, this single vector is the machine's complete memory. That simplicity is the whole selling point.

4. Concatenation $[h_{t-1},x_t]$ — stacking two vectors

The equations always feed the network both the old memory and the new word together. To do that we glue the two vectors into one taller vector.

Why concatenate? So a single matrix (next section) can look at old memory and new input simultaneously and decide based on their combination — e.g. "this word is 'but' and the memory so far is positive → prepare to flip".

5. The linear layer $W\cdot v+b$ — the learnable "mixing board"

Inside every gate you see the pattern $W\cdot v+b$. This is the one operation neural nets are built from.

The parent note has three of these boards: $W_r,W_z,W_h$ (with biases $b_r,b_z,b_h$) — one for the reset gate, one for the update gate, one for the candidate.

6. Sigmoid $\sigma$ — squashing into $[0,1]$ so a number becomes a "dial"

A gate must output a fraction between 0 (fully closed) and 1 (fully open). Raw $W\cdot v+b$ can be any number, huge or negative. We need to squash it.

Why this tool and not another? We want a smooth, always-increasing knob that can never leave $[0,1]$, so its output can be read directly as "keep this fraction". Sigmoid is the classic function with exactly those properties, and being smooth means we can train through it. This is why gates ($z_t$, $r_t$) use $\sigma$.

Here $e$ is Euler's number ($\approx 2.718$); $e^{-x}$ shrinks fast as $x$ grows, which is what drags the output toward 1.

7. The two gates built from sigmoid — $r_t$ and $z_t$

Now that we have $\sigma$, a concatenation, and a mixing board, we can build the two dials the whole topic is named for. Each gate is one linear layer followed by a sigmoid, so its output is a vector of dials in $[0,1]$.

Notice both gates read the same input $[h_{t-1},x_t]$ but have their own weights ($W_r$ vs $W_z$), so they can learn to answer two different questions from the same evidence.

8. Tanh and the candidate memory ${\tilde{h}}_t$

The candidate new memory ${\tilde{h}}_t$ uses tanh, not sigmoid. Different job, different squasher.

Why tanh here and sigmoid there? A gate answers "what fraction, 0 to 1?" → sigmoid. A memory value wants to represent features that can be positive or negative (e.g. "very positive sentiment" = $+0.9$, "very negative" = $-0.9$), so it needs the $[-1,1]$ range centred at 0 → tanh. Centring at 0 also keeps values from drifting, which helps training.

Now we can write the candidate hidden state — the "proposed new memory":

9. Hadamard product $\odot$ — element-wise multiply (the "gating" action)

This symbol is how a dial actually controls a memory vector.

10. The "1" in $(1-z_t)$ — a vector of ones

The final formula uses $(1-z_t)$. But $z_t$ is a vector and $1$ looks like a single number — how do you subtract?

11. Putting the pieces into the mixing formula

Once all the above exist, the final GRU update reads cleanly: $h_t=(1-z_t)\odot {\tilde{h}}_t+z_t\odot h_{t-1}$

Every symbol is now defined: $z_t=\sigma (W_z[h_{t-1},x_t]+b_z)$ is a sigmoid dial-vector, ${\tilde{h}}_t=\tanh (W_h[r_t\odot h_{t-1},x_t]+b_h)$ a tanh candidate (built using the reset gate $r_t$), $h_{t-1}$ the old memory (with $h_0=0$), $\odot$ the per-slot scaling, and $(1-z_t)=1-z_t$ the leftover fraction.

12. Why gates fix vanishing gradients (the motive)

The parent topic exists to beat the vanishing gradient problem. The key symbol there is $\frac{\partial h_t}{\partial h_{t-1}}$: "how much does a tiny change in old memory change new memory?" Multiplied across many steps, if this is $<1$ the effect shrinks toward zero and long-range learning dies. Because $h_t$ contains the term $z_t\odot h_{t-1}$, this derivative gets a $z_t$ contribution — and when $z_t\approx 1$, information (and gradient) travels across many steps almost undamped. That is the whole payoff of building the update gate.

Prerequisite map

Each foundation on the left feeds the GRU cell on the right. Miss any node and an equation will contain a symbol you cannot read.

Equipment checklist

Test yourself — cover the right side and answer aloud.

Related: Hinglish version · builds toward 3.5.06-Bidirectional-RNNs and 3.6.01-Attention-Mechanism.