Visual walkthrough — Gated Recurrent Units (GRU)

---

Step 1 — The problem: a number that must survive

WHAT. Imagine memory as a single number $h$ (think: "how positive is this sentence so far", a value like $+0.8$). At every word we get a chance to change it. We call the value before the current word $h_{t-1}$ and the value after $h_t$. The little subscript $t$ just counts words: $t=1$ is the first word, $t=2$ the second, and so on.

WHY. A plain recurrent network (3.5.01-Basic-RNN-Architecture) rewrites $h$ completely at every word. If $h$ is multiplied by a small number 30 times in a row, it shrinks toward zero — the famous vanishing gradient problem. Our whole goal is to give $h$ a way to pass through unchanged when nothing important happens.

PICTURE. The top track shows a plain RNN: every word squashes the memory a little, so "Alice" fades to nothing by the end. The bottom track is the goal — a lane where the memory can coast untouched.

---

Step 2 — The dial we need: the sigmoid $\sigma$

WHAT. To control how much of anything flows, we need a knob that only ever gives a value between fully-closed ($0$) and fully-open ($1$). That knob is the sigmoid, written $\sigma$:

$\sigma (a)=\frac{1}{1+e^{-a}}$

Read it term by term: $a$ is any real number the network computes (it can be huge negative or huge positive). $e^{-a}$ is the exponential — a positive number that is tiny when $a$ is large and enormous when $a$ is very negative. Dividing $1$ by "$1$ plus that" squeezes everything into the open interval between $0$ and $1$.

WHY this tool and not another? We want a soft gate, not a hard on/off switch. A hard switch (a step function) has zero slope everywhere, so learning by gradients is impossible — the network could never tell which direction to nudge. The sigmoid is the smooth S-shaped stand-in: it can be nearly $0$ or nearly $1$, yet it always has a nonzero slope to learn from.

PICTURE. The S-curve: far left it hugs $0$ (gate shut), far right it hugs $1$ (gate open), and it passes through $0.5$ at the middle.

---

Step 3 — Feeding the gate: concatenate, then weight

WHAT. What number $a$ goes into the sigmoid? We stack the old memory and the new word into one long list and take a weighted sum:

$r_t=\sigma (W_r[h_{t-1},x_t]+b_r)$

• $[h_{t-1},x_t]$ ::: concatenation — literally glue the two lists end to end into one longer list.
• $W_r$ ::: a matrix of weights the network learns; it mixes every input number to produce the gate's raw score.
• $b_r$ ::: a bias, a learned constant that shifts the score up or down.
• $\sigma (\dots )$ ::: squashes that score into a $0$–$1$ tap opening. The subscript $r$ marks this as the reset gate.

WHY. The gate must react to both what we already remember ($h_{t-1}$) and what just arrived ($x_t$). Gluing them together and running one weighted sum lets the network base its decision on the full situation.

PICTURE. The two boxes slide together into one bar, the weight matrix rakes across it, the bias nudges, and the sigmoid tap comes out.

---

Step 4 — The reset gate: should the past even count?

WHAT. The reset gate $r_t$ (built in Step 3) decides how much of the old memory to consult when drafting a new idea. We apply it with the element-wise product $\odot$:

$r_t\odot h_{t-1}$

$\odot$ means: multiply matching entries. If $h_{t-1}=(0.8,-0.2)$ and $r_t=(0.2,0.9)$, then $r_t\odot h_{t-1}=(0.16,-0.18)$. Each memory slot has its own tap.

WHY. Sometimes the past is misleading. In "The movie started well but…", the word "but" flips the meaning. Here the network learns $r_t\approx 0.2$, so the positive history is dialled way down before we compute anything new — we "reset" so the stale positivity does not poison the fresh guess.

PICTURE. Two cases side by side. When $r_t\approx 1$ the past passes through untouched; when $r_t\approx 0$ the past is squeezed to nearly nothing.

---

Step 5 — The candidate: a fresh proposal via $\tanh$

WHAT. Using the reset-muted past, we draft a proposed new memory, the candidate ${\tilde{h}}_t$ (the tilde $\sim$ just means "candidate / not final yet"):

${\tilde{h}}_t=\tanh (W_h[r_t\odot h_{t-1},x_t]+b_h)$

• $r_t\odot h_{t-1}$ ::: the muted past from Step 4.
• glued to $x_t$, raked by weights $W_h$, shifted by bias $b_h$ ::: same recipe as Step 3.
• $\tanh$ ::: squashes the result into the range $-1$ to $+1$.

WHY $\tanh$ and not $\sigma$ here? A gate must be $0$–$1$ (a tap opening can't be negative). But a memory value should be able to be positive or negative — "very positive sentiment" vs. "very negative sentiment." $\tanh$ is the sibling of sigmoid that is centred on zero, running from $-1$ to $+1$, so it can express both directions.

PICTURE. The sigmoid and tanh curves overlaid: same S-shape, but tanh straddles zero so the candidate can swing negative.

---

Step 6 — The update gate mixes old and new

WHAT. We compute a second $0$–$1$ tap, the update gate $z_t$ (same recipe as Step 3, its own weights $W_z,b_z$):

$z_t=\sigma (W_z[h_{t-1},x_t]+b_z)$

Then we blend the old memory and the fresh candidate with it:

$h_t=(1-z_t)\odot {\tilde{h}}_t+z_t\odot h_{t-1}$

• $z_t\odot h_{t-1}$ ::: the fraction $z_t$ of the old memory we keep.
• $(1-z_t)\odot {\tilde{h}}_t$ ::: the leftover fraction $(1-z_t)$ goes to the new candidate.
• The two fractions always add to $1$, so this is a weighted average — nothing is created or destroyed, only shared.

WHY. One knob, two coupled decisions. In an LSTM a forget gate and a separate input gate do this in two moves. GRU notices they are two sides of one coin: keep more old ⇒ accept less new. So it uses a single $z_t$ and its complement $(1-z_t)$. That is the "merged gate" that makes GRU lighter.

PICTURE. A slider: at one end $h_t=h_{t-1}$ (pure keep), at the other $h_t={\tilde{h}}_t$ (pure replace), everywhere in between a proportional mix.

---

Step 7 — The edge cases (all four corners)

WHAT. Two gates, each near $0$ or $1$, give four corners. We must be sure every corner behaves.

$z_t$	$r_t$	Result	Meaning
$\approx 1$	anything	$h_t\approx h_{t-1}$	coast — memory passes through untouched
$\approx 0$	$\approx 1$	$h_t\approx {\tilde{h}}_t$ using full past	rewrite, but informed by history
$\approx 0$	$\approx 0$	$h_t\approx \tanh (W_h[0,x_t]+b_h)$	hard reset — start fresh from the word alone
$\approx 1$	$\approx 0$	$h_t\approx h_{t-1}$	candidate is ignored anyway, so $r_t$ doesn't matter

WHY it matters. Row 1 is the gradient highway: when $z_t\approx 1$ the slope $\partial h_t/\partial h_{t-1}\approx 1$, so the gradient neither shrinks nor explodes across many words — that is precisely how GRU beats the vanishing-gradient wall. Row 3 shows a clean topic change. Row 4 shows why $r_t$ is harmless when we're keeping the old value: its work is discarded, no conflict.

PICTURE. The unit square of $(z_t,r_t)$ with each corner labelled by what the memory does there.

Recall Check the corners

If $z_t=1$, what is $h_t$? ::: Exactly $h_{t-1}$ — the memory coasts, gradient $\approx 1$. If $z_t=0$ and $r_t=0$, what drives $h_t$? ::: Only the current word $x_t$; the past is fully muted — a fresh start. Why is $r_t$ irrelevant when $z_t\approx 1$? ::: The candidate ${\tilde{h}}_t$ is weighted by $(1-z_t)\approx 0$, so whatever $r_t$ did is thrown away.

---

The one-picture summary

Everything at once: the word $x_t$ and old memory $h_{t-1}$ come in, feed two sigmoid taps ($r_t$, $z_t$), the reset tap mutes the past for the tanh candidate ${\tilde{h}}_t$, and the update tap slides between old and new to produce $h_t$. Follow the plum "highway" line to see the gradient's coasting path.

Recall Feynman retelling (say it out loud)

Memory is a set of numbers I want to protect on a long trip. At each word I ask two soft yes/no questions using the sigmoid tap. First question (reset): "should I trust what I already remember while I brainstorm?" — if not, I mute the old memory before drafting. I draft a proposal with tanh so it can be positive or negative. Second question (update): "how much of the old should I keep versus this proposal?" — I take a weighted average, where the two weights $z_t$ and $1-z_t$ always sum to one. When the update tap is fully open on "keep," the memory sails through unchanged and the gradient rides along without shrinking — that's the trick that lets a name like "Alice" survive to the end of a paragraph. LSTM does this with three taps and a spare tank; GRU realises two of the taps were really one, and does the whole thing with two.

Recall Where this leads

The same "let information coast, decide what to attend to" idea reappears, generalised, in attention and is stacked in both directions in bidirectional RNNs. Prefer to read this in Hindi/English mix? See 3.5.05 Gated Recurrent Units (GRU) (Hinglish).