Worked examples — Gated Recurrent Units (GRU)

Before anything else, let me re-list the four rules so this page stands on its own. Read them once; we will refer to them by number.

The heart of every example below is rule (4). Picture it as a slider between two memories.

Look at the slider: the left end is the fresh candidate ${\tilde{h}}_t$, the right end is the old memory $h_{t-1}$. The update gate $z_t$ is the knob position. $z_t=1$ pins the knob to "keep old"; $z_t=0$ pins it to "take new"; anything between blends the two.

---

The scenario matrix

Every GRU cell you will ever hand-compute falls into one of these cells. Each worked example below is tagged with the cell it covers.

#	Cell (scenario class)	What makes it special	Example
A	$z_t\approx 0$, $r_t\approx 1$	fully overwrite, keep past for candidate	Ex 1
B	$z_t\approx 1$ (degenerate =1), any $r_t$	freeze memory — perfect carry	Ex 2
C	$z_t\approx 0$, $r_t\approx 0$ (degenerate)	full reset + full overwrite — "start fresh"	Ex 3
D	$0<z_t<1$, $0<r_t<1$	genuine blend (the everyday case)	Ex 4
E	Negative components / $\tanh$ sign flip	candidate goes negative	Ex 5
F	Limiting behaviour over $T$ steps	gradient highway $\prod z_t$	Ex 6
G	Real-world word problem	sentiment pivot at "but"	Ex 7
H	Exam twist: $z_t$ vs $(1-z_t)$ mix-up	which term multiplies which memory	Ex 8

I will keep the numbers small (1- or 2-dimensional hidden states) so you can check every arithmetic step in your head.

---

1. Rule (2) already gave the candidate. With $r_t=1.0$, the candidate saw the full old memory, so $r_t\odot h_{t-1}=1.0\times 0.8=0.8$ went into the $\tanh$. We are told the result is ${\tilde{h}}_t=-0.5$. Why this step? $r_t=1$ means "the past is relevant while proposing the new memory" — reset does not erase; it only scales the past inside the candidate.
2. Apply rule (4): $h_t=(1-z_t){\tilde{h}}_t+z_t{h}_{t-1}=(1-0)\times (-0.5)+0\times 0.8$. Why this step? Rule (4) is the only equation that produces the output memory.
3. Arithmetic: $=1\times (-0.5)+0=\boxed{-0.5}$. Why this step? $z_t=0$ zeroes the "keep old" term entirely.

Verify: $z_t=0\Rightarrow h_t={\tilde{h}}_t$ exactly, and indeed $h_t=-0.5={\tilde{h}}_t$. The old $0.8$ vanished — matches the forecast: knob at "take new".

---

1. Rule (4): $h_t=(1-1)\times 0.99+1\times 0.42$. Why this step? We test the upper degenerate corner of the slider.
2. Arithmetic: $=0\times 0.99+1\times 0.42=\boxed{0.42}$. Why this step? The $(1-z_t)$ factor is $0$, so the loud candidate is multiplied away completely.
3. Repeat 100 times with $z_t=1$ each step: $h_t=h_{t-1}=\cdots =0.42$ unchanged. Why this step? This is the "gradient highway": if $z_t=1$ always, $\frac{\partial h_t}{\partial h_{t-1}}=1$, so information (and gradient) passes 100 steps with no decay.

Verify: $z_t=1\Rightarrow h_t=h_{t-1}$ identically, independent of ${\tilde{h}}_t$. A gate of exactly $1$ is a perfect memory lock. Matches forecast: the candidate stays out.

---

1. Rule (2) with reset zero: $r_t\odot h_{t-1}=0\times 0.9=0$, so the candidate sees no history: ${\tilde{h}}_t=\tanh (1.2)$. Why this step? $r_t=0$ literally deletes $h_{t-1}$ from inside the candidate — "start fresh from $x_t$ only."
2. Compute the tanh: $\tanh (1.2)\approx 0.83365$. Why this step? $\tanh$ turns the raw score $1.2$ into a bounded content value in $(-1,1)$.
3. Rule (4): $h_t=(1-0.1)(0.83365)+0.1\times 0.9=0.9\times 0.83365+0.09$. Why this step? Update gate is low, so mostly the fresh candidate flows through.
4. Arithmetic: $0.75028+0.09=\boxed{0.84028}$. Why this step? Final blend.

Verify: the survivor of old memory is only the $0.1\times 0.9=0.09$ chunk (about $11\%$ of the answer). Reset ($r_t=0$) killed the past inside the candidate; update ($z_t=0.1$) let only a sliver of raw $h_{t-1}$ leak into the final blend. Two independent knobs, two independent jobs — exactly as the matrix demands.

---

1. Coordinate 1, rule (4): $(1-0.7)(0.1)+0.7(0.6)=0.3\times 0.1+0.42$. Why this step? Rule (4) runs element-wise; each dimension has its own gate value.
2. $=0.03+0.42=\boxed{0.45}$ (coordinate 1).
3. Coordinate 2, rule (4): $(1-0.3)(0.9)+0.3(-0.2)=0.7\times 0.9-0.06$. Why this step? Same rule, second slot.
4. $=0.63-0.06=\boxed{0.57}$ (coordinate 2).
5. Assemble: $h_t=[0.45,0.57]$.

The figure shows both sliders. Coordinate 2 (low $z$) swung far from its old $-0.2$ up to $0.57$; coordinate 1 (high $z$) barely moved from $0.6$ to $0.45$.

Verify: coordinate 1 stayed near old ($0.6\to 0.45$, high $z_t$); coordinate 2 leapt toward candidate ($-0.2\to 0.57$, low $z_t$). Bigger change where $z_t$ is smaller — matches the forecast and the slider picture.

---

1. Candidate via rule (2): ${\tilde{h}}_t=\tanh (-1.5)\approx -0.90515$. Why this step? $\tanh$ is odd: $\tanh (-1.5)=-\tanh (1.5)$, so a negative score gives a negative content value — this is exactly why we use $\tanh$ and not sigmoid for content: memory must be able to point down.
2. Rule (4): $h_t=(1-0.4)(-0.90515)+0.4(0.3)=0.6\times (-0.90515)+0.12$. Why this step? Standard blend.
3. Arithmetic: $-0.54309+0.12=\boxed{-0.42309}$.

Verify: result is negative because the $60\%$ negative candidate ($-0.543$) outweighs the $40\%$ positive old memory ($+0.12$). If instead $z_t\to 1$, $h_t\to 0.3>0$: the sign of $h_t$ is genuinely controlled by the gate, not fixed. Matches forecast.

---

1. Rule (4) with ${\tilde{h}}_t=0$: $h_t=(1-0.95)\times 0+0.95h_{t-1}=0.95h_{t-1}$. Why this step? No new content, so each step just scales the old memory by $z_t$.
2. Iterate 5 times: $h_6=0.95^5\times 1.0$. Why this step? Multiplying by $0.95$ once per step compounds.
3. Compute: $0.95^5\approx 0.77378$. So $\approx 77\%$ of the original signal survives.
4. Plain RNN comparison: $0.4^5=0.01024$, i.e. $\approx 1\%$ survives. Why this step? A vanilla RNN multiplies by ${\tanh }^{\prime }<1$ every step (see 3.5.02-Vanishing-and-Exploding-Gradients), so it decays exponentially fast.

Verify: $0.95^5\approx 0.774$ vs $0.4^5\approx 0.010$ — GRU keeps ~75× more signal. Because $\frac{\partial h_t}{\partial h_{t-1}}=z_t$, the gradient scales as $\prod z_t=0.95^5$, near $1$, not near $0$. This is the whole point of gating over the 3.5.01-Basic-RNN-Architecture. Forecast confirmed.

---

1. Reset acts first (rule 2 input): $r_t\odot h_{t-1}=0.2\times 0.8=0.16$ — the positive past is damped before entering the candidate. Why this step? Low $r_t$ says "the contrast word means old sentiment is less trustworthy for proposing new sentiment."
2. Candidate: ${\tilde{h}}_t=\tanh (-0.4)\approx -0.37995$. Why this step? The pivot pushes the score negative; $\tanh$ yields a mildly negative candidate.
3. Update blend (rule 4): $h_t=(1-0.3)(-0.37995)+0.3(0.8)=0.7\times (-0.37995)+0.24$. Why this step? Low $z_t$ lets most of the new (negative-leaning) candidate through.
4. Arithmetic: $-0.26597+0.24=\boxed{-0.02597}$.

Verify: memory dropped from $+0.8$ all the way to $\approx -0.026$ — it crossed through zero at "but", poised to go fully negative once "boring plot" arrives. Units: sentiment stays in $(-1,1)$ as required ($-0.026$ ✓). The two gates cooperated: reset distrusted the past, update accepted the pivot. This mirrors the LSTM behaviour in 3.5.03-Long-Short-Term-Memory-(LSTM) but with one fewer gate.

---

1. Student's (wrong) formula: $0.9\times (-0.5)+0.1\times 0.5=-0.45+0.05=\boxed{-0.40}$. Why this step? Show the consequence of the swap explicitly.
2. Correct rule (4): $h_t=(1-0.9)(-0.5)+0.9(0.5)=0.1\times (-0.5)+0.45$. Why this step? $(1-z_t)$ multiplies the candidate, $z_t$ multiplies the old state — memorise this ordering.
3. Arithmetic: $-0.05+0.45=\boxed{+0.40}$.

Verify: correct answer $+0.40$ keeps the old positive memory (high $z_t$ ⇒ protect past); the swapped version gives $-0.40$, wrongly discarding it. The sign flips! In the true GRU, $z_t\to 1$ means "remember", so $z_t$ must sit on $h_{t-1}$. Forecast: high $z_t$ protects the old $0.5$.

---

Recall Self-test

With $z_t=0.6$, ${\tilde{h}}_t=1.0$, $h_{t-1}=0$, what is $h_t$? ::: $(1-0.6)(1.0)+0.6(0)=0.4$ Reset gate $r_t=0$ means the candidate ignores what? ::: the old hidden state $h_{t-1}$ (it is multiplied by $0$) Over 5 steps with $z_t=0.9$ and no new content, what fraction of a signal survives? ::: $0.9^5\approx 0.590$ High $z_t$ behaves like which two LSTM gates combined? ::: high forget + low input (keep old, reject new)

Where to go next: 3.5.06-Bidirectional-RNNs runs a GRU forwards and backwards; 3.6.01-Attention-Mechanism later removes the single-state bottleneck entirely.