Gated Recurrent Units (GRU): Exercises — AI-ML deep dive

Throughout, we use the four GRU equations exactly as the parent defined them. Let me restate the vocabulary so every symbol here is earned before use:

The picture above is the mixing law of Step 4 drawn as a slider: $z_{t}$ is the position of the knob between "old memory" and "new candidate".

Level 1 — Recognition

L1.1 In the equation $h_{t} = (1 - z_{t}) ⊙ \tilde{h}_{t} + z_{t} ⊙ h_{t - 1}$ , which term carries the old memory, and what weight multiplies it?

Recall Solution

The old memory is $h_{t - 1}$ , multiplied by $z_{t}$ . The candidate $\tilde{h}_{t}$ gets the complementary weight $(1 - z_{t})$ . Because the two weights add to $1$ , this is a convex combination — a weighted average that never overshoots either input.

L1.2 A GRU cell has $z_{t} = 1$ (in every coordinate). What is $h_{t}$ ?

Recall Solution

$h_{t} = (1 - 1) \tilde{h}_{t} + 1 \cdot h_{t - 1} = h_{t - 1}$ . The cell copies the old state forward unchanged — the "remember everything" extreme.

L1.3 Which gate uses $tanh$ rather than $σ$ , and why does that matter?

Recall Solution

The candidate $\tilde{h}_{t}$ uses $tanh$ . Its output lives in $(- 1, 1)$ , so the candidate is a signed proposal (memory content can be positive or negative). The two gates $r_{t}, z_{t}$ use $σ$ because they are fractions of keeping — they must live in $(0, 1)$ .

Level 2 — Application

For L2 use these tiny 1-D numbers (single hidden unit, so $⊙$ is ordinary multiplication).

L2.1 Given $z_{t} = 0.3$ , $\tilde{h}_{t} = 0.8$ , $h_{t - 1} = - 0.5$ . Compute $h_{t}$ .

Recall Solution

$h_{t} = (1 - 0.3) (0.8) + (0.3) (- 0.5) = 0.7 \cdot 0.8 - 0.15 = 0.56 - 0.15 = 0.41.$ The new state $0.41$ leans toward the positive candidate because we accepted $70%$ of it.

L2.2 A pre-activation into the update gate is $W_{z} [h_{t - 1}, x_{t}] + b_{z} = 2.0$ . Compute $z_{t} = σ (2.0)$ , then use $\tilde{h}_{t} = 1.0, h_{t - 1} = 0.0$ to find $h_{t}$ .

Recall Solution

$z_{t} = σ (2.0) = \frac{1}{1 + e ^{- 2}} = \frac{1}{1 + 0.13534} = 0.8808.$ $h_{t} = (1 - 0.8808) (1.0) + 0.8808 (0.0) = 0.1192.$ High $z_{t}$ means we keep the (zero) old state, so almost none of the strong candidate enters.

L2.3 Reset gate $r_{t} = 0.1$ , old state $h_{t - 1} = 0.9$ . What value does the candidate network see in place of $h_{t - 1}$ ?

Recall Solution

It sees $r_{t} ⊙ h_{t - 1} = 0.1 \times 0.9 = 0.09$ . The reset gate has almost erased the past inside the candidate computation only — the raw $h_{t - 1}$ still survives untouched in Step 4.

Level 3 — Analysis

L3.1 Show that if $z_{t} \to 1$ for every timestep, the backprop factor $\frac{\partial h _{t}}{\partial h _{t - 1}}$ along the carried term approaches $1$ , and explain why that defeats vanishing gradients.

Recall Solution

Differentiate the carried term $z_{t} ⊙ h_{t - 1}$ (treat $z_{t}$ as a gate value near $1$ ): $\frac{\partial h _{t}}{\partial h _{t - 1}}_{carry} = z_{t} \approx 1.$ Over $T$ steps the chain rule multiplies these: the gradient scales like $\prod_{t} z_{t} \approx 1$ . Compare a plain RNN, whose factor is $tanh^{'} (\cdot) \leq 1$ and typically $< 1$ , giving $\prod tanh^{'} \to 0$ exponentially. The near- $1$ product is the gradient highway the parent note mentioned.

L3.2 Two candidate models: Model A always outputs $z_{t} = 0.5$ ; Model B learns $z_{t}$ per timestep. Which is strictly less expressive, and give one sentence type where the difference bites?

Recall Solution

Model A is strictly less expressive: it is forced to blend old and new $50/50$ every step, unable to fully preserve a subject across many filler words. In "Alice … [many clauses] … was tired", Model B can push $z_{t} \to 1$ during filler to keep "Alice", while Model A leaks $50%$ of the identity each step, decaying it as $0. 5^{k}$ after $k$ words.

L3.3 In L2.1 you got $h_{t} = 0.41$ . Recompute with the reset gate acting in Step 4 by mistake (multiply $h_{t - 1}$ by $r_{t} = 0.1$ there) and state numerically how wrong the buggy answer is.

Recall Solution

Buggy: $h_{t} = (0.7) (0.8) + (0.3) (0.1 \times - 0.5) = 0.56 + 0.3 \times (- 0.05) = 0.56 - 0.015 = 0.545.$ Correct was $0.41$ . The bug shifts the state by $0.545 - 0.41 = 0.135$ — a $33%$ error on the mixed part, precisely because it double-applies the reset.

Level 4 — Synthesis

L4.1 A 2-D GRU unit. Reset gate $r_{t} = [0.0, 1.0]$ , old state $h_{t - 1} = [0.6, - 0.4]$ . Compute the vector fed to the candidate, i.e. $r_{t} ⊙ h_{t - 1}$ .

Recall Solution

$r_{t} ⊙ h_{t - 1} = [0.0 \times 0.6, 1.0 \times (- 0.4)] = [0.0, - 0.4] .$ Coordinate 1 is fully reset (starts fresh); coordinate 2 keeps its full history. This per-dimension gating is the whole point of using a vector gate, not a scalar.

L4.2 Design a $z_{t}$ pattern (in words) that makes a GRU behave like a plain feed-forward layer that ignores the past entirely. Then write the resulting $h_{t}$ .

Recall Solution

Force $z_{t} = 0$ for all $t$ . Then $h_{t} = (1 - 0) \tilde{h}_{t} + 0 \cdot h_{t - 1} = \tilde{h}_{t} = tanh (W_{h} [r_{t} ⊙ h_{t - 1}, x_{t}] + b_{h}) .$ If additionally $r_{t} = 0$ , then $\tilde{h}_{t} = tanh (W_{h} [0, x_{t}] + b_{h})$ depends only on $x_{t}$ — a memoryless nonlinear layer. So GRU contains the plain layer as a limiting case.

L4.3 Numeric synthesis. One unit. Inputs give $z_{t} = 0.9$ at three filler steps then $z_{t} = 0.2$ at the decision step. Start $h_{0} = 1.0$ (encodes "Alice"). At every filler step the candidate is $\tilde{h} = 0.0$ ; at the decision step the candidate is $\tilde{h} = - 1.0$ . Track $h_{1}, h_{2}, h_{3}, h_{4}$ .

Recall Solution

Filler steps ( $z = 0.9, \tilde{h} = 0$ ): $h_{t} = (0.1) (0) + 0.9 h_{t - 1} = 0.9 h_{t - 1}$ . $h_{1} = 0.9 \times 1.0 = 0.9, h_{2} = 0.9 \times 0.9 = 0.81, h_{3} = 0.9 \times 0.81 = 0.729.$ Decision step ( $z = 0.2, \tilde{h} = - 1$ ): $h_{4} = (0.8) (- 1.0) + 0.2 \times 0.729 = - 0.8 + 0.1458 = - 0.6542.$ The identity signal survives ( $0.729$ ) through fillers, then the low- $z$ decision step lets the strong new candidate flip the state to $- 0.654$ .

Level 5 — Mastery

L5.1 Prove the update gate couples forget and input behaviour: show that increasing "keep-old" necessarily decreases "accept-new", unlike LSTM's independent $f_{t}, i_{t}$ .

Recall Solution

The old weight is $z_{t}$ ; the new weight is $(1 - z_{t})$ . Their sum is $z_{t} + (1 - z_{t}) = 1$ , a constant. So $\frac{d}{d z _{t}} [new weight] = \frac{d}{d z _{t}} (1 - z_{t}) = - 1$ : any rise in keep-old is an exactly equal fall in accept-new. In LSTM, $f_{t}$ and $i_{t}$ come from separate sigmoids with no such constraint — you can keep old and write new fully, at extra parameter cost. GRU trades that freedom for ~ $25%$ fewer parameters (see 3.5.03-Long-Short-Term-Memory-(LSTM)).

L5.2 Parameter counting. Hidden size $d_{h} = 100$ , input size $d_{x} = 50$ . Count trainable parameters in the GRU (weights + biases for all three of $W_{r}, W_{z}, W_{h}$ ). Then count an LSTM's four gates ( $f, i, o, \tilde{c}$ ) and give the GRU/LSTM ratio.

Recall Solution

Each weight matrix is $d_{h} \times (d_{h} + d_{x}) = 100 \times 150 = 15000$ , plus a bias vector of $100$ . Per gate: $15000 + 100 = 15100$ .

GRU (3 matrices): $3 \times 15100 = 45300$ .
LSTM (4 matrices): $4 \times 15100 = 60400$ .
Ratio $= 45300/60400 = 0.75$ exactly → GRU has $25%$ fewer parameters, confirming the parent's claim.

L5.3 Limiting-behaviour completeness. Fill the table of extreme gate settings and name each regime. Cases: $(z = 1)$ , $(z = 0, r = 1)$ , $(z = 0, r = 0)$ .

Recall Solution

$z_{t}$	$r_{t}$	$h_{t}$ becomes	Regime
$1$	any	$h_{t - 1}$	Freeze: copy memory, ignore input
$0$	$1$	$tanh (W_{h} [h_{t - 1}, x_{t}] + b_{h})$	Full refresh with context: plain RNN update
$0$	$0$	$tanh (W_{h} [0, x_{t}] + b_{h})$	Reset: memoryless, input-only

These three corners span the behaviour space; everything a GRU does is a smooth, per-dimension blend between them. This links forward to how attention removes gating entirely — see 3.6.01-Attention-Mechanism.

Recall Self-test summary

$z_{t}$ high means ::: keep the old state, reject the new candidate The reset gate $r_{t}$ acts only inside ::: the candidate $\tilde{h}_{t}$ (Step 2), never in Step 4 Backprop carry factor per step is ::: $z_{t}$ , so long-memory needs $z_{t} \approx 1$ GRU-to-LSTM parameter ratio is ::: $0.75$ (25% fewer) Prerequisite to revisit ::: 3.5.01-Basic-RNN-Architecture