Gated Recurrent Units (GRU)

What Problem Does GRU Solve?

Standard RNN issue: Gradient vanishes exponentially over many timesteps → can't learn long-range dependencies.

LSTM solution: Separate cell state $c_t$ with gates to control information flow.

GRU insight: Do we really need a separate cell state? What if we combine the hidden state and cell state into one $h_t$ , and merge the forget and input gates into a single update gate? This cuts parameters by ~25% while keeping most of LSTM's power.

The GRU Architecture

GRU has 2 gates operating on hidden state $h_t$ :

Derivation from First Principles

Goal: Update hidden state $h_t$ from $h_{t-1}$ and $x_t$ , controlling information flow.

Step 1: Compute Reset Gate

$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$

Why this step? We need to decide: is the past $h_{t-1}$ relevant for the new candidate we're about to compute? If the sequence context changes (e.g., topic shift), we want $r_t \to 0$ to "reset" and not be biased by irrelevant history.

Step 2: Compute Candidate Hidden State

$\tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$

Why this step?

$r_t \odot h_{t-1}$ element-wise multiplies the old state by the reset gate. If $r_t$ is small, we ignore $h_{t-1}$ .
Concatenate with $x_t$ , pass through tanh to get a candidate update $\tilde{h}_t$ in $[-1, 1]$ .
This is the "proposed new memory" if were to fully replace $h_{t-1}$ .

Step 3: Compute Update Gate

$z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$

Why this step? We need to decide: how much of the old $h_{t-1}$ to keep vs. how much of the new candidate $\tilde{h}_t$ to accept. This is the interpolation weight.

Step 4: Final Hidden State (Linear Interpolation)

$h_t = (1 - z_t) \odot \tilde{h}_t + z_t \odot h_{t-1}$

Why this step?

If $z_t \approx 1$ : $h_t \approx h_{t-1}$ → preserve old memory (like LSTM's high forget gate).
If $z_t \approx 0$ : $h_t \approx \tilde{h}_t$ → accept new candidate (like LSTM's high input gate).
This is a convex combination: $z_t$ acts as a learned interpolation weight per dimension.

Key insight: The update gate $z_t$ merges LSTM's forget and input gates. In LSTM, you have separate $f_t$ (forget) and $i_t$ (input). In GRU, when $z_t$ is high (keep old), it's low for new (reject new), and vice versa—they're coupled as $(1-z_t)$ and $z_t$ .

GRU vs LSTM: Formula Comparison

LSTM	GRU
3 gates: forget $f_t$ , input $i_t$ , output $o_t$	2 gates: update $z_t$ , reset $r_t$
Separate cell state $c_t$ and hidden state $h_t$	Single hidden state $h_t$ (no separate cell)
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$	$h_t = (1-z_t) \odot \tilde{h}_t + z_t \odot h_{t-1}$
$h_t = o_t \odot \tanh(c_t)$	(no output gate, $h_t$ is directly exposed)

Parameter count: GRU has ~25% fewer parameters than LSTM (2 gates vs 3, no separate cell state weights).

Formula Complete GRU Equations

Given input $x_t \in \mathbb{R}^{d_x}$ , hidden state $h_{t-1} \in \mathbb{R}^{d_h}$ :

Reset gate: $r_t = \sigma(W_r [h_{t-1}, x_t] + b_r)$
Candidate: $\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t] + b_h)$
Update gate: $z_t = \sigma(W_z [h_{t-1}, x_t] + b_z)$
New hidden state: $h_t = (1 - z_t) \odot \tilde{h}_t + z_t \odot h_{t-1}$

Where:

$[h_{t-1}, x_t]$ denotes concatenation: $\in \mathbb{R}^{d_h + d_x}$
$\odot$ is element-wise (Hadamard) product
$W_r, W_z W_h \in \mathbb{R}^{d_h \times (d_h + d_x)}$ , $b_r, b_z, b_h \in \mathbb{R}^{d_h}$

Example 1: Sentiment Analysis Sentence

Sequence: "The movie started well but the ending was terrible."

Let's trace one GRU cell at word "but" (timestep $t$ ):

Input: $x_t$ = embedding of "but", $h_{t-1}$ = hidden state after "well" (likely positive sentiment accumulated).

Step 1: Compute $r_t$ : $r_t = \sigma(W_r [h_{t-1}, x_t] + b_r)$ "But" signals a contrast. If the network has learned that "but" often reverses sentiment, $r_t$ might be low (say 0.2), meaning "don't rely heavily on the positive $h_{t-1}$ —we're about to change direction."

Why this step? The reset gate detects the contrast word and decides to partially ignore the positive history.

Step 2: Compute $\tilde{h}_t$ : $\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t] + b_h)$ With $r_t = 0.2$ , we multiply $h_{t-1}$ by 0.2, heavily damping the old positive sentiment. The candidate $\tilde{h}_t$ is now mostly driven by $x_t$ ("but") and will start shifting toward neutral or preparing for negativity.

Why this step? We compute a fresh candidate that's not overly biased by the past positive context.

Step 3: Compute $z_t$ : $z_t = \sigma(W_z [h_{t-1}, x_t] + b_z)$ "But" is a pivot word. The network might learn $z_t$ should be low (say 0.3), meaning "don't keep much of the old positive $h_{t-1}$ —accept more of the new candidate."

Why this step? The update gate decides to shift away from the old sentiment toward the new candidate.

Step 4: Final $h_t$ : $h_t = (1 - 0.3) \odot \tilde{h}_t + 0.3 \odot h_{t-1} = 0.7 \tilde{h}_t + 0.3 h_{t-1}$ The new hidden state is 70% the candidate (which has low influence from past due to reset gate) and 30% the old state. Net effect: sentiment representation shifts from positive toward neutral/negative.

Result: As the sequence continues ("ending was terrible"), the reset gate will again be low, update gate will favor new negative information, and final $h_T$ will represent negative sentiment.

Example 2: Long-Term Dependency (Name Recall)

Sequence: "Alice went to the park. She played with her dog. Later, she went home. __ was tired."

We want to fill the blank with "Alice" (or "She"), requiring memory of "Alice" from the start.

At the blank (timestep $t$ ):

Many timesteps have passed since "Alice" was mentioned.
Intermediate words ("park", "dog", "home") are less relevant to who is tired.

GRU behavior:

For intermediate words (park, dog, home):

$z_t$ is high (say 0.8–0.9): The update gate learns these words don't change the subject identity, so $h_t \approx h_{t-1}$ . The identity "Alice" is preserved through the $(1-z_t)$ term being small—we mostly keep old $h_{t-1}$ .

Why this works: High $z_t$ acts like LSTM's high forget gate + low input gate: "keep old memory, ignore new irrelevant content." Gradient flows backward through the $(1-z_t)$ and $z_t$ terms, which are gated—gradient highway.

At the blank:

$h_{t-1}$ still encodes "Alice" because intermediate $z_t$ were high.
The decoder can extract "Alice" from $h_t$ to generate "She" or "Alice was tired."

Key: GRU's update gate $z_t$ controls the gradient flow. When $z_t \approx 1$ , the gradient $\frac{\partial h_t}{\partial h_{t-1}} \approx 1$ (chain rule), avoiding vanishing gradient.

Common Mistakes

When to Use GRU vs LSTM?

Use GRU	Use LSTM
Default choice for new projects (simpler)	Very long sequences (100+ steps)
Small datasets (fewer params → less overfit)	Complex hierarchical dependencies
Faster training/inference needed	When you have compute budget and want max expressiveness
Performance is close to LSTM empirically	Speech, complex NLP (slight edge in some benchmarks)

Rule of thumb: Start with GRU. If you hit a performance ceiling and have resources, try LSTM. The difference is often marginal (1-2% accuracy) in practice.

Recall Explain to a 12-Year-Old

Imagine you're writing a story in a notebook, one sentence at a time. You have two sticky notes to help you decide what to write next: Sticky Note 1 (Reset Gate): "Should I look back at what I wrote before to get ideas, or should I start fresh?" If the story changes topics (like going from pirates to princesses), you might ignore the old sentences. Low reset = "forget the old stuff for this new idea."

Sticky Note 2 (Update Gate): "Should I keep my current page as-is, or should I erase some and write the new sentence?" If the new sentence is really important, you erase more of the old page (low update gate). If the old stuff is still important, you keep most of it (high update gate).

The GRU does this automatically for every word in a sentence, deciding "how much old memory to keep" and "how much past context to use for new ideas." It's like a smart notebook that knows when to remember and when to move on—perfect for understanding long stories or conversations!

Connections

3.5.03-Long-Short-Term-Memory-(LSTM) — GRU simplifies LSTM by merging gates
3.5.02-Vanishing-and-Exploding-Gradients — GRU solves vanishing gradient via gating
3.5.01-Basic-RNN-Architecture — GRU extends basic RNN with gates for long-term memory
3.5.06-Bidirectional-RNNs — GRU can be used bidirectionally for context from both directions
3.6.01-Attention-Mechanism — Attention can replace/augment GRU for very long sequences

#flashcards/ai-ml

What are the two gates in a GRU and what does each control? :: Reset gate $r_t$ controls how much of past hidden state influences the new candidate. Update gate $z_t$ controls interpolation between old hidden state and new candidate.

Write the GRU final hidden state update equation :: $h_t = (1 - z_t) \odot \tilde{h}_t + z_t \odot h_{t-1}$ , where $z_t$ is update gate, $\tilde{h}_t$ is candidate hidden state.

How does GRU's update gate

z_t

relate to LSTM's forget and input gates?

GRU merges them: high

z_t

means keep old (like LSTM forget=1, input=0), low

z_t

means accept new (like LSTM forget=0, input=1). They're coupled as

z_t

and

(1-z_t)

What is the role of the reset gate

r_t

in the candidate computation?

It gates the previous hidden state:

\tilde{h}_t = \tanh(W_h [r_t \odot h_{t-1}, x_t] + b_h)

. Low

r_t

means ignore past context when forming the new candidate.

Why does GRU avoid vanishing gradients?

The update gate creates a gradient highway: when

z_t \approx 1

\frac{\partial h_t}{\partial h_{t-1}} \approx 1

, so gradients don't decay exponentially over long sequences.

GRU vs LSTM: which has fewer parameters and by roughly how much?

GRU has ~25% fewer parameters (2 gates vs 3, no separate cell state).

If you see the word "but" in sentiment analysis, should the GRU's reset gate be high or low, and why?

Low. "But" signals a contrast/topic shift, so the reset gate should reduce influence of past (positive) sentiment when computing the new candidate.

In a long sequence where intermediate words are irrelevant, should the update gate

z_t

be high or low?

High. High

z_t

means

h_t \approx h_{t-1}

, preserving long-term memory and ignoring irrelevant new information.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

GRU ek simplified version hai LSTM ka. Socho RNN ko long-term memory problem hai—bahut zyada timesteps ke bad gradient vanish ho jata hai, toh network bhool jaata hai purane words. LSTM neisko solve kiya3 gates se aur alag cell state se, lekin thoda complex ho gaya. GRU bolta hai: kya zarurat hai 3 gates ki? Chaliye 2 gates se kaam chalate hain—ek update gate jo decide karta hai kitna purana hidden state rakhna hai aur kitna naya candidate lena hai, aur ek reset gate jo decide karta hai ki naya candidate banate waqt kitna purana context use karna hai.

Result kya hai? GRU me 25% kam parameters hain LSTM se

What Problem Does GRU Solve?

The GRU Architecture

Derivation from First Principles

Step 1: Compute Reset Gate

Step 2: Compute Candidate Hidden State

Step 3: Compute Update Gate

Step 4: Final Hidden State (Linear Interpolation)

GRU vs LSTM: Formula Comparison

Example 1: Sentiment Analysis Sentence

Example 2: Long-Term Dependency (Name Recall)

Common Mistakes

When to Use GRU vs LSTM?

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Sequence Models

Connections