Gated Recurrent Units (GRU)

3.5.5 · AI-ML › Sequence Models

GRU Kis Problem Ko Solve Karta Hai?

Standard RNN ka issue: Gradient bahut zyada timesteps par exponentially vanish ho jaata hai → long-range dependencies seekh nahi sakta.

LSTM ka solution: Separate cell state $c_t$ gates ke saath information flow control karne ke liye.

GRU ki insight: Kya hume sach mein ek separate cell state chahiye? Kya hum hidden state aur cell state ko ek $h_t$ mein combine kar sakte hain, aur forget aur input gates ko ek single update gate mein merge kar sakte hain? Isse parameters ~25% kam ho jaate hain aur LSTM ki zyaadatar power bani rehti hai.

GRU Architecture

GRU mein 2 gates hain jo hidden state $h_t$ par operate karte hain:

First Principles Se Derivation

Goal: Hidden state $h_t$ ko $h_{t-1}$ aur $x_t$ se update karo, information flow control karte hue.

Step 1: Reset Gate Compute Karo

$r_t=\sigma (W_r\cdot [h_{t-1},x_t]+b_r)$

Yeh step kyun? Hume decide karna hai: kya past $h_{t-1}$ un naye candidate ke liye relevant hai jo hum compute karne wale hain? Agar sequence ka context change hota hai (jaise topic shift), toh hum chahte hain $r_t\to 0$ taaki "reset" ho sake aur irrelevant history se bias na aaye.

Step 2: Candidate Hidden State Compute Karo

${\tilde{h}}_t=\tanh (W_h\cdot [r_t\odot h_{t-1},x_t]+b_h)$

Yeh step kyun?

• $r_t\odot h_{t-1}$ purani state ko reset gate se element-wise multiply karta hai. Agar $r_t$ chhota hai, toh hum $h_{t-1}$ ignore karte hain.
• $x_t$ ke saath concatenate karo, tanh se pass karo taaki candidate update ${\tilde{h}}_t$ mila sake $[-1,1]$ mein.
• Yeh "proposed new memory" hai agar hum $h_{t-1}$ ko poori tarah replace karte.

Step 3: Update Gate Compute Karo

$z_t=\sigma (W_z\cdot [h_{t-1},x_t]+b_z)$

Yeh step kyun? Hume decide karna hai: purani $h_{t-1}$ kitni rakhni hai vs. naye candidate ${\tilde{h}}_t$ kitna accept karna hai. Yahi interpolation weight hai.

Step 4: Final Hidden State (Linear Interpolation)

$h_t=(1-z_t)\odot {\tilde{h}}_t+z_t\odot h_{t-1}$

Yeh step kyun?

• Agar $z_t\approx 1$: $h_t\approx h_{t-1}$ → purani memory preserve karo (jaise LSTM ka high forget gate).
• Agar $z_t\approx 0$: $h_t\approx {\tilde{h}}_t$ → naya candidate accept karo (jaise LSTM ka high input gate).
• Yeh ek convex combination hai: $z_t$ ek learned interpolation weight ki tarah kaam karta hai har dimension ke liye.

Key insight: Update gate $z_t$ LSTM ke forget aur input gates ko merge karta hai. LSTM mein, alag $f_t$ (forget) aur $i_t$ (input) hote hain. GRU mein, jab $z_t$ high hota hai (purana rakho), toh naye ke liye low hota hai (naya reject karo), aur vice versa—yeh coupled hain $(1-z_t)$ aur $z_t$ ki tarah.

GRU vs LSTM: Formula Comparison

Parameter count: GRU mein LSTM se ~25% kam parameters hain (2 gates vs 3, koi separate cell state weights nahi).

Example 1: Sentiment Analysis Sentence

Sequence: "The movie started well but the ending was terrible."

Chalte hain ek GRU cell trace karte hain word "but" par (timestep $t$):

• Input: $x_t$ = "but" ka embedding, $h_{t-1}$ = "well" ke baad hidden state (likely positive sentiment accumulated).

Step 1: $r_t$ compute karo: $r_t=\sigma (W_r[h_{t-1},x_t]+b_r)$ "But" ek contrast signal deta hai. Agar network ne seekha hai ki "but" aksar sentiment reverse karta hai, toh $r_t$ low ho sakta hai (say 0.2), matlab "positive $h_{t-1}$ par zyaada rely mat karo—hum direction change karne wale hain."

Yeh step kyun? Reset gate contrast word detect karta hai aur decide karta hai positive history ko partially ignore karna hai.

Step 2: ${\tilde{h}}_t$ compute karo: ${\tilde{h}}_t=\tanh (W_h[r_t\odot h_{t-1},x_t]+b_h)$ $r_t=0.2$ ke saath, hum $h_{t-1}$ ko 0.2 se multiply karte hain, purane positive sentiment ko heavily damp karte hue. Candidate ${\tilde{h}}_t$ ab zyaadatar $x_t$ ("but") se driven hai aur neutral ki taraf shift hone lagega ya negativity ke liye prepare karega.

Yeh step kyun? Hum ek fresh candidate compute karte hain jo past positive context se zyaada biased nahi hai.

Step 3: $z_t$ compute karo: $z_t=\sigma (W_z[h_{t-1},x_t]+b_z)$ "But" ek pivot word hai. Network seekh sakta hai ki $z_t$ low hona chahiye (say 0.3), matlab "purani positive $h_{t-1}$ zyaada mat rakho—naye candidate ka zyaada accept karo."

Yeh step kyun? Update gate decide karta hai ki purane sentiment se hat ke naye candidate ki taraf shift karna hai.

Step 4: Final $h_t$: $h_t=(1-0.3)\odot {\tilde{h}}_t+0.3\odot h_{t-1}=0.7{\tilde{h}}_t+0.3h_{t-1}$ Nayi hidden state 70% candidate hai (jisme reset gate ki wajah se past ka kam influence hai) aur 30% purani state. Net effect: sentiment representation shift karta hai positive se neutral/negative ki taraf.

Result: Jaise sequence aage badhti hai ("ending was terrible"), reset gate phir low hoga, update gate naye negative information ko favor karega, aur final $h_T$ negative sentiment represent karega.

Example 2: Long-Term Dependency (Name Recall)

Sequence: "Alice went to the park. She played with her dog. Later, she went home. __ was tired."

Hum blank mein "Alice" (ya "She") fill karna chahte hain, jo "Alice" ki memory require karta hai sequence ke start se.

Blank par (timestep $t$):

• "Alice" mention hone ke baad bahut saare timesteps guzar chuke hain.
• Intermediate words ("park", "dog", "home") kaun tired hai, iske liye less relevant hain.

GRU ka behavior:

Intermediate words ke liye (park, dog, home):

• $z_t$ high hai (say 0.8–0.9): Update gate seekhta hai ki yeh words subject identity change nahi karte, isliye $h_t\approx h_{t-1}$. Identity "Alice" preserved rahti hai kyunki $(1-z_t)$ term chhoti hai—hum zyaadatar purani $h_{t-1}$ rakhte hain.

Yeh kyun kaam karta hai: High $z_t$ LSTM ke high forget gate + low input gate ki tarah kaam karta hai: "purani memory rakho, naya irrelevant content ignore karo." Gradient $(1-z_t)$ aur $z_t$ terms ke through backward flow karta hai, jo gated hain—gradient highway.

Blank par:

• $h_{t-1}$ abhi bhi "Alice" encode karta hai kyunki intermediate $z_t$ high the.
• Decoder "Alice" ko $h_t$ se extract kar sakta hai "She" ya "Alice was tired" generate karne ke liye.

Key: GRU ka update gate $z_t$ gradient flow control karta hai. Jab $z_t\approx 1$, gradient $\frac{\partial h_t}{\partial h_{t-1}}\approx 1$ (chain rule), vanishing gradient se bachta hai.

Common Mistakes

GRU vs LSTM Kab Use Karein?

Rule of thumb: GRU se shuru karo. Agar performance ceiling aa jaaye aur resources hon, LSTM try karo. Difference aksar marginal hota hai (1-2% accuracy) practice mein.

Connections

• 3.5.03-Long-Short-Term-Memory-(LSTM) — GRU, LSTM ko gates merge karke simplify karta hai
• 3.5.02-Vanishing-and-Exploding-Gradients — GRU, gating ke zariye vanishing gradient solve karta hai
• 3.5.01-Basic-RNN-Architecture — GRU, basic RNN ko long-term memory ke liye gates ke saath extend karta hai
• 3.5.06-Bidirectional-RNNs — GRU ko bidirectionally use kiya ja sakta hai dono directions se context ke liye
• 3.6.01-Attention-Mechanism — Attention, bahut lambi sequences ke liye GRU ko replace/augment kar sakta hai

#flashcards/ai-ml

GRU mein do gates kaun se hain aur har ek kya control karta hai? :: Reset gate $r_t$ control karta hai ki past hidden state naye candidate ko kitna influence kare. Update gate $z_t$ purani hidden state aur naye candidate ke beech interpolation control karta hai.

GRU final hidden state update equation likho :: $h_t=(1-z_t)\odot {\tilde{h}}_t+z_t\odot h_{t-1}$, jahan $z_t$ update gate hai, ${\tilde{h}}_t$ candidate hidden state hai.

Concept Map