Recurrent Neural Networks (RNN) architecture

3.5.1 · AI-ML › Sequence Models

Overview

Recurrent Neural Networks (RNNs) wo neural network architectures hain jo sequential data process karne ke liye design ki gayi hain — ye ek internal hidden state maintain karti hain jo memory ki tarah kaam karta hai. Feedforward networks ke ulat, jo har input ko alag se treat karte hain, RNNs mein connections hote hain jo khud par wapas loop karte hain, jisse time steps ke across information persist ho sakti hai.

Figure — Recurrent Neural Networks (RNN) architecture

Core Architecture Components

Derivation from First Principles

YE architecture kyun hai?

Problem se shuru karo: hum chahte hain ek function $f$ jo sequence ko outputs se map kare, lekin function ko pehle ke inputs "yaad" rehne chahiye.

Step 1: Memory ko ek vector ke roop mein encode karo

Maano $h^{(t)}$ hamaari time $t$ par memory hai
$h^{(t)}$ ko current input $x^{(t)}$ AUR pehli memory $h^{(t - 1)}$ dono par depend karna chahiye

Step 2: Inputs ko linearly combine karo

Sabse general linear combination: $z = W_{h} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h}$
Linear kyun? Kyunki linear maps ko non-linearities ke saath compose karke hum koi bhi non-linear transformation seekh sakte hain

Step 3: Non-linearity add karo

Activation function apply karo: $h^{(t)} = σ (z)$
$tanh$ kyun? Output $[- 1, 1]$ mein hota hai, backprop ke dauran sigmoid ke comparison mein gradients zyada healthy rehte hain, aur ye zero-centered hai

Step 4: Output produce karo

Linear projection: $y^{(t)} = W_{h y} h^{(t)} + b_{y}$
Hidden state se alag kyun? Isse hidden dimension output dimension se alag ho sakta hai; hidden state zyada rich ho sakta hai

Unfolding Through Time

"Recurrence" ko time ke through network ko unfolding karke visualize kiya ja sakta hai:

x^(1)    x^(2)    x^(3)
 |        |        |
 ↓        ↓        ↓
[RNN] → [RNN] → [RNN]
  ↓        ↓        ↓
 y^(1)   y^(2)   y^(3)

Har box SAME weights share karta hai. Boxes ke beech ke arrows $h^{(t - 1)} \to h^{(t)}$ represent karte hain.

Worked example Example 1: Character-Level Language Model

Task: "HELLO" mein next character predict karo Setup:

Vocabulary: {H, E, L, O ...}, size $∣ V ∣ = 26$
Har character ko one-hot encode karo: $x^{(t)} \in R^{26}$
Hidden dimension: $d_{h} = 128$
Output: next character ki probability distribution $P (c_{t + 1} ∣ c_{1}, \dots, c_{t})$

"HEL" ke liye forward pass: $t = 1$ : Input "H" (one-hot: $x^{(1)} = [1, 0, 0, \dots]$ )

$h^{(1)} = tanh (W_{x h} x^{(1)} + b_{h})$ (koi previous state nahi, $h^{(0)} = 0$ )
$y^{(1)} = W_{h y} h^{(1)} + b_{y}$ → softmax → "E" predict karo

Ye step kyun? Scratch se hidden state initialize karo kyunki koi pehla context nahi hai.

$t = 2$ : Input "E" (one-hot: $x^{(2)} = [0, 1, 0, \dots]$ )

$h^{(2)} = tanh (W_{h} h^{(1)} + W_{x h} x^{(2)} + b_{h})$
$y^{(2)} = W_{h y} h^{(2)} + b_{y}$ → softmax → "L" predict karo

Ye step kyun? $W_{hh} h^{(1)}$ "H" dekhne ki information carry karta hai; naye "E" input ke saath combine hota hai.

$t = 3$ : Input "L"

$h^{(3)} = tanh (W_{hh} h^{(2)} + W_{x h} x^{(3)} + b_{h})$
$y^{(3)} = W_{h y} h^{(3)} + b_{y}$ → softmax → "L" predict karo

Ye step kyun? Ab $h^{(3)}$ "ab tak HE dekha" encode karta hai, doosre "L" ko predict karne mein madad karta hai.

Training: Backpropagation Through Time (BPTT)

$W_{hh}$ ke liye Gradient Derive karna:

Chain rule se: $\frac{\partial L}{\partial W _{hh}} = \sum_{t = 1}^{T} \frac{\partial L ^{(t)}}{\partial W _{hh}}$

Time step $t$ ke liye: $\frac{\partial L ^{(t)}}{\partial W _{hh}} = \frac{\partial L ^{(t)}}{\partial y ^{(t)}} \frac{\partial y ^{(t)}}{\partial h ^{(t)}} \frac{\partial h ^{(t)}}{\partial W _{hh}}$

Lekin $h^{(t)}$ depend karta hai $h^{(t - 1)}$ par, jo depend karta hai $h^{(t - 2)}$ par, ... $h^{(1)}$ tak wapas!

Full gradient: $\frac{\partial L ^{(t)}}{\partial W _{hh}} = \sum_{k = 1}^{t} \frac{\partial L ^{(t)}}{\partial y ^{(t)}} \frac{\partial y ^{(t)}}{\partial h ^{(t)}} \frac{\partial h ^{(t)}}{\partial h ^{(k)}} \frac{\partial h ^{(k)}}{\partial W _{hh}}$

Jahan: $\frac{\partial h ^{(t)}}{\partial h ^{(k)}} = \prod_{i = k + 1}^{t} \frac{\partial h ^{(i)}}{\partial h ^{(i - 1)}} = \prod_{i = k + 1}^{t} W_{h}^{T} \cdot diag (tanh^{'} (z^{(i)}))$

Ye kyun matter karta hai? Jacobians ka ye product vanishing/exploding gradients ka karan banta hai.

RNN Variants and Types

Common Mistakes and Fixes

Common mistake Mistake 1: Sequences ke beech Hidden State Reset karna bhool jaana

Galat soch: Pehle training example ka $h^{(T)}$ अगले example ke $h^{(0)}$ ke roop mein reuse karo. Ye sahi kyun lagta hai: "Shayad network examples ke across connections seekh sake?"

Reality: Training examples independent hain. State carry over karne se spurious correlations banti hain. Network un jagahon par patterns dhundne ki koshish karta hai jahan koi pattern hai hi nahi.

Fix: Har naye sequence ki shuruwat mein hamesha $h^{(0)} = 0$ (ya learned vector) initialize karo.

# Wrong
for batch in dataset:
    h = model(batch, h)  # h carries over!
 
# Right
for batch in dataset:
    h = torch.zeros(hidden_size)  # Reset!
    h = model(batch, h)

Common mistake Mistake 2: Long Sequences ke liye BPTT Truncate na karna

Galat soch: Chahe sequence kitni bhi lambi ho, pure sequence ke through backpropagate karo.

Ye sahi kyun lagta hai: "Full gradients zyada accurate hote hain."

Reality: 1000+ length ki sequences ke liye, memory explode ho jaati hai (saari intermediate activations store karni padti hain) aur vanishing gradients waise bhi learning khatam kar dete hain.

Fix: Truncated BPTT: Sirf $k$ steps ke through backprop karo (jaise $k = 20$ ). Pure sequence ke through forward-prop karte raho, lekin har $k$ steps par gradient detach karo.

for t in range(0, T, truncate_len):
    loss = 0
    for in range(truncate_len):
        h = rnn_step(x[t+i], h)
        loss += criterion(h, target[t+i])
    loss.backward()  # Only backprop through truncate_len steps
    h = h.detach()  # Detach for next chunk

Key Properties

Property	Description
Parameter Sharing	Har time step par same weights → variable-length sequences handle karta hai
Hidden State	$h^{(t)}$ memory ki tarah kaam karta hai, time $t$ tak sequence ka summary
Inductive Bias	Ye assume karta hai ki paas ke elements door walon se zyada relevant hain
Limitation	Vanishing gradients → poor long-term dependencies (LSTM/GRU se fix hota hai)

Computational Complexity

Length $T$ ki sequence ke liye:

Forward pass: $O (T \cdot d_{h}^{2})$ (har step par matrix multiplications)
Backward pass (BPTT): $O (T \cdot d_{h}^{2})$ (same, lekin gradients backward flow karte hain)
Space: $O (T \cdot d_{h})$ (backprop ke liye hidden states store karo)

Feedforward se comparison:

Feedforward: Ek input ke liye $O (1)$ , koi sequential dependency nahi
RNN: $O (T)$ sequential operations, time ke across parallelize nahi ho sakta

Ye sequential nature RNN ki Transformers ke comparison mein key limitation hai.

Recall Ek 12 saal ke bachche ko samjhao

Socho tum ek detective story padh rahe ho. Har page tumhe ek clue deta hai. Ek normal robot brain (feedforward network) har page ko alag se dekhega aur jo pehle padha usse bhool jaayega — mystery solve karne ke liye bilkul bekar!

Ek RNN ek detective ki notebook ki tarah hai. Jab tum page 1 padhte ho, tum apni notebook mein important cheezein likhte ho. Page 2 par, tum NAYA clue padhte ho AUR apni notebook check karte ho ki pehle kya seekha tha. Tum is combined information se notebook update karte ho. Page 3? Same cheez — naya clue padho, notebook check karo, update karo.

Notebook "hidden state" ( $h$ ) hai. Detective ka dimaag har page ke liye same thinking rules (weights $W$ ) use karta hai, lekin notebook ke contents badlte rehte hain jaise tum zyada seekhte ho. Last page tak, teri notebook mein POORI kitaab ke clues hain, jo tumhe kehne mein madad karte hain "The butler did it!"

Problem: Agar kitaab 1000 pages lambi hai, toh notebook page 1000 tak page 1 ki cheezein bhool sakti hai — yahi vanishing gradient problem hai. Behtar detectives (LSTM, GRU) ke paas special notebooks hoti hain jo zyada lambe time tak yaad rakh sakti hain!

Connections

Long Short-Term Memory (LSTM) - Gating mechanisms se vanishing gradient solve karta hai
Gated Recurrent Unit (GRU) - Kam parameters ke saath LSTM ka simpler alternative
Backpropagation Through Time (BPTT) - RNNs ke liye training algorithm
Vanishing and Exploding Gradients - Vanilla RNNs ko limit karne wali core problem
Sequence-to-Sequence Models - RNNs use karte hue encoder-decoder architecture
Attention Mechanism - Input sequence ke relevant parts par focus karne deta hai
Transformers - Modern alternative jo sequence processing ko parallelize karta hai
Word Embeddings - RNNs ke saath NLP ke liye common input representation
Gradient Clipping - Exploding gradients rokne ki technique

#flashcards/ai-ml

RNNs mein hidden state ka kya purpose hai? :: Hidden state $h^{(t)}$ memory ki tarah kaam karta hai, jo saare pehle time steps ke baare mein information encode karta hai, jisse network sequence ke across context maintain kar sakta hai.

RNNs time steps ke across weights kyun share karte hain?

Weight sharing RNNs ko fixed number of parameters ke saath variable-length sequences handle karne deta hai, aur ek inductive bias provide karta hai ki sequence ke rules time ke saath nahi badalte (CNNs mein spatial weight sharing ke analogous).

RNN hidden state update equation likho

h^{(t)} = tanh (W_{hh} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h})

jahan

W_{hh}

recurrent weights hain,

W_{x h}

input weights hain, aur

tanh

activation function hai.

RNNs mein vanishing gradients kya cause karta hai?

Backpropagation through time ke dauran, gradients mein Jacobians ke products hote hain

\prod_{i} W_{hh}^{T} \cdot diag (tanh^{'})

. Agar

∣∣ W_{hh} ∣∣ < 1

, toh repeated multiplication gradients ko exponentially shrink kar deta hai, jisse long-term dependencies seekhna impossible ho jaata hai.

Many-to-one aur many-to-many RNN architectures mein kya fark hai?

Many-to-one ek input sequence process karta hai aur ek single output produce karta hai (jaise final hidden state use karke sentiment classification). Many-to-many ek input sequence process karta hai aur ek output sequence produce karta hai (jaise machine translation), potentially alag input/output lengths ke saath.

Har naye sequence ki shuruwat mein

h^{(0)} = 0

initialize kyun karte hain?

Training examples independent hote hain. Pehle sequence ka hidden state carry over karne se spurious correlations banti hain, jisse network un patterns ko seekhne ki koshish karta hai jo alag sequences ke across actually exist nahi karte.

Truncated BPTT kya hai aur isko kyun use karte hain?

Truncated Backpropagation Through Time gradient computation ko full sequence ki jagah

k

time steps tak limit karta hai. Ye long sequences ke liye memory usage aur computation reduce karta hai, jabki pure sequence ke through forward propagation ab bhi allow karta hai.

RNN hidden state se output kaise produce karte hain?

Ek linear projection apply karo:

y^{(t)} = W_{h y} h^{(t)} + b_{y}

, phir task-specific activation apply karo (classification ke liye softmax, regression ke liye kuch nahi). Isse hidden dimension output dimension se alag ho sakta hai.

Sequence length

T

ke liye RNN forward pass ki computational complexity kya hai?

O (T \cdot d_{h}^{2})

jahan

d_{h}

hidden dimension hai. Ye har time step par matrix multiplications ki wajah se hota hai. Feedforward networks ke ulat, RNN operations sequential hain aur time ke across parallelize nahi ho sakte.

RNN hidden state mein sigmoid ki jagah

tanh

activation kyun use karte hain?

tanh

sigmoid ke

[0, 1]

ki jagah

[- 1, 1]

range mein output deta hai (zero-centered), jo backpropagation ke dauran healthier gradients maintain karne mein madad karta hai aur hidden state mein bias shift issues rokta hai.