Bidirectional RNNs

3.5.6 · AI-ML › Sequence Models

Is Problem Ka Solution Kyun Chahiye?

Vanilla RNNs mein, time $t$ par hidden state $h_t$ sirf inputs $x_1,x_2,\dots ,x_t$ ki information encode karta hai. Ye causal tasks ke liye perfect hai (next word predict karna) lekin un tasks ke liye bahut bura hai jahan answer puri context par depend karta hai:

• Named Entity Recognition: "Paris" in "I live in Paris" (city) vs "Paris Hilton visited" (person) – tumhe "Paris" ke baad wale words chahiye
• Machine Translation: Word order alag-alag languages mein alag hoti hai; word 3 translate karne ke liye word 7 jaanna pad sakta hai
• Speech Recognition: Ek phoneme ki identity aksar baad wale phonemes par depend karti hai
• Sentiment Analysis: "This movie is not bad" – "not" sab kuch badal deta hai, lekin "bad" se pehle aata hai

Standard RNNs causally constrained hote hain – ye "aage dekh" nahi sakte. Bidirectional RNNs ye constraint hatate hain jab inference time par poori sequence available ho.

Architecture: Do RNNs, Ek Forward, Ek Backward

Forward Pass Ko First Principles Se Derive Karna

Step-by-step banate hain. Ek standard RNN cell equation se shuru karo:

$h_t=\tanh (W_h{h}_{t-1}+W_{xh}{x}_t+b_h)$

Ye form kyun? Hidden state (1) previous hidden state $h_{t-1}$ (memory) aur (2) current input $x_t$ (nayi information) ka nonlinear combination hai. tanh [-1, 1] tak squash karta hai taaki values explode na karein.

Ab, ek bidirectional architecture ke liye:

Forward RNN (left-to-right process karta hai): ${\overrightarrow{h}}_t=\tanh (W_{\overrightarrow{h}}{\overrightarrow{h}}_{t-1}+W_{\overrightarrow{x}}{x}_t+b_{\overrightarrow{h}})$

• ${\overrightarrow{h}}_0=0$ se start karta hai (ya learned initialization se)
• $t=1$ par: ${\overrightarrow{h}}_1$ sirf $x_1$ par depend karta hai
• $t=2$ par: ${\overrightarrow{h}}_2$, $x_1,x_2$ par depend karta hai (${\overrightarrow{h}}_1$ ke through)
• $t=T$ par: ${\overrightarrow{h}}_T$ poori past sequence $x_1,\dots ,x_T$ encode karta hai

Backward RNN (right-to-left process karta hai): ${\overleftarrow{h}}_t=\tanh (W_{\overleftarrow{h}}{\overleftarrow{h}}_{t+1}+W_{\overleftarrow{x}}{x}_t+b_{\overleftarrow{h}})$

• ${\overleftarrow{h}}_{T+1}=0$ se start karta hai
• $t=T$ par: ${\overleftarrow{h}}_T$ sirf $x_T$ par depend karta hai
• $t=T-1$ par: ${\overleftarrow{h}}_{T-1}$, $x_T,x_{T-1}$ par depend karta hai (${\overleftarrow{h}}_T$ ke through)
• $t=1$ par: ${\overleftarrow{h}}_1$ poori future sequence $x_2,\dots ,x_T$ encode karta hai

Alag weight matrices kyun? Forward aur backward RNNs alag patterns learn kar rahe hain (past vs. future dependencies), isliye unhe independent parameters chahiye. Weights share karne se dono directions ek compromise representation seekhne par majboor ho jaate.

Har timestep par outputs combine karna:

Final output layer (kisi task jaise NER ya POS tagging ke liye): $y_t=\text{softmax}(W_y{h}_t+b_y)=\text{softmax}(W_y[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]+b_y)$

Yahan softmax kyun? Hum har timestep par ek classification decision le rahe hain (jaise "kya ye word PERSON hai ya LOCATION?"), isliye humein classes par ek probability distribution chahiye.

Poora Forward Pass Algorithm

Input sequence $\mathbf{x}=(x_1,x_2,\dots ,x_T)$ diya gaya hai:

1. Initialize: ${\overrightarrow{h}}_0=0$, ${\overleftarrow{h}}_{T+1}=0$

2. Forward pass (left-to-right):

   for t = 1 to T:
       h⃗ₜ = tanh(W_h⃗ h⃗ₜ₋₁ + W_x⃗ xₜ + b_h⃗)
   

3. Backward pass (right-to-left):

   for t = T down to 1:
       h⃖ₜ = tanh(W_h⃖ h⃖ₜ₊₁ + W_x⃖ xₜ + b_h⃖)
   

4. Combine:

   for t = 1 to T:
       hₜ = [h⃗ₜ; h⃖ₜ]
       yₜ = softmax(Wᵧ hₜ + bᵧ)
   

Do alag loops kyun? Forward RNN complete hona chahiye pehle backward RNN outputs compute karne se, kyunki ${\overleftarrow{h}}_t$, ${\overleftarrow{h}}_{t+1}$ par depend karta hai, jiske liye right-to-left processing chahiye. Practice mein, frameworks ise efficiently vectorize kar dete hain.

Worked Example: Named Entity Recognition

Bidirectional RNNs Kab Use Karein (Aur Kab Nahi)

BiRNNs tab use karo jab:

• Inference time par poori sequence available ho (offline processing)
• Task non-causal ho: position $t$ par label/output, positions $>t$ par inputs par depend kar sakti hai
• Examples: NER, POS tagging, machine translation (encoder side), speech recognition (offline), sentiment analysis, question answering (question ya passage encode karna)

BiRNNs tab mat use karo jab:

• Task causal/autoregressive ho: next token predict karna (language modeling, text generation, online speech recognition)
• Kyun? Time $t$ par, tumhare paas $x_{t+1},x_{t+2},\dots$ abhi available nahi hain. Yahan BiRNN use karna "cheating" hoga – tumhara model real-time setting mein deploy nahi ho sakta.
• Real-world constraint: Agar tumhe predictions banana hai jaise sequence aati hai (jaise live captioning, chatbots), tum backward pass run karne ke liye poori sequence ka wait nahi kar sakte.

Computational Considerations

Time complexity:

• Unidirectional RNN: $O(T\cdot d_h^2)$ length $T$ ki sequence aur hidden dimension $d_h$ ke liye
• Bidirectional RNN: $O(2\cdot T\cdot d_h^2)=O(T\cdot d_h^2)$ (same order, lekin 2× constants)

Kyun? Tum do RNNs run kar rahe ho, har ek $T$ timesteps ke saath aur har step par $O(d_h^2)$ kaam (matrix multiply $W_{hh}{h}_{t-1}$).

Space complexity:

• Backpropagation ke liye dono ${\overrightarrow{h}}_1,\dots ,{\overrightarrow{h}}_T$ aur ${\overleftarrow{h}}_1,\dots ,{\overleftarrow{h}}_T$ store karne padte hain
• Total: $O(2\cdot T\cdot d_h)$ memory

Parallelization:

• Forward aur backward passes apne andar parallelizable nahi hain (sequential dependency: $h_t$, $h_{t-1}$ par depend karta hai)
• Lekin forward aur backward RNNs independent hain, isliye agar tumhare paas multiple GPUs hain ya multiple sequences batch-process kar rahe ho, toh inhe parallel compute kar sakte ho
• Transformers se contrast: Transformers saare timesteps parallel process karte hain (self-attention), jo unhe modern hardware par bahut faster banata hai

Backpropagation Through Time (BPTT) for BiRNNs

Training ke dauran, hume saare parameters ke respect mein gradients chahiye: $W_{\overrightarrow{h}},W_{\overrightarrow{x}},W_{\overleftarrow{h}},W_{\overleftarrow{x}},W_y$.

Loss function (jaise sequence tagging ke liye): $\mathcal{L}={\sum }_{t=1}^T\text{CrossEntropy}(y_t,{\hat{y}}_t)$

Gradient flow:

1. $\frac{\partial \mathcal{L}}{\partial h_t}$ compute karo (combined hidden state ke respect mein gradient)
2. Kyunki $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$ hai, $\frac{\partial \mathcal{L}}{\partial {\overrightarrow{h}}_t}$ aur $\frac{\partial \mathcal{L}}{\partial {\overleftarrow{h}}_t}$ mein split karo
3. Forward RNN gradients time mein backward flow karte hain (forward chain ke through right-to-left): $\frac{\partial \mathcal{L}}{\partial {\overrightarrow{h}}_{t-1}}=\frac{\partial \mathcal{L}}{\partial {\overrightarrow{h}}_t}\cdot \frac{\partial {\overrightarrow{h}}_t}{\partial {\overrightarrow{h}}_{t-1}}=\frac{\partial \mathcal{L}}{\partial {\overrightarrow{h}}_t}\cdot W_{\overrightarrow{h}}^T\cdot {\text{tanh}}^{\prime }(\cdots )$
4. Backward RNN gradients time mein forward flow karte hain (backward chain ke through left-to-right): $\frac{\partial \mathcal{L}}{\partial {\overleftarrow{h}}_{t+1}}=\frac{\partial \mathcal{L}}{\partial {\overleftarrow{h}}_t}\cdot \frac{\partial {\overleftarrow{h}}_t}{\partial {\overleftarrow{h}}_{t+1}}=\frac{\partial \mathcal{L}}{\partial {\overleftarrow{h}}_t}\cdot W_{\overleftarrow{h}}^T\cdot {\text{tanh}}^{\prime }(\cdots )$

Ye tricky kyun hai: Tumhare paas do vanishing/exploding gradient problems hain (ek har RNN ke liye). Wohi solutions apply hote hain: LSTMs/GRUs use karo, gradient clipping karo, careful initialization karo.

Bidirectional LSTMs (BiLSTMs)

Practice mein, vanilla BiRNNs vanishing gradients ki wajah se rarely use hote hain. Uski jagah, hum Bidirectional LSTMs (BiLSTMs) ya Bidirectional GRUs (BiGRUs) use karte hain.

Architecture bilkul same hai, bas har RNN cell ko LSTM/GRU cell se replace karte hain:

${\overrightarrow{h}}_t,{\overrightarrow{c}}_t={\text{LSTM}}_{\text{forward}}(x_t,{\overrightarrow{h}}_{t-1},{\overrightarrow{c}}_{t-1})$ ${\overleftarrow{h}}_t,{\overleftarrow{c}}_t={\text{LSTM}}_{\text{backward}}(x_t,{\overleftarrow{h}}_{t+1},{\overleftarrow{c}}_{t+1})$ $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$

LSTMs kyun? Cell state $c_t$ gradients ke liye ek "highway" provide karta hai, jo long sequences par vanishing gradients ko mitigate karta hai. NLP tasks ke liye ye crucial hai jahan dependencies 20+ words tak span kar sakti hain.

Example task: "The movie was not particularly bad, but the acting was terrible" par sentiment analysis.

• "not" par Forward LSTM: "The movie was not" encode karta hai
• "not" par Backward LSTM: "not particularly bad, but the acting was terrible" encode karta hai
• Combined: model seekhta hai ki "not...bad" ek negation hai, lekin "acting was terrible" sentiment ko dominate karta hai

Connections

• Recurrent Neural Networks (RNNs): Bidirectional RNNs do unidirectional RNNs se bane hote hain
• LSTM and GRU Cells: BiLSTMs aur BiGRUs, BiRNNs mein vanishing gradients solve karte hain
• Encoder-Decoder Architecture: BiRNNs commonly encoder mein use hote hain (poora source sentence encode karo) lekin kabhi decoder mein nahi (autoregressive generation)
• Attention Mechanism: Aksar BiRNN outputs ke upar apply hota hai timesteps ko alag-alag weight karne ke liye
• Transformers: BiRNNs ko self-attention se replace karte hain better parallelization ke liye, lekin intuition (full-sequence context) similar hai
• Named Entity Recognition: BiRNNs/BiLSTMs ka canonical application
• Part-of-Speech Tagging: Ek aur sequence labeling task jahan BiRNNs excel karte hain
• Sequence-to-Sequence Models: Encoder mein BiRNNs decoder ko poore input context par attend karne dete hain

#flashcards/ai-ml

Ek unidirectional RNN aur ek Bidirectional RNN mein kya main fark hai? :: Ek unidirectional RNN sequence ko ek direction mein (left-to-right) process karta hai, isliye $h_t$ sirf past inputs $x_1,\dots ,x_t$ par depend karta hai. Ek Bidirectional RNN mein do RNNs hote hain: ek left-to-right process karta hai (${\overrightarrow{h}}_t$), ek right-to-left (${\overleftarrow{h}}_t$), aur har timestep par $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$ hota hai, jo past aur future dono context access deta hai.

Concept Map