Limitations of RNNs motivating transformers — Hinglish notes

4.1.1 · AI-ML › Transformer Architecture

Overview

Recurrent Neural Networks (RNNs) 1990s se lekar 2016 tak sequence modeling ke liye dominant architecture thi, lekin inme kuch fundamental architectural constraints hain jinhe overcome karne ke liye transformers ko explicitly design kiya gaya tha. RNNs kyun fail hote hain ye samajhna is baat ki appreciation ke liye critical hai ki transformer architecture kyun kaam karta hai.

Figure — Limitations of RNNs motivating transformers

Core Problem: Sequential Processing Bottleneck

Limitation 1: Time Ke Across Parallelize Nahi Ho Sakta

Math: Parallelization Kyun Impossible Hai

RNN recurrence relation se shuru karte hain:

$h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})$

Aao $h_{3}$ directly compute karne ki koshish karte hain. Humein chahiye: $h_{3} = tanh (W_{hh} h_{2} + W_{x h} x_{3} + b_{h})$

Lekin $h_{2}$ ke liye chahiye: $h_{2} = tanh (W_{hh} h_{1} + W_{x h} x_{2} + b_{h})$

Aur $h_{1}$ ke liye chahiye: $h_{1} = tanh (W_{hh} h_{0} + W_{x h} x_{1} + b_{h})$

Conclusion: $h_{t}$ compute karne ke liye, humein pehle $h_{1}, h_{2}, \dots, h_{t - 1}$ sequentially ZAROOR compute karne padte hain. "Skip ahead" karne ka koi tarika nahi hai kyunki har hidden state apne previous state ki ek non-linear function hai.

Worked example Concrete Scaling Example

Task: 32 sequences ka ek batch process karo, jinmein se har ek 512 tokens ki hai. RNN Approach (LSTM):

Har sequence ko 512 timesteps sequentially process karne padte hain
Batch size 32 ke saath, hum 32 sequences parallel mein process kar sakte hain
Total sequential steps: 512 steps (sequence length)
5,000 CUDA cores wale GPU par, har timestep mein 4,968 cores idle baithe rehte hain Ye step kyun? Bhale hi hamare paas 32 sequences hain, unhe sab milke time mein aage badhna padta hai. Timestep $t = 100$ par, SAARI sequences $h_{100}$ compute kar rahi hain, lekin isse $h_{101}$ compute karna kuch bhi fast nahi hota.

Transformer Approach:

Saare 32 sequences ke saare 512 positions simultaneously compute hote hain
Total parallel operations: $32 \times 512 = 16, 384$ positions
Saare CUDA cores utilize hote hain
Wall-clock time: modern hardware par ~50× faster

Ye step kyun? Self-attention position 512 ko position 1 ko directly "dekhne" deta hai, bina positions 2-511 ke pehle process hone ka wait kiye.

Limitation 2: Vanishing Gradient Problem

Gradient Flow Derive Karna

Backpropagation through time (BPTT) ke dauran, gradients ko recurrence ke through backward flow karna padta hai:

$\frac{\partial L}{\partial h _{1}} = \frac{\partial L}{\partial h _{T}} \cdot \frac{\partial h _{T}}{\partial h _{T - 1}} \cdot \frac{\partial h _{T - 1}}{\partial h _{T - 2}} \dots \frac{\partial h _{2}}{\partial h _{1}}$

$h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})$ wale vanilla RNN ke liye:

$\frac{\partial h _{t}}{\partial h _{t - 1}} = diag (tanh^{'} (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})) \cdot W_{hh}$

Kyunki $tanh^{'} (z) = 1 - tanh^{2} (z) \in (0, 1]$ , aur typically zyaadatar values ke liye $tanh^{'} (z) < 1$ :

$\frac{\partial h _{t}}{\partial h _{t - 1}} \approx σ_{m a x} (W_{hh}) \cdot γ$

jahan $γ < 1$ , $tanh$ ki average derivative hai.

$T$ timesteps ke upar:

$\frac{\partial h _{T}}{\partial h _{1}} \approx (σ_{m a x} (W_{hh}) \cdot γ)^{T - 1}$

Worked example Ye Learning Kyun Todta Hai

Scenario: Ek RNN ko movie reviews classify karna sikhana (average length: 300 words).

Problem Setup:

Word "not" position 5 par hai: "This movie is not..."
Word "terrible" position 295 par hai: "...actually terrible"
Correct sentiment NEGATIVE hai ("not" ne "terrible" ko negate kiya)

Kya Hota Hai:

Loss end mein compute hoti hai (position 300)
Gradient backward flow karta hai: $\frac{\partial L}{\partial h _{300}}, \frac{\partial L}{\partial h _{299}}, \dots, \frac{\partial L}{\partial h _{5}}$
Jab tak gradient position 5 tak pahunchta hai jahan "not" embedded tha, tab tak wo $(0.95)^{295} \approx 1 0^{- 7}$ se multiply ho chuka hota hai Result: Model learn nahi kar sakta ki position 5 par "not" important hai. Gradient signal itna weak hai ki wo "not" process karne wale parameters update nahi kar sakta.

Ye step kyun? Exponential decay ek mathematical inevitability hai, koi implementation bug nahi. Ye recursive structure mein baked in hai.

Common mistake "LSTMs Vanishing Gradients Solve Kar Dete Hain"

Galat idea: LSTMs ke paas gates aur cell state hain, isliye wo vanishing gradient problem completely solve kar dete hain.

Ye sahi kyun lagta hai: LSTMs ke paas additive updates wala ek "highway" (cell state) hai: $c_{t} = f_{t} ⊙ c_{t - 1} + i_{t} ⊙ \tilde{c}_{t}$ . Isse ek gradient path banta hai jisme $W_{hh}$ se repeated multiplication nahi hoti. Ye galat kyun hai: LSTMs problem ko mitigate karte hain, eliminate nahi karte:

Forget gate $f_{t}$ gradient ko abhi bhi multiply karta hai: $\frac{\partial c _{t}}{\partial c _{t - 1}} = f_{t}$
Agar $f_{t} < 1$ zyaadatar steps par hai (jo hona chahiye, warna cell state explode ho jaata hai), gradients abhi bhi exponentially decay karte hain
LSTMs effective range ko ~10 steps se ~100 steps tak extend karte hain, lekin 1000+ tokens ki sequences ke liye problem wapas aa jaati hai

Fix: Transformers direct connections (self-attention) use karte hain jahan position 1, position 1000 ko directly attend kar sakta hai. 1000 se 1 tak ka gradient sirf ek matrix multiplication se guzarta hai, na ki 999 recursive steps se.

Limitation 3: Fixed Compression Bottleneck

Information Theory Perspective

Ek RNN encoder ko poori input sequence $x_{1}, \dots, x_{T}$ ko ek single fixed-size vector $h_{T}$ (final hidden state) mein compress karna padta hai.

Worked example Machine Translation Failure Case

Task: English se French mein translate karo.

Input (50 words): "The committee discussed the proposal submitted by the research team regarding the implementation of the new methodology for data collection and analysis, which had been reviewed by external consultants who provided detailed feedback on the technical specifications and recommended several modifications to the experimental design to ensure statistical validity..."

RNN Encoder:

Word-by-word process karta hai: $h_{1} \to h_{2} \to \dots \to h_{50}$
Final state $h_{50} \in R^{512}$ ko yaad rakhna hoga:
"committee" (position 1 se subject)
"discussed" (position 2 se main verb)
"proposal" (position 3 se direct object)
Multiple nested clauses
Saari entities aur unke relationships Actually Kya Hota Hai:
Position 50 tak, $h_{50}$ ne positions 1-10 ki details "bhula" di hoti hain
Model sentence ke end ko accurately translate karta hai lekin beginning se subject-verb agreement bhul jaata hai
Translation: "Le comité a discuté... [correct] ...qui avaient été examinés... [wrong gender/number agreement kyunki 'committee' bhul gaya]"

Ye step kyun? Hidden state update $h_{t} = f (h_{t - 1}, x_{t})$ ek lossy compression hai. Har update $x_{t}$ ke liye jagah banane hetu $h_{t - 1}$ ke kuch hisse overwrite kar deta hai. Step 50 tak, step 1 ki information compress hokar gayab ho jaati hai.

Attention Mechanism: Ek Partial Solution

Attention mechanism (Bahdanau et al., 2014) specifically fixed bottleneck address karne ke liye introduce kiya gaya tha:

$α_{ij} = \frac{e x p ( e _{ij} )}{\sum _{k = 1}^{T} e x p ( e _{ik} )}, e_{ij} = score (h_{i}^{dec}, h_{j}^{enc})$

$c_{i} = \sum_{j = 1}^{T} α_{ij} h_{j}^{enc}$

Decoder ab sirf final $h_{T}$ ki bajaye SAARE encoder hidden states $h_{1}, \dots, h_{T}$ ko "dekhne ke liye" wapas ja sakta hai.

Limitation 4: Long-Range Dependencies ke Saath Difficulty

Long-Range Problem ko Quantify Karna

Positions $i$ aur $j$ ke beech dependency sikhne ke baare mein sochon jahan $∣ i - j ∣ = k$ ho.

Ek RNN mein:

Gradient path ki length $k$ hai (must propagate through $k$ intermediate steps)
Gradient magnitude $γ^{k}$ ke roop mein scale hoti hai jahan $γ < 1$
Effective learning distance $k$ ke saath exponentially diminish hoti hai

Ek Transformer mein:

Self-attention ke zariye direct connection: position $i$ , position $j$ ko directly attend karta hai
Gradient path ki length 1 hai (plus network ki depth, lekin sequence length nahi)
Saari distances ke liye learning equally effective hai

Worked example Linguistic Long-Range Dependencies

Task: Coreference resolution

Sentence (ek news article se): "The CEO announced the merger. She said the deal would close in Q3. Her statement emphasized synergies. The board supported her decision."

Dependencies:

"She" (position 10) "CEO" (position 2) ko refer karta hai: distance = 8
"Her" (position 17) "CEO" (position 2) ko refer karta hai: distance = 15
"her" (position 24) "CEO" (position 2) ko refer karta hai: distance = 22 RNN Performance:
Pehla "She": gradient strength $\approx 0.9 5^{8} \approx 0.66$ → 60% baar correctly seekhta hai
"Her": gradient strength $\approx 0.9 5^{15} \approx 0.46$ → 30% baar correctly seekhta hai
Doosra "her": gradient strength $\approx 0.9 5^{22} \approx 0.32$ → 15% baar correctly seekhta hai

Transformer Performance:

Teeno pronouns: "CEO" ko direct attention → 90%+ baar correctly seekhta hai Ye step kyun? Transformer mein, position 24 compute karta hai: $Attention (Q_{24}, K_{,} V_{2}) = softmax (\frac{Q _{24} K _{2}^{T}}{d _{k}}) V_{2}$ Ye ek direct connection hai. Gradient $\frac{\partial L}{\partial V _{2}}$ seedha ek matrix multiply se wapas flow karta hai, 22 recurrent steps se nahi.

Limitation 5: Memory aur Computation Inefficiency

Training Complexity

RNN Training:

Backpropagation (BPTT) ke liye saare hidden states $h_{1}, \dots, h_{T}$ store karne padte hain
Memory: $O (T \cdot d)$ jahan $d$ hidden size hai
Efficiently checkpoint nahi kar sakte kyunki har $h_{t}$ , $h_{t - 1}$ par depend karta hai

Transformer Training:

Koi sequential dependency nahi; activation checkpointing use kar sakte hain
Store karne ki bajaye backward pass ke dauran activations recompute karo
Memory: checkpointing ke saath $O (T \cdot d)$

Transformers Revolutionary Kyun The

Recall Ek 12-saal ke bache ko explain karo

Socho tum ek lambi kahani ek ek word sun ke samajhne ki koshish kar rahe ho, lekin tumhari memory bahut kharab hai. Jab tak tum word 100 sunte ho, tab tak word 1 kya tha ye bhul chuke hote ho. Isi tarah RNNs kaam karte hain - unhe words ek order mein process karne padte hain aur wo bahut pehle ki cheezein bhool jaate hain.

Ab socho tum POORI kahani likhi hui dekh sako, aur jab bhi chaho uska koi bhi hissa dekh sako. Tum ek saath word 1 aur word 100 dekh sako! Tum kai logon ko alag-alag hisse simultaneously padhne bhi de sakte ho. Isi tarah transformers kaam karte hain - wo ek saath saare words "dekh" sakte hain aur kisi bhi word par dhyan de sakte hain, chahe words kitni bhi door hon.

Result? Transformers hain:

Way faster - jaise 100 log alag alag hisse padh rahe hon vs. 1 insaan ek ek word padh raha ho
Better memory - wo words ke beech connections yaad rakh sakte hain chahe wo bahut door hon
Smarter understanding - wo dekh sakte hain ki SAARE words ek doosre se kaise relate karte hain, sirf nearby words se nahi

Transformer Solution: Teen Key Innovations

Self-Attention: Har position directly har doosre position se connect hoti hai $Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V$
Positional Encoding: Kyunki koi sequential processing nahi hai, positions explicitly encode ki jaati hain $P E_{(p os, 2 i)} = sin (p os /1000 0^{2 i / d})$
Parallel Processing: Saare positions simultaneously compute hote hain
- Position 1 apni representation compute karta hai jabki position 1000 apni representation compute karta hai
- Previous timesteps ka koi wait nahi

Connections

Self-Attention Mechanism - Wo core operation jo recurrence ko replace karta hai
Positional Encoding - Transformers recurrence ke bina sequence order kaise track karte hain
Vanishing and Exploding Gradients - Mathematical reason kyun RNNs long sequences par fail hote hain
LSTMs and GRUs - RNN limitations mitigate karne ki pehle ki koshishein
Encoder-Decoder Architecture - Jahan bottleneck problem pehli baar identify ki gayi thi
Backpropagation Through Time - Sequential processing gradient flow ko kyun hurt karta hai
Computational Complexity of Sequence Models - RNN vs Transformer efficiency ki comparison

Flashcards

#flashcards/ai-ml

RNNs mein sequential processing bottleneck kya hai? :: RNNs hidden states iteratively compute karte hain $h_{t} = f (h_{t - 1}, x_{t})$ ke roop mein, ek causal chain banate hain jahan $h_{t}$ tab tak compute nahi ho sakta jab tak $h_{t - 1}$ available na ho. Isse computation $O (T)$ sequential steps ban jaata hai jo parallelize nahi ho sakta.

RNNs time ke across computation parallelize kyun nahi kar sakte?

Kyunki har hidden state

h_{t}

, previous state

h_{t - 1}

ki ek non-linear function hai,

h_{t}

compute karne ka koi tarika nahi hai bina pehle

h_{1}, h_{2}, \dots, h_{t - 1}

sab sequence mein compute kiye.

RNNs ke liye T timesteps mein vanishing gradient formula derive karo :: $\frac{\partial h _{t}}{\partial h _{t - 1}} = diag (tanh^{'} (\cdot)) \cdot W_{hh}$ se shuru karte hue, hamare paas hai $\frac{\partial h _{t}}{\partial h _{t - 1}} \approx σ_{m a x} (W_{hh}) \cdot γ$ jahan $γ < 1$ . $T$ steps mein, gradients $(σ_{m a x} (W_{hh}) \cdot γ)^{T - 1}$ ke roop mein decay karte hain. Agar ye product 1 se kam hai, to gradients exponentially vanish hote hain.

Encoder-decoder RNNs mein compression bottleneck kya hai?

Encoder ko poori input sequence

x_{1}, \dots, x_{T}

ko ek single fixed-size hidden state

h_{T} \in R^{d}

mein compress karna padta hai. Isse ek information bottleneck banta hai: ek

d

-dimensional vector ki limited capacity hoti hai jabki sequence mein

T lo g_{2} V

bits information ho sakti hai.

Transformers RNNs se long sequences par kitna faster ho sakte hain?

Transformers ki

O (1)

parallel time complexity hai RNNs ki

O (T)

sequential time ke comparison mein.

T = 1000

length ki sequences par sufficient parallel hardware ke saath, transformers wall-clock time mein ~1000× faster ho sakte hain.

LSTMs vanishing gradient problem fully kyun solve NAHI karte? :: LSTMs problem ko mitigate karte hain, eliminate nahi karte. Forget gate $f_{t}$ gradient ko abhi bhi multiply karta hai: $\frac{\partial c _{t}}{\partial c _{t - 1}} = f_{t}$ . Agar $f_{t} < 1$ zyaadatar steps par hai, gradients abhi bhi exponentially decay karte hain, bas thoda slow. LSTMs effective range ko ~10 se ~100 steps tak extend karte hain, lekin 1000+ token sequences ke liye problems wapas aati hain.

RNNs ke liye effective context window formula kya hai?

Effective context wo distance

k

ke roop mein define karo jahan gradient 1% tak decay ho jaaye:

γ^{k} = 0.01

, to

k = \frac{l o g ( 0.01 )}{l o g ( γ )}

γ = 0.95

wale vanilla RNN ke liye:

k \approx 90

steps.

γ = 0.99

wale LSTM ke liye:

k \approx 459

steps.

Self-attention long-range dependency problem kaise solve karta hai?

Self-attention saare positions ke beech direct connections create karta hai: position

i

, position

j

ko single matrix operation se attend karta hai, distance

∣ i - j ∣

se independent. Gradient path ki length 1 hai (length

∣ i - j ∣

nahi), distance ke saath exponential decay eliminate karta hai.