Long Short-Term Memory (LSTM) cells

3.5.4 · AI-ML › Sequence Models

The Vanishing Gradient Problem (LSTMs Ki Zaroorat Kyun Hai)

Ek standard RNN mein, hidden state $h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$ hota hai. Backpropagation through time (BPTT) ke dauran, hum compute karte hain:

$\frac{\partial L}{\partial h_{t-k}}=\frac{\partial L}{\partial h_t}{\prod }_{i=1}^k\frac{\partial h_{t-i+1}}{\partial h_{t-i}}$

Yeh step kyun? Chain rule humein har time step par Jacobians multiply karne par majboor karta hai. Har Jacobian $\frac{\partial h_{t-i+1}}{\partial h_{t-i}}$ mein $W_{hh}$ aur $\tanh$ ka derivative hota hai, jo $\le 1$ hai.

Agar $||W_{hh}||<1$ ho, toh product ${\prod }_{i=1}^k{W}_{hh}$ exponentially shrink hota hai jaise $k$ badhta hai (vanishing gradient). Agar $||W_{hh}||>1$ ho, toh yeh explode karta hai. Kisi bhi case mein, long-term dependencies seekhna impossible ho jaata hai.

LSTM ka fix: Cell state $C_t$ time ke through additive updates ke saath flow karta hai, multiplicative nahi. Yeh gradient highways create karta hai jo information preserve karte hain.

LSTM Architecture: The Four Gates Derivation

Ek LSTM mein hota hai:

• Input $x_t$ (current input vector)
• Previous hidden state $h_{t-1}$
• Previous cell state $C_{t-1}$
• Teen gates: forget, input, output
• Outputs: naaya cell state $C_t$ aur hidden state $h_t$

Step 1: Forget Gate (Kya Discard Karna Hai)

$f_t=\sigma (W_f\cdot [h_{t-1},x_t]+b_f)$

Kya hai: Ek vector $[0,1{]}^{n_c}$ mein jahaan $n_c$ cell state dimension hai.

Sigmoid kyun? Hum chahte hain values 0 (completely forget) aur 1 (completely remember) ke beech hon. Sigmoid $\sigma (z)=\frac{1}{1+e^{-z}}$ yahi range output karta hai.

Kaise kaam karta hai: $f_t$ ko $C_{t-1}$ ke saath element-wise multiply kiya jaata hai. Agar $f_t[i]=0$ ho, toh cell state ka dimension $i$ erase ho jaata hai. Agar $f_t[i]=1$ ho, toh preserve hota hai.

Notation: $[h_{t-1},x_t]$ ka matlab hai dono vectors ko concatenate karna: agar $h_{t-1}\in {\mathbb{R}}^{n_h}$ aur $x_t\in {\mathbb{R}}^{n_x}$ ho, toh $[h_{t-1},x_t]\in {\mathbb{R}}^{n_h+n_x}$. Weight matrix $W_f\in {\mathbb{R}}^{n_c\times (n_h+n_x)}$ hoti hai.

Step 2: Input Gate (Kaunsi Nayi Info Store Karni Hai)

$i_t=\sigma (W_i\cdot [h_{t-1},x_t]+b_i)$ ${\tilde{C}}_t=\tanh (W_C\cdot [h_{t-1},x_t]+b_C)$

Kya hai: $i_t$ ek gate vector hai $[0,1{]}^{n_c}$ mein; ${\tilde{C}}_t$ ek candidate cell state hai $[-1,1{]}^{n_c}$ mein.

Do parts kyun? Hum alag karte hain kya add karna hai (${\tilde{C}}_t$) aur kitna add karna hai ($i_t$). Candidate ${\tilde{C}}_t$ $\tanh$ use karke compute hoti hai (outputs $[-1,1]$) kyunki cell states ko positive aur negative dono information represent karne ke liye signed values chahiye.

Yeh step kyun? Gate $i_t$ ke bina, hum hamesha poori candidate add kar dete. Gate network ko seekhne deta hai ki nayi information kab relevant hai (gate open) vs. kab ignore karni hai (gate closed).

Step 3: Cell State Update (Forget + Input Combine Karna)

$C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$

Notation: $\odot$ element-wise (Hadamard) product hai.

Scratch se derivation:

1. Purane cell state $C_{t-1}$ se shuru karo
2. Forget gate se multiply karo: $f_t\odot C_{t-1}$ (relevant memories rakhta hai)
3. Kya add karna hai compute karo: $i_t\odot {\tilde{C}}_t$ (nayi relevant info)
4. Add karo: $C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$

Yeh brilliant kyun hai: Yeh update additive hai, multiplicative nahi! Backpropagation ke dauran:

$\frac{\partial C_t}{\partial C_{t-1}}=f_t$

Gradient $f_t$ ke through flow karta hai bina repeated matrix multiplications ke. Agar $f_t\approx 1$ ho (forget gate open), gradients unchanged flow karte hain. Yahi woh gradient highway hai jo vanishing gradients solve karta hai.

Step 4: Output Gate (Kya Expose Karna Hai)

$o_t=\sigma (W_o\cdot [h_{t-1},x_t]+b_o)$ $h_t=o_t\odot \tanh (C_t)$

Kya hai: $o_t$ control karta hai ki cell state ke kaunse parts hidden state $h_t$ ke roop mein expose hon.

$\tanh (C_t)$ kyun? Cell states time ke saath large ho sakti hain. $\tanh$ stability ke liye $[-1,1]$ par squash karta hai.

Sirf $h_t=C_t$ kyun nahi? Cell state internal memory hai (large, unbounded ho sakti hai). Hidden state woh external representation hai jo hum next layer ko pass karte hain—hum chahte hain yeh controlled aur squashed ho.

Yeh step kyun? Output gate network ko decide karne deta hai: "Mere paas yeh memory hai, lekin kya main ise abhi expose karna chahta hoon?" Jaise, grammatical gender store karna lekin use tabhi output karna jab pronoun generate karna ho.

Complete LSTM Forward Pass

Inputs $x_1,x_2,\dots ,x_T$ diye hue, $h_0=0$, $C_0=0$ initialize karo. $t=1$ se $T$ tak:

Parameter count: Agar $n_x$ = input dim, $n_h$ = hidden dim, $n_c$ = cell dim (usually $n_c=n_h$) ho:

• Har gate mein weight matrix ${\mathbb{R}}^{n_h\times (n_h+n_x)}$ aur bias ${\mathbb{R}}^{n_h}$ hota hai
• 4 sets of parameters (forget, input, candidate, output)
• Total: $4\times [n_h\times (n_h+n_x)+n_h]$ parameters

Vanilla RNN se comparison:

• RNN: gradient har step par $W_{hh}\cdot \text{diag}({\tanh }^{\prime }(h_{t-1}))$ se multiply hota hai → exponential decay
• LSTM: gradient $f_t$ se multiply hota hai (values near 1) → linear flow, koi exponential decay nahi

Cell state ek gradient superhighway ki tarah kaam karta hai—agar forget gates open rahein ($f_t\approx 1$), gradients hundreds of steps peeche minimal degradation ke saath flow karte hain.

LSTMs kyun help karte hain: Word "not" pehle aata hai lekin baad mein "bad" ko affect karta hai. Ek RNN "bad" tak pahunchte pahunchte "not" bhool sakta hai.

LSTM behavior:

• "not" process karte waqt, input gate $i_t$ khulta hai, cell state $C_t$ mein negation store karta hai
• Forget gate $f_t$ "was" ke through high rehta hai (negation mat bhoolo)
• Jab "bad" aata hai, cell state mein abhi bhi "negation" hai, isliye network positive output karta hai

Numerical walkthrough (simplified, 1D):

1. $t=1$ ("not"): $i_1=0.9$, ${\tilde{C}}_1=0.8$ → $C_1=0.72$ (negation store karo)
2. $t=2$ ("bad"): $f_2=0.95$, $C_2=0.95\times 0.72+\dots =0.68$ (negation preserved)
3. Output gate $C_2$ use karke positive sentiment predict karta hai

Yeh step kyun? Additive update $C_2=f_2\odot C_1+\dots$ ka matlab hai $C_2$ directly $C_1$ par depend karta hai minimal information loss ke saath, unlike RNN ka $h_2=\tanh (W{h}_1+\dots )$ jo $h_1$ erase kar sakta hai.

Sequence: $x_1=[,x_2=[,x_3=[,x_4=[x_5=],\dots$

LSTM solution:

• $W_C,W_i$ seekho taaki ${\tilde{C}}_t=+1$ jab $x_t=[$ aur $i_t=1$
• $f_t=1$ hamesha seekho (kabhi mat bhoolo)
• Cell state: $C_1=1,C_2=2,C_3=3,C_4=4,\dots$
• "]" dekhne par, decrement seekho

Yeh kyun kaam karta hai: $C_t=1\cdot C_{t-1}+1\cdot (+1)=C_{t-1}+1$

Cell state literally count kar raha hai! Additive update yeh enable karta hai. Ek vanilla RNN $h_t=\tanh (W{h}_{t-1}+\dots )$ ke saath exact count maintain nahi kar sakta kyunki $\tanh$ values ko $[-1,1]$ par bound kar deta hai.

Kyun sahi lagta hai: "Forget gate" naam ek switch jaisa lagta hai.

Fix: Gates continuous multipliers hain. $f_t=0.6$ ka matlab hai $C_{t-1}$ ka har dimension 0.6 se scale ho jaata hai—yeh partial forgetting hai, random erasure nahi. Network smooth interpolations seekhta hai, hard decisions nahi. Yahi continuity hai isliye LSTMs ko gradient descent se train kiya ja sakta hai.

Steel-man: Yeh confusion isliye hoti hai kyunki hum interpretability chahte hain, aur binary decisions reason karne mein asaan hain. Lekin soft gates network ko nuanced timing seekhne ki flexibility dete hain.

Kyun sahi lagta hai: Dono time ke through information carry karte hain.

Fix:

• $C_t$ unfiltered long-term memory hai (large, unrestricted ho sakti hai)
• $h_t=o_t\odot \tanh (C_t)$ filtered output hai (bounded, task-relevant)

Example: "John went to the store. He bought milk" padhte waqt, cell state store kar sakti hai [John, male, past-tense, store-context]. Lekin "He" ke baad next word generate karte waqt, hidden state sirf [male] expose karta hai (output gate ke control mein) kyunki pronoun consistency ke liye wahi chahiye.

Yeh design kyun? Storage ($C_t$) ko exposure ($h_t$) se alag karna network ko rich internal state rakhne deta hai bina har detail ko output mein force kiye.

Kyun sahi lagta hai: Humne kaha tha ki LSTMs vanishing gradients solve karte hain aur long-term dependencies preserve karte hain.

Fix: LSTMs long dependencies seekh sakte hain agar data support kare, lekin yeh perfectly memorize nahi karte. Forget gate $f_t$ learned hota hai—agar training data ko long-term memory ki zaroorat nahi, network bhoolna seekh leta hai. Saath hi, extremely long sequences (100k+ steps) ke saath cell states abhi bhi saturate ya gradient issues suffer kar sakti hain.

Practical limit: LSTMs ~100-200 steps ki dependencies achhi tarah handle karte hain. Isse zyada ke liye, attention mechanisms ya Transformers use karo.

Backpropagation Through Time in LSTMs

Key advantage: gradients cell state path ke through minimal degradation ke saath flow karte hain.

Gradient flow equation: $\frac{\partial L}{\partial C_{t-k}}=\frac{\partial L}{\partial C_t}{\prod }_{i=t-k+1}^t{f}_i$

Yeh kyun matter karta hai: Agar forget gates average $f_i\approx 0.9$ hon, 100 steps ke baad: ${\prod }_{i=1}^{100}0.9=0.9^{100}\approx 2.7\times 10^{-5}$

Abhi bhi small hai, lekin vanilla RNN se bahut better hai jahaan: ${\prod }_{i=1}^{100}||W_{hh}||\cdot |{\tanh }^{\prime }||<0.25^{100}\approx 10^{-60}$

Yeh step kyun? Forget gate values learned hain, fixed nahi. Network important memories ke liye $f_t$ ko 1 ke paas rakh sakta hai, jisse practice mein $\prod f_i$ bahut larger ho jaata hai.

Ek LSTM ki tarah ek notebook with smart sticky notes rakhna:

• Cell state ($C_t$) tumhari notebook hai jahaan tum important facts likhte ho
• Forget gate ek smart eraser hai—woh sirf woh cheezein erase karta hai jo tumhe ab nahi chahiye
• Input gate ek filter hai—yeh decide karta hai ki nayi info likhne ke layak hai ya nahi
• Output gate ek privacy filter hai—tumhare paas memory hai, lekin tum sirf wahi share karte ho jo abhi relevant hai

Toh agar story hai "The wizard, who wore a purple hat, cast a spell", LSTM:

1. Notebook mein "wizard, purple hat" likhta hai (input gate ise andar aane deta hai)
2. "Wizard" poore time rakhta hai (forget gate kehta hai "erase mat karo")
3. Jab story poochhe "Who cast the spell?", output gate notebook se "wizard" dikhata hai

Notebook (cell state) ka time ke through ek seedha raasta hai—info bina corrupt hue peeche flow kar sakti hai. Isliye LSTMs regular RNNs se behtar yaad rakhte hain!

Order: FIO-C-O, jahaan C (cell state) F aur I ko combine karne ka result hai, aur O sabse last aata hai.

Connections

• 3.5.01-Recurrent-Neural-Networks-(RNN)-basics: LSTMs, RNNs ko gated memory ke saath extend karte hain
• 3.5.02-Backpropagation-Through-Time-(BPTT): Vanilla BPTT kyun fail hota hai, LSTMs ise fix karte hain
• 3.5.03-Vanishing-and-Exploding-Gradientsin-RNNs: Woh core problem jo LSTMs solve karte hain
• 3.5.05-Gated-Recurrent-Units-(GRU): 3 ki jagah 2 gates wala simplified alternative
• 3.5.06-Bidirectional-RNNs: LSTMs ko bidirectional banaya ja sakta hai
• 3.6.01-Attention-Mechanisms: Modern alternative jo aur bhi lambi dependencies handle karta hai
• 4.2.01-Transformers: Bahut saare NLP tasks mein LSTMs ko replace kar diya

#flashcards/ai-ml

Vanilla RNNs kyun nahi kar paate woh problem jo LSTMs solve karte hain? :: Long sequences mein vanishing gradients—RNNs mein gradients exponentially shrink hote hain lekin LSTM cell states ke through additively flow karte hain, long-term dependencies preserve karte hue.

Concept Map