LSTM — gates, cell state

5.6.13 · Coding › Machine Learning (Aerospace Applications)

WHAT is an LSTM?

WHY do we need it? Aerospace mein hum cheezein lambe horizons par predict karte hain — jaise ek wing spar mein lambi vibration history se fatigue accumulation estimate karna, ya ek turbine blade ki remaining useful life (RUL) predict karna. Relevant event (ek micro-crack initiation) hazaaron timesteps pehle hua hoga. Ek plain RNN use bhool jaata hai; ek LSTM use $c_t$ mein hold kar sakta hai.

HOW it works — the four computations

Har timestep par hum previous output aur current input ko concatenate karte hain: $[h_{t-1},x_t]$. Sare gates is vector ke affine maps hain jo ek squashing function se guzarti hain.

Gates ke liye sigmoid kyun? $\sigma \in (0,1)$ ek valve opening fraction ki tarah kaam karta hai: $0$ = puri tarah band (block), $1$ = puri tarah khula (pass). Candidate ke liye tanh kyun? $\tanh \in (-1,1)$ likhne wali value ko positive ya negative hone deta hai, zero par centered — signed features ke liye acha hai.

DERIVATION — why the cell update kills vanishing gradients

Chalo scratch se derive karte hain ki $c_t=f_t\odot c_{t-1}+\dots$ gradients ko survive karne mein kyun help karta hai.

Step 1 — the RNN problem. Ek plain RNN mein, $h_t=\tanh (W{h}_{t-1}+\dots )$. $k$ steps mein backprop $k$ Jacobians ko multiply karta hai: $\frac{\partial h_t}{\partial h_{t-k}}={\prod }_{j=1}^k\mathrm{diag}({\tanh }^{\prime })W.$ Yeh step kyun? Timesteps mein chain rule. Har factor mein $|{\tanh }^{\prime }|\le 1$ hai aur $\Vert W\Vert$ aksar $<1$ hoti hai, toh product $\to 0$ exponentially — gradients vanish ho jaate hain.

Step 2 — the LSTM cell path. Sirf $c$ line ke saath cell recurrence ka derivative lo: $\frac{\partial c_t}{\partial c_{t-1}}=f_t.$ Yeh step kyun? $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$ mein, baaki terms ko inputs maan kar, direct partial sirf $f_t$ hai — koi weight matrix nahi, koi tanh' nahi, sirf ek elementwise gate.

Step 3 — chain along the cell path. $\frac{\partial c_t}{\partial c_{t-k}}={\prod }_{j=t-k+1}^t{f}_j.$ Yeh kyun matter karta hai: agar forget gate $1$ ke paas rehta hai (network choose karta hai yaad rakhna), toh product $1$ ke paas rehta hai — gradient bina kisi kami ke peeche flow karta hai. LSTM seekh sakta hai highway ko khula rakhna. Yahi constant error carousel hai.

Worked Example 1 — one scalar timestep

Scalar states lo. Diya gaya $c_{t-1}=0.5$, aur pre-activations is tarah chosen ki $f_t=0.9,i_t=0.8,o_t=0.6,{\tilde{c}}_t=0.4$.

• Cell: $c_t=f_t{c}_{t-1}+i_t{\tilde{c}}_t=0.9(0.5)+0.8(0.4)=0.45+0.32=0.77$. Kyun? Purani memory ka 90% rakho, naye candidate ka 80% add karo.
• Output: $h_t=o_t\tanh (c_t)=0.6\tanh (0.77)=0.6(0.6469)=0.388$. Kyun? (Squashed) cell content ka 60% read out karo.

Worked Example 2 — "remember forever"

Maano task ko $t=0$ par dekhi gayi ek value 1000 steps tak store karni hai. Agar network $f_t=1,i_t=0$ set kare saare $t>0$ ke liye, toh $c_t=1\cdot c_{t-1}+0=c_{t-1}\Rightarrow c_{1000}=c_0.$ Yeh step kyun? Forget gate poori tarah khula aur input gate band ke saath, memory perfectly copy hoti hai — ek plain RNN literally yeh bina exploding/vanishing ke nahi kar sakta.

Worked Example 3 — aerospace framing

Vibration sensor stream $x_t$ ek LSTM ko feed karta hai jo bearing RUL predict karta hai. $t=200$ par ek rare high-amplitude spike ko $t=5000$ par prediction par influence karna chahiye. Training $i_{200}\approx 1$ push karta hai (spike ko $c$ mein write karo), phir $f_t\approx 1$ baad mein (use hold karo), aur $o_{5000}\approx 1$ (forecasting karte waqt read out karo). Kyun? Gates learned data-dependent switches hain — gradient descent is routing ko automatically discover karta hai.

Common Mistakes

Flashcards

Connections

• Recurrent Neural Networks (RNN) — woh base jise LSTM improve karta hai.
• Vanishing & Exploding Gradients — woh problem jo cell state solve karta hai.
• GRU — Gated Recurrent Unit — sasta cousin (gates merge karta hai).
• Backpropagation Through Time (BPTT) — yeh gradients kaise compute hote hain.
• Sigmoid and Tanh Activations — use hone wale squashing functions.
• Remaining Useful Life (RUL) Prediction — aerospace application.
• Time Series Forecasting — general use case.

Concept Map