LSTM — gates, cell state

WHAT is an LSTM?

WHY do we need it? In aerospace we predict things over long horizons — e.g. estimating fatigue accumulation in a wing spar from a long vibration history, or predicting remaining useful life (RUL) of a turbine blade. The relevant event (a micro-crack initiation) may have happened thousands of timesteps ago. A plain RNN forgets it; an LSTM can hold it in $c_t$.

HOW it works — the four computations

At each timestep we concatenate the previous output and current input: $[h_{t-1}, x_t]$. All gates are affine maps of this vector passed through a squashing function.

Why sigmoid for gates? $\sigma\in(0,1)$ acts like a valve opening fraction: $0$ = fully closed (block), $1$ = fully open (pass). Why tanh for candidate? $\tanh\in(-1,1)$ lets the written value be positive or negative, centered at zero — good for signed features.

DERIVATION — why the cell update kills vanishing gradients

Let's derive from scratch why $c_t = f_t\odot c_{t-1} + \dots$ helps gradients survive.

Step 1 — the RNN problem. In a plain RNN, $h_t = \tanh(W h_{t-1}+\dots)$. Backprop through $k$ steps multiplies $k$ Jacobians: $\frac{\partial h_t}{\partial h_{t-k}} = \prod_{j=1}^{k} \operatorname{diag}(\tanh')\,W.$ Why this step? Chain rule across timesteps. Each factor has $|\tanh'|\le 1$ and $\|W\|$ often $<1$, so the product $\to 0$ exponentially — gradients vanish.

Step 2 — the LSTM cell path. Take the derivative of the cell recurrence along the $c$ line only: $\frac{\partial c_t}{\partial c_{t-1}} = f_t.$ Why this step? In $c_t = f_t\odot c_{t-1} + i_t\odot\tilde c_t$, treating other terms as inputs, the direct partial is just $f_t$ — no weight matrix, no tanh', just an elementwise gate.

Step 3 — chain along the cell path. $\frac{\partial c_t}{\partial c_{t-k}} = \prod_{j=t-k+1}^{t} f_j.$ Why this matters: if the forget gate stays near $1$ (network chooses to remember), the product stays near $1$ — the gradient flows back undiminished. The LSTM can learn to keep the highway open. That's the constant error carousel.

Worked Example 1 — one scalar timestep

Take scalar states. Given $c_{t-1}=0.5$, and pre-activations chosen so that $f_t=0.9,\; i_t=0.8,\; o_t=0.6,\; \tilde c_t=0.4$.

• Cell: $c_t = f_t c_{t-1} + i_t \tilde c_t = 0.9(0.5)+0.8(0.4)=0.45+0.32=0.77$. Why? Keep 90% of old memory, add 80% of the new candidate.
• Output: $h_t = o_t\tanh(c_t)=0.6\tanh(0.77)=0.6(0.6469)=0.388$. Why? Read out 60% of the (squashed) cell content.

Worked Example 2 — "remember forever"

Suppose the task needs to store a value seen at $t=0$ for 1000 steps. If the network sets $f_t=1,\; i_t=0$ for all $t>0$, then $c_t = 1\cdot c_{t-1} + 0 = c_{t-1} \Rightarrow c_{1000}=c_0.$ Why this step? With forget gate wide open and input gate shut, the memory is copied perfectly — a plain RNN literally cannot do this without exploding/vanishing.

Worked Example 3 — aerospace framing

Vibration sensor stream $x_t$ feeds an LSTM predicting bearing RUL. A rare high-amplitude spike at $t=200$ should influence the prediction at $t=5000$. Training pushes $i_{200}\approx 1$ (write the spike into $c$), then $f_t\approx 1$ afterwards (hold it), and $o_{5000}\approx 1$ (read it out when forecasting). Why? The gates are learned data-dependent switches — gradient descent discovers this routing automatically.

Common Mistakes

Flashcards

Connections

• Recurrent Neural Networks (RNN) — the base LSTM improves on.
• Vanishing & Exploding Gradients — the problem the cell state solves.
• GRU — Gated Recurrent Unit — cheaper cousin (merges gates).
• Backpropagation Through Time (BPTT) — how these gradients are computed.
• Sigmoid and Tanh Activations — the squashing functions used.
• Remaining Useful Life (RUL) Prediction — aerospace application.
• Time Series Forecasting — general use case.

Concept Map

Hinglish (regional understanding)