Visual walkthrough — LSTM — gates, cell state

This page is the Hinglish version parent's visual companion. If a word feels unfamiliar, it links back to Recurrent Neural Networks (RNN), Vanishing & Exploding Gradients, Sigmoid and Tanh Activations, or Backpropagation Through Time (BPTT).

---

Step 1 — What "memory across time" even means

WHAT. Picture a row of boxes, one per instant in time: $t=0,1,2,\dots$. Each box holds a number that is the network's memory right then. An arrow carries memory from one box to the next.

WHY start here. Before we can ask "why do things get forgotten," we need a picture of memory being passed along a chain. That chain is the heart of every recurrent network.

PICTURE. Look at the figure: the boxes are the timesteps, the orange arrows are the "pass memory forward" links. The value inside each box is written $m_t$ — just "the memory at time $t$." Nothing more yet: $m$ for memory, subscript $t$ for which instant.

---

Step 2 — The plain-RNN way: overwrite every step

WHAT. A vanilla RNN builds the new memory by squashing a mix of the old memory and the new input. In symbols, with $x_t$ the incoming data at time $t$:

$$m_t=\tanh (\underset{\text{one weight}}{\underbrace{w}}\cdot \underset{\text{old memory}}{\underbrace{m_{t-1}}}+\underset{\text{new input}}{\underbrace{u{x}_t}}).$$

Let us earn each piece:

• $w$ — a single fixed number the net multiplies the old memory by (a "gain").
• $m_{t-1}$ — the memory one tick ago.
• $u{x}_t$ — the fresh input scaled by another number $u$.
• $\tanh$ — the squashing curve from Sigmoid and Tanh Activations. It takes any real number and gently bends it into the band $(-1,1)$.

WHY $\tanh$ and not just leave the sum alone? Without a squash, the running sum could blow up to huge numbers. $\tanh$ keeps every memory value polite — inside $(-1,1)$ — so the network stays numerically stable. That is the only reason it is there.

PICTURE. The figure shows the $\tanh$ curve. Notice two things we will need: (1) it flattens out at the edges, and (2) its slope (how steep it is) is largest in the middle and nearly $0$ at the edges. Slope will be the villain of Step 3.

---

Step 3 — Why the plain RNN forgets: a product that shrinks to zero

WHAT. To learn, the network asks: "if I nudge the old memory $m_{t-k}$, how much does today's memory $m_t$ change?" That sensitivity is written $\frac{\partial m_t}{\partial m_{t-k}}$ — read it as "the change in $m_t$ per unit change in $m_{t-k}$." The symbol $\partial$ is just "a tiny change in," borrowed from Backpropagation Through Time (BPTT).

Because memory passes through $k$ boxes to get from $t-k$ to $t$, the chain rule multiplies the per-step sensitivities together:

$$\frac{\partial m_t}{\partial m_{t-k}}=\prod _{j=1}^k\underset{\text{slope, }\le 1}{\underbrace{{\tanh }^{\prime }(\cdot )}}\cdot \underset{\text{gain}}{\underbrace{w}}.$$

Here $\prod$ means "multiply all these together," and ${\tanh }^{\prime }$ is the slope of the $\tanh$ curve we flagged in Step 2 — a number that is at most $1$ and usually well below it.

WHY this is fatal. Multiply $k$ numbers each smaller than $1$ and the result races to $0$. If ${\tanh }^{\prime }\cdot w\approx 0.5$, then after $20$ steps you have $0.5^{20}\approx 10^{-6}$. The signal from $20$ ticks ago is essentially gone — the vanishing gradient of Vanishing & Exploding Gradients.

PICTURE. The figure plots $0.5^k$ against $k$. Watch the plum curve nose-dive: by $k=15$ the memory link is invisible. This is why a plain RNN cannot remember a spike from $5000$ steps ago.

Recall

Why does a plain RNN forget long ago events? ::: Its time-to-time sensitivity is a product of many factors each $\le 1$ (slopes of $\tanh$ times the weight), which shrinks toward $0$ exponentially.

---

Step 4 — The fix: a second, ADDITIVE memory line

WHAT. Instead of overwriting memory through a squash, the LSTM keeps a separate line $c_t$ — the cell state — and updates it by adding, not rewriting:

$$c_t=\underset{\text{keep-dial}}{\underbrace{f_t}}\odot c_{t-1}+\underset{\text{new stuff written in}}{\underbrace{i_t\odot {\tilde{c}}_t}}.$$

New symbols, each earned:

• $c_t$ — the cell state, a memory value carried straight along a "conveyor belt."
• $f_t$ — the forget gate, a dial between $0$ and $1$: how much of the old memory to keep.
• $i_t,{\tilde{c}}_t$ — the input gate and candidate; together they are "how much new stuff to add." We build them properly in Step 6. For now treat their product as a single quantity: "the write."
• $\odot$ — elementwise multiply (multiply matching slots). For scalars it is ordinary multiplication.

WHY additive? Because the dangerous operation in Step 3 was multiplying through a squash. Addition has no squash and no weight in the memory path. The old memory can simply ride forward and have a little added to it — like dropping a note on a moving conveyor belt instead of rewriting a sticky note.

PICTURE. The figure draws the conveyor belt: $c_{t-1}$ enters on the left, the forget-dial $f_t$ scales it, the "write" is added on top, and $c_t$ leaves on the right — straight line, no bend.

---

Step 5 — The derivation payoff: the sensitivity is just $f_t$

WHAT. Redo the Step-3 question, but on the cell line. Ask: "how much does $c_t$ change if I nudge $c_{t-1}$?" Differentiate $c_t=f_t\odot c_{t-1}+(\text{write})$, treating the write as an input:

$$\frac{\partial c_t}{\partial c_{t-1}}=f_t.$$

That is the whole answer. No weight matrix. No ${\tanh }^{\prime }$. Just the dial.

WHY this changes everything. Chain it across $k$ steps exactly as before:

$$\frac{\partial c_t}{\partial c_{t-k}}=\prod _{j=t-k+1}^t{f}_j.$$

Now the multiplied factors are the forget gates themselves. If the network chooses $f_j\approx 1$ (keep everything), the product stays near $1$ — the gradient flows back undiminished across thousands of steps. This is the constant error carousel.

PICTURE. The figure compares two products side by side: the plum RNN curve $0.5^k$ (dies) versus the teal LSTM curve $1.0^k$ (flat at $1$). Same horizontal axis, radically different fate.

---

Step 6 — Where the dials come from (building the gates)

WHAT. A gate is not a hand-set number — the network computes it from what it currently knows. Concatenate the previous readout $h_{t-1}$ and the new input $x_t$ into one stacked vector $z_t=[h_{t-1},x_t]$, then run an affine map through a squash:

$$f_t=\sigma (W_f{z}_t+b_f),i_t=\sigma (W_i{z}_t+b_i),o_t=\sigma (W_o{z}_t+b_o).$$

• $\sigma$ — the sigmoid from Sigmoid and Tanh Activations: $\sigma (x)=\frac{1}{1+e^{-x}}$, squeezing any number into $(0,1)$.
• $W_f,b_f$ — a weight matrix and bias; "$Wz+b$" is just "scale and shift the inputs."
• $o_t$ — the output gate (used in Step 7).

The candidate uses $\tanh$ instead:

$${\tilde{c}}_t=\tanh (W_c{z}_t+b_c)\in (-1,1).$$

WHY sigmoid for gates, tanh for the candidate? A gate is a valve opening: $0$ = shut, $1$ = wide open — so it must live in $(0,1)$, and sigmoid is the curve that maps everything there. A candidate is a value to store, which might be positive or negative, so it uses $\tanh$'s signed band $(-1,1)$.

PICTURE. The figure overlays sigmoid (teal, penned into $(0,1)$ — a valve) and tanh (orange, into $(-1,1)$ — a signed value), with the $0$ and $1$ guide lines marked.

---

Step 7 — Reading the memory out (the output gate)

WHAT. The cell state $c_t$ can grow large (it is a running additive sum). We do not feed it out raw. Instead:

$$h_t=\underset{\text{read-dial}}{\underbrace{o_t}}\odot \underset{\text{squashed memory}}{\underbrace{\tanh (c_t)}}.$$

• $h_t$ — the hidden state, the vector that actually leaves the cell and becomes the prediction (and part of next step's $z_{t+1}$).
• $\tanh (c_t)$ — squashes the possibly-huge cell value back into $(-1,1)$ so downstream layers stay stable.
• $o_t$ — the output gate deciding how much of that to reveal now.

WHY two different memories? $c_t$ is the protected archive (updated additively, never squashed on the way through). $h_t$ is the filtered readout — what you choose to say out loud this instant. Only $h_t$ leaves the cell.

PICTURE. The figure shows the cell belt on top and the readout branch dropping down: $c_t\to \tanh \to$ times $o_t\to h_t$.

---

Step 8 — Edge cases: check every dial setting

We must make sure no reader hits a corner we skipped. Each row of the figure is one extreme.

$f_t$	$i_t$	Behaviour	Meaning
$1$	$0$	$c_t=c_{t-1}$	Copy forever. Perfect memory (Worked Example 2).
$0$	$1$	$c_t={\tilde{c}}_t$	Full reset. Behaves memoryless — the "safe default" trap.
$0$	$0$	$c_t=0$	Wipe. Memory erased, nothing written.
$1$	$1$	$c_t=c_{t-1}+{\tilde{c}}_t$	Accumulate. Can grow unbounded → why $h_t$ needs $\tanh$.

WHY show all four. The gates are learned data-dependent switches; training can land on any of these. The dangerous one is $f_t=0$: it silently turns your LSTM back into a forgetful RNN. That is why LSTMs use a positive forget-gate bias ($b_f\approx 1$ to $2$), so early on $f_t\approx \sigma (1-2)\approx 0.73$–$0.88$ — remember by default.

PICTURE. The figure draws the four conveyor belts side by side, one per row of the table, so you see the memory copied, reset, wiped, or piled up.

---

The one-picture summary

Everything above collapses into a single diagram: the cell belt runs straight across (additive, sensitivity $=f_t$), three sigmoid dials tap into it (forget scales the old, input+candidate add the new, output reads it out), and the hidden state drops out the bottom. The straightness of the belt is the reason gradients survive.

Recall Feynman: the whole walkthrough in plain words

A normal memory robot rewrites its one note every second by squashing it — and squashing quietly shaves off a little of the learning signal each time, so after a few dozen seconds the signal is gone. The LSTM's trick is a second note on a conveyor belt. It never rewrites this note by squashing; it just scales the old note by a "keep dial" and adds a little new stuff. Because the belt is straight (scale-and-add, no squash, no hidden weight), asking "how much did the old note affect today's note?" gives back just the keep-dials multiplied together — and if the robot keeps those dials open, the answer stays big even after thousands of seconds. Three small dials run the show: forget (how much old to keep), input (how much new to add), output (how much to read aloud). The read-aloud value gets one gentle squash so it never explodes. That straight belt is the entire reason LSTMs remember what plain RNNs forget.

---

Connections

• Recurrent Neural Networks (RNN) — the forgetful baseline of Steps 1–3.
• Vanishing & Exploding Gradients — the disease cured in Step 5.
• Sigmoid and Tanh Activations — the two curves in Step 6.
• Backpropagation Through Time (BPTT) — where the $\partial c_t/\partial c_{t-k}$ product lives.
• GRU — Gated Recurrent Unit — the same belt idea with fewer dials.
• Remaining Useful Life (RUL) Prediction — the spike-at-200, read-at-5000 story.
• Time Series Forecasting — the general home of all this.