Visual walkthrough — Long Short-Term Memory (LSTM) cells

This is the visual companion to Long Short-Term Memory (LSTM) cells. It exists because that note showed the cure for vanishing gradients; here we see the cure.

---

Step 0 — The problem we are escaping

Before we build anything, look at what goes wrong without an LSTM.

A plain recurrent network keeps one memory vector, the hidden state $h_t$ (a list of numbers describing "what I remember at step $t$"). At each new step it is completely rewritten:

$h_t=\tanh (\underset{\text{grid on old memory}}{\underbrace{W_{hh}}}\underset{\text{old memory}}{\underbrace{h_{t-1}}}+\underset{\text{grid on new input}}{\underbrace{W_{xh}}}\underset{\text{current input}}{\underbrace{x_t}}+\underset{\text{learned offset}}{\underbrace{b_h}})$

Reading it term by term: $W_{hh}$ is a fixed grid of learned numbers applied to the old memory $h_{t-1}$; $W_{xh}$ is a second grid applied to the current input $x_t$; $b_h$ is a learned offset that shifts the total; and $\tanh$ (a function that gently bends every number into the range $-1$ to $+1$) squashes the result. The point to notice is the first term: the old memory is passed through the same grid $W_{hh}$ every step. Multiplying by the same grid over and over is like multiplying a number by $0.7$ again and again: $0.7,0.49,0.34,\cdots \to 0$. The early memory fades geometrically.

Figure s01 — A curve of $0.7^k$ against the number of steps $k$ we look back. The red dot marks the strong early memory; by twelve steps the surviving signal is essentially zero. This is the geometric fading a plain RNN cannot avoid.

Everything below is us earning that sentence.

---

Step 1 — The conveyor belt: the cell state $C_t$

WHAT. We introduce a second memory vector called the cell state $C_t$, a list of $n_c=n_h$ numbers (say 128 of them) that runs straight along the top of the cell. Read the subscript $t$ as "at time step $t$".

WHY. We want a channel where information can travel from step to step untouched unless we deliberately touch it. Think of a horizontal conveyor belt: a box placed on it at step 1 rides to step 100 without anyone lifting it off.

PICTURE. The red line is the belt. Notice it goes straight across — no grid $W$, no $\tanh$ sitting on the main road.

Figure s02 — The cell state drawn as a red conveyor belt running straight from $C_{t-1}$ on the left to $C_t$ on the right. Boxes of memory sit on it and ride across untouched: no weight grid and no $\tanh$ live on this main road, which is exactly why memory can survive many steps.

We will build the machinery that reads and writes the belt out of small, understandable parts. The first part is a valve.

---

Step 2 — A valve: the sigmoid gate

WHAT. A gate is a vector of numbers each between $0$ and $1$. We get those numbers from the sigmoid function, written $\sigma$:

$\sigma (z)=\frac{1}{1+e^{-z}}$

Here $z$ is any real number and $e\approx 2.718$. Let's read the formula: when $z$ is very negative, $e^{-z}$ is huge, so the fraction is tiny → output near $0$. When $z$ is very positive, $e^{-z}$ is near $0$, so the fraction is near $1$. At $z=0$ we get exactly $\frac{1}{2}$.

WHY sigmoid and not something else? We need a dial that lands smoothly anywhere in $[0,1]$ — $0$ meaning "block this", $1$ meaning "let it all through", and everything between meaning "let a fraction through". A hard on/off switch (step function) has no slope, so gradient descent could never nudge it. Sigmoid is the smooth switch.

PICTURE. The S-curve. The red dot at $z=0$ shows the midpoint; the flat tails are the "fully closed" and "fully open" ends.

Figure s03 — The sigmoid curve $\sigma (z)$ rising smoothly from $0$ (far left, "valve closed") to $1$ (far right, "valve open"). The red dot at $z=0$ marks the exact midpoint value $0.5$. Every gate in an LSTM is one of these smooth valves.

The dial takes its instructions from the current situation. The "situation" is the previous worker's report $h_{t-1}$ glued to the new input $x_t$.

---

Step 3 — The forget gate: how much of the old belt to keep

WHAT. First we glue two vectors together. $[h_{t-1},x_t]$ means concatenation: stack the old hidden state (length $n_h$) on top of the input (length $n_x$) to get one longer list of length $n_h+n_x$. Then:

$$\underset{\text{keep-dial, }\mathrm{0..1}}{\underbrace{f_t}}=\sigma (\underset{\text{learned grid}}{\underbrace{W_f}}\underset{\text{old report + new input}}{\underbrace{[h_{t-1},x_t]}}+\underset{\text{learned offset}}{\underbrace{b_f}})$$

Term by term: $W_f$ is a grid of learned weights (shape $n_h\times (n_h+n_x)$) that turns the situation into a raw score; $b_f$ is a learned offset (length $n_h$) that shifts that score up or down; $\sigma$ crushes it into a dial between $0$ and $1$. The result $f_t$ has one dial per belt lane.

WHY. Some memories should be dropped when a new fact contradicts them (e.g. the sentence's subject switches from singular to plural). The forget gate decides, per lane, how much of the old belt survives.

PICTURE. The belt reaching a row of red valves; $f_t=1$ valves stay wide open (memory flows on), $f_t=0$ valves pinch shut (memory erased).

Figure s04 — The memory belt (black) meeting a row of red forget valves. Each valve's shade shows its opening $f_t$: fully open at $f_t=1$ (memory flows on), nearly shut at $f_t=0.2$, closed at $f_t=0$ (that lane erased). The kept memory is the element-wise product $f_t\odot C_{t-1}$.

Element-wise, the belt after forgetting is $f_t\odot C_{t-1}$, where $\odot$ means "multiply matching entries" ($[a,b]\odot [c,d]=[ac,bd]$).

---

Step 4 — The input gate: what new thing to write, and how much

WHAT. Two pieces work together. First a candidate — a proposal of new content:

$${\tilde{C}}_t=\tanh (W_C[h_{t-1},x_t]+b_C)$$

We use $\tanh$ (range $-1$ to $+1$) here, not sigmoid, because content can be negative — memory should be able to say "the sentiment is negative" as a $-0.8$, not just "present/absent". Then a dial deciding how much of that proposal to actually write:

$$i_t=\sigma (W_i[h_{t-1},x_t]+b_i)$$

WHY two separate pieces? We split what (${\tilde{C}}_t$, signed content) from how much ($i_t$, a $\mathrm{0..1}$ dial). This lets the network form a strong candidate yet write only a whisper of it, or vice-versa.

PICTURE. Below the belt, ${\tilde{C}}_t$ (black arrow, a signed proposal) passes through the red $i_t$ valve before being allowed onto the belt.

Figure s05 — A signed candidate ${\tilde{C}}_t=\tanh (\cdot )$ rises as a black arrow from below and must pass the red input valve $i_t$ before joining the belt. The amount that actually gets written is the element-wise product $i_t\odot {\tilde{C}}_t$.

The amount actually written is $i_t\odot {\tilde{C}}_t$.

---

Step 5 — The additive update: the whole point

WHAT. Combine "what we kept" and "what we wrote":

$$C_t=\underset{\text{kept old memory}}{\underbrace{f_t\odot C_{t-1}}}+\underset{\text{new memory}}{\underbrace{i_t\odot {\tilde{C}}_t}}$$

WHY this is the hero equation. Look at how $C_t$ depends on $C_{t-1}$: it is a plus, not a chain of matrix multiplies. Ask "if I nudge $C_{t-1}$, how much does $C_t$ move?" — that sensitivity is:

$$\frac{\partial C_t}{\partial C_{t-1}}=f_t$$

Nothing else. During backpropagation the gradient travelling from step $t$ to step $t-1$ is simply multiplied by $f_t$. If the network keeps $f_t\approx 1$, the gradient passes through unchanged — across 1 step or across 500. That is the gradient highway that plain RNNs lack (there the gradient is multiplied by $W\cdot \text{diag}({\tanh }^{\prime })$ every step and decays geometrically, exactly the fading from Step 0).

PICTURE. Two streams merging with a + node onto the belt, and the tiny red label $f_t$ on the arrow going backward in time — the only thing the gradient meets.

Figure s06 — Kept memory $f_t\odot C_{t-1}$ (red belt from the left) meets new memory $i_t\odot {\tilde{C}}_t$ (rising arrow) at a + node, producing $C_t$. The backward arrow underneath carries the label "gradient backward $\times f_t$ only": travelling from $C_t$ to $C_{t-1}$ the learning signal is scaled by nothing but $f_t$.

---

Step 6 — The output gate: what to reveal

WHAT. The belt may hold big numbers; the outside world wants a tidy, bounded report. So we squash and filter:

$$o_t=\sigma (W_o[h_{t-1},x_t]+b_o),h_t=\underset{\text{reveal-dial}}{\underbrace{o_t}}\odot \underset{\text{squashed belt}}{\underbrace{\tanh (C_t)}}$$

Term by term: $W_o$ is a learned grid (shape $n_h\times (n_h+n_x)$); $b_o$ is a learned offset (length $n_h$) that shifts the raw score before the sigmoid, exactly like $b_f,b_i,b_C$ in the earlier gates; $\sigma$ turns the result into the reveal-dial $o_t$. Then $\tanh (C_t)$ compresses the belt into $[-1,1]$ for stability, and $o_t$ chooses per lane which parts of that compressed memory to expose as $h_t$.

WHY separate memory from output? The network might store a fact ("the subject is female") for many steps but only reveal it at the moment a pronoun is generated. The belt keeps it; the output gate decides when to speak it.

PICTURE. The belt tapped from above: $\tanh$ box then a red $o_t$ valve producing the worker $h_t$, which then loops down to become next step's report.

Figure s07 — The belt $C_t$ is tapped from above: it first passes through a $\tanh$ box (squashing to $[-1,1]$), then through the red output valve $o_t$, giving $h_t=o_t\odot \tanh (C_t)$. This $h_t$ is the tidy summary handed outside and looped back as the next step's report.

---

Step 7 — Edge and degenerate cases

Never leave the reader guessing what happens at the extremes. Each dial is in $[0,1]$, so the corners are:

Situation	Setting	What the belt does
Pure memory	$f_t=1,i_t=0$	$C_t=C_{t-1}$ — belt frozen, perfect recall (the counting task)
Full reset	$f_t=0,i_t=1$	$C_t={\tilde{C}}_t$ — old memory wiped, replaced by candidate (behaves like a plain RNN cell)
Wipe to zero	$f_t=0,i_t=0$	$C_t=0$ — forget everything AND write nothing; the belt is blanked, a fresh start
Ignore step	$f_t=1,i_t=0,o_t=0$	belt kept but nothing revealed: $h_t=0$, silent memory
First step	$C_0=0,h_0=0$	belt starts empty; $C_1=i_1\odot {\tilde{C}}_1$

Counting, seen on the belt. With $f_t=1$ and $i_t\odot {\tilde{C}}_t=+1$ each time a "[" arrives, the update is literally

$C_t=1\cdot C_{t-1}+1=C_{t-1}+1,$

so the belt reads $1,2,3,4,\dots$ — an exact counter. A plain RNN cannot do this: $\tanh$ caps its memory at $1$, so it can never store the number $4$.

Figure s08 — The cell state value plotted step by step as the tokens "[ [ [ [ ] ]" arrive. With the forget gate held at $1$ and a $+1$ written on each "[", the red belt climbs $1,2,3,4$ then falls as brackets close — the LSTM is literally counting, something a $\tanh$-bounded RNN cannot do.

---

The one-picture summary

Here is the entire cell: belt across the top, three red valves ($f_t,i_t,o_t$) reaching up to it, the + merge that makes memory additive, and $h_t$ leaving to loop back. Trace the red belt with your finger — it never passes through a weight grid, and that is the whole secret.

Figure s09 — The whole LSTM cell on one belt: $C_{t-1}$ enters at the left, is dimmed by the red forget valve $f_t$, merges at the + node with the input contribution $i_t\odot {\tilde{C}}_t$ to become $C_t$, and is tapped through $\tanh$ and the red output valve $o_t$ to produce $h_t$, which loops back as the next step's report. The unbroken red belt — never routed through a weight grid — is the whole secret that lets memory and gradient survive across many time steps.

Recall Feynman retelling — say it like a story

Picture a conveyor belt carrying boxes of memory. At every tick of the clock three little valves lean over the belt. The forget valve decides how much of each old box to keep — it dims them, it never throws dice. The input valve lets a freshly written box (which can be positive or negative, because a fact can be "yes" or "no") drop onto the belt, and the two are simply added together — kept memory plus new memory. Because it's addition, a box placed at the start can ride all the way to the end untouched, which also means the learning signal (how the final error $L$ depends on this memory) riding backward only gets multiplied by the forget dial each step — a number the network keeps near $1$ when it wants to remember — so nothing fades away. Finally the output valve peeks at the belt (after gently squashing it with $\tanh$) and hands a tidy summary $h_t$ to the outside. Freeze the forget valve open and stop writing, and the belt becomes a perfect counter. That belt, and the plus sign that feeds it, is the entire reason LSTMs remember.

Recall Quick self-test

Why is $\tanh$ used for the candidate but $\sigma$ for the gates? ::: Candidate content must be signed ($-1$ to $+1$), so $\tanh$; gates are fractions of "how much", so $\sigma$ ($0$ to $1$). What is $\partial C_t/\partial C_{t-1}$ and why does it matter? ::: It equals $f_t$; the gradient is only scaled by this per step, giving a highway instead of geometric decay. With $f_t=1,i_t=0$, what is $C_t$? ::: $C_t=C_{t-1}$ — the belt is frozen, perfect memory. With $f_t=0,i_t=0$, what is $C_t$? ::: $C_t=0$ — the belt is completely blanked, a fresh start mid-sequence. Why can't a vanilla RNN count to 4? ::: Its memory passes through $\tanh$ each step, bounding it to $[-1,1]$, so it can't hold the value $4$.

---

Where next: the GRU merges these gates into fewer valves; bidirectional RNNs run two belts in opposite directions; and the belt idea is eventually replaced by direct look-back in attention and Transformers.