Worked examples — Long Short-Term Memory (LSTM) cells

This page runs an LSTM cell through every kind of situation it can meet: gates fully open, fully closed, half-open, cell states that grow, cell states that vanish, the degenerate "brand-new cell" case, and a couple of exam twists. If you have not yet met the gate equations, read Long Short-Term Memory (LSTM) cells first — this page assumes those six lines and then stress-tests them.

Everything below uses 1-dimensional cells ($n_c=1$) so every number is something you can hold in your head. The formulas are identical in higher dimensions — you just apply them element-by-element.

Recall The six lines we will keep reusing

$f_t=\sigma (W_f[h_{t-1},x_t]+b_f),i_t=\sigma (W_i[h_{t-1},x_t]+b_i)$ ${\tilde{C}}_t=\tanh (W_C[h_{t-1},x_t]+b_C),o_t=\sigma (W_o[h_{t-1},x_t]+b_o)$ $C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t,h_t=o_t\odot \tanh (C_t)$ Here $\sigma (z)=\frac{1}{1+e^{-z}}$ squashes any real number into $(0,1)$, and $\tanh (z)$ squashes into $(-1,1)$. The symbol $\odot$ is element-wise multiply. Since we work in 1D, $\odot$ is just ordinary multiplication.

---

The scenario matrix

Think of an LSTM step as three knobs — forget $f_t$, input $i_t$, output $o_t$ — each of which can be near $0$, near $1$, or somewhere in between, plus the state of the memory $C_{t-1}$ (empty, small, large). Every real behaviour is some combination. Here is the full grid we will cover:

#	Cell class	$f_t$	$i_t$	${\tilde{C}}_t$	$o_t$	What it models
A	Pure carry (gradient highway)	$\approx 1$	$\approx 0$	—	—	remember forever, ignore input
B	Pure overwrite	$\approx 0$	$\approx 1$	new	—	wipe old, store fresh
C	Partial blend	$0.6$	$0.5$	mixed	—	smooth interpolation
D	Sealed memory	$1$	$1$	—	$\approx 0$	store now, expose later
E	Degenerate start	any	any	any	any	$C_0=0,h_0=0$ cold boot
F	Counting / accumulation	$1$	$1$	$+1$	—	additive integer count
G	Saturation limit	$1$	$1$	$+1$	$1$	$C_t\to \infty$, why $\tanh$ caps $h_t$
H	Gradient decay contrast	$<1$ vs $1$	—	—	—	LSTM vs vanilla RNN over many steps
I	Real-world word problem	learned	learned	—	learned	negation in sentiment
J	Exam twist	trick	trick	—	trick	"what $b_f$ forces remembering?"

The examples below are labelled with the cell(s) they hit. Between them they touch every row.

---

Example A — Pure carry: the gradient highway

Forecast: guess before reading — does the memory change at all? What number does the gradient carry?

1. Apply the cell update. $C_t=f_t{C}_{t-1}+i_t{\tilde{C}}_t=1\cdot 0.72+0\cdot {\tilde{C}}_t=0.72$. Why this step? The whole point of the forget/input pair is that with $f_t=1,i_t=0$ the memory is copied unchanged — no matrix, no $\tanh$ in the path.
2. Differentiate. From $C_t=f_t{C}_{t-1}+\dots$, treating $f_t$ as fixed, $\frac{\partial C_t}{\partial C_{t-1}}=f_t=1$. Why this step? This is the "superhighway" claim from the parent note made concrete. A gradient multiplied by $1$ at every step neither shrinks nor grows.

Verify: over $k=100$ steps of pure carry, the gradient is $f_t^{100}=1^{100}=1$ — no decay. Compare with a vanilla RNN weight $0.9$: $0.9^{100}\approx 2.7\times 10^{-5}$, effectively zero. This is exactly the vanishing gradient the LSTM defeats.

---

Example B — Pure overwrite

Forecast: roughly, how much of the old $0.72$ survives?

1. Old contribution. $f_t{C}_{t-1}=0.02\times 0.72=0.0144$. Why this step? A near-zero forget gate means the past is almost erased — only $1.44\%$ leaks through.
2. New contribution. $i_t{\tilde{C}}_t=0.98\times (-0.5)=-0.49$. Why this step? An open input gate lets nearly the whole candidate in; the sign is negative, showing cell states carry signed information ($\tanh$ output).
3. Combine. $C_t=0.0144+(-0.49)=-0.4756$.

Verify: the result is dominated by the fresh $-0.5$, with a tiny trace of the old positive value nudging it up. That matches "overwrite, don't blend." Sign flipped from $+$ to $-$ — the memory now stores the opposite fact.

---

Example C — Partial blend (the continuity case)

Forecast: is $f_t=0.6$ "forgetting 40% of the cell" or something else?

1. Kept memory. $0.60\times 0.60=0.36$. Why this step? $f_t=0.6$ scales the memory by $0.6$ — every dimension shrinks smoothly to $60\%$, it does not randomly delete $40\%$ of dimensions.
2. Added candidate. $0.50\times 0.80=0.40$. Why this step? Only half the candidate is admitted, showing the gate as a dimmer switch, not an on/off toggle.
3. Sum. $C_t=0.36+0.40=0.76$.

Verify: both parts are strictly between full-keep and full-drop, so $C_t=0.76$ sits between the old $0.60$ and the candidate $0.80$ — a genuine interpolation. Because everything is smooth, the whole thing is differentiable, which is why gradient descent can train it.

---

Example D — Sealed memory: store now, reveal later

Forecast: the memory is strong — will the hidden state be strong too?

1. Squash the cell. $\tanh (0.9)\approx 0.7163$. Why this step? $h_t$ never reads $C_t$ raw; it reads $\tanh (C_t)$ so the exposed value is bounded to $(-1,1)$.
2. Apply output gate. $h_t=o_t\tanh (C_t)=0.01\times 0.7163\approx 0.00716$. Why this step? A closed output gate means "I hold this memory but choose not to show it now" — think of storing grammatical gender and only revealing it when a pronoun is needed.

Verify: memory $C_t=0.9$ is large, yet $h_t\approx 0.007$ is tiny — the fact is retained internally but invisible externally. This is the cell/hidden-state split. Later, if $o_t$ opens to $\approx 1$, $h_t$ jumps to $\approx 0.716$ without $C_t$ changing at all.

---

Example E — Degenerate cold start ($C_0=0,h_0=0$)

Forecast: what happens to the forget term when there is nothing to forget?

1. Forget term vanishes. $f_1{C}_0=0.5\times 0=0$. Why this step? This is the degenerate case: no matter what $f_1$ is, multiplying by $C_0=0$ gives $0$. On the first step the forget gate is irrelevant.
2. Only the input term survives. $C_1=0+i_1{\tilde{C}}_1=0.8\times 0.4=0.32$. Why this step? The very first memory is built purely from the input path — there is no past to carry.
3. Hidden state. $h_1=o_1\tanh (C_1)=0.6\times \tanh (0.32)=0.6\times 0.30951\approx 0.18571$.

Verify: with an empty start the cell equals exactly $i_1{\tilde{C}}_1$, and $h_1$ is a squashed, gated version of it. Everything reduces cleanly — no division, no undefined quantity, so the cold boot is well-behaved.

---

Example F — Counting: additive accumulation

Forecast: can a bounded-activation network hold the exact number $4$?

1. Recurrence. $C_t=1\cdot C_{t-1}+1\cdot 1=C_{t-1}+1$. Why this step? With those learned gate values the update collapses to plain integer addition — the cell literally becomes a counter.
2. Unroll. $C_1=1,C_2=2,C_3=3,C_4=4$. Why this step? Each additive step increments by exactly one; there is no $\tanh$ on the memory path, so nothing caps the value.

Verify: $C_4=4$ exactly. A vanilla RNN with $h_t=\tanh (W{h}_{t-1}+\dots )$ can never reach $4$ because $\tanh \in (-1,1)$ — it would saturate near $1$ (shown next). The additive cell path is what makes exact counting possible.

---

Example G — Saturation limit: why $h_t$ needs $\tanh$

Forecast: does $h_{20}$ blow up to $20$ as well?

1. Squash the huge cell. $\tanh (20)\approx 1.0000000$ (to machine precision, $1-4\times 10^{-18}$). Why this step? $\tanh$ of any large positive number is essentially $1$. This is exactly why the hidden path uses $\tanh$: it protects downstream layers from an unbounded conveyor belt.
2. Output. $h_{20}=o_t\tanh (C_{20})=1\times 1=1$.
3. Limit. As $C_t\to \infty$, $\tanh (C_t)\to 1$, so $h_t\to o_t$. Symmetrically $C_t\to -\infty \Rightarrow h_t\to -o_t$. Why this step? This is the limiting-behaviour cell of the matrix: the memory can be arbitrarily large, yet the exposed state is always trapped in $[-o_t,o_t]\subseteq (-1,1)$.

Verify: $C_{20}=20$ but $h_{20}=1$ — the internal counter keeps rising while the external signal is safely capped. The memory remembers "twenty"; the world just sees "very positive."

---

Example H — Gradient decay: LSTM vs vanilla RNN

Forecast: $0.99$ vs $0.9$ — a $9\%$ difference per step. How different are they after 50 steps?

1. LSTM path. $\frac{\partial L}{\partial C_{t-50}}\propto {\prod }_i{f}_i=0.99^{50}\approx 0.6050$. Why this step? The cell-state gradient multiplies only by the forget gate each step — no weight matrix, no ${\tanh }^{\prime }$ shrinking it.
2. Vanilla RNN path. $\propto 0.9^{50}\approx 0.005154$. Why this step? The RNN gradient multiplies by $W_{hh}\cdot {\tanh }^{\prime }$ each step, here abstracted to $0.9$, and ${\tanh }^{\prime }\le 1$ makes decay worse.
3. Compare. LSTM keeps $\sim 60\%$; RNN keeps $\sim 0.5\%$ — over 100× stronger signal.

Verify: $0.99^{50}/0.9^{50}\approx 117$. The LSTM preserves roughly two orders of magnitude more gradient, which is precisely why it learns long-range dependencies. See 3.5.03-Vanishing-and-Exploding-Gradientsin-RNNs for the RNN side.

---

Example I — Real-world word problem: "not bad" is positive

Forecast: the word "not" is three tokens before "bad" — will the network still know it saw "not"?

1. Store negation. At "not", $C=i\tilde{C}=0.9\times 0.8=0.72$. Why this step? Input gate opens to write the fact "a negation just happened" into memory.
2. Carry through "was". $C\leftarrow 0.95\times 0.72+0=0.684$. Why this step? Forget gate stays high so the negation persists nearly intact across an irrelevant word.
3. Still present at "bad". $C\leftarrow 0.95\times 0.684=0.6498$. Why this step? When "bad" arrives, the cell still holds $\approx 0.65$ of negation, so the network flips "bad" into "positive overall."

Verify: starting at $0.72$, after two $\times 0.95$ carries the memory is $0.72\times 0.95^2=0.6498$ — over $90\%$ retained. The negation survived the gap, which is exactly what an RNN would fail to do. Contrast the GRU, which merges these gates, and attention, which skips the carry entirely.

---

Example J — Exam twist: what bias forces "remember by default"?

Forecast: should $b_f$ be positive, negative, or zero?

1. Set up the equation. We need $\sigma (b_f)=\frac{1}{1+e^{-b_f}}=0.95$. Why this step? At initialization the weighted term is $\approx 0$, so the gate value is decided entirely by the bias.
2. Invert the sigmoid. $b_f={\sigma }^{-1}(0.95)=\ln \frac{0.95}{1-0.95}=\ln 19\approx 2.9444$. Why this step? The inverse sigmoid (the "logit") answers "which input produces this output?" — the same undo-question logic as arctan for tan.
3. Interpret. A positive $b_f\approx +3$ pushes forget gates near $1$, so untrained LSTMs default to keeping memory, giving gradients a head start.

Verify: $\sigma (\ln 19)=\frac{1}{1+e^{-\ln 19}}=\frac{1}{1+1/19}=\frac{19}{20}=0.95$ exactly. This is the well-known "forget-gate bias = 1 or 2" initialization trick, here solved precisely.

---

Where this goes next: the same gating idea, simplified, becomes the GRU; run two LSTMs in opposite directions and you get bidirectional RNNs; drop the recurrence entirely for attention and Transformers.