Foundations — LSTM — gates, cell state

This page assumes nothing. Before you can read the parent note LSTM — gates, cell state, you need to recognise a handful of symbols on sight. We build each one from a picture, in an order where every symbol only uses ones already defined.

0. What a "vector" and a "timestep" mean

Before any equation, two ideas the parent note silently assumes.

Look at the figure: the grid shows three rows (labelled "ch 1", "ch 2", "ch 3" on the vertical axis) and four columns (labelled $t=1$ to $t=4$ on the horizontal axis). The orange box frames a single column — that column is one input vector $x_t$. Reading down a column gives you all channels at one instant; reading across a row shows one channel evolving over time.

1. The input symbol $x_t$ and the two memories $h_t$, $c_t$

Now we can name the three vectors that flow through the machine.

Why two memories? A plain RNN keeps only one note ($h_t$) and rewrites it every tick, smearing out old information. The LSTM adds $c_t$ as a "conveyor belt" that can carry a message for thousands of ticks without being overwritten. That extra line is the entire reason LSTMs beat RNNs.

Look at the figure: the horizontal axis is the timestep $t$. The thick blue line near the top is the cell state $c_t$ — notice it stays almost flat, and the pale blue arrows between ticks show information sliding forward with little change (a "highway"). The thin gray line along the bottom is the hidden state $h_t$ — it jumps up and down every tick because it is rewritten each time. The flatness of the blue line is exactly what "protected memory" looks like.

2. The bracket $[h_{t-1},x_t]$ — concatenation, and the name $z_t$

Why concatenate? Each gate needs to look at both what the machine remembers ($h_{t-1}$) and what just arrived ($x_t$) to make its decision. Gluing them together into $z_t$ lets a single weight layer see everything at once.

3. Weight matrix $W$, bias $b$, and the "affine map"

Why do we need it? The gates must be learnable. The numbers in $W$ and $b$ are exactly what training adjusts so the machine discovers, on its own, when to remember and when to forget.

4. The squashers: $\sigma$ (sigmoid) and $\tanh$

Affine maps can output any number, huge or tiny. We need to squash them into a controlled range. Two squashers appear. Both first need one constant.

Look at the figure: the horizontal axis is the raw input value (the number coming out of $Wz+b$); the vertical axis is the squashed output. The blue curve is $\sigma$ — it hugs the dotted line at $1$ on the right and $0$ on the left, and the blue dot marks $\sigma (0)=0.5$. The orange curve is $\tanh$ — it saturates at the dotted lines $+1$ and $-1$, and the orange dot marks $\tanh (0)=0$, i.e. it passes through the origin. Both are flat (saturated) at the ends and steep in the middle.

Why sigmoid for a gate? A number in $(0,1)$ is a perfect valve-opening fraction: $0$ = fully shut (block everything), $1$ = fully open (pass everything), $0.5$ = half open. Why tanh for the candidate note? Because a new memory value might need to be positive or negative; $\tanh$ allows signed, zero-centred writes. More on both in Sigmoid and Tanh Activations.

5. The elementwise product $\odot$

Why elementwise and not matrix multiply? A gate decides, per channel, how much to let through. Channel 5's valve should touch only channel 5's memory — no mixing. That independence is exactly what $\odot$ gives.

Look at the figure: the horizontal axis lists four channels (ch1–ch4); the vertical axis is value. For each channel there are three bars — gray is the memory value before gating, blue is the gate value (printed above each bar, all in $0$ to $1$), and green is the result $\text{gate}\odot \text{memory}$. The red arrow points at ch3, where the gate is $0.0$: its green result bar vanishes — that channel is blacked out. Where the gate is $1.0$ (ch1) the green bar equals the gray bar exactly; where it is $0.5$ the green bar is halved.

6. Naming the four gate outputs

Now every ingredient is defined — $z_t$, $W$, $b$, $\sigma$, $\tanh$, elementwise application — so the parent's four names finally make sense, with their affine arguments spelled out in full.

The whole update, in words you can now decode: $c_t=\underset{\text{keep old}}{\underbrace{f_t\odot c_{t-1}}}+\underset{\text{add new}}{\underbrace{i_t\odot {\tilde{c}}_t}},h_t=\underset{\text{read out (bounded)}}{\underbrace{o_t\odot \tanh (c_t)}}.$

Prerequisite map

Equipment checklist

Read a term inside :::; say it aloud first, then reveal to check yourself.

Connections

• Parent: LSTM — gates, cell state — where these symbols are used in anger.
• Recurrent Neural Networks (RNN) — the single-memory machine LSTM extends.
• Sigmoid and Tanh Activations — the two squashers in depth.
• Vanishing & Exploding Gradients — why the protected memory line matters.
• Backpropagation Through Time (BPTT) — how $W$ and $b$ get learned.
• GRU — Gated Recurrent Unit — a leaner variant of the same idea.
• Time Series Forecasting · Remaining Useful Life (RUL) Prediction — where the data comes from.