Long Short-Term Memory (LSTM) cells: Foundations

Before you can read the parent note Long Short-Term Memory (LSTM) cells comfortably, you must own every symbol it throws at you. This page builds each one from nothing, in the order they depend on each other. Nothing here assumes you have seen an RNN or calculus before.

1. A vector: a bag of numbers with a fixed length

The parent note writes things like $x_{t}$ , $h_{t}$ , $C_{t}$ . Each of these is a vector.

The picture: think of a vector as a column of little boxes, each box holding one number. If someone says " $x_{t} \in R^{n_{x}}$ ", read it as: " $x_{t}$ is a column of $n_{x}$ boxes, each holding a real number."

Figure — Long Short-Term Memory (LSTM) cells

Why the topic needs it: at each moment the LSTM sees an input (a word, a sensor reading) encoded as a list of numbers — a vector. Its memory and output are also lists of numbers. Everything is vectors of boxes.

$R$ ::: the set of all real numbers (any number on the number line, e.g. $- 3.2$ , $0$ , $7$ ).
$R^{n}$ ::: the set of all vectors with exactly $n$ real-number boxes.

2. The subscript $t$ : a clock, not a multiplication

The picture: lay the sequence out left-to-right like frames of a film. $x_{t - 1}$ is the frame just before $x_{t}$ . The subscript never means "multiply by $t$ ".

Why the topic needs it: LSTMs process one tick at a time and pass memory forward from tick $t - 1$ to tick $t$ . Without a clock symbol we could not talk about "before" and "after".

3. Hidden state $h_{t}$ and cell state $C_{t}$ : two kinds of memory

Two of the vectors have special names because they persist across ticks.

The picture: imagine a conveyor belt running left to right through every tick (that is $C_{t}$ ). At each tick a small window lets a controlled amount of the belt peek out — that peek is $h_{t}$ .

$h_{0}$ and $C_{0}$ ::: the starting memories before any input; both are set to the all-zeros vector.
Why two separate memories? ::: $C_{t}$ can grow large and carry raw memory; $h_{t}$ is kept bounded and controlled because it feeds the next computation and the final answer.

Why the topic needs it: the entire trick of the LSTM (see 3.5.03-Vanishing-and-Exploding-Gradientsin-RNNs) is keeping $C_{t}$ gently edited so that information — and the learning signal — survives across many ticks.

4. The sigmoid $σ$ : a knob squeezed into $[0, 1]$

The parent writes $σ (\dots)$ for every gate. Here is what it is and why.

The picture: an S-shaped curve. Very negative input → output near $0$ (gate shut). Very positive input → output near $1$ (gate wide open). Input $0$ → output exactly $0.5$ (half open).

$e$ ::: Euler's number, about $2.718$ ; $e^{- z}$ shrinks fast as $z$ grows, which is what bends the S-curve.
$σ (0)$ ::: $0.5$ .
As $z \to + \infty$ , $σ (z) \to$ ::: $1$ .
As $z \to - \infty$ , $σ (z) \to$ ::: $0$ .

5. The $tanh$ : a squasher into $[- 1, 1]$ that allows negatives

The candidate memory $\tilde{C}_{t}$ and the exposed memory use $tanh$ , not sigmoid.

The picture: an S-curve like sigmoid, but shifted and stretched so it passes through $0$ and runs from $- 1$ (bottom) to $+ 1$ (top).

$tanh (0)$ ::: $0$ .
Why not use sigmoid for the candidate content? ::: content must be able to be negative; sigmoid can never output a negative number.

6. Element-wise product $⊙$ : box-by-box multiply

The cell update uses $⊙$ , the Hadamard (element-wise) product.

The picture: line up two columns of boxes side by side; multiply each pair straight across. No mixing between different positions.

$f_{t} ⊙ C_{t - 1}$ where $f_{t} = [0, 0, 0]$ ::: the zero vector — memory completely erased.
Why must $f_{t}$ and $C_{t - 1}$ have the same length? ::: because $⊙$ pairs boxes position-by-position; unequal lengths have unpaired boxes.

7. Concatenation $[h_{t - 1}, x_{t}]$ : stack two columns into one

The picture: glue one column of boxes on top of the other to form a single, longer column.

Why the topic needs it: every gate looks at both the past hidden state and the current input together. Stacking them lets a single weight matrix mix all that information at once.

Length of $[h_{t - 1}, x_{t}]$ if $n_{h} = 3$ , $n_{x} = 5$ ::: $8$ .

8. Weight matrix $W$ and matrix–vector multiply: the learnable mixer

Every gate is $σ (W \cdot [\dots] + b)$ . The $W$ is where learning lives.

The picture: think of $W$ as a mixing board. Each output box has a row of dials in $W$ that say how strongly it listens to each input box. Training turns these dials.

If $W_{f} \in R^{n_{c} \times (n_{h} + n_{x})}$ , it turns an input of length $n_{h} + n_{x}$ into an output of length ::: $n_{c}$ .
What does " $\cdot$ " mean in $W_{f} \cdot [h_{t - 1}, x_{t}]$ ? ::: matrix–vector multiplication (rows of $W$ dotted with the stacked column).

9. Loss $L$ and gradients $\frac{\partial L}{\partial \cdot}$ : the "which way to nudge?" arrow

The parent note's gradient-highway argument uses $\frac{\partial L}{\partial C _{t - 1}}$ .

The picture: stand on a hilly landscape where height = loss. The gradient is the arrow pointing uphill; training walks the opposite way (downhill).

$\prod_{i = 1}^{k}$ ::: "multiply these $k$ things together" (capital-pi = repeated product).
$\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}$ tells us gradient through the belt is just ::: multiplied by the forget gate $f_{t}$ (near $1$ = signal survives).

10. How it all feeds the topic

Once these pieces click, the LSTM equations read as plain English, and the follow-ons — 3.5.05-Gated-Recurrent-Units-(GRU), 3.5.06-Bidirectional-RNNs, 3.6.01-Attention-Mechanisms, 4.2.01-Transformers — reuse the exact same vocabulary.

Equipment checklist

Test yourself — cover the right side and answer aloud.

What is a vector and its dimension? ::: An ordered list of numbers; the dimension is how many numbers it holds.
What does the subscript $t$ in $h_{t}$ mean? ::: The clock tick / time step — NOT multiplication.
Difference between cell state $C_{t}$ and hidden state $h_{t}$ ? ::: $C_{t}$ is the lightly-edited long-term memory belt; $h_{t}$ is its filtered, squashed short-term output.
Range and shape of $σ (z)$ , and why gates use it? ::: Output in $(0, 1)$ , S-shaped; gates need a "fraction to let through" from none to all.
Range of $tanh (z)$ and why the candidate uses it? ::: $(- 1, 1)$ ; memory content must be able to be negative.
What does $⊙$ do? ::: Multiplies two same-length vectors box-by-box (element-wise).
What is $[h_{t - 1}, x_{t}]$ ? ::: The two vectors stacked into one taller vector of length $n_{h} + n_{x}$ .
What does the weight matrix $W$ do and what is learned? ::: Mixes the stacked input into a new vector via weighted sums; $W$ and bias $b$ are the learned dials.
What is the loss $L$ and the gradient $\frac{\partial L}{\partial C _{t - 1}}$ ? ::: $L$ is a single wrongness score; the gradient says which way to nudge each box of $C_{t - 1}$ to lower $L$ .
Why is $\frac{\partial C _{t}}{\partial C _{t - 1}} = f_{t}$ good news? ::: The backward signal through the belt is only scaled by $f_{t}$ (near $1$ ), so it survives across many ticks.

Recall Quick self-check answer

If $f_{t} = [1, 0, 0.5]$ and $C_{t - 1} = [2, - 4, 6]$ , then $f_{t} ⊙ C_{t - 1} = [2, 0, 3]$ .