3.5.4 · D5Sequence Models
Question bank — Long Short-Term Memory (LSTM) cells
True or false — justify
A forget gate value of means the cell erases 60% of its dimensions at random
False. scales every dimension it applies to by — it is a smooth partial fade of magnitude, not a random dropout of dimensions.
The cell state is bounded to just like the hidden state
False. is not squashed — it can grow unbounded (this is what lets it count). Only passes through , so the output is bounded, not the memory.
If the forget gate stays near for many steps, gradients flow back almost undiminished
True. Since , a chain of gates near multiplies to something near — a linear (not exponential) path, which is the gradient highway.
An LSTM is completely immune to vanishing gradients
False. It mitigates them: if the network learns small forget gates , the product of 's can still shrink. LSTMs make long-memory possible and learnable, not guaranteed.
The candidate uses instead of sigmoid because memory needs to represent both positive and negative information
True. outputs , so a cell dimension can encode "presence of negation" (positive) or its opposite (negative); a sigmoid could only ever add non-negative content.
The input gate and the candidate are the same thing computed differently
False. The candidate is what new content to add; the input gate is how much of it to let in. They multiply: .
An LSTM reduces to a plain summing accumulator exactly when , , and the candidate is held at the constant for every step
True. Under those three conditions the update collapses to — a pure counter, exactly the behaviour a vanilla RNN's cannot sustain because saturates at .
and always have the same values
False. , so they match only if the output gate and already lives in . In general is a gated, squashed view of .
Concatenating before each gate lets the gates depend on both past state and current input
True. That is the whole point: every gate decision is a function of what we remember () and what we just saw (), which is why one weight matrix per gate spans columns.
Spot the error
"Because and can be tuned, a vanilla RNN never has vanishing gradients."
Error: even with , the product of many Jacobians shrinks exponentially when the recurrent matrix ; tuning one matrix cannot stop an exponential over hundreds of steps.
"."
Error: the update is additive, . Multiplying everything ( throughout) would destroy the gradient highway and reintroduce vanishing.
"The output gate decides what to store in the cell."
Error: storage is done by the forget gate and input gate . The output gate decides only what to expose as ; the memory in is unchanged by it.
"We use sigmoid for the candidate so its values stay in ."
Error: the candidate uses , giving , so memory can hold signed information. Sigmoid would forbid negative content.
", so LSTMs have the same gradient as RNNs."
Error: for the LSTM cell path (element-wise, near ), with no repeated recurrent-matrix multiplication by — that is precisely the difference.
"An LSTM has three weight matrices — one per gate."
Error: there are four parameter sets — the forget, input, and output gates plus the candidate — each with its own and bias.
"Since gates output , LSTMs make hard yes/no decisions about memory."
Error: gates are continuous multipliers (sigmoid output anywhere in ). Continuity is required so gradients can flow through them during training; hard decisions would be non-differentiable.
Why questions
Why is the cell-state update additive rather than passing through a nonlinearity like the hidden state does?
Because addition gives with no squashing derivative shrinking it, so gradients survive long chains — a nonlinearity would reintroduce the derivative factor that kills RNNs.
Why do we squash with before forming , but never squash itself?
The cell must be free to grow (e.g. to count), so we leave it unbounded internally; but the exposed value passed downstream should be bounded for numerical stability, so we squash only at the output.
Why use a separate input gate instead of always adding the full candidate ?
Without the cell would absorb new content every step regardless of relevance; the gate lets the network learn when incoming information matters (gate open, ) versus when to protect existing memory (gate closed, ).
Why can an LSTM count exactly to large numbers when a vanilla RNN cannot?
Counting needs , an unbounded additive accumulator; the RNN's saturates at , so it physically cannot represent an ever-growing count.
Why does the forget gate depend on the current input and not just on ?
Whether to keep a memory often depends on what just arrived (e.g. a new sentence may make old context irrelevant), so the gate reads to make a context-aware decision.
Edge cases
If for one dimension at one step, what happens to that memory and its gradient?
That dimension of is fully erased, and since there, the gradient path through it is cut — the network deliberately drops both the value and its history.
If at every step, what does the LSTM output, and is memory lost?
Every , so downstream sees nothing, but still updates internally — memory is retained, just never exposed. It behaves like silent internal bookkeeping.
At with and , what determines the first gates?
They depend only on , so the biases and the -columns of the weights fully set the initial behaviour — the past-state columns contribute nothing on step one.
What if for many steps while ?
The cell is frozen: it neither absorbs new content nor forgets old, so indefinitely — perfect long-term storage, useful for carrying a fact (like negation) across a long span.
If but and is large every step, can blow up?
Yes — unbounded additive growth can drive to large magnitudes; then saturates near , so stays bounded but the cell can suffer saturation and stalled learning if unregulated.
Compared with a GRU, why might an LSTM keep a memory and hide it simultaneously?
The LSTM's separate output gate lets it store a fact in while emitting nothing about it, decoupling memory from exposure — a GRU, lacking a distinct output gate, ties what it remembers more tightly to what it emits.
Recall Fast self-check
One-sentence reason for each below. Additive update ::: gives , the gradient highway. on candidate ::: signed memory content. No squash on ::: lets it grow / count. Output gate ::: chooses exposure, not storage. ::: element-wise product, entry by entry. Gates use sigmoid ::: every gate entry lies in .
Related traps live in 3.5.03-Vanishing-and-Exploding-Gradientsin-RNNs (the problem this all fixes) and forward in 3.6.01-Attention-Mechanisms / 4.2.01-Transformers (which sidestep recurrence entirely). See also 3.5.06-Bidirectional-RNNs.