Exercises — Long Short-Term Memory (LSTM) cells

(a) forget gate $f_t$ — it multiplies $C_{t-1}$, so it scales the old memory. (b) output gate $o_t$ — it multiplies $\tanh (C_t)$ to build $h_t$, the exposed value. (c) input gate $i_t$ — it multiplies the candidate ${\tilde{C}}_t$ before it is added. Anchor picture: figure below. Forget is the valve on the incoming belt, input is the valve on the new-cargo chute, output is the window letting you peek at the belt.

$f_t,i_t,o_t\in [0,1]$ — all pass through sigmoid, whose graph rises from $0$ to $1$. ${\tilde{C}}_t\in [-1,1]$ — passes through tanh, which is signed so the memory can go up or down. $\tanh (C_t)\in [-1,1]$ — squashed for the same signed-but-bounded reason. Note $C_t$ itself has no bound — it is the raw conveyor belt and may grow past $\pm 1$.

Gates answer "what fraction to keep?" — a fraction lives in $[0,1]$, exactly sigmoid's range. The candidate answers "what value to write?" — memory content can be positive or negative, so we need the signed range $[-1,1]$ that tanh gives.

Apply the cell update directly: $C_t=f_t{C}_{t-1}+i_t{\tilde{C}}_t=0.90(0.80)+0.50(0.60)=0.72+0.30=\mathrm{1.02.}$ Notice $C_t=1.02>1$: the belt is unbounded, confirming Exercise 1.2.

$h_t=o_t\tanh (C_t)=0.70\times 0.7699\approx \mathrm{0.5389.}$ So even though the belt holds $1.02$, the world only sees $\approx 0.539$ — squashed by tanh, then masked by $o_t$.

$\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.13534}\approx \mathrm{0.8808.}$ Interpretation: about $88\%$ of each old-memory coordinate survives this step.

The update collapses to $C_t=1\cdot C_{t-1}+1\cdot 1=C_{t-1}+1$: $C_1=0+1=1,C_2=1+1=2,C_3=2+1=3.$ The cell state is literally an integer counter — impossible for a vanilla RNN whose $\tanh$ caps values at $1$.

Differentiate $C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$ with respect to $C_{t-1}$. The second term does not contain $C_{t-1}$, so its derivative is $0$. The first term is element-wise multiplication by the constant $f_t$, whose derivative is $f_t$ itself: $\frac{\partial C_t}{\partial C_{t-1}}=f_t.$ Why it wins: across $k$ steps the RNN multiplies $k$ copies of $W_{hh}\mathrm{diag}({\tanh }^{\prime })$, and since $|{\tanh }^{\prime }|\le 1$ and $\Vert W_{hh}\Vert$ is generic, that product decays (or blows up) exponentially. The LSTM instead multiplies $f_t{f}_{t-1}\cdots$; if the network learns $f\approx 1$, that product stays near $1$ over hundreds of steps — a gradient highway.

LSTM: $f^{100}=0.99^{100}\approx 0.3660$ — about $37\%$ survives. RNN: $0.9^{100}\approx 2.66\times 10^{-5}$ — effectively $0$. The gap of roughly four orders of magnitude is the reason LSTMs learn long dependencies.

With no writes, $C_t=0.5C_{t-1}\Rightarrow C_t=0.5^t{C}_0$. Below $1\%$ means $0.5^t<0.01$. Take logs: $t>\frac{\ln 0.01}{\ln 0.5}=\frac{-4.6052}{-0.6931}\approx 6.6439$. So the memory dips under $1\%$ at $t=7$. A half-open forget gate is a fast leak — long memory needs $f$ near $1$, not near $0.5$.

Inside the bracket: $n_h+n_x=256+100=356$; then $n_h(n_h+n_x)=256\times 356=91136$; add the bias $n_h=256$ to get $91392$. Multiply by the $4$ gate/candidate blocks: $4\times 91392=365568.$ So $365,568$ trainable parameters — four times a same-size vanilla RNN block, the price of the gates.

(a) On seeing $[$: open the input gate and write $+1$, forget nothing (there's nothing yet): $f=0,i=1,\tilde{C}=+1\Rightarrow C=0\cdot 0+1\cdot 1=1.$ (b) Every later step: freeze — keep the old value, write nothing: $f=1,i=0\Rightarrow C=1\cdot 1+0=1.$ The memory is latched at $1$ permanently. This is exactly why $\frac{\partial C_t}{\partial C_{t-1}}=f=1$ gives a lossless gradient path for the stored bit.

$C_1=0\cdot 0+0.9\times 0.8=\mathrm{0.72.}$ $C_2=0.95\times 0.72+(-0.02)=0.684-0.02=\mathrm{0.664.}$ The negation stored at step 1 ($0.72$) survives almost intact to step 2 ($0.664\approx C_1$), because $f_2\approx 1$. That preserved value is what flips "bad" into an overall positive judgement.

Gates first (all sigmoids): $f_t=\sigma (1.0)=\frac{1}{1+e^{-1}}\approx 0.7311,i_t=\sigma (0.5)\approx 0.6225,o_t=\sigma (-0.2)\approx \mathrm{0.4502.}$ Candidate: ${\tilde{C}}_t=\tanh (0.8)\approx \mathrm{0.6640.}$ Cell update: $C_t=f_t{C}_{t-1}+i_t{\tilde{C}}_t=0.7311(0.3)+0.6225(0.6640)\approx 0.2193+0.4133=\mathrm{0.6327.}$ Hidden state: $\tanh (0.6327)\approx 0.5599$, so $h_t=o_t\tanh (C_t)=0.4502\times 0.5599\approx \mathrm{0.2521.}$ Full answer: $f_t\approx 0.731,i_t\approx 0.623,{\tilde{C}}_t\approx 0.664,C_t\approx 0.633,o_t\approx 0.450,h_t\approx \mathrm{0.252.}$

The LSTM's cell update $C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$ is additive and unbounded — set $f=1,i=1,\tilde{C}=1$ and it becomes $C_t=C_{t-1}+1$, a pure integer accumulator (Exercise 2.4). The vanilla RNN update $h_t=\tanh (W{h}_{t-1}+\dots )$ is saturating: no matter how many $[$ it sees, $\tanh$ clamps the state into $[-1,1]$, so it can never represent "4". The additive highway is what turns memory from a squashed blur into an exact register. This is the same additive structure that later inspired the residual/skip connections used in 4.2.01-Transformers.

The GRU ties keep-fraction and write-fraction together: $f_t=1-z_t$ and $i_t=z_t$ so the two coefficients sum to $1$. For $z_t=0.3$: $f_t=1-0.3=0.7$ and $i_t=0.3$, and indeed $f_t+i_t=1.0$. The LSTM is strictly more expressive because it can choose $f_t$ and $i_t$ independently (e.g. $f=1,i=1$ for counting, which violates $f+i=1$), at the cost of the extra parameters counted in Exercise 4.1.

Range of a gate output ::: $[0,1]$ because it is a sigmoid. The single-step cell gradient $\partial C_t/\partial C_{t-1}$ ::: equals $f_t$. Fraction of gradient surviving 100 steps at $f=0.99$ ::: about $0.366$. LSTM params for $n_x=100,n_h=256$ ::: $365,568$. GRU-matching LSTM gates when $z_t=0.3$ ::: $f_t=0.7,i_t=0.3$.