Exercises — LSTM — gates, cell state

The two carried vectors are the ==cell state $c_t$ and the hidden state $h_t$==.

• $c_t$ is the protected long-term memory (the "conveyor belt"), updated additively.
• $h_t$ is the short-term / output vector; it is what actually leaves the cell for prediction and feeds the next step's gates. Only $h_t$ leaves — never the raw $c_t$.

The additive line is the cell update: $c_t=\underset{\text{keep}}{\underbrace{f_t\odot c_{t-1}}}+\underset{\text{write}}{\underbrace{i_t\odot {\tilde{c}}_t}}.$ The two products are multiplicative gating. Forgetting happens in $f_t\odot c_{t-1}$ (scaling the old memory down); writing happens in $i_t\odot {\tilde{c}}_t$ (scaling the proposed new value). The plus sign in the middle is the whole trick — memory is added to, not overwritten.

Cell: $c_t=f_t{c}_{t-1}+i_t{\tilde{c}}_t=0.9(0.5)+0.8(0.4)=0.45+0.32=0.77$. Output: $h_t=o_t\tanh (c_t)=0.6(0.6469)=0.388$ (to 3 dp). Reading: we kept $90\%$ of old memory, added $80\%$ of the candidate, then read out $60\%$ of the squashed result.

$\sigma (0)=\frac{1}{1+e^0}=\frac{1}{2}=0.5$. A gate at $0.5$ passes half of whatever it multiplies through. It is the "half-open valve" point on the red curve in the figure above.

$\tanh (-5)=-0.99991\approx -1.000$. So this candidate proposes writing a value near $-1$: it will push the memory downward. This is exactly why the candidate uses $\tanh$ and not $\sigma$ — it must be able to decrease memory, not only increase it.

Elementwise ($\odot$) means work coordinate by coordinate.

• Entry 1: $0.5(1.0)+1.0(0.4)=0.5+0.4=0.9$.
• Entry 2: $0.9(-2.0)+0.0(0.7)=-1.8+0=-1.8$. So $c_t=(0.9,-1.8)$. Entry 2 wrote nothing ($i=0$) and just decayed the old value a little ($f=0.9$).

Start from $c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$. Treating the gates and candidate as inputs (they depend on $h_{t-1}$, but we follow the direct $c$-highway), differentiate w.r.t. $c_{t-1}$: $\frac{\partial c_t}{\partial c_{t-1}}=f_t.$ There is no weight matrix and no ${\tanh }^{\prime }$ factor — just the gate. Chaining $k$ steps, $\frac{\partial c_t}{\partial c_{t-k}}={\prod }_{j=t-k+1}^t{f}_j.$ If the forget gates stay near $1$, the product stays near $1$, so gradients travel back undiminished — the constant error carousel. Contrast a plain RNN, whose factor is $\mathrm{diag}({\tanh }^{\prime })W$ with magnitude typically $<1$, so its product $\to 0$ exponentially. See Vanishing & Exploding Gradients and Backpropagation Through Time (BPTT).

The factor is $f^k$.

• $0.9^{50}\approx 0.00515$.
• $0.9^{200}\approx 7.06\times 10^{-10}$. After 50 steps it is already tiny; after 200 it is effectively gone. So $f$ near but below $1$ still vanishes, just slower than a plain RNN. Genuine long memory needs $f$ very close to $1$ (this is why we bias $f$ high — see 3.3).

$f=\sigma (b_f)$.

• $\sigma (1)=\frac{1}{1+e^{-1}}=0.731$.
• $\sigma (2)=\frac{1}{1+e^{-2}}=0.881$. Both are close to $1$, so at the start of training the network remembers by default. This keeps the cell highway open and gradients alive during the crucial early epochs, before the gates have learned anything. Initialising $b_f=0$ would give $f=0.5$ — half the memory erased every step — and training stalls.

With $f_t=1,i_t=0$: $c_t=1\cdot c_{t-1}+0\cdot {\tilde{c}}_t=c_{t-1}$. By induction $c_{1000}=c_{999}=\cdots =c_0$. Perfect copy. A plain RNN has $h_t=\tanh (W{h}_{t-1}+\dots )$: to hold a value it would need $\tanh (Wh)=h$ exactly for the right $h$, and even then the backward Jacobian $\mathrm{diag}({\tanh }^{\prime })W$ shrinks it. There is no "identity valve" — memory is always reprocessed. The LSTM's plus-sign gives it one for free.

Choose $f_t=0$ (erase old) and $i_t=1$ (write candidate fully): $c_t=0\cdot c_{t-1}+1\cdot {\tilde{c}}_t={\tilde{c}}_t.$ The cell is overwritten by the candidate — this is deliberately RNN-like behaviour, useful when a genuinely new regime starts (e.g. a maintenance event resets a machine's history).

• $t=200$ (capture): $i_{200}\approx 1$ (write the spike into $c$), and let ${\tilde{c}}_{200}$ encode its magnitude. $o_{200}$ is task-dependent — nonzero only if a prediction is also needed here.
• $200<t<5000$ (hold): $f_t\approx 1$, $i_t\approx 0$ — keep the stored spike, write nothing new over it. Set $o_t\approx 0$ during this stretch: we do not want the stored spike leaking into the output yet, so the read valve stays shut until release. (If intermediate forecasts are needed, $o_t$ may reflect other channels, but the spike's channel stays gated.)
• $t=5000$ (release): $o_{5000}\approx 1$ so $h_{5000}=o\odot \tanh (c)$ actually reveals the stored info to the forecast head. Gradient descent discovers this routing automatically because the gates are data-dependent learned switches; see Time Series Forecasting.

Set $f_t=0.7$ and $i_t=0.3$ (constant). Then $c_t=0.7c_{t-1}+0.3{\tilde{c}}_t,$ an exponential moving average of the candidates. Memory of an old input decays as $0.7^k$: it fades gradually rather than being kept forever (large $f$) or reset (small $f$). This is useful for smoothing noisy sensor streams. Note $f+i$ need not equal $1$ in a real LSTM — here we chose them to sum to $1$ on purpose to get a clean average.

$\tanh (8)=1.0000$ (to 4 dp; exactly $0.99999977\dots$). So $h_t=0.5(1.0000)=0.5$. The candidate's $\tanh$ bounds what we write in one step; but $c_t$ is a running additive sum and can grow unbounded over many steps (here $8$). The output's $\tanh$ re-squashes the accumulated cell to $(-1,1)$ so downstream layers always see bounded, stable values. Two different jobs — not redundant. See Sigmoid and Tanh Activations.

$c_t=1.0(-3.0)+1.0(-0.5)=-3.5$. $\tanh (-3.5)=-0.99818$, so $h_t=0.8(-0.99818)=-0.7985$ (4 dp). $c_t<0$ and $h_t<0$: they share a sign because $\tanh$ is an odd function ($\tanh (-x)=-\tanh (x)$) and the output gate $o_t\in (0,1)$ is always non-negative, so it never flips sign. A negative memory reads out as a negative hidden value.

$c_t=0\cdot c_{t-1}+0\cdot {\tilde{c}}_t=0$. $h_t=0\cdot \tanh (0)=0$. The cell is fully wiped and emits nothing — a memoryless dead unit. It is essentially never a good learned state (it destroys all gradient flow through that cell), which is precisely why we bias $f$ positive at init to steer away from this corner. Useful only as a deliberate "silence this channel" behaviour.

• $f=1$: $1^k=1$ for all $k$ → gradient preserved forever (perfect memory highway). Limit $=1$.
• $0<f<1$: $f^k\to 0$ but only polynomially-slowly compared to how fast a plain RNN dies; still vanishes eventually. Limit $=0$. (E.g. $0.99^k$ survives ~hundreds of steps.)
• $f=0$: $0^k=0$ for every $k\ge 1$ → gradient killed in one step. Limit $=0$ (instantly). The whole art is keeping $f$ as close to $1$ as the task allows for as long as the memory is needed. This is the analysis backbone of Backpropagation Through Time (BPTT) for LSTMs.

LSTM copy: $f_t=1,i_t=0\Rightarrow c_t=c_{t-1}$. GRU copy: a GRU uses $h_t=(1-u_t)h_{t-1}+u_t{\tilde{h}}_t$. Setting $u_t=0$ gives $h_t=h_{t-1}$ — same perfect copy. The GRU ties keep-vs-write into one gate $u_t$ (write fraction) with its complement $(1-u_t)$ (keep fraction), and has no separate output gate (it exposes $h_t$ directly). Fewer gates → fewer weight matrices → fewer parameters, at the cost of less independent control. Same constant-error-carousel idea, leaner hardware — attractive for embedded avionics.