Foundations — Vanishing gradients in RNNs

This page builds every symbol used in Vanishing gradients in RNNs from nothing. If you have never seen $h_t$, $\tanh$, a derivative, or $\prod$ before, start at line one and do not skip.

1. A sequence and its time index $t$

The picture shows a row of boxes left-to-right. Each box is one time step. The topic constantly says things like "many time steps ago" — that just means a box far to the left of the current one. The gap between two positions is written $(T-t)$, the number of steps between step $t$ and the final step $T$.

2. Vectors, and the bold-ish letters $x_t$, $h_t$

Left figure: a single dot on a number line = one number (a scalar). Right figure: an arrow / column of stacked cells = a vector, a bundle of numbers travelling together.

Why the topic needs it: the whole problem is that information in $h_t$ (like "the subject was singular") must survive from an early step to a late one. If the memory-to-memory link is weak, that fact is lost.

3. Weights $W$, and what a matrix does

The double subscript reads right-to-left: $W_{hh}$ maps $h$ to $h$; $W_{xh}$ maps $x$ to $h$.

The figure shows a small arrow being turned into a longer, rotated arrow by "multiply by $W$". The key thing to feel: multiplying by the same $W$ over and over either keeps stretching (arrows grow) or keeps shrinking (arrows collapse). That repeated stretch/shrink is the seed of both exploding and vanishing gradients.

See Backpropagation Through Time for where these weights are updated.

4. The activation $\tanh$ — a squashing function

Notice the flat tails in the figure. When $z$ is large the curve is nearly flat — moving $z$ barely changes the output. "Flat" is the villain of the next section.

5. The derivative — "how steep is the curve here?"

Why this tool and not another? Learning means adjusting knobs to reduce error. To know which way to turn a knob, we need "if I change this input a little, does the error go up or down, and how fast?" — that is the derivative. No other tool answers the sensitivity question.

The figure overlays $\tanh (z)$ (top) with its derivative ${\tanh }^{\prime }(z)=1-{\tanh }^2(z)$ (bottom, a hump).

This single fact — the multiplier is between 0 and 1 — is what makes the tanh factor a shrinking factor in the topic.

6. The chain rule — passing the signal hand-to-hand

The picture is a relay race: the baton (the gradient signal) passes $h_t\to h_{t+1}\to \cdots \to h_T$. Each handoff multiplies by that step's local sensitivity. Multiplying many numbers, each below 1, drives the total toward zero. That is vanishing gradients in one sentence.

Why the topic needs the chain rule: the loss lives at the end ($t=T$) but the parameters live at every step. The chain rule is the only bridge that carries blame from the end back to the beginning.

7. The product symbol $\prod$ and the norm $\Vert \cdot \Vert$

Key rule the topic leans on: $\Vert AB\Vert \le \Vert A\Vert \Vert B\Vert$ — the size of a product is at most the product of sizes. This lets us bound a scary product by simple powers.

8. Exponential decay ${\gamma }^{T-t}$

That collapse is exactly why plain RNNs fail on long dependencies, and why LSTM Architecture, GRU Architecture, Residual Connections and Attention Mechanisms were invented to give the gradient a shortcut.

Prerequisite map

Equipment checklist