Vanishing gradients in RNNs

Overview

The vanishing gradient problem occurs when gradients become exponentially small as they propagate backward through time in recurrent neural networks, making it impossible to learn long-term dependencies. This is the fundamental obstacle that prevented RNNs from being effective on real-world sequential tasks until LSTM and GRU architectures were developed.

The Mathematical Root Cause

To train via gradient descent, we need $\frac{\partial L}{\partial W_{hh}}$ where $L$ is our loss. By the chain rule, gradients must flow backward through all time steps.

Derivation from First Principles

Let's derive why gradients vanish. Consider a simple RNN with loss $L$ at time step $T$.

Step 1: Chain rule through time

To update parameters at time $t$, we need: $\frac{\partial L}{\partial h_t} = \frac{\partial L}{\partial h_T} \cdot \frac{\partial h_T}{\partial h_t}$

Why this step? The loss at time $T$ depends on $h_t$ only through the chain of hidden states $h_t \to h_{t+1} \to \cdots \to h_T$. We must accumulate gradients across this entire path.

Step 2: Expand the chain

$\frac{\partial h_T}{\partial h_t} = \frac{\partial h_T}{\partial h_{T-1}} \cdot \frac{\partial h_{T-1}}{\partial h_{T-2}} \cdots \frac{\partial h_{t+1}}{\partial h_t}$

$= \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}$

Why this step? Each hidden state depends directly only on the previous one, so we chain all the local derivatives together.

Step 3: Compute the local derivative

From $h_k = \tanh(W_{hh}h_{k-1} + W_{xh}x_k + b_h)$:

$\frac{\partial h_k}{\partial h_{k-1}} = \text{diag}(\tanh'(z_k)) \cdot W_{hh}$

where $z_k = W_{hh}h_{k-1} + W_{xh}x_k + b_h$ and $\tanh'(z) = 1 - \tanh^2(z)$.

Why this step? The chain rule for the composition $\tanh(\cdot)$ applied to a linear transformation gives us the derivative of the activation times the weight matrix.

Step 4: Bound the product

Since $\tanh'(z) \leq 1$ for all $z$, each diagonal entry is at most 1. Therefore:

$\left\| \frac{\partial h_k}{\partial h_{k-1}} \right\| \leq \| W_{hh} \|$

Why this step? The derivative of tanh is bounded (each diagonal entry $\leq 1$), and matrix norms satisfy $\|AB\| \leq \|A\| \cdot \|B\|$. So the tanh factor can only shrink the norm, leaving $\|W_{hh}\|$ as the effective per-step multiplier.

Step 5: The exponential decay

For a product of $(T-t)$ terms:

$\left\| \frac{\partial h_T}{\partial h_t} \right\| \leq \| W_{hh} \|^{T-t}$

If $\|W_{hh}\| < 1$ (whether by default initialization or by weight decay driving the spectral norm below 1), then:

$\left\| \frac{\partial h_T}{\partial h_t} \right\| \leq \gamma^{T-t}$

where $\gamma < 1$. This decays exponentially in the time gap $(T-t)$.

Important distinction: The two mechanisms act on different factors. The spectral norm $\|W_{hh}\|$ is a property of the weight matrix and is what must be $< 1$ for this bound to produce decay. Tanh saturation does not change $\|W_{hh}\|$—instead it shrinks the $\text{diag}(\tanh'(z_k))$ factor toward $0$, which multiplies on top of the $\|W_{hh}\|$ term. Both push in the same (shrinking) direction, but they are separate contributions and must not be conflated.

Why this step? Multiplying a number less than 1 by itself many times gives exponential decay. After just 10 time steps with $\gamma = 0.9$, the gradient is $0.9^{10} \approx 0.35$. After 50 steps: $0.9^{50} \approx 0.005$. The gradient effectively vanishes.

Setup: We have a sequence of length 50, and we want to see how a gradient at $t=50$ affects parameter updates at $t=0$.

Computation: $\frac{\partial h_{50}}{\partial h_0} = 0.9^{50} = 5.15 \times 10^{-3}$

Interpretation: A gradient of magnitude 1.0 at the output becomes 0.005 by the time it reaches the beginning. If our learning rate is 0.01, the parameter update is $0.01 \times 0.005 = 0.00005$—essentially zero. The network cannot learn dependencies that span 50 steps.

With typical RNN: Add $\tanh$ saturation on top. If inputs are large, $\tanh'(z) \approx 0.1$ per step. Now the combined per-step multiplier (tanh factor $\times$ weight factor) is: $\text{Effective multiplier per step} = 0.1 \times 0.9 = 0.09$ $0.09^{50} \approx 2 \times 10^{-53}$

This is smaller than machine epsilon for float32. The gradient has numerically vanished. Note the $0.1$ came from tanh's derivative and the $0.9$ from the weight norm—separate factors that multiply.

$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$

Key difference: The gradient path through the cell state is: $\frac{\partial C_t}{\partial C_{t-1}} = f_t$

This is additive, not multiplicative through many layers. The forget gate $f_t$ is learned and can be close to 1, creating an uninterrupted gradient highway.

Why this step? Instead of exponential decay from repeated multiplication by small numbers, we have a learned gate that controls how much of the old information (and its gradient) to preserve. When $f_t \approx 1$, gradients flow nearly unchanged across many time steps.

When and Why It Happens

All three usually occur together in practice, but they act on different factors in the product.

Common Mistakes and Misconceptions

Why it's wrong: ReLU removes the shrinking tanh-derivative factor (making the local derivative either $0$ or $1$), but that does not guarantee explosion. Whether gradients explode still depends entirely on the spectral norm of $W_{hh}$: if $\|W_{hh}\| > 1$, the product $\prod \text{diag}(\text{ReLU}'(z_k)) W_{hh}$ can now grow because the tanh damping is gone; if $\|W_{hh}\| < 1$, gradients can still vanish. So ReLU removes one safety cushion (activation damping) without fixing the underlying credit-assignment problem—it does not inherently cause explosion, it just makes the weight spectral norm the sole controlling factor.

The fix: You typically need gradient clipping and careful (e.g., identity/orthogonal) initialization of $W_{hh}$ with ReLU RNNs, and even then the architecture doesn't solve the credit assignment problem. LSTMs/GRUs with carefully gated additions are the proper solution.

Why it's wrong: Gradients at different time steps decay at different rates. Early steps have vanished gradients ($10^{-50}$) while recent steps have reasonable gradients ($10^{-2}$). A single learning rate cannot fix both: large enough for early steps means exploding updates for recent steps.

The fix: Architectural solutions (LSTM/GRU) or adaptive optimizers (Adam, RMSprop) that maintain per-parameter learning rates. Even better: both together.

Why it's wrong: BatchNorm normalizes across the batch dimension, but vanishing gradients in RNNs occur across the time dimension. Normalizing $h_t$ across different sequences doesn't fix the temporal gradient flow within a single sequence. Layer normalization (across features at each time step) helps more, but still doesn't create gradient highways like LSTM gates do.

The fix: Use layer normalization + LSTM/GRU architecture. The gates are essential.

Practical Implications

1. Maximum learnable dependency length In practice, vanilla RNNs can only learn dependencies spanning 5-10 time steps. For language (need 20-50 token context) or time series (seasonal patterns over hundreds of steps), they fail completely.

2. Training symptoms

• Loss plateaus early
• Parameters barely change after first few epochs
• Recent outputs learn, but early sequence context is ignored
• Validation performance terrible on tasks requiring memory

3. When you still see this in modern architectures Even LSTMs can experience vanishing gradients if:

• Sequences are extremely long (1000+ steps)
• Forget gates saturate to 0 due to poor initialization
• Very deep RNN stacks (many layers)

Solution Landscape

Now imagine you need to figure out who messed up the message. You start at the end and work backward, asking each person "how much did you change it?" But each person only remembers a tiny bit of what they did (like "I changed it by 10%"). When you multiply 0.9 × 0.9 × 0.9 fifty times, you get a number so small it's basically zero.

That's vanishing gradients: the "feedback signal" about what went wrong becomes so tiny by the time it reaches the beginning of the sequence that the network can't learn anything about the early parts.

LSTMs are like giving each person a notepad where they can write down the original message and pass THAT along too, not just their modified version. Now you have two paths: the telephone game (which still gets garbled) and the notepad (which stays accurate). The network learns to use the notepad for important long-term information.

Connections

• Backpropagation Through Time - The training algorithm where this occurs
• LSTM Architecture - The primary solution to vanishing gradients
• GRU Architecture - A simplified alternative to LSTM
• Exploding Gradients - The opposite problem when $\|W_{hh}\| > 1$
• Gradient Clipping - Addresses exploding but not vanishing
• Attention Mechanisms - A way to bypass sequential processing entirely
• Residual Connections - Similar additive gradient highways in feedforward networks
• Activation Functions - Tanh saturation is a key contributor

#flashcards/ai-ml

What is the vanishing gradient problem in RNNs? :: When gradients become exponentially small as they backpropagate through time, making it impossible to learn long-term dependencies because parameter updates for early time steps approach zero.

What is the maximum dependency length vanilla RNNs can learn? :: Approximately 5-10 time steps in practice. Beyond this, exponential gradient decay makes parameter updates too small for effective learning, even though theoretically RNNs can model arbitrarily long dependencies.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)