Vanishing gradients in RNNs: Question bank — AI-ML deep dive

This is a reasoning gym for Vanishing gradients in RNNs. Each item below is a trap that a fast reader falls into. Cover the answer with your hand, commit to a position out loud, then reveal. If your reason doesn't match the reason given, you had the right answer for the wrong cause — which is the trap.

Before you start, three plain-words reminders so nothing below uses a symbol you haven't been handed:

Spectral norm $∥ W_{hh} ∥$ — the largest stretch factor the weight matrix $W_{hh}$ can apply to any vector. Think "the worst-case zoom the matrix does." When it is below $1$ , repeatedly applying the matrix shrinks vectors.
tanh derivative $tanh^{'} (z) = 1 - tanh^{2} (z)$ — how steep the tanh curve is at input $z$ . It is $1$ at the centre ( $z = 0$ ) and slides toward $0$ as $∣ z ∣$ grows ("saturation").
Gradient path — the chain of multiplications the loss signal passes through as it travels backward from the last time step to an early one (see Backpropagation Through Time).

True or false — justify

Long sequences are the only cause of vanishing gradients.

False — sequence length only sets how many factors multiply; the per-step factors (

∥ W_{hh} ∥

and

tanh^{'}

) decide whether each multiplication shrinks. A short sequence with tiny factors still vanishes.

∥ W_{hh} ∥ = 1

exactly, gradients are guaranteed to be stable.

False — the bound

∥ W_{hh} ∥^{T - t}

only stops weight-driven decay; the

tanh^{'} (z_{k}) \leq 1

factors still multiply on top and drag the product toward zero, so gradients can still vanish.

Vanishing gradients mean the network's forward predictions are also tiny.

False — the forward pass and backward pass are separate; hidden states can be perfectly healthy while the backward gradient signal to early steps has decayed to near zero.

Using tanh guarantees gradients vanish.

False — tanh only contributes a shrinking factor (

tanh^{'} \leq 1

); if

∥ W_{hh} ∥

were large enough it could still explode. tanh biases toward shrinking but does not guarantee it alone.

The vanishing gradient bound

∥ W_{hh} ∥^{T - t}

is an equality.

False — it is an upper bound (from

∥ A B ∥ \leq ∥ A ∥∥ B ∥

and

tanh^{'} \leq 1

); the true magnitude is usually smaller because the tanh factors shrink it further.

A learned forget gate near

1

in an LSTM keeps the cell-state gradient path near

1

True — the cell-state Jacobian

\partial C_{t} / \partial C_{t - 1} = f_{t}

is additive-gated, so

f_{t} \approx 1

passes the gradient through almost unchanged, forming the "gradient highway."

Exploding and vanishing gradients are the same phenomenon with a sign flip.

False — they are the same mechanism (repeated multiplication) with opposite outcomes: vanishing when the per-step factor

< 1

, exploding when

> 1

. See Exploding Gradients.

Vanishing gradients can be detected by watching the loss value alone.

False — the loss may plateau for many reasons; the telltale sign is that gradients w.r.t. early-time parameters are orders of magnitude smaller than those w.r.t. recent-time parameters.

Spot the error

"tanh saturation reduces

∥ W_{hh} ∥

, so the weight norm shrinks over training."

Error: saturation shrinks the separate

diag (tanh^{'} (z_{k}))

factor, not the weight matrix.

∥ W_{hh} ∥

is a fixed property of the weights and is untouched by which region of tanh you sit in.

"Since

tanh^{'} (z) \leq 1

, the product of Jacobians can never exceed

∥ W_{hh} ∥^{T - t}

, so gradients can never explode with tanh."

Error: the

\leq 1

tanh factor caps the tanh contribution, but

∥ W_{hh} ∥^{T - t}

itself grows without bound when

∥ W_{hh} ∥ > 1

, so tanh RNNs can still explode.

"Switching to ReLU fixes vanishing because

ReLU^{'} = 1

Error: ReLU only removes the tanh damping factor; whether gradients vanish or explode now depends entirely on

∥ W_{hh} ∥

. See the mistake note in Vanishing gradients in RNNs and consider Gradient Clipping.

"Use a learning rate of

1 0^{5}

to cancel a

1 0^{- 5}

gradient."

Error: different time steps decay at different rates, so one giant learning rate that rescues early steps blows up the recent steps. Adaptive optimizers or architecture fixes are needed instead.

"BatchNorm normalizes activations, so it fixes vanishing gradients in RNNs."

Error: BatchNorm normalizes across the batch dimension; vanishing happens across the time dimension within one sequence, which BatchNorm never touches.

"The gradient

\partial h_{T} / \partial h_{t}

is a single number, so we can just clip it."

Error: it is a matrix (a Jacobian), and clipping is applied to the assembled gradient's norm, not to this intermediate — clipping fixes explosion, not vanishing.

Why questions

Why do we multiply Jacobians rather than add them when going back in time?

Because each hidden state depends on the previous one through function composition, and the chain rule turns composition into a product of local derivatives.

Why does a per-step factor below

1

cause exponential (not linear) decay?

Multiplying by the same sub-one factor

(T - t)

times gives

γ^{T - t}

; repeated multiplication compounds, which is exponential in the number of steps.

Why does BPTT make the vanishing problem worse for distant dependencies specifically?

The exponent

(T - t)

is exactly the time gap, so the further apart the cause and effect, the more shrinking factors multiply — near dependencies barely decay, far ones vanish.

Why can an LSTM escape the exponential trap that a plain RNN cannot?

Its cell-state gradient path is additive and gated (

\partial C_{t} / \partial C_{t - 1} = f_{t}

) rather than a forced product through squashing nonlinearities, so a learned

f_{t} \approx 1

keeps the path near

1

Why do GRUs also help despite having fewer gates than LSTMs?

Their update gate similarly creates an additive skip for the hidden state, giving a near-identity gradient path when the gate says "keep old state."

Why do residual connections and attention address the same underlying issue in other architectures?

Both create short gradient paths (an identity skip, or direct attention weights) so the signal need not traverse a long product of shrinking Jacobians.

Why does saturation (

tanh^{'} \to 0

) and a small spectral norm push in the same direction yet count as two causes?

Both are factors

< 1

in the product so both shrink the gradient, but they live in different multiplicands (

diag (tanh^{'})

W_{hh}

) and can be changed independently.

Edge cases

What happens to the gradient path when the time gap

T - t = 0

(loss at the same step)?

There is no product at all —

\partial h_{T} / \partial h_{t}

is the identity, so there's nothing to decay; vanishing needs at least one intervening step.

What if

∥ W_{hh} ∥

is exactly

1

but all inputs are near zero (tanh at its centre,

tanh^{'} \approx 1

Both factors are

\approx 1

, so the gradient neither vanishes nor explodes — the rare stable "sweet spot" that identity/orthogonal initialization deliberately targets.

What if the forget gate

f_{t}

in an LSTM learns a value near

0

Then the cell-state path multiplies by

\approx 0

, deliberately erasing old information and its gradient — vanishing here is a feature, letting the model forget irrelevant history.

What happens with an extremely long sequence but a per-step factor of exactly

1

The gradient stays at magnitude

1

no matter how long the sequence, showing that length alone is harmless — it is the factor's size, not the count, that decides the fate.

What is the behaviour when

∥ W_{hh} ∥ > 1

but tanh is deeply saturated so

tanh^{'} \approx 0.05

The two factors fight; if the product per step (e.g.

1.2 \times 0.05 = 0.06

) is below

1

the gradient still vanishes — saturation can rescue stability by damping an otherwise-exploding weight.

Recall Self-test: name the two independent factors

The two per-step shrinking factors are the ==spectral norm $∥ W_{hh} ∥$ and the $diag (tanh^{'} (z_{k}))$ == activation-derivative diagonal.