Vanishing and exploding gradients: Question bank

This is a rapid-fire set of conceptual traps for the parent topic. No heavy arithmetic here — that lives in the computation decks. Each item is a one-line reveal: read the prompt, commit to an answer out loud, then check. If your reasoning differs from the answer's reasoning (not just the verdict), that's the misconception to fix.

Before the traps, four definitions so every symbol below is earned.

Definition The words this deck reuses

Layers $L$ := the total number of weight layers stacked from input to output. Numbered $1$ (nearest the input) up to $L$ (nearest the loss). "Deep" just means $L$ is large.
Pre-activation $z^{(l)}$ := the raw weighted sum a neuron computes before the activation is applied, $z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)}$ . The activation then outputs $a^{(l)} = σ (z^{(l)})$ . Every $z^{(l)}$ below is this quantity.
Forward activation $a^{(l)}$ := the value a neuron outputs as data flows input → loss. When a trap says "activation" it means this forward value.
Backward gradient (the "error signal" of backprop) := the quantity $δ^{(l)} = \partial L / \partial z^{(l)}$ that flows loss → input. Every trap below that says "the gradient" or "the backward signal" means this, never the forward activation. The whole problem is about what happens to this backward gradient as it travels back through all $L$ layers.

Formula Where the per-layer gain

$γ$ comes from (the $L - 1$ exponent, made precise) One backprop link (from the parent note) is $δ^{(l)} = (W^{(l + 1)})^{⊤} δ^{(l + 1)} ⊙ σ^{'} (z^{(l)})$ . Here $diag (σ^{'} (z^{(l - 1)}))$ just means "put the per-neuron slopes $σ^{'} (z^{(l - 1)})$ down the diagonal of a matrix so that multiplying by it scales each coordinate of the incoming gradient by that neuron's slope" — it is the matrix form of the elementwise product $⊙ σ^{'}$ . Unrolling from the loss (layer $L$ ) down to layer $1$ crosses $L - 1$ links, so $δ^{(1)} = (\prod_{l = 2}^{L} (W^{(l)})^{⊤} diag (σ^{'} (z^{(l - 1)}))) δ^{(L)} .$ Take the spectral (operator-2) norm $∥ \cdot ∥$ — the largest factor by which a matrix can stretch any vector, i.e. its largest singular value. Two facts make the bound clean: (i) the norm of a product $\leq$ product of norms, and (ii) a diagonal matrix's spectral norm is its largest diagonal entry, so $diag (σ^{'}) = max_{i} ∣ σ^{'} (z_{i}) ∣ \leq max σ^{'}$ . With exactly $L - 1$ factors, $δ^{(1)} \leq γ^{L - 1} δ^{(L)}, γ := (max_{l} (W^{(l)})^{⊤}) \cdot max σ^{'} .$ Read $γ$ exactly: it is the worst-case (maximum over all layers) spectral norm of a weight matrix, times the steepest activation slope $max σ^{'}$ . When the parent note writes the shorthand $γ \approx ∥ W ∥ \cdot max σ^{'}$ , the symbol $∥ W ∥$ there stands for "the spectral norm of a typical layer's weight matrix" — a stand-in for the exact $max_{l} ∥ (W^{(l)})^{⊤} ∥$ above. So $γ$ itself is an exact upper bound; only the $∥ W ∥$ shorthand is the approximation, used when all layers are assumed similar. The $L - 1$ exponent is the count of links crossed, nothing more.

True or false — justify

Sigmoid networks vanish only because the weights are initialized too small.

False — even with ideally scaled weights (spectral norm exactly

1

, so the weight factor neither shrinks nor grows the gradient), the sigmoid derivative is still capped at

0.25

, so

γ \leq 0.25

; the activation-slope factor

max σ^{'}

alone drives

γ^{L - 1}

toward zero regardless of init.

A vanishing gradient means the network has converged to a good minimum.

False — a frozen early layer can coexist with high loss; later layers still receive a healthy backward gradient while early ones simply stop updating. Convergence needs low loss, not small gradients.

Exploding gradients and vanishing gradients cannot happen in the same network.

False — the bound

γ^{L - 1}

is a single worst case, but the actual per-link factors vary block by block; some layers can have spectral-norm-times-slope

> 1

(locally amplifying the backward gradient) while others have it

< 1

(locally shrinking it), so the same net can both explode in some sub-paths and vanish in others.

Using ReLU everywhere completely removes the vanishing/exploding problem.

False — ReLU's slope is

1

(good) but its slope is

0

for negative inputs, so "dead neurons" still zero the backward gradient, and large weights still push

γ > 1

and explode.

Making the network shallower reduces the exponent, so it always improves stability.

False — fewer layers shrinks the exponent

L - 1

, but if

γ

itself is far from

1

the net still vanishes or explodes catastrophically (e.g.

γ = 1.5

over even 10 links gives

\approx 58 \times

); fewer layers only reduces how fast the deviation compounds, it does not fix a bad

γ

Gradient clipping fixes vanishing gradients.

False — clipping only caps large gradients (exploding). It does nothing when the backward gradient is already near zero; a tiny gradient stays tiny after clipping.

Batch Normalization helps by keeping forward activations in a range where

σ^{'}

is healthy.

True — by re-centering and rescaling

z^{(l)}

, it keeps pre-activations away from the flat saturated tails where

σ^{'} \to 0

, indirectly nudging the

max σ^{'}

factor of

γ

toward

1

(this is built-in normalization — see the "modern alternatives" edge cases below).

If the loss becomes NaN, the gradients almost certainly vanished.

False — NaN comes from numbers blowing up to infinity, i.e. exploding backward gradients (

γ > 1

compounded over depth), not shrinking ones.

Residual connections work by making each layer's function smaller.

False — they work by adding an identity path so the layer Jacobian becomes

I + (\dots)

, guaranteeing a gain-

\approx 1

route for the backward gradient even if

f

itself is small.

SELU / self-normalizing nets and Leaky-ReLU are irrelevant to this problem.

False — Leaky-ReLU gives negatives a small nonzero slope so they never fully die; SELU is engineered so activations self-normalize (mean/variance drift back toward a fixed point), pinning

γ

near

1

without an explicit BatchNorm layer.

Spot the error

"Since backprop adds the error contributions from each layer, deep nets accumulate large gradients — that's why they explode."

The backward gradient is passed by multiplication of Jacobians, not addition; the product of

L - 1

factors is what compounds exponentially, which is why depth is dangerous. Recall backprop is the chain rule, and the chain rule multiplies derivatives — see Backpropagation.

"He initialization uses

Var (W) = 2/ n_{in}

because ReLU doubles the signal."

Wrong reason — the factor of

2

compensates for ReLU zeroing roughly half the forward activations (halving their variance); the

2

restores it, keeping

γ \approx 1

. He is for ReLU, Xavier for tanh/sigmoid — see Weight Initialization.

"Sigmoid's derivative reaches its maximum of

1

z = 0

The maximum is

0.25

, not

1

, occurring at

z = 0

where

σ = 0.5

; the cap at

0.25

is precisely why sigmoid promotes vanishing.

"To fix vanishing gradients, just crank up the learning rate so the tiny gradients take bigger steps."

Gradient magnitude differs by many orders across layers (

1 0^{- 6}

1 0^{0}

); one global learning rate cannot balance both — a rate big enough for early layers would destabilize later ones. The learning rate multiplies the gradient in the update step; it cannot un-vanish a gradient — see Gradient Descent.

"The gradient for an

L

-layer net has

L

multiplicative factors."

It has

L - 1

factors — propagating from layer

L

down to layer

1

crosses

L - 1

layer-to-layer links, so the bound scales like

γ^{L - 1}

(exactly the count in the unrolled product above).

"tanh is immune to vanishing because its derivative can equal

1

Its derivative reaches

1

only at $z = 0$ ; away from zero it drops fast, and in the saturated tails

tanh^{'} \to 0

, so tanh still vanishes for large-magnitude pre-activations.

Why questions

Why does the exponent being

L - 1

rather than exactly

L

barely matter in practice?

Because the dependence on depth is exponential — whether it's

γ^{L}

γ^{L - 1}

, if

γ \neq = 1

the value still races toward

0

\infty

; the base

γ

is what you must control, not the off-by-one exponent.

Why does the same property (depth

L

) that makes networks powerful also make them unstable?

Depth stacks many multiplicative factors: more layers means richer representations and a longer product

γ^{L - 1}

that amplifies any deviation of

γ

from

1

Why do we check per-layer gradient norms instead of just the total loss?

Total loss can be high while the early layers' backward gradients are near zero; only per-layer norms reveal that the early layers are frozen while later ones still learn.

Why is

γ \approx 1

the design target rather than

γ

as large as possible?

γ < 1

vanishes and

γ > 1

explodes; only

γ \approx 1

keeps the backward gradient's magnitude roughly preserved across all layers, which is the whole point of every fix.

Why do LSTM/GRU gates help specifically in recurrent nets?

An RNN reuses the same recurrence over many time steps, so its effective depth is the sequence length; the additive cell state gives a near-

1

recurrence, avoiding compounded shrinkage over long sequences — see LSTM and GRU.

Why does ReLU "dominate deep nets" despite having no upper bound on its output?

Its derivative is exactly

1

for positive inputs, so it neither shrinks nor amplifies the backward gradient per layer — the factor stays at

1

, keeping

γ

from collapsing the way sigmoid's

0.25

does.

Edge cases

What happens to

γ^{L - 1}

when

γ

is exactly

1

It stays at

1

for any depth — the backward gradient is perfectly preserved, the idealized stability regime every trick tries to approximate.

If a ReLU neuron receives only negative pre-activations for all training data, what is its gradient contribution?

Zero — its slope is

0

on negatives, so it is a "dead neuron"; its factor is

0

, killing the backward gradient through that path. Leaky-ReLU avoids exactly this by giving negatives a small nonzero slope.

What does the gain look like at the very first training step versus deep into training?

Init controls

γ

at step 0 (keeping it

\approx 1

), but as weights grow during training the spectral norm

∥ W ∥

can drift, pushing

γ

away — which is why normalization (BatchNorm or a self-normalizing SELU net) matters throughout, not just at start.

Consider a network with a single hidden layer (

L = 2

). How severe is the vanishing/exploding problem?

Barely present — the product has just

L - 1 = 1

factor, so there is almost no compounding; the instability is fundamentally a deep-network phenomenon.

For sigmoid at the extreme input

z \to \pm \infty

, what happens to

σ^{'}

and thus the gradient?

σ^{'} \to 0

in both tails, so the backward gradient through that neuron vanishes entirely — this saturation is exactly what normalization prevents by keeping

z

near

0

If every weight matrix has spectral norm slightly below

1

(say

0.9

) with ReLU, is the net safe?

No — with

γ \approx 0.9

γ^{L - 1}

still decays exponentially for large

L

(e.g.

0. 9^{50} \approx 0.005

), so deep enough nets still vanish; "slightly below

1

" is not "

\approx 1

Can the per-link factor effectively flip the gradient's sign, and does that change the vanish/explode verdict?

The bound uses

γ \geq 0

(a spectral norm, always non-negative), which controls only magnitude; the actual factor

(W^{(l)})^{⊤} diag (σ^{'})

is a matrix that can rotate or reflect the gradient vector, so individual coordinates may change sign link to link — but the magnitude bound

γ^{L - 1}

still governs vanish/explode, so sign flips affect direction, never the verdict.

Does a self-normalizing activation (SELU) remove the need for careful init entirely?

No — SELU's fixed-point behavior only holds under its matching "LeCun-normal" init and when layers stay dense; break those assumptions and the activation drifts, so init still matters.

Recall One-line summary of every trap

Every item reduces to: is the per-layer gain $γ$ (worst-case spectral norm $∥ W ∥$ times $max σ^{'}$ ) close to $1$ , and did depth multiply it $L - 1$ times? Backward gradient vanishes if $γ < 1$ , explodes if $γ > 1$ , stable only near $γ = 1$ .