Vanishing and exploding gradients: Visual walkthrough

Everything below assumes you can multiply numbers. That's it. Every symbol is earned before it's used.

Step 1 — One number, passed backward

WHAT. Imagine a signal — just a length of arrow — being handed from one room to the next. Each room does one thing: it multiplies the arrow's length by a fixed number. Call that number $g$ (for gain). If $g = 0.5$ the arrow leaves half as long; if $g = 2$ it leaves twice as long.

WHY. In a neural network, "passing the signal backward" during Backpropagation is exactly this: the error at one layer becomes the error at the previous layer after being multiplied by some factor. Before we drown in matrices, we look at the simplest possible version — a single number multiplied over and over.

PICTURE. Look at figure below: one arrow enters on the right, passes through three rooms, and each room shrinks it by the same gain $g = 0.5$ .

Figure — Vanishing and exploding gradients

Step 2 — Many rooms: the signal is a product of gains

WHAT. If the arrow goes through $n$ rooms, and each room has gain $g$ , the arrow's final length is the starting length times $g$ multiplied by itself $n$ times: $final = n rooms g \times g \times \dots \times g \times start = g^{n} start .$

Term by term:

$g^{n}$ — the gain $g$ raised to the power $n$ , i.e. multiplied by itself $n$ times. The little raised $n$ counts the rooms.
$start$ — the signal's original length, before any room touched it.

WHY. We use a power ( $g^{n}$ ) and not, say, $g \times n$ , because each room multiplies — it does not add. Multiplying repeatedly is what turns a gentle per-room effect into a violent overall effect. That distinction is the entire lesson: repeated multiplication is explosive or crushing, never gentle.

PICTURE. The figure plots $g^{n}$ against the number of rooms $n$ for three gains. Watch how the curves fan apart.

Step 3 — What is really being multiplied in a network

WHAT. In a real layer the signal isn't multiplied by a bare number $g$ . It's multiplied by two things:

the layer's weight $W$ (how strongly neurons connect), and
the slope of the activation function, written $σ^{'}$ (how much the neuron's output changes when its input nudges).

So the per-layer gain is $γ \approx weight size ∥ W ∥_{2} \times activation slope max σ^{'} .$

Term by term:

$γ$ (Greek letter "gamma") — our real gain, the honest version of $g$ from Steps 1–2.
$∥ W ∥_{2}$ — the spectral (operator) norm of the weight matrix. See the definition box below: it is the largest factor by which $W$ can stretch any input vector. We use this norm — not "average size" — for a precise reason revealed in Step 6.
$σ^{'}$ — the derivative (slope) of the activation function $σ$ . We need the slope because backprop asks "if the input wiggles a little, how much does the output wiggle?" — that ratio is exactly a derivative.
$max σ^{'}$ — the largest slope the activation ever has, our worst-case factor.

WHY the slope? The chain rule of Backpropagation multiplies the incoming error by "output-wiggle per input-wiggle." That's the definition of a derivative. We choose the derivative — not the function value itself — because we're tracking sensitivity, not the signal's raw value.

PICTURE. The figure shows a neuron's activation curve (violet) and its slope (orange). Where the curve is flat, the slope $σ^{'}$ is near zero — a tiny gain. Where the curve is steep, the slope is large.

Step 4 — The chain rule stacks these gains

WHAT. Backprop connects the error at layer $l$ to the error at the next layer $l + 1$ : $δ^{(l)} = (W^{(l + 1)})^{⊤} δ^{(l + 1)} ⊙ σ^{'} (z^{(l)}) .$

Term by term:

$δ^{(l)}$ — the error signal at layer $l$ (our arrow, now written properly). The superscript $(l)$ just labels which layer.
$(W^{(l + 1)})^{⊤}$ — the next layer's weights, transposed (flipped) so they push the signal backward instead of forward. (Transposing does not change the spectral norm: $∥ W^{⊤} ∥_{2} = ∥ W ∥_{2}$ .)
$δ^{(l + 1)}$ — the error one layer closer to the output; it arrives first and we pull it back.
$⊙$ — element-wise multiply: multiply matching entries, one for each neuron.
$σ^{'} (z^{(l)})$ — the activation slope at this layer, our per-neuron gain from Step 3.

WHY. Layer $l$ can only affect the loss through layer $l + 1$ (its output is the next layer's input). So to find its blame, we take $l + 1$ 's blame and route it back — pulling through the weights, then scaling by the local slope. Each backward hop = one multiplication by (weight × slope) = one factor of $γ$ .

PICTURE. Each arrow-hop between layers is one multiplication. Count the hops.

Step 5 — Unroll all the way: $L - 1$ factors

WHAT. Apply Step 4 repeatedly, from the last layer $L$ down to the first layer $1$ : $δ^{(1)} = (\prod_{l = 2}^{L} (W^{(l)})^{⊤} diag (σ^{'} (z^{(l - 1)}))) δ^{(L)} .$

Term by term:

$δ^{(1)}$ — the error at the first layer (the one that learns basic features; the one we fear will freeze).
$\prod_{l = 2}^{L}$ — the product symbol: multiply everything inside for $l = 2, 3, \dots, L$ .
$diag (σ^{'})$ — the slopes arranged on a diagonal so they multiply cleanly; think "one gain per neuron."
$δ^{(L)}$ — the error at the last layer, where backprop starts.

Going from layer $L$ to layer $1$ crosses $L - 1$ links, so the product has $L - 1$ factors. Each factor is roughly $γ$ .

WHY. This is Step 2 in disguise: many rooms, one gain each. The network's depth $L$ is the number of rooms.

PICTURE. The full backward chain, with each of the $L - 1$ links labelled by a $γ$ factor.

Step 6 — Take the size: $γ^{L - 1}$ appears

WHAT. Measure the size of both sides using the spectral norm from Step 3. The key tool is submultiplicativity: for the spectral norm, the size of a product is at most the product of the sizes, $∥ A B ∥_{2} \leq ∥ A ∥_{2} ∥ B ∥_{2} and ∥ A v ∥ \leq ∥ A ∥_{2} ∥ v ∥.$ This holds by the very definition of $∥ A ∥_{2}$ as the worst-case stretch: $A$ can never stretch $B v$ by more than its worst-case factor $∥ A ∥_{2}$ . Applying it to all $L - 1$ factors, and using $max σ^{'}$ to bound each $diag (σ^{'})$ block: $δ^{(1)} \leq (\prod_{l = 2}^{L} (W^{(l)})^{⊤}_{2} diag (σ^{'})_{2}) δ^{(L)} \leq γ^{L - 1} δ^{(L)} .$

Term by term:

$δ^{(1)}$ — how large the first layer's error signal is.
$γ^{L - 1}$ — the gain raised to the number of links: exactly the power curve from Step 2.
$δ^{(L)}$ — the last layer's error size, our starting length.

WHY. Submultiplicativity is what lets us collapse a messy matrix product into a single power of one number. All the structure boils down to one power of $γ$ — precisely because the spectral norm never lets a product amplify more than the factors would separately.

PICTURE. The three fates return — now on the axis of network depth $L$ .

Formula From error signal to actual gradient

The gradient we descend on is not $δ^{(1)}$ alone. For a weight matrix, the chain rule gives $\nabla_{W^{(1)}} L = δ^{(1)} (a^{(0)})^{⊤}, so \nabla_{W^{(1)}} L \leq δ^{(1)} \cdot a^{(0)} .$ Here $a^{(0)}$ is the layer's input activations; $∥ a^{(0)} ∥$ is their size. So the gradient carries an extra activation-size factor on top of $δ^{(1)}$ . In practice normalization and Weight Initialization keep $∥ a^{(0)} ∥$ roughly constant across layers, so it does not grow or shrink with depth — it is a bounded constant we can normalize away. That is why the depth dependence lives entirely in $γ^{L - 1}$ : $\nabla_{W^{(1)}} L \sim γ^{L - 1}, γ \approx ∥ W ∥_{2} \cdot max σ^{'} .$ Stability demands $γ \approx 1$ . Every fix — Weight Initialization, ReLU, Batch Normalization, skip connections, gates — exists to nudge $γ$ toward $1$ .

Step 7 — Edge and degenerate cases

WHAT & WHY. We must show every scenario, or the reader hits one we skipped.

$γ = 0$ (dead neuron). If a neuron's slope is $0$ (e.g. ReLU on a negative input), its factor is $0$ . One zero factor kills the whole product — that neuron's gradient path is frozen. This is the "dead ReLU" problem: ReLU avoids shrinking but can zero out entirely.
$γ$ exactly $1$ . The flat line: signal passes untouched no matter how deep. This is the design target, achieved by He/Xavier init and skip connections.
Sigmoid ceiling $γ \leq 0.25$ . Sigmoid's slope never exceeds $0.25$ , so even with $∥ W ∥_{2} = 1$ the gain caps at $0.25$ — guaranteed vanishing in deep nets. That single number doomed pre-2010 deep networks.
Negative per-layer gain ( $γ < 0$ ). A weight can be negative, so a real per-layer factor can be negative. For the magnitude bounds above nothing changes — we took $∣ γ ∣$ via norms, and $∥ W ∥_{2} \geq 0$ always. But the sign flips: an odd number of negative factors flips the sign of the propagated error, so a neuron may receive a gradient pointing the "wrong way" relative to a naive expectation. This does not cause vanishing or exploding by itself, but it makes the sign of the update oscillate across layers and can slow or destabilise Gradient Descent. Takeaway: size is governed by $∣ γ ∣^{L - 1}$ ; sign is governed by the parity of the negative factors.
Mixed gains (why the geometric mean decides it). Real layers have different gains $γ_{2}, \dots, γ_{L}$ . The product is $P = \prod_{l = 2}^{L} γ_{l}$ . Define the geometric mean $\overset{γ}{ˉ} = (\prod_{l} ∣ γ_{l} ∣)^{1/ (L - 1)}$ — the single equal factor that would give the same product. By construction $∣ P ∣ = \overset{γ}{ˉ}^{L - 1}$ , so the product behaves exactly like a uniform gain of $\overset{γ}{ˉ}$ . Hence $\overset{γ}{ˉ} < 1 \Rightarrow$ vanish, $\overset{γ}{ˉ} > 1 \Rightarrow$ explode, $\overset{γ}{ˉ} = 1 \Rightarrow$ preserved. The geometric (not arithmetic) mean is the right average precisely because the factors multiply: taking $lo g ∣ P ∣ = \sum_{l} lo g ∣ γ_{l} ∣$ turns the product into a sum, and the average of the logs is the log of the geometric mean.

PICTURE. Four mini-panels: dead ( $γ = 0$ ), sigmoid ceiling ( $γ = 0.25$ ), preserved ( $γ = 1$ ), exploding ( $γ = 1.5$ ).

The one-picture summary

The whole story is one arrow, multiplied $L - 1$ times by a gain $γ$ . Below $1$ (in size) it dies; above $1$ it explodes; at $1$ it survives; and if any factor is negative, its sign flips. Everything else is engineering to hit that $1$ .

Recall Feynman retelling — say it back in plain words

Picture handing a note down a long corridor of rooms. Each room photocopies the note at some zoom setting $γ$ . If every room zooms out to 50%, after ten rooms the note is a speck you can't read — that's a vanishing gradient, and the first room (the early layer) never learns what to fix. If every room zooms to 150%, after twenty rooms the note is a wall-sized blur — that's an exploding gradient, and training blows up (NaN). The only safe zoom is 100% ( $γ = 1$ ), where the note arrives readable no matter how long the corridor. Now: the zoom setting is really the weight's worst-case stretch $∥ W ∥_{2}$ (its spectral norm) times the activation's slope $σ^{'}$ . The reason many rooms multiply cleanly into one power is submultiplicativity — no matrix can amplify more than its worst-case stretch, so the whole chain is bounded by the product of stretches. When rooms have different zooms, the outcome is set by the geometric mean of the zooms, because multiplying-then-averaging is the same as averaging-the-logs. A negative zoom is like a photocopier that also flips the note upside-down: it doesn't change the size but flips the sign, and an odd number of flips confuses the update direction. Sigmoid's slope caps at $0.25$ , forcing a permanent zoom-out — that's why sigmoid vanishes. ReLU keeps slope $1$ , and Weight Initialization tunes $∥ W ∥_{2}$ so the product stays near $1$ . Skip connections literally add a "photocopy at 100%" side-path so the note always has one clean copy. That's the entire theory of vanishing and exploding gradients — one number, raised to the depth.

Recall Quick self-check

Why a power $γ^{L - 1}$ and not a product $γ \cdot (L - 1)$ ? ::: Because layers multiply the signal, not add to it; repeated multiplication is a power. What two ingredients make up $γ$ ? ::: The spectral norm $∥ W ∥_{2}$ (worst-case stretch of $W$ ) and the activation slope $max σ^{'}$ . Which norm property lets us bound a matrix product by a power? ::: Submultiplicativity: $∥ A B ∥_{2} \leq ∥ A ∥_{2} ∥ B ∥_{2}$ . What extra factor sits between $δ^{(1)}$ and the actual weight gradient? ::: The input-activation size $∥ a^{(0)} ∥$ , kept roughly constant by normalization so it drops out of the depth scaling. For unequal per-layer gains, which average decides vanish vs explode? ::: The geometric mean $\overset{γ}{ˉ}$ ; $\overset{γ}{ˉ} < 1$ vanish, $> 1$ explode. What does a negative per-layer gain do? ::: Leaves the magnitude bound unchanged but flips the sign of the propagated error (parity of negative factors). What does a single $γ = 0$ factor do to the product? ::: Kills it entirely — that path's gradient is zero (dead neuron). What is the one safe value of $∣ γ ∣$ ? ::: $∣ γ ∣ \approx 1$ : the signal is preserved at any depth.