Worked examples — Vanishing and exploding gradients

This page is the "grab-bag of cases" companion to Vanishing and exploding gradients. The parent note built the one master equation; here we exercise it against every situation you could meet: shrinking, growing, borderline, zero, degenerate, real-world, and an exam twist. Guess each answer before reading the steps.

Recall The one formula we keep reusing

For an $L$-layer network the early-layer gradient scales like $\Vert {\nabla }_{W^{(1)}}L\Vert \sim {\gamma }^{L-1},\gamma \approx \underset{\text{weight size (spectral norm)}}{\underbrace{\Vert W\Vert }}\cdot \underset{\text{activation slope}}{\underbrace{\max {\sigma }^{\prime }}}.$ Here $\gamma$ (Greek letter "gamma") is just one number: how much the error signal gets multiplied by each time it crosses one layer. If $\gamma >1$ the signal grows, if $\gamma <1$ it shrinks, if $\gamma =1$ it is preserved. $L$ is the number of layers, so $L-1$ is the number of layer-to-layer links the signal must cross. That is the whole engine. Every example below just plugs numbers into ${\gamma }^{L-1}$.

---

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The examples that follow each fill one cell — together they cover the whole grid.

Cell	What makes it distinct	Regime	Example
A	$\gamma <1$ (small gain)	Vanish	Ex 1
B	$\gamma >1$ (large gain)	Explode	Ex 2
C	$\gamma \approx 1$ (balanced by init)	Stable	Ex 3
D	$\gamma =1$ exactly, limit as $L\to \infty$	Boundary	Ex 4
E	Degenerate: ${\sigma }^{\prime }=0$ (dead ReLU / saturation)	Zero gradient	Ex 5
F	Choosing init to force $\gamma =1$ (solve for $\Vert W\Vert$)	Design	Ex 6
G	Real-world: NaN loss, diagnose & clip	Explode + fix	Ex 7
H	Exam twist: mixed layers, per-link product	Mixed	Ex 8

The figure below plots the gradient scaling ${\gamma }^{L-1}$ (vertical axis, on a log scale) against the number of layers $L$ (horizontal axis), for four values of the per-layer gain $\gamma$. Read it like this: the red curve ($\gamma =0.2$, Cell A) dives straight down — vanishing; the orange curve ($\gamma =1.4$, Cell B) climbs straight up — exploding; the green horizontal line ($\gamma =1.0$, Cells C/D) stays flat at $1$ forever — perfectly preserved; the blue curve ($\gamma =0.99$, Cell D) looks flat at first but slowly bends downward, showing that even a hair below $1$ eventually vanishes at great depth. The dashed gray line marks the "signal preserved" level of $1$.

---

Cell A — the classic vanish

1. Compute the per-layer gain. $\gamma =\Vert W\Vert \cdot \max {\sigma }^{\prime }=0.8\times 0.25=\mathrm{0.2.}$ Why this step? $\gamma$ is the single number that decides everything; we always find it first. Here it is well below $1$, so we already know we are vanishing.
2. Count the links. $L=15$ layers $\Rightarrow L-1=14$ links between them. Why this step? The signal is multiplied once per link, not per layer — off-by-one here is the most common error.
3. Raise to the power. ${\gamma }^{L-1}=0.2^{14}\approx 1.64\times 10^{-10}.$ Why this step? Each of the 14 links multiplies by $0.2$; compounding $0.2$ fourteen times is what makes it astronomically small.

Verify: $0.2=1/5$, so $0.2^{14}=5^{-14}$. Since $5^{14}\approx 6.1\times 10^9$, the reciprocal is $\approx 1.6\times 10^{-10}$ — matches. The layer-1 gradient is ten billion times smaller. It is frozen.

---

Cell B — the classic explode

1. Find $\gamma$. Given directly: $\gamma =1.4>1$, so we expect growth. Why this step? Sign of $\gamma -1$ tells the regime before any arithmetic.
2. Links: $L-1=19$. Why this step? Same link-counting rule; a 20-layer net has 19 gaps.
3. Compound. $1.4^{19}\approx 597.$ Why this step? A "modest" $1.4$ per link compounds to nearly six hundred times across 19 links — this is why "it looks close to 1" is a trap.

Verify: use logarithms as an independent check. $\ln (1.4)\approx 0.33647$, times $19$ gives $6.393$. Exponentiating back, $e^{6.393}\approx 597$ — the same value as the direct power $1.4^{19}$. So the gradient is scaled by roughly $600\times$: weights update wildly, loss $\to$ NaN. See Gradient Descent for why an oversized step overshoots.

---

Cell C — balanced by good init

1. Gain. He init restores the variance ReLU throws away, giving $\gamma \approx 1$. Why this step? This is the design goal from the parent note — every good init exists to push $\gamma$ to $1$.
2. Links: $L-1=49$.
3. Power. $1.0^{49}=\mathrm{1.0.}$ Why this step? $1$ raised to any power is $1$ — depth becomes free.

Verify: $1^{49}=1$ trivially. Neither vanish nor explode; a 50-layer net is trainable. Contrast Ex 1: same idea, different $\gamma$, opposite fate.

Note — a related but distinct fix. Careful init is one way to reach $\gamma \approx 1$. Residual (skip) connections achieve stable gradients by a different mechanism: the skip makes each layer compute $a^{(l)}=a^{(l-1)}+f(a^{(l-1)})$, so its local sensitivity is $I+(\dots )$. The identity $I$ gives the gradient a guaranteed additive "pass-through" path of gain exactly $1$, even if the learned branch $f$ has a poorly-scaled gain. That is different from tuning the multiplicative gain of a single dense layer to $1$; ResNets add an extra route rather than fix the existing one. Both are used together in practice.

---

Cell D — the exact boundary and its limit

1. Exact boundary. ${\lim }_{L\to \infty }{1}^{L-1}=1$. Why this step? At exactly $1$ the product is a constant — the only value that survives infinite depth.
2. Just below. ${\lim }_{L\to \infty }0.99^{L-1}=0$. Why this step? Any base strictly less than $1$, raised to ever-larger powers, still collapses to $0$ — just more slowly. The boundary is a knife-edge.
3. How slow? Solve $0.99^{L-1}=0.5$ for the "half-life in layers": $L-1=\frac{\ln 0.5}{\ln 0.99}\approx 69$. Why this step? It quantifies "slow": every ~69 layers the signal halves. Deep enough, even $0.99$ vanishes.

Verify: $\ln 0.5=-0.6931$, $\ln 0.99=-0.01005$; ratio $\approx 68.97\approx 69$. So $0.99$ is not safe at extreme depth — only exactly $1$ is. This is why practitioners chase $\gamma =1$, not "close to $1$".

---

Cell E — the degenerate zero

1. Insert the zero factor. The product of per-link gains includes one factor with slope $0$: ${\gamma }^{(\text{that link})}=\Vert W\Vert \cdot 0=0.$ Why this step? A single zero in a product zeroes the entire product — unlike a sum.
2. Propagate. $\prod =(\dots )\times 0\times (\dots )=0.$ Why this step? Every layer upstream of the dead one gets multiplied through this $0$, so they all receive exactly zero gradient.
3. Interpret. Those upstream layers cannot learn — the dead neuron severs the signal path. Why this step? This is the "dead ReLU" failure the parent's third mistake warns about; slope-$1$ does not save you if the neuron is on the flat side.

Verify: $x\cdot 0=0$ for any finite $x$, so the whole product is $0$ regardless of the other weights. Fixes: LeakyReLU (small nonzero slope) or Batch Normalization to keep pre-activations off the flat zone.

---

Cell F — design: solve for the init

Here $n_{in}$ = the number of inputs into a layer (the length of the vector $a^{(l-1)}$ it receives) and $n_{out}$ = the number of outputs it produces (the length of $a^{(l)}$). For a square hidden layer $n_{in}=n_{out}=n$.

1. Write the target. Require $\gamma =\Vert W\Vert \cdot \max {\sigma }^{\prime }=1$. Why this step? We are inverting the master formula — solving for the design variable instead of predicting an outcome.
2. Plug the slope. $\max {\sigma }^{\prime }=1$, so $\Vert W\Vert \cdot 1=1\Rightarrow \Vert W\Vert =1$. Why this step? At $z\approx 0$ tanh does not shrink the signal, so the spectral norm of the weights alone must be $1$.
3. How an init recipe delivers $\Vert W\Vert \approx 1$. We cannot set the spectral norm directly with a random draw, but we can control the entry-wise variance. Xavier sets $\mathrm{Var}(W_{ij})=\frac{2}{n_{in}+n_{out}}$; for a square layer $n_{in}=n_{out}=n$ this is $\frac{1}{n}$. Why this step? Random-matrix theory says: for an $n\times n$ matrix whose entries are independent with variance $\frac{1}{n}$, the largest singular value concentrates near ${\sigma }_{\max }\approx 2\sqrt{n}\cdot \sqrt{\frac{1}{n}}=2$ — of order $1$, not exploding, but not exactly $1$ either. Key caveat: entry-wise variance only controls $\Vert W\Vert$ indirectly and approximately. To pin $\Vert W\Vert =1$ exactly you would need spectral normalization (divide $W$ by its measured largest singular value). Xavier/He get $\Vert W\Vert$ into the safe order-1 band; they do not guarantee the exact value.

Verify: the goal $\Vert W\Vert =1$ follows from $\Vert W\Vert \cdot 1=1$ (Step 2, exact algebra). The Xavier recipe with $n=100$ gives $\mathrm{Var}=2/200=0.01$, std $=0.1$, and predicted spectral norm $\approx 2\sqrt{100}\cdot 0.1=2$ — order $1$ as claimed, and honestly not exactly $1$, which is why spectral normalization exists when you need $\gamma$ locked precisely.

---

Cell G — real-world diagnosis

1. Recognise the regime. Loss rising super-linearly then NaN ⇒ exploding gradients ($\gamma >1$), matching Ex 2's pattern. Why this step? Diagnosis before treatment — vanishing would show flat loss, not blow-up.
2. Apply clipping. The rule is $g\leftarrow g\cdot \frac{\text{threshold}}{\Vert g\Vert }$ when $\Vert g\Vert >\text{threshold}$. New norm: $\Vert g_{\text{clipped}}\Vert =\Vert g\Vert \cdot \frac{10}{\Vert g\Vert }=10.$ Why this step? By construction clipping sets the norm exactly to the threshold; the direction (unit vector) is untouched.
3. Shrink factor. $\frac{\Vert g\Vert }{\text{threshold}}=\frac{5000}{10}=500.$ Why this step? It quantifies how violent the explosion was — a $500\times$ overshoot, tamed to a safe step.

Verify: $g\cdot \frac{10}{5000}$ has norm $5000\cdot \frac{10}{5000}=10$. ✓ Direction preserved since we scaled by a positive scalar. See Backpropagation for where in the pipeline the clip is inserted (after the backward pass, before the optimizer step).

---

Cell H — exam twist: mixed layers

1. Multiply the actual factors. The chain rule multiplies them (see Cell E's box), so $\prod =1.2\times 0.5\times 2.0\times 0.9\times 0.5=\mathrm{0.54.}$ Why this step? When layers differ we cannot use one ${\gamma }^{L-1}$; we must take the true product of per-link gains. Working left to right: $1.2\times 0.5=0.6$, $\times 2.0=1.2$, $\times 0.9=1.08$, $\times 0.5=0.54$.
2. Compare to a naive average. Arithmetic mean of gains $=(1.2+0.5+2.0+0.9+0.5)/5=1.02$, which raised to the 5th power gives $1.02^5\approx 1.10$ — predicting growth. Why this step? Shows the trap: the arithmetic mean over-predicts. Products care about the geometric mean, not the arithmetic one.
3. Check with geometric mean. $\sqrt[5]{0.54}\approx 0.885$, and $0.885^5=0.54$. Why this step? Confirms the correct summary statistic for compounding gains is the geometric mean; it reproduces the true product exactly, while the arithmetic mean does not.

Verify: direct product $1.2\cdot 0.5\cdot 2.0\cdot 0.9\cdot 0.5=0.54$. ✓ Actual behaviour: mild vanishing ($0.54<1$), the opposite of the naive-average prediction of growth. Exam lesson: multiply the real factors, never average them.

---

Active recall

Recall Test yourself

What is $\sigma$ on this page, and what is ${\sigma }^{\prime }$? ::: $\sigma$ is the activation function (sigmoid/tanh/ReLU); ${\sigma }^{\prime }$ is its derivative — the slope backprop multiplies by at each neuron. Which norm does $\Vert W\Vert$ mean on this page, and why that one? ::: The spectral norm (largest singular value); it is exactly the worst-case factor by which $W^{\top }$ can grow or shrink the error vector. Which single number decides vanish vs explode, and what value is the goal? ::: The per-layer gain $\gamma \approx \Vert W\Vert \cdot \max {\sigma }^{\prime }$; goal $\gamma =1$. For $\gamma =0.2$, $L=15$, order of magnitude of shrink? ::: $0.2^{14}\approx 10^{-10}$. For $\gamma =1.4$, $L=20$, the scaling factor? ::: $1.4^{19}\approx 597$. Why is $\gamma =0.99$ still unsafe at extreme depth? ::: Any base $<1$ still tends to $0$ as $L\to \infty$; only exactly $1$ survives (half-life ~69 layers). One dead-ReLU layer (${\sigma }^{\prime }=0$) affects which layers? ::: All layers upstream of it — a zero factor zeroes the whole product. Does Xavier/He init guarantee $\Vert W\Vert =1$ exactly? ::: No — it puts $\Vert W\Vert$ in the order-1 band via entry-wise variance; exact control needs spectral normalization. Clipping $\Vert g\Vert =5000$ to threshold $10$ gives what norm and shrink factor? ::: Norm $10$; shrink factor $500$; direction unchanged. For mixed per-link gains, which mean summarises the product? ::: The geometric mean, not the arithmetic mean.