Exercises — Vanishing and exploding gradients

• Vanishing: $\gamma <1$, so ${\gamma }^{L-1}\to 0$ as depth $L$ grows — the early-layer gradient shrinks toward zero and those layers stop updating.
• Exploding: $\gamma >1$, so ${\gamma }^{L-1}\to \infty$ — the early-layer gradient blows up, weight updates become huge, and the loss can go to NaN. WHAT it looks like: on the s01 figure, vanishing is the curve hugging the floor on the left; exploding is the curve shooting off the top on the right.

${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$. This is a downward parabola in the variable $\sigma \in (0,1)$, maximized when $\sigma =0.5$. That gives ${\sigma }^{\prime }=0.5\times 0.5=\boxed{0.25}$, occurring at $z=0$ (because $\sigma (0)=0.5$).

$L-1$ factors. Going from the last layer $L$ back to layer $1$ crosses $L-1$ layer-to-layer links, and each link inserts exactly one factor $(W^{(l)}{)}^{\top }\mathrm{diag}({\sigma }^{\prime })$. Hence the bound ${\gamma }^{L-1}$.

$\gamma =\Vert W\Vert \cdot \max {\sigma }^{\prime }=1\times 0.25=0.25$. With $L-1=9$ links: ${\gamma }^9=0.25^9=\frac{1}{4^9}=\frac{1}{262144}\approx 3.8\times 10^{-6}.$ The first layer receives a gradient roughly four million times smaller than the last — effectively frozen.

$L-1=20$, so $1.5^{20}\approx 3325$. The gradient is amplified ~3300× through the stack. Updates overshoot, weights swing wildly, and loss typically diverges to NaN. Fix: smaller init or gradient clipping.

$\mathrm{Var}(W)=2/512=0.00390625$. Standard deviation: ${\sigma }_W=\sqrt{2/512}=\sqrt{0.00390625}=\mathrm{0.0625.}$ So each weight is drawn from a distribution with std $\approx 0.0625$. The $\times 2$ compensates for ReLU zeroing about half the activations (see Weight Initialization).

The ratio of norms is ${\gamma }^{29}=\frac{\Vert {\delta }^{(1)}\Vert }{\Vert {\delta }^{(30)}\Vert }=\frac{10^{-6}}{1}=10^{-6}$. Take logs: $29\ln \gamma =\ln (10^{-6})=-6\ln 10$, so $\ln \gamma =\frac{-6\ln 10}{29}\approx -0.4763,\gamma =e^{-0.4763}\approx \mathrm{0.621.}$ A per-layer gain of about $0.62$ — well below $1$ — fully explains the six-order-of-magnitude collapse.

Arithmetically $\gamma =\Vert W\Vert \cdot \max {\sigma }^{\prime }=4\times 0.25=1$ — the maximum-slope case is balanced. But $\max {\sigma }^{\prime }=0.25$ only holds at $z=0$. Large weights push pre-activations $z$ to big magnitudes, where ${\sigma }^{\prime }\to 0$ (saturation), so the actual per-neuron slope collapses far below $0.25$. Net effect: $\gamma$ still drops below 1 for most neurons, and you have added saturation. Inflating $\Vert W\Vert$ to fix vanishing sigmoids trades one failure for another; the real fix is normalization or a non-saturating activation.

$\frac{\partial a^{(l)}}{\partial a^{(l-1)}}=I+\frac{\partial f}{\partial a^{(l-1)}}.$ The identity $I$ guarantees a "gain-1 path": even if $\partial f/\partial a^{(l-1)}$ has tiny norm, the product across layers still contains $I\cdot I\cdots I=I$, so the gradient cannot vanish to zero along that path. This is exactly why ResNets train at 50+ layers.

Choice: ReLU activations + He initialization + optional Batch Norm.

• ReLU gives $\max {\sigma }^{\prime }=1$ (slope 1 for positives), so the activation factor is $1$, not $0.25$ — removes the sigmoid's built-in shrink.
• He init sets $\mathrm{Var}(W)=2/n_{in}$ so that $\Vert W\Vert$ contributes a factor $\approx 1$ after ReLU has zeroed half the units; the $\times 2$ restores the halved variance. Net $\gamma \approx 1\times 1=1$.
• Then ${\gamma }^{40-1}=1^{39}=1$: depth stops mattering for signal size.
• Batch Norm keeps $z$ centered so ReLU units don't collectively die, protecting the realized slope.

Unrolling 100 steps gives $99$ links (or $\approx 100$; the exponential is the point): $0.9^{99}\approx e^{99\ln 0.9}=e^{99\times (-0.10536)}=e^{-10.43}\approx 2.96\times 10^{-5}.$ The gradient shrinks by ~$10^{-5}$ across time — early time steps learn almost nothing. Fix: LSTM/GRU gates, whose additive cell-state update gives a near-$1$ recurrence (an $I$-like path through time), just like residual connections do through depth.

Clipping rescales $g\leftarrow g\cdot \frac{\tau }{\Vert g\Vert }$ only when $\Vert g\Vert >\tau$ — it caps large gradients. Exploding means $\gamma >1$ produced a huge norm, which clipping tames. Vanishing produces a tiny norm ($\gamma <1$), never exceeding the threshold, so clipping never triggers; worse, it cannot manufacture signal that has already collapsed to $\sim 10^{-6}$. Vanishing needs structural fixes (init/activation/skip), not post-hoc rescaling.

(a) From layer 50 to layer 10 is $40$ links. Norm ratio $=0.014/1.0=0.014$. ${\gamma }^{40}=0.014\Rightarrow 40\ln \gamma =\ln (0.014)=-4.269\Rightarrow \ln \gamma =-0.10673,\gamma =e^{-0.10673}\approx \mathrm{0.899.}$ A per-link gain $\approx 0.90$ — consistently below 1. (Sanity check: $0.9^{10}\approx 0.349$ predicts layer 40 $\approx 0.35$ ✓; $0.9^{40}\approx 0.0148$ ✓.) (b) Classic vanishing gradient: later layers learn (near-1 gradients), early layers are near-frozen ($0.014$), so the loss plateaus high — this is the parent note's "high loss can coexist with vanishing gradients." (c) Priority fixes: (1) add residual/skip connections — the $I$ path forces per-link gain $\approx 1$ regardless of depth (biggest structural lever for 50 layers). (2) add Batch Normalization to keep activations in the healthy-slope region so realized $\gamma$ rises toward 1. (Re-init to He would also help if not already done.)

Write each backward link as a linear map $M^{(l)}=(W^{(l+1)}{)}^{\top }\mathrm{diag}({\sigma }^{\prime }(z^{(l)}))$, so ${\delta }^{(l)}=M^{(l)}{\delta }^{(l+1)}$. Unrolling from $L$ to $1$: ${\delta }^{(1)}=\left({\prod }_{l=1}^{L-1}{M}^{(l)}\right){\delta }^{(L)}.$ Take the norm and use (i) sub-multiplicativity of the operator norm, $\Vert AB\Vert \le \Vert A\Vert \Vert B\Vert$: $\Vert {\delta }^{(1)}\Vert \le ({\prod }_{l=1}^{L-1}\Vert M^{(l)}\Vert )\Vert {\delta }^{(L)}\Vert .$ Then use (ii) the uniform per-link bound $\Vert M^{(l)}\Vert \le \gamma$ for every link. There are exactly $L-1$ links, so: $\Vert {\delta }^{(1)}\Vert \le {\gamma }^{L-1}\Vert {\delta }^{(L)}\Vert .■$ The two inequalities are: (i) norm of a product $\le$ product of norms, and (ii) each factor's norm $\le \gamma$. See Backpropagation for the recurrence itself.