Vanishing and exploding gradients

What is actually happening

WHAT is multiplied? In a plain feed-forward net, each layer computes $a^{(l)} = \sigma\!\big(z^{(l)}\big), \qquad z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}.$ Backprop pushes the error $\delta^{(l)} = \partial L/\partial z^{(l)}$ backward using the chain rule.

Derivation from first principles

Let the "error at layer $l$" be $\delta^{(l)} = \dfrac{\partial L}{\partial z^{(l)}}$.

Step 1 — link consecutive layers. Since $z^{(l+1)} = W^{(l+1)} \sigma(z^{(l)}) + b^{(l+1)}$, the chain rule gives $\delta^{(l)} = \Big(W^{(l+1)}\Big)^{\!\top}\,\delta^{(l+1)} \;\odot\; \sigma'\!\big(z^{(l)}\big).$ Why this step? Error at layer $l$ only reaches the loss through layer $l+1$, so we take $l+1$'s error, pull it back through the weights $(W^{(l+1)})^\top$, and scale by how much this neuron's output changes with its input, $\sigma'(z^{(l)})$.

Step 2 — unroll across $L$ layers. Applying Step 1 repeatedly from the last layer down to layer 1: $\delta^{(1)} = \left(\prod_{l=2}^{L} \Big(W^{(l)}\Big)^{\!\top} \operatorname{diag}\!\big(\sigma'(z^{(l-1)})\big)\right)\delta^{(L)}.$ Why this step? Every layer inserts one more multiplicative factor. Going from layer $L$ down to layer $1$ crosses $L-1$ links, so there are $L-1$ factors in the product.

Step 3 — bound the magnitude (the key insight). Take norms. If every factor satisfies $\big\|(W^{(l)})^\top \operatorname{diag}(\sigma')\big\| \le \gamma$, then $\big\|\delta^{(1)}\big\| \;\le\; \gamma^{\,L-1}\,\big\|\delta^{(L)}\big\|.$ Why this step? Norm of a product $\le$ product of norms, and there are $L-1$ factors. Now the behaviour is obvious:

• $\gamma < 1 \Rightarrow \gamma^{L-1}\to 0$ → vanish.
• $\gamma > 1 \Rightarrow \gamma^{L-1}\to \infty$ → explode.
• $\gamma \approx 1 \Rightarrow$ signal preserved. This is the whole design goal.

Why activation choice matters

$\sigma_{\text{sigmoid}}'(z) = \sigma(z)\big(1-\sigma(z)\big) \le \tfrac14,\qquad \tanh'(z) = 1 - \tanh^2(z) \le 1.$

Why sigmoid vanishes: its derivative never exceeds $0.25$. So even with perfectly scaled weights, $\gamma \le 0.25 \cdot \|W\|$; through many layers the $0.25$ factors compound to near zero. ReLU has derivative $1$ for positive inputs, so it does not shrink the signal (its factor is exactly $1$), which is why it dominates deep nets.

Fixes (the 80/20 that solves most cases)

1. Weight initialization — scale $W$ so that variance is preserved.
   • Xavier/Glorot (for tanh/sigmoid): $\operatorname{Var}(W)=\dfrac{2}{n_{in}+n_{out}}$.
   • He (for ReLU): $\operatorname{Var}(W)=\dfrac{2}{n_{in}}$ (extra $\times 2$ because ReLU zeroes half the inputs). Why: keeps $\|W\|\cdot\max\sigma' \approx 1$ at the start.
2. Non-saturating activations — ReLU / LeakyReLU (slope $1$).
3. Batch/Layer Normalization — re-centers/scales $z^{(l)}$ each layer so $\sigma'$ stays in a healthy range.
4. Residual (skip) connections — $a^{(l)} = a^{(l-1)} + f(a^{(l-1)})$ gives the Jacobian an $I + (\dots)$ term, so the gradient always has a "$1$" path (factor $\approx 1$).
5. Gradient clipping (for exploding) — rescale $g \leftarrow g\cdot \dfrac{\text{threshold}}{\|g\|}$ if $\|g\|>$ threshold.
6. LSTM/GRU gates (for RNNs) — additive cell state gives a near-$1$ recurrence.

Worked examples

Common mistakes

Active recall

Connections

• Backpropagation — the chain-rule product that creates the problem.
• Activation Functions — sigmoid/tanh/ReLU and their derivatives.
• Weight Initialization — Xavier & He as $\gamma$-control.
• Batch Normalization — keeps pre-activations well-scaled.
• Residual Networks (ResNet) — identity paths for gradient flow.
• LSTM and GRU — additive cell state fixing the recurrent version.
• Gradient Descent — how these gradients get used to update weights.

Concept Map

Hinglish (regional understanding)