Foundations — Vanishing and exploding gradients

This page assumes you have seen none of the notation in the parent note. We build every symbol from the ground up, in the order the story needs them. By the end, the equation $\Vert {\delta }^{(1)}\Vert \le {\gamma }^{L-1}\Vert {\delta }^{(L)}\Vert$ should read like a plain English sentence.

0 — What a layer is, drawn

Before any symbol: a neural network is a stack of layers. Each layer takes a list of numbers, mixes them, bends them, and hands the result to the next layer. "Deep" just means many layers stacked.

Why we need this: the whole topic is about what happens as a signal travels from layer $L$ back to layer $1$. Without a way to number layers we cannot even ask "how many multiplications happened."

1 — Vectors, and the symbol $a^{(l)}$

Inside a layer, the numbers travel in a bunch. A vector is just an ordered list of numbers, like $[0.3,-1.2,0.8]$. Picture it as several parallel wires, each carrying one number.

2 — The weighted sum, and the symbols $W$, $b$, $z$

Each neuron takes the previous layer's numbers, multiplies each by an importance factor, adds them up, and adds a constant. That is a weighted sum.

Why we need $z$ separately from $a$: the multiplication that causes vanishing/exploding happens partly through $W$ and partly through the bending we apply to $z$. We must name the number before bending to talk about both.

The tool used here is matrix multiplication — not ordinary multiplication — precisely because a layer takes many inputs to many outputs at once. A matrix is the compact way to write "every output is a weighted combination of every input." See Weight Initialization for how the size of the numbers in $W$ is chosen.

3 — The activation function $\sigma$ and its slope ${\sigma }^{\prime }$

After the weighted sum we bend the value with a function called $\sigma$ (Greek "sigma"). Bending is what lets a network learn curves, not just straight lines.

The tool here is the derivative, chosen because backpropagation literally asks "rate of change of output per change of input" at every neuron — and that is the definition of a derivative. More on choices in Activation Functions.

4 — The loss $L$ and the gradient $\nabla$

The network's mistake is measured by a single number, the loss. Training means shrinking it.

Why we need this: "the gradient vanishes" means ${\nabla }_{W^{(1)}}L$ is nearly all zeros — layer 1 has no direction to move, so it stops learning. Everything ties back to Gradient Descent, which steps the weights along this gradient.

5 — The error signal ${\delta }^{(l)}$ and the chain rule

Backprop does not recompute everything from scratch at each layer. It carries a running quantity backward.

Each backward step multiplies by a $W^{\top }$ and a ${\sigma }^{\prime }$. Two multipliers, one per link. See Backpropagation for the full derivation of this recursion.

6 — Products, norms, and the gain $\gamma$

Going from layer $L$ all the way to layer 1 chains $L-1$ such steps. Multiplying $L-1$ factors is the source of all the trouble.

The tool here is the exponential ${\gamma }^{L-1}$: whenever the same factor is applied repeatedly, the result is that factor raised to a power. That is why a tiny per-layer bias (say $0.25$ or $1.5$) becomes astronomically small or large over many layers. Fixes like Batch Normalization and Residual Networks (ResNet) exist purely to pin $\gamma$ near $1$; LSTM and GRU do the same for recurrent nets.

Prerequisite map

Equipment checklist