Backpropagation algorithm derivation

WHY does backprop exist?

WHAT problem does it solve? A neural network has millions of weights. To train with gradient descent we need $\frac{\partial L}{\partial w}$ for every weight $w$. Computing each derivative independently by the chain rule would repeat the same sub-expressions billions of times.

WHY it's fast: The gradient of a loss w.r.t. an early-layer weight contains as a factor the gradient w.r.t. later layers. So if we compute later gradients first and cache them, every earlier gradient is cheap. This caching-of-shared-subproblems is exactly dynamic programming over the computational graph.

Setup and notation

Consider layer $l$ (from $1$ to $L$). For layer $l$:

• $W^{(l)}$ = weight matrix, $b^{(l)}$ = bias vector
• $z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$ — the pre-activation
• $a^{(l)} = \sigma(z^{(l)})$ — the activation ($\sigma$ = activation function)
• $a^{(0)} = x$ (input), $a^{(L)} = \hat y$ (output)
• $L(a^{(L)}, y)$ = loss

Deriving the four equations from scratch

Step 1 — Output layer error $\delta^{(L)}$

HOW: By the chain rule, the loss depends on $z^{(L)}$ only through $a^{(L)} = \sigma(z^{(L)})$:

$\delta^{(L)}_j = \frac{\partial L}{\partial z^{(L)}_j} = \frac{\partial L}{\partial a^{(L)}_j}\cdot \frac{\partial a^{(L)}_j}{\partial z^{(L)}_j}$

Why this step? $a^{(L)}_j = \sigma(z^{(L)}_j)$ is the only path from $z^{(L)}_j$ to the loss, so we chain through it.

Since $a^{(L)}_j=\sigma(z^{(L)}_j)$, the second factor is $\sigma'(z^{(L)}_j)$. In vector form:

$\boxed{\;\delta^{(L)} = \nabla_a L \odot \sigma'(z^{(L)})\;}$

where $\odot$ is elementwise product (Hadamard).

Step 2 — Recurrence: $\delta^{(l)}$ from $\delta^{(l+1)}$

WHY a recurrence? $z^{(l)}$ affects the loss only through the next layer's pre-activations $z^{(l+1)}$. So:

$\delta^{(l)}_j = \frac{\partial L}{\partial z^{(l)}_j} = \sum_k \frac{\partial L}{\partial z^{(l+1)}_k}\cdot \frac{\partial z^{(l+1)}_k}{\partial z^{(l)}_j} = \sum_k \delta^{(l+1)}_k \cdot \frac{\partial z^{(l+1)}_k}{\partial z^{(l)}_j}$

Why the sum over $k$? $z^{(l)}_j$ fans out to every neuron $k$ in layer $l+1$; total sensitivity = sum over all those paths (multivariable chain rule).

Now compute $\frac{\partial z^{(l+1)}_k}{\partial z^{(l)}_j}$. Since $z^{(l+1)}_k = \sum_i W^{(l+1)}_{ki}\, a^{(l)}_i + b^{(l+1)}_k$ and $a^{(l)}_i = \sigma(z^{(l)}_i)$:

$\frac{\partial z^{(l+1)}_k}{\partial z^{(l)}_j} = W^{(l+1)}_{kj}\,\sigma'(z^{(l)}_j)$

Substituting:

$\delta^{(l)}_j = \Big(\sum_k W^{(l+1)}_{kj}\,\delta^{(l+1)}_k\Big)\sigma'(z^{(l)}_j)$

In matrix form:

$\boxed{\;\delta^{(l)} = \big((W^{(l+1)})^\top \delta^{(l+1)}\big)\odot \sigma'(z^{(l)})\;}$

Step 3 — Gradient w.r.t. weights

$z^{(l)}_j = \sum_i W^{(l)}_{ji} a^{(l-1)}_i + b^{(l)}_j$, so $\frac{\partial z^{(l)}_j}{\partial W^{(l)}_{ji}} = a^{(l-1)}_i$:

$\frac{\partial L}{\partial W^{(l)}_{ji}} = \frac{\partial L}{\partial z^{(l)}_j}\cdot\frac{\partial z^{(l)}_j}{\partial W^{(l)}_{ji}} = \delta^{(l)}_j\, a^{(l-1)}_i$

$\boxed{\;\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)}\,(a^{(l-1)})^\top\;}$

Why this outer product? Each weight $W_{ji}$ connects input $i$ to neuron $j$; its gradient = (blame of $j$) × (activity of $i$).

Step 4 — Gradient w.r.t. biases

$\frac{\partial z^{(l)}_j}{\partial b^{(l)}_j}=1$, so

$\boxed{\;\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}\;}$

Worked example 1 — a 2-2-1 network by hand

Input $x=\begin{bmatrix}1\\0\end{bmatrix}$, target $y=1$. Sigmoid $\sigma$, MSE loss $L=\tfrac12(a^{(2)}-y)^2$.

Weights: $W^{(1)}=\begin{bmatrix}0.1&0.2\\0.3&0.4\end{bmatrix}$, $b^{(1)}=0$, $W^{(2)}=[0.5\ \ 0.6]$, $b^{(2)}=0$.

Forward:

• $z^{(1)}=W^{(1)}x=\begin{bmatrix}0.1\\0.3\end{bmatrix}$, $a^{(1)}=\sigma(z^{(1)})=\begin{bmatrix}0.525\\0.574\end{bmatrix}$. Why? $\sigma(0.1)=0.525$, $\sigma(0.3)=0.574$.
• $z^{(2)}=0.5(0.525)+0.6(0.574)=0.607$, $a^{(2)}=\sigma(0.607)=0.647$.

Backward:

• $\nabla_a L = a^{(2)}-y = 0.647-1=-0.353$. Why? $\frac{d}{da}\tfrac12(a-y)^2=a-y$.
• $\sigma'(z^{(2)})=a^{(2)}(1-a^{(2)})=0.647(0.353)=0.228$.
• $\delta^{(2)} = -0.353\times 0.228 = -0.0805$ (BP1).
• $\frac{\partial L}{\partial W^{(2)}}=\delta^{(2)}(a^{(1)})^\top=[-0.0423,\ -0.0462]$ (BP3). Why? multiply the scalar $\delta$ by each activation.
• $\delta^{(1)}=(W^{(2)})^\top\delta^{(2)}\odot\sigma'(z^{(1)})$. $(W^{(2)})^\top\delta^{(2)}=\begin{bmatrix}-0.0403\\-0.0483\end{bmatrix}$; $\sigma'(z^{(1)})=\begin{bmatrix}0.249\\0.244\end{bmatrix}$; so $\delta^{(1)}=\begin{bmatrix}-0.0100\\-0.0118\end{bmatrix}$ (BP2).

Now a gradient-descent step subtracts $\eta\cdot$ these gradients. Done — one full pass.

Worked example 2 — why sharing saves work

For $W^{(1)}$ we needed $\delta^{(1)}$, which reused $\delta^{(2)}$ (already computed). If instead we chained fully from $L$ to $W^{(1)}$ we'd recompute the path through layer 2 again. In a 100-layer net that reuse is the difference between linear and exponential cost.

Flashcards

Connections

• Chain rule (multivariable) — the mathematical engine of every step.
• Gradient descent — consumes the gradients backprop produces.
• Activation functions and their derivatives — supplies $\sigma'(z)$ (sigmoid, ReLU, tanh).
• Computational graphs and autodiff — backprop is reverse-mode automatic differentiation.
• Vanishing and exploding gradients — direct consequence of the repeated $\odot\sigma'$ and $W^\top$ products.
• Loss functions (MSE, cross-entropy) — determine $\nabla_a L$ in BP1.

Concept Map

Hinglish (regional understanding)