Visual walkthrough — Backpropagation algorithm derivation

---

Step 0 — The picture we are working on

WHAT. A tiny neural network is a chain of boxes. Numbers flow left→right (the forward pass), get compared to a target, and a single number called the loss $L$ measures "how wrong we were."

WHY draw it first. Every symbol on this page lives on one wire or one box of this diagram. If you can point at where a symbol sits, the algebra stops being scary.

PICTURE. Below: circles are neurons (little number-holders). Each arrow carries a weight $w$ — a dial that multiplies whatever passes along it.

The superscript $(l)$ just means "belonging to layer $l$." It is a label, not a power.

---

Step 1 — The one quantity we chase: delta $\delta$

WHAT. We invent one bookkeeping number per neuron, called delta: ${\delta }_j^{(l)}\equiv \frac{\partial L}{\partial z_j^{(l)}}$

Reading the symbols. The $\partial$ (curly-d) means "how much does the top wiggle when we wiggle the bottom a tiny bit, holding everything else still." So ${\delta }_j^{(l)}$ answers: "if I nudge the input-voltage of neuron $j$ in layer $l$ by a hair, how much does the final loss change?" That is exactly how much blame neuron $j$ deserves.

WHY this quantity and not $\partial L/\partial w$ directly? Because — as the picture shows — every weight feeding into neuron $j$ shares the same downstream story. If we know the blame at $z_j$ once, we get all those weight-gradients for free. $\delta$ is the shared sub-answer we cache. This "solve-once, reuse-everywhere" is dynamic programming over the graph.

PICTURE. A single neuron with a "blame meter" reading ${\delta }_j^{(l)}$ hanging off its input wire.

---

Step 2 — Blame at the very end (BP1)

WHAT. Start where blame is easiest to see: the output neuron. The loss $L$ touches $z^{(L)}$ through exactly one path: $z^{(L)}\to a^{(L)}\to L$. One path means one plain chain rule, no sums: ${\delta }_j^{(L)}=\underset{\text{how loss reacts to output}}{\underbrace{\frac{\partial L}{\partial a_j^{(L)}}}}\cdot \underset{={\sigma }^{\prime }(z_j^{(L)})}{\underbrace{\frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}}}$

Term by term.

• $\frac{\partial L}{\partial a_j^{(L)}}$ — "if the output brightens a hair, how much worse is the loss?" For MSE $L=\frac{1}{2}(a-y{)}^2$ this is simply $a^{(L)}-y$: the raw error.
• ${\sigma }^{\prime }(z_j^{(L)})$ — the slope of the squashing curve at this neuron's voltage. It gates the error: a flat region of $\sigma$ lets almost no blame through.

Stacking all $j$ into vectors: $\boxed{{\delta }^{(L)}={\nabla }_{a}L\odot {\sigma }^{\prime }(z^{(L)})}\text{(BP1)}$ Here $\odot$ = Hadamard product = multiply matching entries (entry 1 × entry 1, entry 2 × entry 2). We use it, not matrix-multiply, because each neuron's error and its own slope pair up one-to-one.

WHY ${\sigma }^{\prime }$ can silence a neuron. Look at the picture: on the flat tails of the S-curve, ${\sigma }^{\prime }\approx 0$, so blame is multiplied by nearly zero. That single fact is the seed of vanishing gradients.

---

Step 3 — Why blame must fan-IN when we go backward

WHAT. Now step one layer earlier. Here the plain chain rule breaks: $z_j^{(l)}$ does not reach the loss through one path. It fans out to every neuron $k$ in the next layer, so it reaches $L$ through many paths.

WHY a sum appears. The multivariable chain rule says: total sensitivity = add up the sensitivity along each path. So ${\delta }_j^{(l)}={\sum }_k\underset{\text{blame arriving from neuron }k}{\underbrace{{\delta }_k^{(l+1)}}}\cdot \underset{\text{how strongly }j\text{ feeds }k}{\underbrace{\frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}}}$

PICTURE. Forward, one neuron fans out to many (splits). Backward, those many blames fan in to that one neuron (merge). The green arrows reverse the blue ones.

---

Step 4 — The backward wire is the transpose (BP2)

WHAT. We need the "how strongly $j$ feeds $k$" factor. Since $z_k^{(l+1)}={\sum }_i{W}_{ki}^{(l+1)}{a}_i^{(l)}+b_k^{(l+1)}$ and $a_i^{(l)}=\sigma (z_i^{(l)})$, wiggling $z_j^{(l)}$ only touches the $i=j$ term: $\frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}=W_{kj}^{(l+1)}{\sigma }^{\prime }(z_j^{(l)})$

Plug in and pull ${\sigma }^{\prime }(z_j^{(l)})$ out of the sum (it doesn't depend on $k$): ${\delta }_j^{(l)}=\left({\sum }_k{W}_{kj}^{(l+1)}{\delta }_k^{(l+1)}\right){\sigma }^{\prime }(z_j^{(l)})$

The key visual. Forward we used row $k$, column $i$ of $W$: entry $W_{ki}$. Backward the sum ${\sum }_k{W}_{kj}{\delta }_k$ uses the column $j$ — i.e. we read the matrix the other way, which is exactly the transpose $W^{\top }$ (rows↔columns). In matrix form: $\boxed{{\delta }^{(l)}=((W^{(l+1)}{)}^{\top }{\delta }^{(l+1)})\odot {\sigma }^{\prime }(z^{(l)})}\text{(BP2)}$

Read it aloud, pointing at the picture:

• $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}$ — route next layer's blame backward through the flipped weight grid.
• $\odot {\sigma }^{\prime }(z^{(l)})$ — gate it by each neuron's own local slope.

---

Step 5 — From blame to weight-gradients (BP3, BP4)

WHAT. We wanted $\partial L/\partial W$, not $\delta$. But now it's one short step. A weight $W_{ji}^{(l)}$ enters $z_j^{(l)}$ multiplied by $a_i^{(l-1)}$, so $\partial z_j^{(l)}/\partial W_{ji}^{(l)}=a_i^{(l-1)}$. Chain once more: $\frac{\partial L}{\partial W_{ji}^{(l)}}=\underset{\text{blame of neuron }j}{\underbrace{{\delta }_j^{(l)}}}\cdot \underset{\text{how active input }i\text{ was}}{\underbrace{a_i^{(l-1)}}}$

WHY this shape (outer product). A weight is guilty in proportion to (the blame of the neuron it feeds) × (how loud the input it carried was). A weight carrying a silent input ($a_i=0$) gets zero gradient — it couldn't have caused anything. $\boxed{\frac{\partial L}{\partial W^{(l)}}={\delta }^{(l)}(a^{(l-1)}{)}^{\top }}(\text{BP3})\boxed{\frac{\partial L}{\partial b^{(l)}}={\delta }^{(l)}}(\text{BP4})$

The bias gradient is just $\delta$ because $\partial z_j/\partial b_j=1$ (the bias is added straight in, weight of one). These gradients feed straight into Gradient descent.

---

Step 6 — Edge cases the picture must cover

WHY. A derivation you can trust must survive its degenerate inputs. Three that bite in practice:

1. Dead neuron (${\sigma }^{\prime }(z)=0$). On the flat tail of the S-curve the gate in BP1/BP2 multiplies by $\approx 0$. Blame stops. Multiply many such gates down a deep net → vanishing gradient. The picture shows blame shrinking each backward hop.
2. Silent input ($a_i^{(l-1)}=0$). BP3 gives that weight gradient $0$ — correct: an input that never fired can't be blamed.
3. ReLU kink ($z=0$). For $\sigma (z)=\max (0,z)$, ${\sigma }^{\prime }$ is $1$ for $z>0$, $0$ for $z<0$, undefined at exactly $z=0$. In code we simply pick $0$ (or $1$) there — a measure-zero event. This is also why you must evaluate ${\sigma }^{\prime }$ at $z$, not $a$: for ReLU the sign of $z$ is the whole story, and it's lost once you only keep $a$.

---

The one-picture summary

WHAT. Two sweeps over the same graph. Blue goes forward computing $z,a$. Yellow/red comes backward computing $\delta$, and at each neuron BP3/BP4 harvest the weight and bias gradients. The whole algorithm is: forward to fill the wires, backward to pour the blame.

Recall Feynman retelling — the whole walkthrough in plain words

A network is a relay of message-passers (Step 0). We want to know how much each dial (weight) is to blame for the final mistake. Instead of tracing each dial's guilt from scratch, we give every neuron a single blame-meter $\delta$ (Step 1). We fill it first at the exit, where blame is a plain one-path chain rule: raw error times the local slope ${\sigma }^{\prime }$ (Step 2, BP1). Then we move backward. A neuron's voltage affected many neurons ahead, so its blame is the sum of blames coming back along every wire (Step 3) — and reading the weight grid backward is transposing it, then we gate by the neuron's own slope (Step 4, BP2). Once each neuron holds its blame, a dial's guilt is just (blame of the neuron it feeds) × (loudness of the input it carried) — an outer product (Step 5, BP3), and the bias just inherits $\delta$ (BP4). We check the corners: flat curves kill blame (vanishing), silent inputs get zero guilt, and ReLU's kink is why ${\sigma }^{\prime }$ must be read at $z$ (Step 6). Two passes, everything reused — that's backprop.

Recall Self-test

Where does $\delta$ live, on the $z$ wire or the $a$ wire? ::: On the pre-activation $z$: ${\delta }^{(l)}=\partial L/\partial z^{(l)}$. Why a sum in BP2 but not BP1? ::: Output neuron reaches the loss by one path (no sum); a hidden neuron fans out to many next-layer neurons (multivariable chain rule → sum). What geometric operation turns the forward weight grid into the backward one? ::: Transpose (swap rows and columns) — reading the same matrix in the reverse direction. Why does a silent input give a weight gradient of zero? ::: BP3 multiplies $\delta$ by $a_i^{(l-1)}$; if that input was $0$, the weight carried nothing and gets no blame. Why evaluate ${\sigma }^{\prime }$ at $z$ not $a$? ::: The slope is a function of the pre-activation; for ReLU it depends on $\text{sign}(z)$, information lost if you keep only $a$.