Exercises — Backpropagation algorithm derivation

WHAT we're asked: just match names to jobs.

• The gradient for a weight matrix is BP3: $\frac{\partial L}{\partial W^{(l)}}={\delta }^{(l)}(a^{(l-1)}{)}^{\top }$.
• The starting error at the output is BP1: ${\delta }^{(L)}={\nabla }_{a}L\odot {\sigma }^{\prime }(z^{(L)})$.

WHY: Gradient descent updates parameters ($W$, $b$), so it needs BP3/BP4. But BP3 needs $\delta$, and the whole $\delta$ chain has to start somewhere — that start is BP1 at the last layer, because the last layer is the only one that touches the loss directly.

• $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}$ — "take the next layer's blame and route it backward through the weights, flipped." The flip (transpose) is because blame travels outputs→inputs.
• $\odot$ — "elementwise gate," multiply position-by-position, no summing across.
• ${\sigma }^{\prime }(z^{(l)})$ — "how sensitive this neuron's output was to its own input at the value it actually saw." If the neuron was in a flat part of $\sigma$, this is near $0$ and it blocks the blame.

Forward. $a=\sigma (0.5)=\frac{1}{1+e^{-0.5}}=0.6225$. Backward, WHY these two factors: $z$ reaches $L$ only through $a$, so chain rule gives $\delta =\frac{\partial L}{\partial a}\cdot \frac{\partial a}{\partial z}$.

• $\frac{\partial L}{\partial a}=a-y=0.6225-0=0.6225$ (derivative of $\frac{1}{2}(a-y{)}^2$).
• $\frac{\partial a}{\partial z}={\sigma }^{\prime }(0.5)=a(1-a)=0.6225(0.3775)=0.2350$. $\delta =0.6225\times 0.2350=\mathrm{0.1463.}$ That is exactly BP1 for a one-neuron output layer.

WHAT: apply BP2 for one neuron: $\delta =(\text{routed blame})\times {\sigma }^{\prime }(z)$. WHY the sign matters: ReLU${}^{\prime }(z)=0$ when $z<0$. Here $z=-2<0$, so: $\delta =0.9\times {\text{ReLU}}^{\prime }(-2)=0.9\times 0=0.$ The neuron was "off," so it forwarded a constant $0$; changing its input changes nothing downstream, hence zero blame. This dead-zone is why very negative pre-activations kill gradients — see Vanishing and exploding gradients.

WHY biases are easy: $\partial z_j^{(l)}/\partial b_j^{(l)}=1$, so BP4 says the bias gradient equals the delta. No extra multiply. From the parent note's worked forward+backward pass:

• ${\delta }^{(2)}=-0.0805\Rightarrow \frac{\partial L}{\partial b^{(2)}}=-\mathrm{0.0805.}$
• ${\delta }^{(1)}=\left[\begin{array}{c}-0.0100\\ -0.0118\end{array}\right]\Rightarrow \frac{\partial L}{\partial b^{(1)}}=\left[\begin{array}{c}-0.0100\\ -0.0118\end{array}\right].$

Read it: the bias "absorbs" exactly the blame of its neuron, because a bias shifts $z$ one-for-one.

Update rule (Gradient descent): $W\leftarrow W-\eta \partial L/\partial W$. $W_{\text{new}}^{(2)}=[0.5,0.6]-1.0\cdot [-0.0423,-0.0462]=[0.5423,0.6462].$ Did we improve? The gradient was negative, meaning "increasing $W^{(2)}$ decreases loss." We increased $W^{(2)}$, so loss should drop. Concretely a larger $W^{(2)}$ raises $z^{(2)}$, raises $a^{(2)}$ toward the target $y=1$ (we were at $0.647$, target $1$). ✔ moves toward the target.

BP2 for one neuron: $\delta =(\text{routed blame})\cdot {\sigma }^{\prime }(z)$.

• Saturated neuron ($z=6$): $a=\sigma (6)=0.9975$, ${\sigma }^{\prime }(6)=a(1-a)=0.9975\times 0.0025=0.00247$. ${\delta }_{\text{sat}}=2.0\times 0.00247=\mathrm{0.00494.}$
• Central neuron ($z=0$): ${\sigma }^{\prime }(0)=0.5\times 0.5=0.25$. ${\delta }_{\text{center}}=2.0\times 0.25=\mathrm{0.5.}$ The point: same incoming blame, but the saturated neuron passes back $\approx 100\times$ less. Stack many such layers and the product of tiny ${\sigma }^{\prime }$ factors vanishes — that is Vanishing and exploding gradients in one line.

Shapes are the whole proof.

• $W^{(l+1)}$ maps activations of layer $l$ (length $3$) → pre-activations of layer $l+1$ (length $2$). A matrix that maps length-$3$ to length-$2$ is $2\times 3$. ✔
• Try $W^{(l+1)}{\delta }^{(l+1)}$: that is $(2\times 3)\cdot (3\times 1)$ — but ${\delta }^{(l+1)}$ has length $2$, not $3$, so the inner dimensions ($3$ vs $2$) don't match. Illegal. ✘
• Try $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}$: that is $(3\times 2)\cdot (2\times 1)=(3\times 1)$. Inner dims $2=2$ match, output length $3$. ✔
• Now $\odot {\sigma }^{\prime }(z^{(l)})$: ${\sigma }^{\prime }(z^{(l)})$ has length $3$ (one per layer-$l$ neuron). Elementwise product of two length-$3$ vectors is length $3$. ✔ Legal, and gives the required ${\delta }^{(l)}$ of length $3$.

Conclusion: the transpose is not a stylistic choice — it is the only linear map that returns blame to the correct number of neurons. We say the forward map $W$ and the backward map $W^{\top }$ are adjoints of each other: an "adjoint" is just the linear-algebra name for the transpose partner that sends a signal back through the reverse wiring. Forward pushes inputs→outputs; its adjoint $W^{\top }$ pushes blame outputs→inputs.

Two pieces of BP1: ${\delta }^{(L)}=\frac{\partial L}{\partial a}\cdot {\sigma }^{\prime }(z)$.

• $\frac{\partial L}{\partial a}=-\frac{y}{a}+\frac{1-y}{1-a}=\frac{-y(1-a)+(1-y)a}{a(1-a)}=\frac{a-y}{a(1-a)}.$
• ${\sigma }^{\prime }(z)=a(1-a).$ Multiply: ${\delta }^{(L)}=\frac{a-y}{a(1-a)}\cdot a(1-a)=a-y.$ WHY this is beautiful: the $a(1-a)$ that causes vanishing gradients in the sigmoid cancels exactly against the cross-entropy denominator. So the output layer never saturates its own learning signal — a strong reason to pair sigmoid with cross-entropy rather than MSE. Compare with Exercise 2.1, where MSE kept the ${\sigma }^{\prime }$ factor alive. (See Loss functions (MSE, cross-entropy).)

==...==) BP4 says the bias gradient equals the delta ${\delta }^{(l)}$ because $\partial z/\partial b=1$. The backward pass uses $W^{\top }$ because forward and backward maps are adjoints (transposes) of each other. For sigmoid + cross-entropy the output error collapses to ==$a-y$==. A neuron with very large $|z|$ passes back a tiny gradient because ${\sigma }^{\prime }(z)\approx 0$.