Exercises — Backpropagation — chain rule, gradient computation

The chain rule. For nested (composed) functions, sensitivities multiply: if $x$ wiggles by a tiny amount, $f_1$ wiggles by $f_1^{\prime }$ times that, then $f_2$ wiggles by $f_2^{\prime }$ times that, and so on. Chaining these scalings gives $\frac{dL}{dx}=f_3^{\prime }\cdot f_2^{\prime }\cdot f_1^{\prime }.$ A sum appears only when $x$ reaches $L$ through several separate paths (the multivariable chain rule); a straight chain is one path, so it is a pure product. See Chain Rule.

Differentiate $\frac{1}{2}(\hat{y}-t{)}^2$ with respect to $\hat{y}$: the $\frac{1}{2}$ and the power-2 cancel, leaving $\frac{\partial L}{\partial \hat{y}}=\hat{y}-t.$ It is the seed because it is the very first number of the backward pass — the "how wrong are we, and in which direction" quantity that every later gradient is grown from.

The forward step multiplies inputs by the matrix $W$ (a linear map). The derivative of a linear map $Wa$ with respect to its input $a$ is the transpose $W^{\top }$ (rows and columns swapped), not the inverse. Backprop is differentiation, and differentiating a linear map transposes it; nothing here undoes/inverts anything.

Walk the orange arrows of Figure 1: $z_1=w_{1}x+b_1=0.5(1)+0=0.5,a_1=\text{ReLU}(0.5)=0.5,$ $z_2=w_2{a}_1+b_2=2(0.5)+0=1.0,\hat{y}=z_2=1.0,$ $L=\frac{1}{2}(\hat{y}-t{)}^2=\frac{1}{2}(1-0{)}^2=\mathrm{0.5.}$

Seed then chain. Since $\hat{y}=z_2$, $\partial \hat{y}/\partial z_2=1$, so ${\delta }_2=\frac{\partial L}{\partial z_2}=\frac{\partial L}{\partial \hat{y}}\cdot 1=(\hat{y}-t)=1.$ Because $z_2=w_2{a}_1+b_2$: $\partial z_2/\partial w_2=a_1$ and $\partial z_2/\partial b_2=1$. Hence $\frac{\partial L}{\partial w_2}={\delta }_2{a}_1=1\times 0.5=0.5,\frac{\partial L}{\partial b_2}={\delta }_2=1.$ Read it as error out × signal in: the gradient of $w_2$ is its downstream error ($1$) times its own input ($a_1=0.5$).

Error flows back through $w_2$ and is scaled by the local slope ${\sigma }^{\prime }(z_1)$. Since $z_1=0.5>0$, ${\sigma }^{\prime }(z_1)=1$: ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\times 2\times 1=2.$ Then with $\partial z_1/\partial w_1=x$ and $\partial z_1/\partial b_1=1$: $\frac{\partial L}{\partial w_1}={\delta }_{1}x=2\times 1=2,\frac{\partial L}{\partial b_1}={\delta }_1=2.$

Forward: $z_1=-3(1)+0=-3$, so $a_1=\text{ReLU}(-3)=0$, $z_2=2(0)+0=0$, $\hat{y}=0$, $L=\frac{1}{2}(0-0{)}^2=0$. Backward: ${\delta }_2=\hat{y}-t=0$. Even the seed is zero. And the slope ${\sigma }^{\prime }(z_1)=0$ because $z_1<0$. So ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0\times 2\times 0=0,\frac{\partial L}{\partial w_1}={\delta }_{1}x=0.$ All layer-1 gradients vanish. The ReLU is in its flat region — the unit is "dead." No error can pass a zero slope, so learning stalls for $w_1,b_1$. This is a baby case of Vanishing Gradients.

The chain rule factors $\partial L/\partial w$ into a product of local derivatives. Each $\delta$ is computed once and reused for every weight feeding that node: ${\delta }_1$ is built from ${\delta }_2$ (already known), and every weight's gradient is $\delta \times (\text{its input})$. So one backward sweep reuses cached quantities and touches each edge once — total work is proportional to the number of edges, i.e. one forward pass' worth. The naïve finite-difference method instead re-runs the whole net per weight: $O(\#\text{weights})$ passes. This reuse is exactly what Automatic Differentiation mechanises. See Computational Graphs.

$\frac{\partial L}{\partial a}=u{\delta }_A+v{\delta }_B.$ The value $a$ influences $L$ along two independent paths. The multivariable chain rule says total sensitivity is the sum of per-path sensitivities — each path's contribution is (weight on that path) × (error at that path's target). Look at Figure 2: the two magenta arrows meet at $a$ and their blames add.

Forward: $z_1=0.5(1)+(-1)(2)+0=0.5-2=-1.5$. ReLU clamps: $a_1=\max (0,-1.5)=0$. Then $z_2=2(0)=0$, $\hat{y}=0$, $L=\frac{1}{2}(0{)}^2=0$. Backward: ${\delta }_2=\hat{y}-t=0$; slope ${\sigma }^{\prime }(z_1)=0$ (since $z_1<0$); ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0$. The weight gradient is the outer product "error × input": $\partial L/\partial w_1={\delta }_1{x}^{\top }=0\cdot [1,2]=[0,0]$, and $\partial L/\partial b_1={\delta }_1=0$. Every gradient is zero — again a dead unit. This shows the vector rule reproduces the scalar logic elementwise.

Forward: $z_1=0.5(1)+1(2)=2.5>0$, so $a_1=2.5$; $z_2=2(2.5)=5$, $\hat{y}=5$, $L=\frac{1}{2}(5-0{)}^2=12.5$. Backward: ${\delta }_2=\hat{y}-t=5$. $\partial L/\partial w_2={\delta }_2{a}_1=5(2.5)=12.5$, $\partial L/\partial b_2=5$. ${\sigma }^{\prime }(z_1)=1$ (positive), so ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=5(2)(1)=10$. Vector weight gradient (outer product): $\frac{\partial L}{\partial w_1}={\delta }_1{x}^{\top }=10[1,2]=[10,20],\frac{\partial L}{\partial b_1}={\delta }_1=10.$ Notice each component is ${\delta }_1\times$ (the corresponding input) — the weight touching the bigger input ($x_2=2$) gets the bigger gradient.

Update rule $w\leftarrow w-\eta \partial L/\partial w$: $w_2^{\text{new}}=2-0.01(12.5)=1.875,b_2^{\text{new}}=0-0.01(5)=-\mathrm{0.05.}$ Sanity: with $a_1=2.5$ still, new $\hat{y}=1.875(2.5)-0.05=4.6875-0.05=4.6375$, new $L=\frac{1}{2}(4.6375{)}^2\approx 10.753$. Since $10.753<12.5$, the loss decreased — gradient descent stepped downhill, as intended.

Forward: $z_1=0.5$, $a_1=\sigma (0.5)=\frac{1}{1+e^{-0.5}}\approx 0.622459$. Then $z_2=2(0.622459)=1.244918$, $\hat{y}=1.244918$, $L=\frac{1}{2}(1.244918{)}^2\approx 0.774910$. Backward: ${\delta }_2=\hat{y}-t\approx 1.244918$. Local slope ${\sigma }^{\prime }(z_1)=a_1(1-a_1)=0.622459(0.377541)\approx 0.234988$. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)\approx 1.244918\times 2\times 0.234988\approx \mathrm{0.585062.}$ $\frac{\partial L}{\partial w_1}={\delta }_{1}x\approx \mathrm{0.585062.}$ Saturation: as $|z_1|\to \infty$, $\sigma (z_1)\to 0$ or $1$, so ${\sigma }^{\prime }(z_1)=\sigma (1-\sigma )\to 0$. Since ${\delta }_1$ carries a factor ${\sigma }^{\prime }(z_1)$, the gradient is throttled toward $0$ — the classic sigmoid Vanishing Gradients problem. The peak slope is only ${\sigma }^{\prime }(0)=0.25$, so even at best sigmoids shrink error by $\frac{1}{4}$ per layer.

With the sigmoid net, $L(w_1)=\frac{1}{2}(2\sigma (w_1){)}^2=2\sigma (w_1{)}^2$ (since $x=1,b=0,w_2=2$). $L_{+}=2\sigma (0.5001{)}^2$, $L_{-}=2\sigma (0.4999{)}^2$. Numerically the slope estimate $\frac{L_{+}-L_{-}}{2ϵ}\approx 0.58506$, matching L5.1's $0.585062$ to five decimals. ✓ Central vs forward: the central formula's error shrinks like ${ϵ}^2$ (the odd, first-order error terms cancel by symmetry), while a one-sided forward difference errs like $ϵ$. So the central estimate is far more accurate for the same $ϵ$.

${\delta }_{\text{front}}={\delta }_{\text{seed}}\cdot r^n.$

• $r=0.25$: $r^n\to 0$ geometrically — vanishing gradients; deep front layers barely learn.
• $r=1$: $r^n=1$ — error passes intact, the healthy regime that skip-connections and careful initialisation try to engineer.
• $r=2$: $r^n\to \infty$ — exploding gradients; updates blow up unless clipped. This single product $r^n$ is the reason depth is delicate, and why surrogate nets for aerodynamic data (Neural Network Surrogate Models (CFD)) need normalisation and residual paths to stay in the $r\approx 1$ band.