Worked examples — Backpropagation — chain rule, gradient computation

---

The scenario matrix

Before working anything, let us list every distinct situation a backprop problem can be in. Each row is a "case class"; each worked example below is tagged with the cell it covers.

#	Case class	What is special	Covered by
A	Linear output, positive slope	ReLU with $z>0$, ${\sigma }^{\prime }=1$ — the "clean" case	Ex 1
B	Dead ReLU ($z<0$)	${\sigma }^{\prime }(z)=0$ — the unit passes zero error backward	Ex 2
C	Sigmoid, mid-range	${\sigma }^{\prime }$ is a fraction $\in (0,\frac{1}{4}]$ — error shrinks	Ex 3
D	Saturated sigmoid	${\sigma }^{\prime }\to 0$ — the vanishing-gradient failure	Ex 4
E	Fan-in node (weight feeds two paths)	must sum over paths (multivariable rule)	Ex 5
F	Zero / degenerate input ($x=0$ or $a=0$)	weight gradient collapses to $0$	Ex 6
G	Sign bookkeeping ($t>\hat{y}$ vs $t<\hat{y}$)	error signal changes sign, so does every gradient	Ex 7
H	Real-world word problem (aerospace surrogate)	translate physics → net → gradient	Ex 8
I	Exam twist (shared weight / weight tying)	one weight used twice ⇒ gradients add	Ex 9

We will reuse this tiny network throughout unless stated otherwise:

$z_1=w_{1}x+b_1,a_1=\sigma (z_1),z_2=w_2{a}_1+b_2,\hat{y}=z_2,L=\frac{1}{2}(\hat{y}-t{)}^2.$

Here $x$ is the input number, $w$ are weights (multipliers), $b$ are biases (offsets), $\sigma$ is the activation function (the per-unit nonlinear bend), $\hat{y}$ is the network's guess, $t$ is the true target, and $L$ is the loss (how wrong we are). The picture below is the map we walk backward on.

---

Ex 1 — Case A: linear/ReLU, positive slope (the clean baseline)

Forecast: Guess the sign of both gradients before reading. (Hint: $\hat{y}$ will overshoot the target $0$, so we expect positive gradients that push weights down.)

1. Forward, cache everything. $z_1=0.5\cdot 1+0=0.5$; since $z_1>0$, $a_1=\sigma (0.5)=0.5$; $z_2=2\cdot 0.5+0=1.0$; $\hat{y}=1.0$; $L=\frac{1}{2}(1-0{)}^2=0.5$. Why this step? Backward slopes (${\sigma }^{\prime }(z_1)$) and inputs ($a_1$) are read from these cached numbers — recomputing wastes work.
2. Seed. $\partial L/\partial \hat{y}=\hat{y}-t=1-0=1$. Why? $L=\frac{1}{2}(\hat{y}-t{)}^2\Rightarrow dL/d\hat{y}=(\hat{y}-t)$.
3. Layer 2. ${\delta }_2=\partial L/\partial z_2=1\cdot 1=1$ (output is linear, slope $1$). Then $\partial L/\partial w_2={\delta }_2{a}_1=1\cdot 0.5=0.5$. Why? weight gradient = error out × signal in.
4. Cross the nonlinearity. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\cdot 2\cdot 1=2$. Why? error flows back through $w_2$ ($\times 2$), then is scaled by the local slope ${\sigma }^{\prime }(z_1)=1$.
5. Layer 1. $\partial L/\partial w_1={\delta }_{1}x=2\cdot 1=2$.

Verify: Bump $w_1$ by $ϵ=10^{-4}$: $\hat{y}=1.0002$, $L\approx 0.50020$, so $\Delta L/ϵ\approx 2.0$ ✓. Both gradients positive as forecast — gradient descent will subtract them, lowering the weights toward a better fit.

---

Ex 2 — Case B: the Dead ReLU (error hits a wall)

Forecast: Will $w_1$ receive any gradient at all?

1. Forward. $z_1=-3\cdot 1+1=-2$. Since $z_1<0$, $a_1=\max (0,-2)=0$. Why note this? ReLU has switched off — its output is stuck at $0$.
2. Local slope. ${\sigma }^{\prime }(z_1)=0$ because for $z<0$ the ReLU graph is flat. Why it matters: a flat function has zero derivative — nudging $z_1$ does nothing to $a_1$.
3. Backward. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=(\dots )\times 0=0$, hence $\partial L/\partial w_1={\delta }_{1}x=0$. Why? The $\odot {\sigma }^{\prime }$ factor annihilates the error — no signal reaches $w_1$.

Verify: Bump $w_1$ by $ϵ$: $z_1$ moves to $-2+ϵ$ (still $<0$), so $a_1$ stays $0$, $\hat{y}$ stays $0$, $L$ unchanged $\Rightarrow \Delta L/ϵ=0$ ✓.

---

Ex 3 — Case C: sigmoid mid-range (error shrinks by a fraction)

Forecast: At $z_1=0$ the sigmoid is at its steepest. What is that maximum slope?

1. Forward. $z_1=0$; $a_1=\sigma (0)=\frac{1}{1+1}=0.5$; $z_2=1\cdot 0.5=0.5$; $\hat{y}=0.5$. Why? We need $a_1=\sigma (z_1)$ to evaluate the slope.
2. Local slope. ${\sigma }^{\prime }(0)=0.5(1-0.5)=0.25$. Why this tool (product $\sigma (1-\sigma )$)? The sigmoid's derivative can be written using its own output, so once we cached $a_1$ we get the slope for free — a big reason sigmoids were popular.
3. Seed & backward. $\partial L/\partial \hat{y}=0.5-0=0.5$; ${\delta }_2=0.5$; then ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0.5\cdot 1\cdot 0.25=0.125$. Why? the incoming error $0.5$ is throttled to a quarter by the slope.

Verify: ${\delta }_1=0.125$. Finite-difference $\partial L/\partial z_1$ around $z_1=0$: bump $ϵ=10^{-4}$, $L$ changes by $\approx 0.125ϵ$ ✓. Note ${\sigma }^{\prime }$ never exceeds $0.25$ — every sigmoid layer shrinks the error by at least $4\times$.

---

Ex 4 — Case D: saturated sigmoid (the vanishing gradient)

Forecast: Big $z$ means $\sigma \approx 1$ — is the slope big or tiny?

1. Forward. $\sigma (8)=\frac{1}{1+e^{-8}}\approx 0.99966$. Why? We need the output to compute the slope.
2. Slope. ${\sigma }^{\prime }(8)=0.99966(1-0.99966)\approx 0.000335$. Why this matters: the S-curve is flat far from the origin — the unit is saturated.
3. Backward. Whatever error arrives, ${\delta }_1=(\dots )\times 0.000335$ is crushed by a factor of $\sim 3000$. Why? Stack a few such layers and the product of tiny slopes $\to 0$ — gradients vanish.

Verify: ${\sigma }^{\prime }(8)\approx 3.35\times 10^{-4}$ ✓. Contrast with Ex 3's $0.25$: same function, $\sim 750\times$ weaker gradient just from a larger $z$. This is exactly why ReLU replaced sigmoids in deep nets — see Vanishing Gradients.

---

Ex 5 — Case E: fan-in node (sum over paths)

Forecast: $u$ influences $L$ through two wires ($p$ and $q$). Do we pick one or add them?

1. Forward. $u=2$; $p=6$; $q=-2$; $s=4$; $L=\frac{1}{2}(16)=8$.
2. Seed & split. $\partial L/\partial s=s=4$. Since $s=p+q$: $\partial L/\partial p=4$, $\partial L/\partial q=4$. Why? $\partial s/\partial p=1$, $\partial s/\partial q=1$.
3. Multivariable chain rule at the fan-in. $u$ reaches $L$ via $p$ and $q$, so we sum the two path contributions: $\frac{\partial L}{\partial u}=\frac{\partial L}{\partial p}\frac{\partial p}{\partial u}+\frac{\partial L}{\partial q}\frac{\partial q}{\partial u}=4\cdot 3+4\cdot (-1)=12-4=8.$ Why sum, not choose? Each path carries independent sensitivity; total sensitivity is their sum — this is the graph fan-in rule.
4. To the input. $\partial L/\partial x=\partial L/\partial u\cdot \partial u/\partial x=8\cdot 1=8$.

Verify: Note $s=3u-u=2u$, so $L=\frac{1}{2}(2u{)}^2=2u^2$, giving $dL/du=4u=8$ at $u=2$ ✓ — the closed form confirms the summed backprop.

---

Ex 6 — Case F: zero / degenerate input (gradient collapses)

Forecast: If the input is zero, does $w_1$ still learn?

1. Forward. $z_1=w_1\cdot 0+b_1=0.5$; $a_1=0.5$ (ReLU, $z_1>0$); $z_2=2\cdot 0.5=1$; $\hat{y}=1$; $L=0.5$.
2. Backward error at $z_1$. ${\delta }_2=\hat{y}-t=1$; ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\cdot 2\cdot 1=2$.
3. Weight vs bias gradient. $\frac{\partial L}{\partial w_1}={\delta }_{1}x=2\cdot 0=0,\frac{\partial L}{\partial b_1}={\delta }_1\cdot 1=2.$ Why the split? $w_1$'s gradient is error × its input, and its input is $0$ — so $w_1$ gets no update. The bias's "input" is the constant $1$, so it still learns.

Verify: Bump $w_1$: $z_1=w_1\cdot 0+0.5$ is independent of $w_1$, so $L$ never changes $\Rightarrow \partial L/\partial w_1=0$ ✓. Bump $b_1$ by $ϵ$: $\hat{y}$ rises by $2ϵ$, $L$ by $\approx 2ϵ$, so $\partial L/\partial b_1=2$ ✓.

---

Ex 7 — Case G: sign bookkeeping (undershoot vs overshoot)

Forecast: In Ex 1 the guess was too high; here it is too low. Which way flips?

1. Forward. Identical to Ex 1: $a_1=0.5$, $\hat{y}=1$, but now $L=\frac{1}{2}(1-2{)}^2=0.5$.
2. Seed changes sign. $\partial L/\partial \hat{y}=\hat{y}-t=1-2=-1$. Why? The error signal is $\hat{y}-t$; undershooting makes it negative.
3. Propagate. ${\delta }_2=-1$; $\partial L/\partial w_2={\delta }_2{a}_1=-1\cdot 0.5=-0.5$. Why the flip? Gradient descent subtracts the gradient: a negative gradient means "increase $w_2$" — exactly right, since $\hat{y}$ is too small.

Verify: $\partial L/\partial w_2=-0.5$, the negation of Ex 1's $+0.5$ ✓. Finite difference: bump $w_2$ by $ϵ$, $\hat{y}=1+0.5ϵ$, $L=\frac{1}{2}(1-2+0.5ϵ{)}^2\approx 0.5-0.5ϵ$, slope $-0.5$ ✓. Sign of the seed sets the sign of every downstream gradient.

---

Ex 8 — Case H: aerospace word problem (a lift surrogate)

Forecast: Is the model over- or under-predicting lift, and does $w$ go up or down?

1. Forward. $z=6\cdot 0.1+0.1=0.7$; ReLU $\Rightarrow {\hat{C}}_L=0.7$; $L=\frac{1}{2}(0.7-0.9{)}^2=\frac{1}{2}(0.04)=0.02$. Why? We must evaluate the prediction before we can measure error.
2. Seed. $\partial L/\partial {\hat{C}}_L=0.7-0.9=-0.2$ (we underpredict lift).
3. Cross ReLU. $z=0.7>0\Rightarrow {\sigma }^{\prime }=1$, so $\delta =-0.2\cdot 1=-0.2$.
4. Weight gradient. $\partial L/\partial w=\delta \alpha =-0.2\cdot 0.1=-0.02$. Why? input into $w$ is $\alpha$.
5. Update direction. Gradient descent: $w\leftarrow w-\eta (-0.02)$ — the negative gradient raises $w$, steepening the lift curve so ${\hat{C}}_L$ climbs toward $0.9$. ✓ physically sensible.

Verify: $\partial L/\partial w=-0.02$; $\partial L/\partial b=\delta =-0.2$. Finite difference on $w$: bump $ϵ$, ${\hat{C}}_L=0.7+0.1ϵ$, $L=\frac{1}{2}(-0.2+0.1ϵ{)}^2\approx 0.02-0.02ϵ$, slope $-0.02$ ✓.

---

Ex 9 — Case I: exam twist (a weight used twice / weight tying)

Forecast: $w$ appears twice — is $\partial L/\partial w$ one term or a sum of two?

1. Forward. $y_1=2\cdot 1=2$; $y_2=2\cdot 2=4$; $L=\frac{1}{2}(16)=8$.
2. Seed. $\partial L/\partial y_2=y_2=4$.
3. Two uses ⇒ two paths. Treat each occurrence of $w$ as a distinct copy $w_a$ (in $y_1=w_{a}x$) and $w_b$ (in $y_2=w_b{y}_1$), then add their gradients (multivariable chain rule):
   • via $w_b$: $\partial L/\partial w_b=\partial L/\partial y_2\cdot y_1=4\cdot 2=8$.
   • via $w_a$: $\partial L/\partial y_1=\partial L/\partial y_2\cdot w=4\cdot 2=8$; then $\partial L/\partial w_a=8\cdot x=8$. $\frac{\partial L}{\partial w}=8+8=16.$ Why sum? A shared weight is a fan-in node in the graph — you sum the gradient contributed by each place it is used.
4. Update. $w\leftarrow w-\eta \cdot 16$.

Verify: Closed form: $y_2=w\cdot (wx)=w^{2}x$, so $L=\frac{1}{2}{w}^4{x}^2$; $dL/dw=2w^3{x}^2=2\cdot 8\cdot 1=16$ ✓. Ignoring one occurrence would give $8$ — half the true gradient, the classic exam trap.

---

Recall

Connections

• Backpropagation (parent) — the machinery these examples exercise.
• Chain Rule — the engine behind every "sum over paths".
• Gradient Descent — consumes the gradients we computed to update weights.
• Vanishing Gradients — Cases B and D are its root causes.
• Activation Functions — ReLU vs sigmoid drove the slope factors above.
• Computational Graphs — fan-in nodes (Ex 5, Ex 9) are graph structure.
• Automatic Differentiation — automates exactly this bookkeeping.
• Neural Network Surrogate Models (CFD) — Ex 8's aerospace context.