Worked examples — Backpropagation algorithm derivation

We only use the four boxed equations. As a quick reminder (each is derived in the parent):

Here $\odot$ means elementwise multiply (multiply matching entries, no summing). $z$ is the pre-activation (before the nonlinearity), $a=\sigma (z)$ is the activation (after it). $\delta$ is the blame — how sensitive the loss is to a neuron's pre-activation. See Chain rule (multivariable), Activation functions and their derivatives, and Loss functions (MSE, cross-entropy) for the pieces we lean on.

---

The scenario matrix

Every case backprop can throw at you falls into one of these cells. The right column names the example that covers it.

#	Case class	What's special	Covered by
A	Positive error, sigmoid	prediction too low, $\delta <0$	Ex 1
B	Negative error, sigmoid	prediction too high, $\delta >0$	Ex 2
C	Zero error (degenerate)	perfect prediction, gradient vanishes	Ex 3
D	Saturated sigmoid (limiting)	$z$ huge, ${\sigma }^{\prime }\to 0$, blame dies	Ex 4
E	ReLU, live neuron	gate $=1$, blame passes untouched	Ex 5
F	ReLU, dead neuron ($z<0$)	gate $=0$, blame blocked	Ex 5
G	Multi-output + cross-entropy	vector ${\nabla }_{a}L$, softmax simplification	Ex 6
H	Real-world word problem	translating a story into BP1–BP4	Ex 7
I	Exam twist: $W^{\top }$ vs $W$	pick the shape that type-checks	Ex 8

The sign of ${\delta }^{(L)}$ is set entirely by BP1's first factor ${\nabla }_{a}L$, because for sigmoid the gate ${\sigma }^{\prime }(z)=a(1-a)$ is always positive (between $0$ and $\frac{1}{4}$). Look at the sign map:

---

Example 1 — Cell A: positive error, sigmoid

Forecast: the prediction will land below $1$, so we are under-shooting. Guess: is $\delta$ negative or positive?

1. Forward. $z=0.5\cdot 2=1$, so $a=\sigma (1)=0.7311$. Why this step? We cannot assign blame before we know what the network actually output.
2. BP1 first factor. ${\nabla }_{a}L=a-y=0.7311-1=-0.2689$. Why this step? $\frac{d}{da}\frac{1}{2}(a-y{)}^2=a-y$; this is the raw "how wrong, and which way."
3. BP1 gate. ${\sigma }^{\prime }(z)=a(1-a)=0.7311\cdot 0.2689=0.1966$. Why this step? BP1 requires ${\sigma }^{\prime }$ at $z$; for sigmoid this equals $a(1-a)$.
4. BP1. $\delta =-0.2689\times 0.1966=-0.05287$. Why this step? This is BP1 for a single neuron.
5. BP3. $\partial L/\partial w=\delta a^{(0)}=-0.05287\times 2=-0.10574$. Why this step? Gradient = blame $\times$ input activity.

Verify: $\delta <0$ ✓ matches "we under-shot, so nudging $z$ up lowers loss, meaning $\partial L/\partial z<0$." Gradient descent does $w\leftarrow w-\eta (-0.1057)$ = increase $w$, which raises $z$, raises $a$ toward $1$. Sane.

---

Example 2 — Cell B: negative error (over-shoot), sigmoid

Forecast: now the target is $0$ but we output $0.7311$ — we over-shoot. Sign of $\delta$?

1. Reuse forward: $a=0.7311$ (unchanged, weights identical). Why? The forward pass does not depend on $y$.
2. BP1 first factor. ${\nabla }_{a}L=a-y=0.7311-0=0.7311$. Why? Positive because we are above target.
3. Gate. ${\sigma }^{\prime }(z)=0.1966$ (same as Ex 1).
4. BP1. $\delta =0.7311\times 0.1966=0.14374$.

Verify: $\delta >0$ ✓. Descent: $w\leftarrow w-\eta (\delta \cdot a^{(0)})$ decreases $w$, lowers $z$, lowers $a$ toward $0$. Correct direction. Compare with Ex 1: same $|z|$, opposite target, opposite sign — the gate never flips sign, only the error does. This is exactly figure s01's message.

---

Example 3 — Cell C: zero error (degenerate, gradient vanishes)

Forecast: with nothing to fix, what should the gradient be?

1. BP1 first factor. ${\nabla }_{a}L=a-y=0.7311-0.7311=0$. Why? Perfect prediction ⇒ zero raw error.
2. BP1. $\delta =0\times 0.1966=0$. Why? A zero factor kills the product regardless of the gate.
3. BP3. $\partial L/\partial w=0\times 2=0$.

Verify: At the loss minimum the gradient must be $0$ ✓. Gradient descent takes a zero step — the weight stays put, as it should at an optimum. Note the degeneracy: the gate ${\sigma }^{\prime }=0.1966$ is still healthy; the vanishing comes purely from the error, not from saturation. Contrast with Ex 4, where the opposite happens.

---

Example 4 — Cell D: saturated sigmoid (limiting behaviour)

Forecast: the error is huge — surely the gradient is huge and it fixes fast? Guess yes/no.

1. Forward. $a=\sigma (8)=0.99966$. Why? Large positive $z$ pushes sigmoid near $1$.
2. BP1 first factor. ${\nabla }_{a}L=a-y=0.99966-0=0.99966$ (near-maximal error).
3. Gate. ${\sigma }^{\prime }(z)=a(1-a)=0.99966\cdot 0.00034=0.00034$. Why? Deep in saturation the sigmoid is flat, so its slope is nearly $0$.
4. BP1. $\delta =0.99966\times 0.00034=0.00034$.

Verify: Despite a near-maximal error, $\delta \approx 0.00034$ is tiny — the flat gate strangled it. This is the mechanism of Vanishing and exploding gradients: a saturated sigmoid learns almost nothing. Sanity check the limit: as $z\to \infty$, $a\to 1$, $a(1-a)\to 0$, so $\delta \to 0$ for any bounded error ✓. Compare Ex 3 (zero $\delta$ from zero error) vs here (near-zero $\delta$ from a dead gate) — two very different roads to a vanishing gradient. Cross-entropy loss dodges this; see Ex 6.

---

Example 5 — Cells E & F: ReLU live vs dead neuron

Recall ReLU: $\sigma (z)=\max (0,z)$, with derivative ${\sigma }^{\prime }(z)=1$ if $z>0$ and ${\sigma }^{\prime }(z)=0$ if $z<0$ (a switch, not a squash). This is why BP2 needs $z$, not $a$ — the sign of $z$ is the whole story.

Forecast: which of the two blame values survives the ReLU gate?

1. Gate vector. ${\sigma }^{\prime }(z^{(1)})=\left[\begin{array}{c}1\\ 0\end{array}\right]$ since $5>0$, $-2<0$. Why this step? ReLU's derivative is the indicator "is $z$ positive?".
2. BP2 elementwise gate. ${\delta }^{(1)}=u\odot {\sigma }^{\prime }(z^{(1)})=\left[\begin{array}{c}0.30\cdot 1\\ -0.40\cdot 0\end{array}\right]=\left[\begin{array}{c}0.30\\ 0\end{array}\right]$. Why? $\odot$ multiplies entrywise; the dead neuron's gate is $0$.

Verify: Neuron 1 (live) passes its blame untouched (gate $=1$, cell E). Neuron 2 (dead) contributes $\delta =0$, so by BP3 all its incoming weight gradients are $0$ (cell F) — it learns nothing this step. Units/shape check: $u$ length $2$, gate length $2$, ${\delta }^{(1)}$ length $2$ ✓. This is the "dying ReLU" phenomenon: a neuron stuck at $z<0$ receives no gradient and may never recover.

---

Example 6 — Cell G: multi-output softmax + cross-entropy

Forecast: cross-entropy + softmax is famous for a clean $\delta$. What do you think it collapses to?

1. Softmax forward. $a_j=e^{z_j}/{\sum }_k{e}^{z_k}$. Numerators: $e^{2.0}=7.389$, $e^{1.0}=2.718$, $e^{0.1}=1.105$; sum $=11.212$. So $a=\left[\begin{array}{c}0.6590\\ 0.2424\\ 0.0986\end{array}\right]$. Why this step? Softmax turns logits into a probability vector (entries positive, sum to 1).
2. The identity. For softmax + cross-entropy, BP1 collapses to ${\delta }^{(L)}=a-y$ (the messy Jacobian-times-gradient cancels exactly). Why this step? This is the celebrated simplification — no separate $\odot {\sigma }^{\prime }$ gate survives.
3. Plug in. ${\delta }^{(L)}=\left[\begin{array}{c}0.6590\\ 0.2424\\ 0.0986\end{array}\right]-\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}-0.3410\\ 0.2424\\ 0.0986\end{array}\right]$.

Verify: Entries of ${\delta }^{(L)}$ sum to $0$: $-0.3410+0.2424+0.0986=0$ ✓ (softmax outputs sum to 1, one-hot sums to 1, difference sums to 0). Sign check: true class has $\delta <0$ (push its logit up), wrong classes have $\delta >0$ (push down). Compare with Ex 4: here there is no vanishing gate — cross-entropy is specifically designed so a confident-but-wrong output still gets a large $\delta$. That's why it beats MSE for classification.

---

Example 7 — Cell H: real-world word problem

Forecast: the link-count feature is bigger (3 vs 0.5) — will its gradient dominate?

1. Forward. $z=0.4\cdot 3+0.2\cdot 0.5+(-1)=1.2+0.1-1=0.3$; $a=\sigma (0.3)=0.5744$. Why? Standard weighted sum plus bias, then squash to a "spam probability."
2. BP1. ${\nabla }_{a}L=a-y=0.5744-1=-0.4256$; gate ${\sigma }^{\prime }=a(1-a)=0.5744\cdot 0.4256=0.2445$; so $\delta =-0.4256\cdot 0.2445=-0.10406$. Why? Under-confident on a true-spam email ⇒ negative blame ⇒ we want to raise $z$.
3. BP3. $\partial L/\partial w=\delta a^{(0)}=\left[\begin{array}{c}-0.10406\cdot 3\\ -0.10406\cdot 0.5\end{array}\right]=\left[\begin{array}{c}-0.31219\\ -0.05203\end{array}\right]$. Why? Gradient = blame $\times$ feature value; the bigger feature scales the blame more.

Verify: Both gradients negative ⇒ Gradient descent increases both weights ⇒ raises $z$ ⇒ classifies future similar emails as more-spam. Correct. The link-count weight's gradient magnitude ($0.312$) is $6\times$ the caps-ratio's ($0.052$) — exactly the feature ratio $3/0.5=6$ ✓. So yes, the larger-valued feature dominates the update, as forecast.

---

Example 8 — Cell I: exam twist, $W^{\top }$ vs $W$ by type-checking

Forecast: guess before reading — which shape produces a length-4 result?

1. List the shapes. $W^{(l+1)}$: $2\times 4$. $(W^{(l+1)}{)}^{\top }$: $4\times 2$. ${\delta }^{(l+1)}$: $2\times 1$. Why this step? Matrix multiply $(m\times n)(n\times 1)$ needs the inner dimensions to match and yields $m\times 1$.
2. Test $W^{(l+1)}{\delta }^{(l+1)}$: $(2\times 4)(2\times 1)$ — inner dims $4\ne 2$, illegal. Rejected.
3. Test $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}$: $(4\times 2)(2\times 1)$ — inner dims $2=2$ ✓, result $4\times 1$. This is the length-4 vector BP2 wants. Why? Only the transpose type-checks and has the right output length.

Verify: BP2's before-gate factor is $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}$, giving length $4=$ size of layer $l$ ✓. This is the adjoint direction: forward $W$ maps layer-$l$ ($4$) → layer-$(l+1)$ ($2$); backward the transpose maps blame $2\to 4$, exactly reversing the arrows. Numeric sanity: with ${\delta }^{(l+1)}=\left[\begin{array}{c}1\\ 1\end{array}\right]$ and $W^{(l+1)}=\left[\begin{array}{cccc}1& 2& 3& 4\\ 5& 6& 7& 8\end{array}\right]$, $(W^{(l+1)}{)}^{\top }{\delta }^{(l+1)}=\left[\begin{array}{c}6\\ 8\\ 10\\ 12\end{array}\right]$ — a length-4 vector, as required.

---

Recall Which cell had a vanishing gradient from the

error, and which from the gate? Zero error ::: Ex 3 (Cell C) — perfect prediction, ${\nabla }_{a}L=0$. Dead gate ::: Ex 4 (Cell D, saturated sigmoid) and Ex 5 F (dead ReLU) — ${\sigma }^{\prime }\approx 0$.

Recall What makes softmax+cross-entropy immune to the saturation problem of Ex 4?

::: BP1 collapses to ${\delta }^{(L)}=a-y$ with no surviving $\odot {\sigma }^{\prime }$ gate, so a confident-wrong output still gets a large error signal.