Visual walkthrough — Backpropagation — chain rule, gradient computation

We use exactly the network from the parent, all numbers scalar so nothing hides:

$z_1=w_{1}x+b_1,a_1=\sigma (z_1),z_2=w_2{a}_1+b_2,\hat{y}=z_2,L=\frac{1}{2}(\hat{y}-t{)}^2.$

Before any calculus, meet the cast.

---

Step 1 — Draw the computation as a chain of boxes

WHAT. A neural net is not a mysterious blob — it is a sequence of small operations, each one taking a number in and pushing a number out. We draw it as a computational graph: every box is one operation, every wire carries one number.

WHY. If we can see which number flows into which box, we can later see which box is to blame when the answer is wrong. Backprop is nothing but reading this same picture right-to-left.

PICTURE. Read the arrows left→right. The input $x$ enters, gets scaled by $w_1$ and shifted by $b_1$ to make $z_1$ (the "pre-activation"). The nonlinearity $\sigma$ squashes $z_1$ into $a_1$ (the "activation"). Layer 2 repeats the pattern to make $\hat{y}$ (our prediction). Finally $L$ compares $\hat{y}$ to the target $t$.

---

Step 2 — Forward pass: fill every wire with a number, and remember it

WHAT. Run the graph left→right with concrete numbers and store every value on every wire. Using the parent's numbers: $x=1,t=0,w_1=0.5,b_1=0,w_2=2,b_2=0$, and $\sigma =$ ReLU (ReLU means "keep positives, zero out negatives").

WHY store them? The backward step will need $a_1$ (the input to layer 2) and the slope ${\sigma }^{\prime }(z_1)$. Both depend on numbers we compute now. Throwing them away means recomputing later — wasteful. Caching is the trick that makes backprop cheap.

$\underset{0.5}{\underbrace{z_1}}=\underset{0.5}{\underbrace{w_1}}\underset{1}{\underbrace{x}}+\underset{0}{\underbrace{b_1}},\underset{0.5}{\underbrace{a_1}}=\sigma (0.5),\underset{1.0}{\underbrace{z_2}}=\underset{2}{\underbrace{w_2}}\underset{0.5}{\underbrace{a_1}}+\underset{0}{\underbrace{b_2}},\underset{1}{\underbrace{\hat{y}}}=z_2,\underset{0.5}{\underbrace{L}}=\frac{1}{2}(1-0{)}^2.$

PICTURE. Same graph, now every wire wears its numeric tag. The little "cache" tray under the graph holds the two values ($a_1=0.5$ and ${\sigma }^{\prime }(z_1)=1$) the backward pass will come back for.

---

Step 3 — Seed the error at the very end

WHAT. Start the backward journey by asking one question at the output: if $\hat{y}$ nudged up a little, how much does $L$ change? That number is $\partial L/\partial \hat{y}$.

WHY here? Only the final box, the loss, directly knows "how wrong" we are. So the blame originates at the end and travels backward. Differentiating $L=\frac{1}{2}(\hat{y}-t{)}^2$:

$\frac{\partial L}{\partial \hat{y}}=\underset{\text{how far off, with sign}}{\underbrace{(\hat{y}-t)}}=1-0=1.$

The $\frac{1}{2}$ and the power $2$ conspire: $\frac{d}{d\hat{y}}\frac{1}{2}{u}^2=u$, so the messy square becomes a clean $(\hat{y}-t)$. If we predicted too high, this is positive (push $\hat{y}$ down); too low, negative.

PICTURE. A single red arrow appears at the far right, pointing backward out of $L$, carrying the value $1$. This is the "error signal" born at the output.

---

Step 4 — Cross layer 2's linear box: split the error into three

WHAT. The pre-activation $z_2$ is built from three ingredients: $z_2=w_2{a}_1+b_2$. Define the error at $z_2$ as ${\delta }_2\equiv \partial L/\partial z_2$. Since $\hat{y}=z_2$ exactly, ${\delta }_2=\partial L/\partial \hat{y}\cdot 1=1$. Now hand this error to each ingredient.

WHY three arrows? The chain rule says: to get $L$'s sensitivity to any ingredient, multiply the incoming error ${\delta }_2$ by how that ingredient affects $z_2$.

$\frac{\partial L}{\partial w_2}={\delta }_2\cdot \underset{=a_1}{\underbrace{\frac{\partial z_2}{\partial w_2}}}=\underset{{\delta }_2}{\underbrace{1}}\cdot \underset{a_1}{\underbrace{0.5}}=0.5,\frac{\partial L}{\partial b_2}={\delta }_2\cdot \underset{=1}{\underbrace{\frac{\partial z_2}{\partial b_2}}}=1,\frac{\partial L}{\partial a_1}={\delta }_2\cdot \underset{=w_2}{\underbrace{\frac{\partial z_2}{\partial a_1}}}=1\cdot 2=2.$

Read each term-by-term:

• Weight gets its input. $\partial z_2/\partial w_2=a_1$ because $w_2$ multiplies $a_1$. So a weight's gradient is error out × signal in.
• Bias gets a plain 1. $b_2$ is just added, so nudging it moves $z_2$ one-for-one.
• The activation $a_1$ gets $w_2$. The error keeps travelling backward through the weight — this is the wire that reaches earlier layers.

PICTURE. The red error arrow hits the layer-2 box and forks into three: a pink branch down to $w_2$ (labelled $\times a_1$), a pink branch down to $b_2$ (labelled $\times 1$), and a blue branch continuing left toward $a_1$ (labelled $\times w_2$).

---

Step 5 — Cross the nonlinearity: multiply by the local slope

WHAT. We have the error arriving at $a_1$ ($=2$). But we want the error at $z_1$, i.e. before the squash. Recall $a_1=\sigma (z_1)$, so:

${\delta }_1\equiv \frac{\partial L}{\partial z_1}=\underset{2}{\underbrace{\frac{\partial L}{\partial a_1}}}\cdot \underset{\text{local slope}}{\underbrace{{\sigma }^{\prime }(z_1)}}.$

WHY the slope? ${\sigma }^{\prime }(z_1)$ is how steeply the activation responds right where we are sitting. If the unit is in a flat region (${\sigma }^{\prime }\approx 0$), a nudge to $z_1$ barely changes $a_1$, so almost no blame passes through — this is exactly the seed of vanishing gradients. For ReLU with $z_1=0.5>0$, the slope is $1$ (ReLU is a straight line of slope 1 on the positive side), so:

${\delta }_1=2\cdot 1=2.$

PICTURE. A small ReLU graph shows the kink at $0$; a dot sits at $z_1=0.5$ on the slope-1 arm. The blue error arrow passes through a "gate" labelled $\times {\sigma }^{\prime }(z_1)=1$ and comes out the other side still carrying $2$.

---

Step 6 — Finish at layer 1: same rule, one layer earlier

WHAT. We now have ${\delta }_1=2$, the error at $z_1$. The last box, $z_1=w_{1}x+b_1$, has the same shape as layer 2, so the same rule applies:

$\frac{\partial L}{\partial w_1}={\delta }_1\cdot \underset{=x}{\underbrace{\frac{\partial z_1}{\partial w_1}}}=2\cdot 1=2,\frac{\partial L}{\partial b_1}={\delta }_1\cdot \underset{=1}{\underbrace{\frac{\partial z_1}{\partial b_1}}}=2.$

WHY this matters. Notice we did not re-run the network. The whole backward pass reused ${\delta }_2$, then ${\delta }_1$, then multiplied by cached inputs. That reuse is the reason backprop costs one backward sweep, not one sweep per weight — the recursion just repeats "error out × signal in" at each stage.

PICTURE. The graph now has all gradients written under their weights and biases in yellow, with the red→blue error arrow shown sweeping fully from right to left across the network.

---

Step 7 — The degenerate case: a "dead" ReLU unit

WHAT. Everything above assumed $z_1>0$. What if the same input had produced $z_1<0$? Then ReLU outputs $a_1=0$ and its slope is ${\sigma }^{\prime }(z_1)=0$.

WHY it's worth its own step. Look at Step 5's gate: multiplying by ${\sigma }^{\prime }=0$ kills the error. Then

${\delta }_1=\frac{\partial L}{\partial a_1}\cdot 0=0\Rightarrow \frac{\partial L}{\partial w_1}=0,\frac{\partial L}{\partial b_1}=0.$

So $w_1$ and $b_1$ receive zero gradient this step — they don't learn from this example at all. The downstream weight $w_2$ still gets a gradient (its input $a_1=0$ though, so $\partial L/\partial w_2={\delta }_2\cdot 0=0$ too here!). This is the "dying ReLU" phenomenon and the exact mechanism behind vanishing gradients.

PICTURE. The same ReLU graph, but the dot now sits on the flat left arm ($z_1<0$). The gate reads $\times 0$, and the error arrow stops dead at the gate — no blue arrow continues to layer 1.

---

Step 8 — Why this scales: fan-in nodes sum their arrows

WHAT. Our toy had one wire per layer. Real nets have a unit's output feeding many downstream units. When error comes back along several paths into the same node, we add them.

WHY add? Each downstream path is an independent way $L$ depends on that node. The multivariable chain rule says total sensitivity = sum of path sensitivities:

$\frac{\partial L}{\partial a}={\sum }_i\frac{\partial L}{\partial z_i^{\text{next}}}\cdot \frac{\partial z_i^{\text{next}}}{\partial a}.$

In matrix form this sum is the $W^{\top }\delta$ of the parent's boxed equation — each output row of $W^{\top }$ dots against all downstream errors, i.e. it sums the paths automatically. (Computing these products efficiently and correctly at scale is what automatic differentiation libraries do for you.)

PICTURE. A node $a$ splits forward into two units; two blue error arrows return and merge at $a$ with a "$+$" symbol, showing the fan-in sum.

---

The one-picture summary

Everything above compressed: the forward pass (yellow, left→right) fills the wires; the backward pass (red seed → blue error → pink weight/bias gradients, right→left) is one sweep. Each weight's gradient is error out × signal in; each nonlinearity multiplies the error by its local slope; fan-ins sum.

Recall Feynman: retell the whole walkthrough

I drew the network as a line of boxes and ran a real number through it, writing the number on every wire and keeping two of them in my pocket (the input $a_1$ and the slope). At the end the loss told me how wrong I was — that's the error, a single red number. I walked that number backwards. At each box I handed it to the box's dials: a weight's blame is the error times the number that flowed into it; a bias's blame is just the error; and to keep travelling I pushed the error back through the weight and shrank it by how steep the squash was right there. If a unit was asleep (flat slope), the blame died at its door and that piece didn't learn. Where one unit fed several others, I added up all the blame coming back. One walk forward, one walk back — and now every dial knows exactly which way to turn. That single backward walk is backpropagation, and it hands its numbers straight to gradient descent.

Connections

• Chain Rule — the single rule every step above secretly uses.
• Computational Graphs — the box picture we drew in Step 1.
• Activation Functions — where the ${\sigma }^{\prime }$ slope in Steps 5 & 7 comes from.
• Vanishing Gradients — the dead-unit degenerate case of Step 7.
• Automatic Differentiation — how Step 8's sums are done at scale.
• Neural Network Surrogate Models (CFD) — the aerospace payoff of cheap gradients.
• Gradient Descent — consumes the gradients this walk produces.