Backpropagation — chain rule, gradient computation

WHY does backprop exist?

WHY it works: derivatives of a composition factor into a product of local derivatives (chain rule). If we cache each layer's output during the forward pass, each local derivative is cheap and we multiply them together going backward.

WHAT is the chain rule (the engine)

Deriving backprop from scratch

Take a tiny 1-neuron-per-layer net (all scalars for clarity):

$z_1 = w_1 x + b_1,\quad a_1=\sigma(z_1),\quad z_2=w_2 a_1 + b_2,\quad \hat y=z_2,\quad L=\tfrac12(\hat y - t)^2.$

Step 1 — forward pass (cache everything). Compute $z_1,a_1,z_2,\hat y,L$ and store them. Why this step? Backward local derivatives (like $\sigma'(z_1)$ ) need the cached activations; recomputing them would waste work.

Step 2 — seed the output gradient. $\frac{\partial L}{\partial \hat y} = \hat y - t.$ Why this step? $L=\tfrac12(\hat y-t)^2 \Rightarrow dL/d\hat y = (\hat y - t)$ . This scalar is the "error signal" we propagate.

Step 3 — push through layer 2. Define $\delta_2 \equiv \partial L/\partial z_2 = \partial L/\partial\hat y \cdot 1 = (\hat y - t)$ . $\frac{\partial L}{\partial w_2}= \delta_2\,\frac{\partial z_2}{\partial w_2}=\delta_2\, a_1,\qquad \frac{\partial L}{\partial b_2}=\delta_2.$ Why this step? $z_2=w_2 a_1+b_2$ , so $\partial z_2/\partial w_2=a_1$ , $\partial z_2/\partial b_2=1$ . A weight's gradient = (error at its output) × (its input).

Step 4 — cross the nonlinearity. Backprop the error into $a_1$ , then into $z_1$ : $\frac{\partial L}{\partial a_1}=\delta_2\, w_2,\qquad \delta_1\equiv\frac{\partial L}{\partial z_1}=\frac{\partial L}{\partial a_1}\,\sigma'(z_1)=\delta_2\, w_2\,\sigma'(z_1).$ Why this step? Error flows back through the weight $w_2$ (transpose direction) and gets modulated by the local slope $\sigma'(z_1)$ .

Step 5 — layer 1 gradients. $\frac{\partial L}{\partial w_1}=\delta_1 x,\qquad \frac{\partial L}{\partial b_1}=\delta_1.$ Same pattern as Step 3 — that's the recursion.

The general recursion (vector/matrix form)

For layer $\ell$ with $z^{(\ell)}=W^{(\ell)}a^{(\ell-1)}+b^{(\ell)}$ , $a^{(\ell)}=\sigma(z^{(\ell)})$ :

Formula Backprop equations

$\delta^{(L)}=\nabla_{\hat y}L \odot \sigma'(z^{(L)})$ $\boxed{\;\delta^{(\ell)}=\big(W^{(\ell+1)\top}\delta^{(\ell+1)}\big)\odot\sigma'(z^{(\ell)})\;}$ $\frac{\partial L}{\partial W^{(\ell)}}=\delta^{(\ell)}\,a^{(\ell-1)\top},\qquad \frac{\partial L}{\partial b^{(\ell)}}=\delta^{(\ell)}$ WHY $W^\top$ : forward multiplies by $W$ ; the derivative of a linear map is its transpose, so error flows back through $W^\top$ . WHY $\odot\sigma'$ : each unit scales its incoming error by its own activation slope. WHY outer product $\delta a^\top$ : $\partial(Wa)/\partial W_{ij}=a_j$ picks out input $j$ for output $i$ .

Worked example (numbers)

Let $x=1,\ t=0$ , $w_1=0.5,\ b_1=0,\ w_2=2,\ b_2=0$ , $\sigma=$ ReLU (so $\sigma'(z)=1$ for $z>0$ ).

Forward: $z_1=0.5$ , $a_1=0.5$ , $z_2=1.0$ , $\hat y=1$ , $L=\tfrac12(1)^2=0.5$ .
$\delta_2=\hat y-t=1$ . → $\partial L/\partial w_2=\delta_2 a_1=0.5$ , $\partial L/\partial b_2=1$ . Why: gradient of $w_2$ = error(1) × its input $a_1$ (0.5).
$\delta_1=\delta_2 w_2 \sigma'(z_1)=1\cdot 2\cdot 1=2$ . → $\partial L/\partial w_1=\delta_1 x=2$ , $\partial L/\partial b_1=2$ . Why: error(1) travels back through $w_2$ (×2), slope is 1, then × input $x$ (1).

Common mistakes (steel-manned)

Feynman

Recall Explain to a 12-year-old

Imagine a line of people passing a wrong answer back down a hallway. The last person sees how wrong the answer is (that's the error). They whisper to the person behind them how much their choice mattered — bigger doors (weights) pass more blame, and a sleepy person (a unit that's "off") passes almost none. By the time the whisper reaches the front, everyone knows exactly how much to change. Doing it backwards means we only whisper once instead of asking every person "what if you'd chosen differently?"

Active recall

Backprop is fundamentally which calculus rule applied backward?

The chain rule (multivariable), reusing cached forward values.

Formula for a weight-matrix gradient in layer ℓ?

\partial L/\partial W^{(\ell)} = \delta^{(\ell)} a^{(\ell-1)\top}

(error × input, outer product).

How does the error signal δ propagate from layer ℓ+1 to ℓ?

\delta^{(\ell)} = (W^{(\ell+1)\top}\delta^{(\ell+1)}) \odot \sigma'(z^{(\ell)})

Why transpose

W

and not invert it during backprop?

The derivative of a linear map

Wa

w.r.t.

a

W^\top

; backprop is differentiation, not inversion.

At a node whose output feeds several paths, how do you combine gradients?

Sum the gradients over all outgoing paths (multivariable chain rule).

Why must you cache activations in the forward pass?

Backward local derivatives (e.g.

\sigma'(z)

, inputs

a

) need those cached values; recomputing wastes work.

Seed gradient for MSE loss

L=\tfrac12(\hat y - t)^2

\partial L/\partial \hat y = \hat y - t

What does

\odot \sigma'(z)

do physically?

Scales each unit's incoming error by its own activation slope; saturated units pass ~0 error.

Bias gradient in a layer equals what?

\partial L/\partial b^{(\ell)} = \delta^{(\ell)}

(since

\partial z/\partial b = 1

Why is backprop essential for aerospace ML (e.g. surrogate aerodynamic models)?

It makes training deep nets on huge CFD/flight datasets computationally feasible by getting all gradients in one sweep.

Connections

Gradient Descent — backprop supplies the gradients it consumes.
Chain Rule — the mathematical engine.
Computational Graphs — the structure backprop traverses.
Activation Functions — supply the $\sigma'(z)$ factors.
Vanishing Gradients — caused by repeated small $\sigma'$ products.
Automatic Differentiation — reverse-mode AD is generalized backprop.
Neural Network Surrogate Models (CFD) — aerospace use case.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Backpropagation ka core idea bilkul simple hai: neural network ek badi composed function hai (function ke andar function ke andar function), aur hume har weight ke liye gradient chahiye taaki gradient descent se training ho sake. Agar hum har weight ko thoda-thoda badal ke poora network dobara chalayein, toh millions of weights ke liye ye kaam bahut slow ho jaata. Backprop chain rule ko ulta (backward) lagaakar sirf ek forward aur ek backward pass mein saare gradients nikaal deta hai — isliye deep learning practical bana.

Kaise? Forward pass mein hum har layer ke $z$ aur activation $a$ ko cache kar lete hain. Phir output pe error signal $\delta = \hat y - t$ banate hain, aur usko peeche ki taraf bhejte hain. Har layer cross karte waqt teen cheezein hoti hain: error ko $W^\top$ se multiply karo (transpose, kyunki forward mein $W$ se multiply hua tha — inverse nahi, transpose!), phir activation ka slope $\sigma'(z)$ se multiply karo, aur weight ka gradient nikaalne ke liye us error ko us weight ke input se multiply karo. Yaad rakho: har weight ka gradient = uska error out × uska signal in.

Common galti: log $\sigma'(z)$ multiply karna bhool jaate hain, ya $W^{-1}$ lene lagte hain — dono galat. Aur jahan ek node kai jagah jaata hai, wahan saare paths ke gradients add karne padte hain (multivariable chain rule).

Aerospace mein ye kyun important hai? Jab hum CFD ya flight-test ke huge data pe neural network surrogate model train karte hain (jaise drag/lift predict karna bina full simulation ke), tab backprop hi hai jo lakhon parameters ko efficiently tune karta hai. Iske bina modern ML aerospace design loops chal hi nahi sakte.

Backpropagation — chain rule, gradient computation

WHY does backprop exist?

WHAT is the chain rule (the engine)

Deriving backprop from scratch

The general recursion (vector/matrix form)

Worked example (numbers)

Common mistakes (steel-manned)

Feynman

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Machine Learning (Aerospace Applications)

Connections