Backpropagation — chain rule, gradient computation

5.6.8 · Coding › Machine Learning (Aerospace Applications)

Backprop exist kyun karta hai?

Yeh kaam kyun karta hai: ek composition ke derivatives factor ho jaate hain ek local derivatives ke product mein (chain rule). Agar hum forward pass ke dauran har layer ka output cache kar lein, toh har local derivative sasta hai aur hum unhe backward jaate hue multiply karte jaate hain.

Chain rule KYA hai (engine)

Backprop scratch se derive karna

Ek chhota sa 1-neuron-per-layer net lo (clarity ke liye saare scalars):

$z_{1} = w_{1} x + b_{1}, a_{1} = σ (z_{1}), z_{2} = w_{2} a_{1} + b_{2}, \overset{y}{^} = z_{2}, L = \frac{1}{2} (\overset{y}{^} - t)^{2} .$

Step 1 — forward pass (sab kuch cache karo). $z_{1}, a_{1}, z_{2}, \overset{y}{^}, L$ compute karo aur unhe store karo. Yeh step kyun? Backward local derivatives (jaise $σ^{'} (z_{1})$ ) ko cached activations chahiye; unhe recompute karna kaam waste karega.

Step 2 — output gradient seed karo. $\frac{\partial L}{\partial y ^} = \overset{y}{^} - t .$ Yeh step kyun? $L = \frac{1}{2} (\overset{y}{^} - t)^{2} \Rightarrow d L / d \overset{y}{^} = (\overset{y}{^} - t)$ . Yeh scalar woh "error signal" hai jo hum propagate karte hain.

Step 3 — layer 2 se push karo. $δ_{2} \equiv \partial L / \partial z_{2} = \partial L / \partial \overset{y}{^} \cdot 1 = (\overset{y}{^} - t)$ define karo. $\frac{\partial L}{\partial w _{2}} = δ_{2} \frac{\partial z _{2}}{\partial w _{2}} = δ_{2} a_{1}, \frac{\partial L}{\partial b _{2}} = δ_{2} .$ Yeh step kyun? $z_{2} = w_{2} a_{1} + b_{2}$ , isliye $\partial z_{2} / \partial w_{2} = a_{1}$ , $\partial z_{2} / \partial b_{2} = 1$ . Ek weight ka gradient = (uske output par error) × (uska input).

Step 4 — nonlinearity cross karo. Error ko $a_{1}$ mein, phir $z_{1}$ mein backprop karo: $\frac{\partial L}{\partial a _{1}} = δ_{2} w_{2}, δ_{1} \equiv \frac{\partial L}{\partial z _{1}} = \frac{\partial L}{\partial a _{1}} σ^{'} (z_{1}) = δ_{2} w_{2} σ^{'} (z_{1}) .$ Yeh step kyun? Error weight $w_{2}$ ke through wapas flow karta hai (transpose direction) aur local slope $σ^{'} (z_{1})$ se modulate hota hai.

Step 5 — layer 1 gradients. $\frac{\partial L}{\partial w _{1}} = δ_{1} x, \frac{\partial L}{\partial b _{1}} = δ_{1} .$ Step 3 jaisa hi pattern hai — yahi recursion hai.

General recursion (vector/matrix form)

Layer $ℓ$ ke liye jahan $z^{(ℓ)} = W^{(ℓ)} a^{(ℓ - 1)} + b^{(ℓ)}$ , $a^{(ℓ)} = σ (z^{(ℓ)})$ :

Worked example (numbers ke saath)

Maano $x = 1, t = 0$ , $w_{1} = 0.5, b_{1} = 0, w_{2} = 2, b_{2} = 0$ , $σ =$ ReLU (toh $σ^{'} (z) = 1$ for $z > 0$ ).

Forward: $z_{1} = 0.5$ , $a_{1} = 0.5$ , $z_{2} = 1.0$ , $\overset{y}{^} = 1$ , $L = \frac{1}{2} (1)^{2} = 0.5$ .
$δ_{2} = \overset{y}{^} - t = 1$ . → $\partial L / \partial w_{2} = δ_{2} a_{1} = 0.5$ , $\partial L / \partial b_{2} = 1$ . Kyun: $w_{2}$ ka gradient = error(1) × uska input $a_{1}$ (0.5).
$δ_{1} = δ_{2} w_{2} σ^{'} (z_{1}) = 1 \cdot 2 \cdot 1 = 2$ . → $\partial L / \partial w_{1} = δ_{1} x = 2$ , $\partial L / \partial b_{1} = 2$ . Kyun: error(1) $w_{2}$ (×2) ke through wapas travel karta hai, slope 1 hai, phir × input $x$ (1).

Common mistakes (steel-manned)

Feynman

Recall Ek 12-saal ke bacche ko samjhao

Socho ek hallway mein logon ki line hai jo ek galat jawaab wapas neeche pass kar rahi hai. Aakhiri insaan dekhta hai answer kitna galat hai (wahi error hai). Woh apne peeche wale insaan ko whisper karta hai ki unki apni choice ne kitna matter kiya — bade darwaaze (weights) zyada blame pass karte hain, aur ek neenda insaan (ek unit jo "off" hai) lagbhag kuch nahi pass karta. Jab tak whisper aage pahunch jaati hai, sabko pata chal jaata hai kitna change karna hai. Ulta karna matlab hai ki hum sirf ek baar whisper karte hain, har insaan se yeh poochne ke bajaaye "agar tumne alag choose kiya hota toh kya hota?"

Active recall

Backprop fundamentally kaun sa calculus rule hai jo backward apply hota hai?

Chain rule (multivariable), cached forward values ko reuse karta hua.

Layer ℓ mein weight-matrix gradient ka formula kya hai?

\partial L / \partial W^{(ℓ)} = δ^{(ℓ)} a^{(ℓ - 1) ⊤}

(error × input, outer product).

Error signal δ layer ℓ+1 se ℓ tak kaise propagate hota hai?

δ^{(ℓ)} = (W^{(ℓ + 1) ⊤} δ^{(ℓ + 1)}) ⊙ σ^{'} (z^{(ℓ)})

Backprop mein

W

ko invert kyun nahi karte, transpose kyun karte hain?

Linear map

W a

ki derivative w.r.t.

a

W^{⊤}

hoti hai; backprop differentiation hai, inversion nahi.

Ek node jiska output kaafi saare paths ko feed karta hai, wahan gradients kaise combine karte hain?

Saare outgoing paths par gradients ka sum karo (multivariable chain rule).

Forward pass mein activations cache kyun karne chahiye?

Backward local derivatives (jaise

σ^{'} (z)

, inputs

a

) ko woh cached values chahiye; recomputing kaam waste karta hai.

MSE loss

L = \frac{1}{2} (\overset{y}{^} - t)^{2}

ke liye seed gradient kya hai?

\partial L / \partial \overset{y}{^} = \overset{y}{^} - t

⊙ σ^{'} (z)

physically kya karta hai?

Har unit ke incoming error ko uski apni activation slope se scale karta hai; saturated units ~0 error pass karti hain.

Ek layer mein bias gradient kya hota hai?

\partial L / \partial b^{(ℓ)} = δ^{(ℓ)}

(kyunki

\partial z / \partial b = 1

Aerospace ML (jaise surrogate aerodynamic models) ke liye backprop kyun zaroori hai?

Yeh ek sweep mein saare gradients lekar huge CFD/flight datasets par deep nets train karna computationally feasible banata hai.

Connections

Gradient Descent — backprop woh gradients supply karta hai jo yeh consume karta hai.
Chain Rule — mathematical engine.
Computational Graphs — woh structure jise backprop traverse karta hai.
Activation Functions — $σ^{'} (z)$ factors supply karte hain.
Vanishing Gradients — repeated small $σ^{'}$ products ki wajah se hota hai.
Automatic Differentiation — reverse-mode AD hi generalized backprop hai.
Neural Network Surrogate Models (CFD) — aerospace use case.