Worked examples — Backpropagation — chain rule, gradient computation

5.6.8 · D3 · Coding › Machine Learning (Aerospace Applications) › Backpropagation — chain rule, gradient computation

---

Scenario matrix

Kuch bhi work karne se pehle, chalte hain har alag situation list karein jo backprop problem mein ho sakti hai. Har row ek "case class" hai; neeche har worked example us cell ke saath tagged hai jo wo cover karta hai.

#	Case class	Kya khaas hai	Covered by
A	Linear output, positive slope	ReLU with $z>0$, ${\sigma }^{\prime }=1$ — the "clean" case	Ex 1
B	Dead ReLU ($z<0$)	${\sigma }^{\prime }(z)=0$ — unit zero error backward pass karta hai	Ex 2
C	Sigmoid, mid-range	${\sigma }^{\prime }$ ek fraction hai $\in (0,\frac{1}{4}]$ — error shrink karta hai	Ex 3
D	Saturated sigmoid	${\sigma }^{\prime }\to 0$ — vanishing-gradient failure	Ex 4
E	Fan-in node (weight do paths ko feed karta hai)	paths par sum karna padta hai (multivariable rule)	Ex 5
F	Zero / degenerate input ($x=0$ ya $a=0$)	weight gradient $0$ ho jaata hai	Ex 6
G	Sign bookkeeping ($t>\hat{y}$ vs $t<\hat{y}$)	error signal ka sign badalta hai, aur saath mein har gradient ka bhi	Ex 7
H	Real-world word problem (aerospace surrogate)	physics → net → gradient translate karo	Ex 8
I	Exam twist (shared weight / weight tying)	ek weight do baar use hota hai ⇒ gradients add hote hain	Ex 9

Hum poore time yahi chhota network reuse karenge jab tak kuch aur na kaha jaaye:

$z_1=w_{1}x+b_1,a_1=\sigma (z_1),z_2=w_2{a}_1+b_2,\hat{y}=z_2,L=\frac{1}{2}(\hat{y}-t{)}^2.$

Yahan $x$ input number hai, $w$ weights hain (multipliers), $b$ biases hain (offsets), $\sigma$ activation function hai (per-unit nonlinear bend), $\hat{y}$ network ki guess hai, $t$ true target hai, aur $L$ loss hai (kitna galat hain hum). Neeche ki picture wo map hai jis par hum backward chalte hain.

---

Ex 1 — Case A: linear/ReLU, positive slope (clean baseline)

Forecast: Padhne se pehle dono gradients ka sign guess karo. (Hint: $\hat{y}$ target $0$ ko overshoot karega, toh hum expect karte hain positive gradients jo weights ko neeche push karein.)

1. Forward, sab kuch cache karo. $z_1=0.5\cdot 1+0=0.5$; kyunki $z_1>0$ hai, $a_1=\sigma (0.5)=0.5$; $z_2=2\cdot 0.5+0=1.0$; $\hat{y}=1.0$; $L=\frac{1}{2}(1-0{)}^2=0.5$. Yeh step kyun? Backward slopes (${\sigma }^{\prime }(z_1)$) aur inputs ($a_1$) inhi cached numbers se padte hain — recompute karna time waste hai.
2. Seed. $\partial L/\partial \hat{y}=\hat{y}-t=1-0=1$. Kyun? $L=\frac{1}{2}(\hat{y}-t{)}^2\Rightarrow dL/d\hat{y}=(\hat{y}-t)$.
3. Layer 2. ${\delta }_2=\partial L/\partial z_2=1\cdot 1=1$ (output linear hai, slope $1$). Phir $\partial L/\partial w_2={\delta }_2{a}_1=1\cdot 0.5=0.5$. Kyun? weight gradient = error out × signal in.
4. Nonlinearity cross karo. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\cdot 2\cdot 1=2$. Kyun? error $w_2$ ke through backward flow karta hai ($\times 2$), phir local slope ${\sigma }^{\prime }(z_1)=1$ se scale hota hai.
5. Layer 1. $\partial L/\partial w_1={\delta }_{1}x=2\cdot 1=2$.

Verify: $w_1$ ko $ϵ=10^{-4}$ bump karo: $\hat{y}=1.0002$, $L\approx 0.50020$, toh $\Delta L/ϵ\approx 2.0$ ✓. Dono gradients positive jaise forecast kiya tha — gradient descent inhe subtract karega, weights ko better fit ki taraf le jaayega.

---

Ex 2 — Case B: Dead ReLU (error ek wall se takraata hai)

Forecast: Kya $w_1$ ko koi bhi gradient milega?

1. Forward. $z_1=-3\cdot 1+1=-2$. Kyunki $z_1<0$ hai, $a_1=\max (0,-2)=0$. Yeh kyun note karein? ReLU switch off ho gaya — uska output $0$ par stuck hai.
2. Local slope. ${\sigma }^{\prime }(z_1)=0$ kyunki $z<0$ ke liye ReLU graph flat hai. Kyun matter karta hai: flat function ka derivative zero hota hai — $z_1$ ko nudge karna $a_1$ par kuch nahi karta.
3. Backward. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=(\dots )\times 0=0$, isliye $\partial L/\partial w_1={\delta }_{1}x=0$. Kyun? $\odot {\sigma }^{\prime }$ factor error ko annihilate kar deta hai — $w_1$ tak koi signal nahi pahunchta.

Verify: $w_1$ ko $ϵ$ bump karo: $z_1$ moves to $-2+ϵ$ (abhi bhi $<0$ hai), toh $a_1$ $0$ par rehta hai, $\hat{y}$ $0$ par rehta hai, $L$ unchanged $\Rightarrow \Delta L/ϵ=0$ ✓.

---

Ex 3 — Case C: sigmoid mid-range (error ek fraction se shrink karta hai)

Forecast: $z_1=0$ par sigmoid apni steepest point par hai. Woh maximum slope kya hai?

1. Forward. $z_1=0$; $a_1=\sigma (0)=\frac{1}{1+1}=0.5$; $z_2=1\cdot 0.5=0.5$; $\hat{y}=0.5$. Kyun? Slope evaluate karne ke liye hume $a_1=\sigma (z_1)$ chahiye.
2. Local slope. ${\sigma }^{\prime }(0)=0.5(1-0.5)=0.25$. Yeh tool (product $\sigma (1-\sigma )$) kyun? Sigmoid ki derivative ko uske apne output se likha ja sakta hai, isliye ek baar $a_1$ cache ho jaaye toh slope free mein milta hai — yeh ek badi wajah thi jab sigmoids popular the.
3. Seed & backward. $\partial L/\partial \hat{y}=0.5-0=0.5$; ${\delta }_2=0.5$; phir ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0.5\cdot 1\cdot 0.25=0.125$. Kyun? aata hua error $0.5$, slope se quarter tak throttle ho jaata hai.

Verify: ${\delta }_1=0.125$. $z_1=0$ ke around $\partial L/\partial z_1$ finite-difference karo: $ϵ=10^{-4}$ bump karo, $L$ $\approx 0.125ϵ$ change hota hai ✓. Note karo ${\sigma }^{\prime }$ kabhi $0.25$ se zyada nahi hoti — har sigmoid layer error ko kam se kam $4\times$ shrink karti hai.

---

Ex 4 — Case D: saturated sigmoid (vanishing gradient)

Forecast: Bada $z$ matlab $\sigma \approx 1$ — kya slope bada hai ya tiny?

1. Forward. $\sigma (8)=\frac{1}{1+e^{-8}}\approx 0.99966$. Kyun? Slope compute karne ke liye output chahiye.
2. Slope. ${\sigma }^{\prime }(8)=0.99966(1-0.99966)\approx 0.000335$. Kyun matter karta hai: S-curve origin se door flat hai — unit saturated hai.
3. Backward. Chahe koi bhi error aaye, ${\delta }_1=(\dots )\times 0.000335$, $\sim 3000$ ke factor se crush ho jaata hai. Kyun? Aisi kuch layers stack karo aur tiny slopes ka product $\to 0$ — gradients vanish ho jaate hain.

Verify: ${\sigma }^{\prime }(8)\approx 3.35\times 10^{-4}$ ✓. Ex 3 ke $0.25$ se compare karo: same function, sirf bade $z$ se $\sim 750\times$ kamzor gradient. Yehi wajah hai kyun ReLU ne deep nets mein sigmoids replace kiye — dekho Vanishing Gradients.

---

Ex 5 — Case E: fan-in node (paths par sum)

Forecast: $u$ $L$ ko do wires ($p$ aur $q$) se influence karta hai. Kya hum ek choose karein ya add karein?

1. Forward. $u=2$; $p=6$; $q=-2$; $s=4$; $L=\frac{1}{2}(16)=8$.
2. Seed & split. $\partial L/\partial s=s=4$. Kyunki $s=p+q$ hai: $\partial L/\partial p=4$, $\partial L/\partial q=4$. Kyun? $\partial s/\partial p=1$, $\partial s/\partial q=1$.
3. Fan-in par multivariable chain rule. $u$ $L$ tak $p$ aur $q$ ke zariye pahunchta hai, isliye hum do path contributions ko sum karte hain: $\frac{\partial L}{\partial u}=\frac{\partial L}{\partial p}\frac{\partial p}{\partial u}+\frac{\partial L}{\partial q}\frac{\partial q}{\partial u}=4\cdot 3+4\cdot (-1)=12-4=8.$ Sum kyun, choose kyun nahi? Har path independent sensitivity carry karta hai; total sensitivity unka sum hai — yeh hai graph fan-in rule.
4. Input tak. $\partial L/\partial x=\partial L/\partial u\cdot \partial u/\partial x=8\cdot 1=8$.

Verify: Note karo $s=3u-u=2u$ hai, toh $L=\frac{1}{2}(2u{)}^2=2u^2$ hai, aur $dL/du=4u=8$ at $u=2$ ✓ — closed form summed backprop ko confirm karta hai.

---

Ex 6 — Case F: zero / degenerate input (gradient collapse)

Forecast: Agar input zero hai, toh kya $w_1$ phir bhi seekhega?

1. Forward. $z_1=w_1\cdot 0+b_1=0.5$; $a_1=0.5$ (ReLU, $z_1>0$); $z_2=2\cdot 0.5=1$; $\hat{y}=1$; $L=0.5$.
2. $z_1$ par backward error. ${\delta }_2=\hat{y}-t=1$; ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\cdot 2\cdot 1=2$.
3. Weight vs bias gradient. $\frac{\partial L}{\partial w_1}={\delta }_{1}x=2\cdot 0=0,\frac{\partial L}{\partial b_1}={\delta }_1\cdot 1=2.$ Split kyun? $w_1$ ka gradient error × uska input hai, aur uska input $0$ hai — toh $w_1$ ko koi update nahi milta. Bias ka "input" constant $1$ hai, isliye woh abhi bhi seekhta hai.

Verify: $w_1$ bump karo: $z_1=w_1\cdot 0+0.5$ $w_1$ se independent hai, toh $L$ kabhi nahi badlega $\Rightarrow \partial L/\partial w_1=0$ ✓. $b_1$ ko $ϵ$ bump karo: $\hat{y}$ $2ϵ$ badhta hai, $L$ $\approx 2ϵ$ badhta hai, toh $\partial L/\partial b_1=2$ ✓.

---

Ex 7 — Case G: sign bookkeeping (undershoot vs overshoot)

Forecast: Ex 1 mein guess too high thi; yahan too low hai. Kaunsa flip hoga?

1. Forward. Ex 1 jaisa hi: $a_1=0.5$, $\hat{y}=1$, lekin ab $L=\frac{1}{2}(1-2{)}^2=0.5$.
2. Seed ka sign badalta hai. $\partial L/\partial \hat{y}=\hat{y}-t=1-2=-1$. Kyun? Error signal hai hi $\hat{y}-t$; undershoot karna ise negative banata hai.
3. Propagate. ${\delta }_2=-1$; $\partial L/\partial w_2={\delta }_2{a}_1=-1\cdot 0.5=-0.5$. Flip kyun? Gradient descent gradient subtract karta hai: negative gradient matlab "increase $w_2$" — exactly sahi, kyunki $\hat{y}$ too small hai.

Verify: $\partial L/\partial w_2=-0.5$, Ex 1 ke $+0.5$ ka negation ✓. Finite difference: $w_2$ ko $ϵ$ bump karo, $\hat{y}=1+0.5ϵ$, $L=\frac{1}{2}(1-2+0.5ϵ{)}^2\approx 0.5-0.5ϵ$, slope $-0.5$ ✓. Seed ka sign har downstream gradient ka sign set karta hai.

---

Ex 8 — Case H: aerospace word problem (ek lift surrogate)

Forecast: Kya model lift over- ya under-predict kar raha hai, aur $w$ upar jaayega ya neeche?

1. Forward. $z=6\cdot 0.1+0.1=0.7$; ReLU $\Rightarrow {\hat{C}}_L=0.7$; $L=\frac{1}{2}(0.7-0.9{)}^2=\frac{1}{2}(0.04)=0.02$. Kyun? Prediction evaluate karna padega pehle, tab hi error measure ho sakta hai.
2. Seed. $\partial L/\partial {\hat{C}}_L=0.7-0.9=-0.2$ (hum lift underpredict kar rahe hain).
3. ReLU cross karo. $z=0.7>0\Rightarrow {\sigma }^{\prime }=1$, toh $\delta =-0.2\cdot 1=-0.2$.
4. Weight gradient. $\partial L/\partial w=\delta \alpha =-0.2\cdot 0.1=-0.02$. Kyun? $w$ mein jaata input $\alpha$ hai.
5. Update direction. Gradient descent: $w\leftarrow w-\eta (-0.02)$ — negative gradient $w$ ko raise karta hai, lift curve ko steep karta hai taaki ${\hat{C}}_L$ $0.9$ ki taraf chadhe. ✓ physically sensible.

Verify: $\partial L/\partial w=-0.02$; $\partial L/\partial b=\delta =-0.2$. $w$ par finite difference: $ϵ$ bump karo, ${\hat{C}}_L=0.7+0.1ϵ$, $L=\frac{1}{2}(-0.2+0.1ϵ{)}^2\approx 0.02-0.02ϵ$, slope $-0.02$ ✓.

---

Ex 9 — Case I: exam twist (ek weight do baar use hona / weight tying)

Forecast: $w$ do baar appear karta hai — kya $\partial L/\partial w$ ek term hai ya do ka sum?

1. Forward. $y_1=2\cdot 1=2$; $y_2=2\cdot 2=4$; $L=\frac{1}{2}(16)=8$.
2. Seed. $\partial L/\partial y_2=y_2=4$.
3. Do uses ⇒ do paths. $w$ ki har occurrence ko ek distinct copy maano — $w_a$ (in $y_1=w_{a}x$) aur $w_b$ (in $y_2=w_b{y}_1$) — phir unke gradients add karo (multivariable chain rule):
   • via $w_b$: $\partial L/\partial w_b=\partial L/\partial y_2\cdot y_1=4\cdot 2=8$.
   • via $w_a$: $\partial L/\partial y_1=\partial L/\partial y_2\cdot w=4\cdot 2=8$; phir $\partial L/\partial w_a=8\cdot x=8$. $\frac{\partial L}{\partial w}=8+8=16.$ Sum kyun? Shared weight graph mein ek fan-in node hai — aap har jagah se contribute hua gradient sum karte hain.
4. Update. $w\leftarrow w-\eta \cdot 16$.

Verify: Closed form: $y_2=w\cdot (wx)=w^{2}x$ hai, toh $L=\frac{1}{2}{w}^4{x}^2$ hai; $dL/dw=2w^3{x}^2=2\cdot 8\cdot 1=16$ ✓. Ek occurrence ignore karne se $8$ milta — true gradient ka aadha — yeh classic exam trap hai.

---

Recall

Connections

• Backpropagation (parent) — woh machinery jo yeh examples exercise karti hain.
• Chain Rule — har "sum over paths" ke peeche ka engine.
• Gradient Descent — hamare compute kiye gradients consume karke weights update karta hai.
• Vanishing Gradients — Cases B aur D iske root causes hain.
• Activation Functions — ReLU vs sigmoid ne upar ke slope factors drive kiye.
• Computational Graphs — fan-in nodes (Ex 5, Ex 9) graph structure hain.
• Automatic Differentiation — exactly isi bookkeeping ko automate karta hai.
• Neural Network Surrogate Models (CFD) — Ex 8 ka aerospace context.