Visual walkthrough — Backpropagation — chain rule, gradient computation

5.6.8 · D2 · Coding › Machine Learning (Aerospace Applications) › Backpropagation — chain rule, gradient computation

Hum bilkul wahi network use karte hain jo parent mein tha, saare numbers scalar hain taaki kuch bhi chhupe nahi:

$z_1=w_{1}x+b_1,a_1=\sigma (z_1),z_2=w_2{a}_1+b_2,\hat{y}=z_2,L=\frac{1}{2}(\hat{y}-t{)}^2.$

Koi bhi calculus se pehle, characters se milte hain.

---

Step 1 — Computation ko boxes ki chain ki tarah draw karo

KYA. Ek neural net koi mysterious blob nahi hai — yeh chhoti-chhoti operations ka ek sequence hai, har ek ek number andar leta hai aur ek number bahar dhakelta hai. Hum ise ek computational graph ki tarah draw karte hain: har box ek operation hai, har wire ek number carry karta hai.

KYUN. Agar hum dekh sakein ki kaun sa number kaun se box mein jaata hai, toh hum baad mein dekh sakte hain ki kaun sa box kasoorwar hai jab jawab galat ho. Backprop is usi picture ko right-to-left padhne ke alawa kuch nahi hai.

PICTURE. Arrows ko left→right padho. Input $x$ andar aata hai, $w_1$ se scale hota hai aur $b_1$ se shift hoke $z_1$ banta hai ("pre-activation"). Nonlinearity $\sigma$ $z_1$ ko $a_1$ mein squeeze karti hai ("activation"). Layer 2 yahi pattern repeat karke $\hat{y}$ banata hai (hamaari prediction). Aakhir mein $L$ $\hat{y}$ ko target $t$ se compare karta hai.

---

Step 2 — Forward pass: har wire ko ek number se bharo, aur yaad rakho

KYA. Graph ko concrete numbers ke saath left→right chalao aur har wire par har value store karo. Parent ke numbers use karte hue: $x=1,t=0,w_1=0.5,b_1=0,w_2=2,b_2=0$, aur $\sigma =$ ReLU (ReLU ka matlab hai "positives rakho, negatives zero karo").

Store kyun karte hain? Backward step ko $a_1$ (layer 2 ka input) aur slope ${\sigma }^{\prime }(z_1)$ chahiye hoga. Dono un numbers par depend karte hain jo hum abhi compute karte hain. Unhe phenk dene ka matlab hai baad mein recompute karna — faaizool. Caching woh trick hai jo backprop ko sasta banati hai.

$\underset{0.5}{\underbrace{z_1}}=\underset{0.5}{\underbrace{w_1}}\underset{1}{\underbrace{x}}+\underset{0}{\underbrace{b_1}},\underset{0.5}{\underbrace{a_1}}=\sigma (0.5),\underset{1.0}{\underbrace{z_2}}=\underset{2}{\underbrace{w_2}}\underset{0.5}{\underbrace{a_1}}+\underset{0}{\underbrace{b_2}},\underset{1}{\underbrace{\hat{y}}}=z_2,\underset{0.5}{\underbrace{L}}=\frac{1}{2}(1-0{)}^2.$

PICTURE. Wahi graph, ab har wire apna numeric tag pehne hua hai. Graph ke neeche ek chhoti "cache" tray hai jisme woh do values ($a_1=0.5$ aur ${\sigma }^{\prime }(z_1)=1$) hain jinhein backward pass waapas lene aayega.

---

Step 3 — Error ko bilkul end par seed karo

KYA. Backward safar output par ek sawaal poochh ke shuru karo: agar $\hat{y}$ thoda sa upar khisak jaaye, toh $L$ kitna badlega? Woh number hai $\partial L/\partial \hat{y}$.

YAHAN KYUN? Sirf aakhri box, loss, seedha jaanta hai "hum kitne galat" hain. Toh blame end par janam leta hai aur peechhe travel karta hai. $L=\frac{1}{2}(\hat{y}-t{)}^2$ ko differentiate karte hue:

$\frac{\partial L}{\partial \hat{y}}=\underset{\text{kitna door hai, sign ke saath}}{\underbrace{(\hat{y}-t)}}=1-0=1.$

$\frac{1}{2}$ aur power $2$ milke kaam karte hain: $\frac{d}{d\hat{y}}\frac{1}{2}{u}^2=u$, toh uljha hua square ek saaf $(\hat{y}-t)$ ban jaata hai. Agar humne zyada predict kiya, toh yeh positive hai ($\hat{y}$ neeche karo); kam predict kiya, toh negative.

PICTURE. $L$ ke bilkul daayein ek akela lal arrow peechhe ki taraf point karta hua dikhta hai, value $1$ carry karte hua. Yeh output par paida hone wala "error signal" hai.

---

Step 4 — Layer 2 ke linear box ko cross karo: error ko teen mein baanto

KYA. Pre-activation $z_2$ teen ingredients se bana hai: $z_2=w_2{a}_1+b_2$. Error at $z_2$ ko define karo ${\delta }_2\equiv \partial L/\partial z_2$. Kyunki $\hat{y}=z_2$ exactly hai, ${\delta }_2=\partial L/\partial \hat{y}\cdot 1=1$. Ab yeh error har ingredient ko do.

TEEN ARROWS KYUN? Chain rule kehta hai: kisi bhi ingredient ke liye $L$ ki sensitivity paane ke liye, incoming error ${\delta }_2$ ko us ingredient ka $z_2$ par effect se multiply karo.

$\frac{\partial L}{\partial w_2}={\delta }_2\cdot \underset{=a_1}{\underbrace{\frac{\partial z_2}{\partial w_2}}}=\underset{{\delta }_2}{\underbrace{1}}\cdot \underset{a_1}{\underbrace{0.5}}=0.5,\frac{\partial L}{\partial b_2}={\delta }_2\cdot \underset{=1}{\underbrace{\frac{\partial z_2}{\partial b_2}}}=1,\frac{\partial L}{\partial a_1}={\delta }_2\cdot \underset{=w_2}{\underbrace{\frac{\partial z_2}{\partial a_1}}}=1\cdot 2=2.$

Har term term-by-term padho:

• Weight ko apna input milta hai. $\partial z_2/\partial w_2=a_1$ kyunki $w_2$, $a_1$ se multiply hota hai. Toh ek weight ka gradient hai error out × signal in.
• Bias ko seedha 1 milta hai. $b_2$ sirf add hota hai, toh ise nudge karne se $z_2$ one-for-one move karta hai.
• Activation $a_1$ ko $w_2$ milta hai. Error weight ke through peechhe chalta rehta hai — yeh woh wire hai jo pehle ki layers tak pahunchti hai.

PICTURE. Lal error arrow layer-2 box se takraata hai aur teen mein fork ho jaata hai: ek pink branch $w_2$ ki taraf neeche (labelled $\times a_1$), ek pink branch $b_2$ ki taraf neeche (labelled $\times 1$), aur ek blue branch $a_1$ ki taraf baayein continue karta hua (labelled $\times w_2$).

---

Step 5 — Nonlinearity cross karo: local slope se multiply karo

KYA. Hamare paas $a_1$ par pahunchta hua error hai ($=2$). Lekin hum $z_1$ par error chahte hain, yaani squash se pehle. Yaad karo $a_1=\sigma (z_1)$, toh:

${\delta }_1\equiv \frac{\partial L}{\partial z_1}=\underset{2}{\underbrace{\frac{\partial L}{\partial a_1}}}\cdot \underset{\text{local slope}}{\underbrace{{\sigma }^{\prime }(z_1)}}.$

SLOPE KYUN? ${\sigma }^{\prime }(z_1)$ hai activation kitni steeply respond karti hai bilkul wahan jahan hum baithe hain. Agar unit ek flat region mein hai (${\sigma }^{\prime }\approx 0$), toh $z_1$ ko nudge karne se $a_1$ mein barely koi badlaav aata hai, toh almost koi blame pass-through nahi hota — yahi vanishing gradients ka seed hai. ReLU ke liye jab $z_1=0.5>0$, slope $1$ hai (ReLU positive side par slope-1 ki seedhi line hai), toh:

${\delta }_1=2\cdot 1=2.$

PICTURE. Ek chhota ReLU graph $0$ par kink dikhata hai; ek dot slope-1 arm par $z_1=0.5$ par baitha hai. Blue error arrow ek "gate" se guzarta hai jis par $\times {\sigma }^{\prime }(z_1)=1$ likha hai aur doosri taraf abhi bhi $2$ carry karta hua nikalta hai.

---

Step 6 — Layer 1 par finish karo: wahi rule, ek layer pehle

KYA. Ab hamare paas ${\delta }_1=2$ hai, $z_1$ par error. Aakhri box, $z_1=w_{1}x+b_1$, layer 2 jaisi hi shape rakhta hai, toh wahi rule apply hoga:

$\frac{\partial L}{\partial w_1}={\delta }_1\cdot \underset{=x}{\underbrace{\frac{\partial z_1}{\partial w_1}}}=2\cdot 1=2,\frac{\partial L}{\partial b_1}={\delta }_1\cdot \underset{=1}{\underbrace{\frac{\partial z_1}{\partial b_1}}}=2.$

YEH KYUN MATTER KARTA HAI. Notice karo ki humne network dobara nahi chalaya. Poore backward pass ne ${\delta }_2$, phir ${\delta }_1$ reuse kiya, phir cached inputs se multiply kiya. Yahi reuse woh reason hai ki backprop ek backward sweep leta hai, weights ki sankhya ke barabar sweeps nahi — recursion bas "error out × signal in" repeat karta rehta hai har stage par.

PICTURE. Graph mein ab saare gradients unke weights aur biases ke neeche yellow mein likhe hain, aur lal→neela error arrow poore network mein right to left sweep karte hua dikhaya gaya hai.

---

Step 7 — Degenerate case: ek "dead" ReLU unit

KYA. Upar sab kuch $z_1>0$ assume karta tha. Kya hoga agar wahi input $z_1<0$ produce karta? Tab ReLU $a_1=0$ output karta hai aur uska slope ${\sigma }^{\prime }(z_1)=0$ hai.

APNE STEP KI ZAROORAT KYUN? Step 5 ka gate dekho: ${\sigma }^{\prime }=0$ se multiply karne par error mar jaata hai. Phir

${\delta }_1=\frac{\partial L}{\partial a_1}\cdot 0=0\Rightarrow \frac{\partial L}{\partial w_1}=0,\frac{\partial L}{\partial b_1}=0.$

Toh $w_1$ aur $b_1$ ko is step mein zero gradient milta hai — woh is example se bilkul nahi seekhte. Downstream weight $w_2$ ko abhi bhi gradient milta hai (lekin uska input $a_1=0$ hai, toh $\partial L/\partial w_2={\delta }_2\cdot 0=0$ bhi yahan!). Yahi "dying ReLU" phenomenon hai aur vanishing gradients ke peechhe ka exact mechanism.

PICTURE. Wahi ReLU graph, lekin dot ab flat left arm par hai ($z_1<0$). Gate par $\times 0$ likha hai, aur error arrow gate par bilkul ruk jaata hai — layer 1 ki taraf koi blue arrow continue nahi karta.

---

Step 8 — Yeh scale kyun karta hai: fan-in nodes apne arrows add karte hain

KYA. Hamare toy mein ek wire per layer tha. Real nets mein ek unit ka output kai downstream units ko feed karta hai. Jab error kai paths se usi node mein waapas aata hai, hum unhe add karte hain.

ADD KYUN? Har downstream path ek independent tarika hai jisse $L$ us node par depend karta hai. Multivariable chain rule kehta hai: total sensitivity = path sensitivities ka sum:

$\frac{\partial L}{\partial a}={\sum }_i\frac{\partial L}{\partial z_i^{\text{next}}}\cdot \frac{\partial z_i^{\text{next}}}{\partial a}.$

Matrix form mein yeh sum parent ki boxed equation ka $W^{\top }\delta$ hai — $W^{\top }$ ki har output row saare downstream errors se dot karta hai, yaani woh automatically paths add kar deta hai. (Yeh products efficiently aur sahi tarike se scale par compute karna wahi kaam hai jo automatic differentiation libraries tumhare liye karti hain.)

PICTURE. Ek node $a$ aage do units mein split hota hai; do blue error arrows waapas aate hain aur $a$ par ek "$+$" symbol ke saath merge hote hain, fan-in sum dikhate hue.

---

Ek-picture summary

Upar sab kuch compress karke: forward pass (yellow, left→right) wires bharta hai; backward pass (red seed → blue error → pink weight/bias gradients, right→left) ek sweep hai. Har weight ka gradient hai error out × signal in; har nonlinearity error ko apni local slope se multiply karti hai; fan-ins add karte hain.

Recall Feynman: poora walkthrough dobara batao

Maine network ko boxes ki ek line ki tarah draw kiya aur ek real number iske through chalaya, har wire par number likhta gaya aur unmen se do apni jeb mein rakhta gaya (input $a_1$ aur slope). End mein loss ne bataya main kitna galat tha — woh error hai, ek akela lal number. Maine woh number peechhe ki taraf chalaya. Har box par maine use box ke dials ko diya: ek weight ki blame hai error times woh number jo usmein aaya tha; ek bias ki blame sirf error hai; aur aage travel karte rehne ke liye maine error ko weight ke through peechhe dhakela aur ise utna shrink kiya jitna steep woh squash wahan tha. Agar koi unit so raha tha (flat slope), toh blame uske darwaze par mar gayi aur woh hissa nahi seekha. Jahan ek unit ne kai logon ko feed kiya, maine saari blame waapas aake jo aayi add kar li. Ek walk forward, ek walk back — aur ab har dial ko pata hai bilkul kis taraf ghoomna hai. Woh ek backward walk backpropagation hai, aur woh apne numbers seedhe gradient descent ko deta hai.

Connections

• Chain Rule — woh akela rule jo upar har step secretly use karta hai.
• Computational Graphs — woh box picture jo humne Step 1 mein draw ki.
• Activation Functions — jahan se Steps 5 & 7 mein ${\sigma }^{\prime }$ slope aata hai.
• Vanishing Gradients — Step 7 ka dead-unit degenerate case.
• Automatic Differentiation — kaise Step 8 ke sums scale par kiye jaate hain.
• Neural Network Surrogate Models (CFD) — saste gradients ka aerospace payoff.
• Gradient Descent — yeh walk jo gradients produce karta hai unhe consume karta hai.