Exercises — Backpropagation — chain rule, gradient computation

Chain rule. Nested (composed) functions ke liye, sensitivities multiply hoti hain: agar $x$ thoda sa wiggle kare, toh $f_1$ wiggle karta hai $f_1^{\prime }$ times us amount se, phir $f_2$ wiggle karta hai $f_2^{\prime }$ times us se, aur aise aage. In scalings ko chain karne se milta hai $\frac{dL}{dx}=f_3^{\prime }\cdot f_2^{\prime }\cdot f_1^{\prime }.$ Sum tab aata hai jab $x$, $L$ tak alag-alag kai paths se pahunchta hai (multivariable chain rule); ek seedha chain ek hi path hai, isliye yeh pure product hai. Dekho Chain Rule.

$\frac{1}{2}(\hat{y}-t{)}^2$ ko $\hat{y}$ ke respect mein differentiate karo: $\frac{1}{2}$ aur power-2 cancel ho jaate hain, bacha $\frac{\partial L}{\partial \hat{y}}=\hat{y}-t.$ Yeh seed isliye hai kyunki yeh backward pass ka sabse pehla number hai — "hum kitne galat hain, aur kis direction mein" wali quantity, jisse baad ke saare gradients ugaaye jaate hain.

Forward step inputs ko matrix $W$ se multiply karta hai (ek linear map). Input $a$ ke respect mein linear map $Wa$ ki derivative transpose $W^{\top }$ hai (rows aur columns swap ho jaate hain), inverse nahi. Backprop differentiation hai, aur ek linear map ko differentiate karna use transpose karta hai; yahan kuch bhi undo/invert nahi ho raha.

Figure 1 ke orange arrows walk karo: $z_1=w_{1}x+b_1=0.5(1)+0=0.5,a_1=\text{ReLU}(0.5)=0.5,$ $z_2=w_2{a}_1+b_2=2(0.5)+0=1.0,\hat{y}=z_2=1.0,$ $L=\frac{1}{2}(\hat{y}-t{)}^2=\frac{1}{2}(1-0{)}^2=\mathrm{0.5.}$

Seed phir chain. Kyunki $\hat{y}=z_2$ hai, $\partial \hat{y}/\partial z_2=1$, isliye ${\delta }_2=\frac{\partial L}{\partial z_2}=\frac{\partial L}{\partial \hat{y}}\cdot 1=(\hat{y}-t)=1.$ Kyunki $z_2=w_2{a}_1+b_2$ hai: $\partial z_2/\partial w_2=a_1$ aur $\partial z_2/\partial b_2=1$. Isliye $\frac{\partial L}{\partial w_2}={\delta }_2{a}_1=1\times 0.5=0.5,\frac{\partial L}{\partial b_2}={\delta }_2=1.$ Isko aise padho — error out × signal in: $w_2$ ka gradient uska downstream error ($1$) times uska apna input ($a_1=0.5$) hai.

Error $w_2$ se back flow karta hai aur local slope ${\sigma }^{\prime }(z_1)$ se scale hota hai. Kyunki $z_1=0.5>0$ hai, ${\sigma }^{\prime }(z_1)=1$: ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=1\times 2\times 1=2.$ Phir $\partial z_1/\partial w_1=x$ aur $\partial z_1/\partial b_1=1$ ke saath: $\frac{\partial L}{\partial w_1}={\delta }_{1}x=2\times 1=2,\frac{\partial L}{\partial b_1}={\delta }_1=2.$

Forward: $z_1=-3(1)+0=-3$, toh $a_1=\text{ReLU}(-3)=0$, $z_2=2(0)+0=0$, $\hat{y}=0$, $L=\frac{1}{2}(0-0{)}^2=0$. Backward: ${\delta }_2=\hat{y}-t=0$. Seed bhi zero hai. Aur slope ${\sigma }^{\prime }(z_1)=0$ hai kyunki $z_1<0$ hai. Toh ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0\times 2\times 0=0,\frac{\partial L}{\partial w_1}={\delta }_{1}x=0.$ Saare layer-1 gradients vanish ho jaate hain. ReLU apne flat region mein hai — unit "dead" hai. Koi error zero slope se pass nahi kar sakta, toh $w_1,b_1$ ke liye learning ruk jaati hai. Yeh Vanishing Gradients ka ek baby case hai.

Chain rule $\partial L/\partial w$ ko local derivatives ke product mein factor karta hai. Har $\delta$ ek baar compute hota hai aur us node ko feed karne wale har weight ke liye reuse hota hai: ${\delta }_1$ ${\delta }_2$ se build hota hai (jo pehle se known hai), aur har weight ka gradient $\delta \times (\text{uska input})$ hai. Toh ek backward sweep cached quantities reuse karta hai aur har edge ko ek baar touch karta hai — total kaam edges ki sankhya ke proportional hai, yaani ek forward pass jitna. Naive finite-difference method instead poore net ko per weight re-run karta hai: $O(\#\text{weights})$ passes. Yahi reuse hai jo Automatic Differentiation mechanise karta hai. Dekho Computational Graphs.

$\frac{\partial L}{\partial a}=u{\delta }_A+v{\delta }_B.$ Value $a$, $L$ ko do independent paths se influence karta hai. Multivariable chain rule kehta hai total sensitivity per-path sensitivities ka sum hai — har path ka contribution (us path par weight) × (us path ke target par error) hai. Figure 2 dekho: do magenta arrows $a$ par milte hain aur unke blames add hote hain.

Forward: $z_1=0.5(1)+(-1)(2)+0=0.5-2=-1.5$. ReLU clamp karta hai: $a_1=\max (0,-1.5)=0$. Phir $z_2=2(0)=0$, $\hat{y}=0$, $L=\frac{1}{2}(0{)}^2=0$. Backward: ${\delta }_2=\hat{y}-t=0$; slope ${\sigma }^{\prime }(z_1)=0$ (kyunki $z_1<0$); ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=0$. Weight gradient outer product "error × input" hai: $\partial L/\partial w_1={\delta }_1{x}^{\top }=0\cdot [1,2]=[0,0]$, aur $\partial L/\partial b_1={\delta }_1=0$. Har gradient zero hai — phir ek dead unit. Yeh dikhata hai ki vector rule scalar logic ko elementwise reproduce karta hai.

Forward: $z_1=0.5(1)+1(2)=2.5>0$, toh $a_1=2.5$; $z_2=2(2.5)=5$, $\hat{y}=5$, $L=\frac{1}{2}(5-0{)}^2=12.5$. Backward: ${\delta }_2=\hat{y}-t=5$. $\partial L/\partial w_2={\delta }_2{a}_1=5(2.5)=12.5$, $\partial L/\partial b_2=5$. ${\sigma }^{\prime }(z_1)=1$ (positive), toh ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)=5(2)(1)=10$. Vector weight gradient (outer product): $\frac{\partial L}{\partial w_1}={\delta }_1{x}^{\top }=10[1,2]=[10,20],\frac{\partial L}{\partial b_1}={\delta }_1=10.$ Notice karo ki har component ${\delta }_1\times$ (corresponding input) hai — bade input ($x_2=2$) ko touch karne wala weight bada gradient pata hai.

Update rule $w\leftarrow w-\eta \partial L/\partial w$: $w_2^{\text{new}}=2-0.01(12.5)=1.875,b_2^{\text{new}}=0-0.01(5)=-\mathrm{0.05.}$ Sanity check: $a_1=2.5$ abhi bhi hai, new $\hat{y}=1.875(2.5)-0.05=4.6875-0.05=4.6375$, new $L=\frac{1}{2}(4.6375{)}^2\approx 10.753$. Kyunki $10.753<12.5$ hai, loss decrease hua — gradient descent downhill step liya, jaise intended tha.

Forward: $z_1=0.5$, $a_1=\sigma (0.5)=\frac{1}{1+e^{-0.5}}\approx 0.622459$. Phir $z_2=2(0.622459)=1.244918$, $\hat{y}=1.244918$, $L=\frac{1}{2}(1.244918{)}^2\approx 0.774910$. Backward: ${\delta }_2=\hat{y}-t\approx 1.244918$. Local slope ${\sigma }^{\prime }(z_1)=a_1(1-a_1)=0.622459(0.377541)\approx 0.234988$. ${\delta }_1={\delta }_2{w}_2{\sigma }^{\prime }(z_1)\approx 1.244918\times 2\times 0.234988\approx \mathrm{0.585062.}$ $\frac{\partial L}{\partial w_1}={\delta }_{1}x\approx \mathrm{0.585062.}$ Saturation: jab $|z_1|\to \infty$, $\sigma (z_1)\to 0$ ya $1$, toh ${\sigma }^{\prime }(z_1)=\sigma (1-\sigma )\to 0$. Kyunki ${\delta }_1$ mein ${\sigma }^{\prime }(z_1)$ ka factor hai, gradient $0$ ki taraf throttle ho jaata hai — classic sigmoid Vanishing Gradients problem. Peak slope sirf ${\sigma }^{\prime }(0)=0.25$ hai, toh best case mein bhi sigmoids error ko $\frac{1}{4}$ per layer shrink karte hain.

Sigmoid net ke saath, $L(w_1)=\frac{1}{2}(2\sigma (w_1){)}^2=2\sigma (w_1{)}^2$ (kyunki $x=1,b=0,w_2=2$). $L_{+}=2\sigma (0.5001{)}^2$, $L_{-}=2\sigma (0.4999{)}^2$. Numerically slope estimate $\frac{L_{+}-L_{-}}{2ϵ}\approx 0.58506$, jo L5.1 ke $0.585062$ se paanch decimals tak match karta hai. ✓ Central vs forward: central formula ka error ${ϵ}^2$ ki tarah shrink karta hai (odd, first-order error terms symmetry se cancel ho jaate hain), jabki one-sided forward difference $ϵ$ ki tarah err karta hai. Toh central estimate same $ϵ$ ke liye bahut zyada accurate hai.

${\delta }_{\text{front}}={\delta }_{\text{seed}}\cdot r^n.$

• $r=0.25$: $r^n\to 0$ geometrically — vanishing gradients; deep front layers barely learn karte hain.
• $r=1$: $r^n=1$ — error intact pass hota hai, healthy regime jo skip-connections aur careful initialisation engineer karne ki koshish karte hain.
• $r=2$: $r^n\to \infty$ — exploding gradients; updates blow up ho jaate hain jab tak clip na karein. Yeh single product $r^n$ hi woh reason hai kyun depth delicate hai, aur kyun aerodynamic data ke surrogate nets (Neural Network Surrogate Models (CFD)) ko $r\approx 1$ band mein rehne ke liye normalisation aur residual paths ki zaroorat hoti hai.