Backpropagation algorithm derivation

3.1.9 · AI-ML › Neural Network Fundamentals

Backprop EXISTS kyun karta hai?

YE problem kya solve karta hai? Ek neural network mein millions of weights hote hain. Gradient descent se train karne ke liye hume har weight $w$ ke liye $\frac{\partial L}{\partial w}$ chahiye. Har derivative ko independently chain rule se compute karna same sub-expressions ko billions baar repeat karega.

YE fast kyun hai: Ek early-layer weight ke w.r.t. loss ke gradient mein later layers ke w.r.t. gradient ek factor ke roop mein contained hota hai. Toh agar hum later gradients pehle compute karein aur unhe cache karein, toh har earlier gradient sasta ho jaata hai. Yeh shared-subproblems ki caching exactly dynamic programming hai computational graph pe.

Setup aur notation

Consider karo layer $l$ ($1$ se $L$ tak). Layer $l$ ke liye:

• $W^{(l)}$ = weight matrix, $b^{(l)}$ = bias vector
• $z^{(l)}=W^{(l)}{a}^{(l-1)}+b^{(l)}$ — pre-activation
• $a^{(l)}=\sigma (z^{(l)})$ — activation ($\sigma$ = activation function)
• $a^{(0)}=x$ (input), $a^{(L)}=\hat{y}$ (output)
• $L(a^{(L)},y)$ = loss

Chaar equations ko scratch se derive karna

Step 1 — Output layer error ${\delta }^{(L)}$

KAISE: Chain rule se, loss $z^{(L)}$ par depend karta hai sirf $a^{(L)}=\sigma (z^{(L)})$ ke through:

${\delta }_j^{(L)}=\frac{\partial L}{\partial z_j^{(L)}}=\frac{\partial L}{\partial a_j^{(L)}}\cdot \frac{\partial a_j^{(L)}}{\partial z_j^{(L)}}$

Yeh step kyun? $a_j^{(L)}=\sigma (z_j^{(L)})$ hi $z_j^{(L)}$ se loss tak ka ek maatra path hai, isliye hum iske through chain karte hain.

Kyunki $a_j^{(L)}=\sigma (z_j^{(L)})$, doosra factor ${\sigma }^{\prime }(z_j^{(L)})$ hai. Vector form mein:

$\boxed{{\delta }^{(L)}={\nabla }_{a}L\odot {\sigma }^{\prime }(z^{(L)})}$

jahaan $\odot$ elementwise product hai (Hadamard).

Step 2 — Recurrence: ${\delta }^{(l)}$ from ${\delta }^{(l+1)}$

Recurrence kyun? $z^{(l)}$ loss ko affect karta hai sirf next layer ke pre-activations $z^{(l+1)}$ ke through. Toh:

${\delta }_j^{(l)}=\frac{\partial L}{\partial z_j^{(l)}}={\sum }_k\frac{\partial L}{\partial z_k^{(l+1)}}\cdot \frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}={\sum }_k{\delta }_k^{(l+1)}\cdot \frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}$

$k$ pe sum kyun? $z_j^{(l)}$ layer $l+1$ ke har neuron $k$ mein fan out hota hai; total sensitivity = un sabhi paths pe sum (multivariable chain rule).

Ab $\frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}$ compute karo. Kyunki $z_k^{(l+1)}={\sum }_i{W}_{ki}^{(l+1)}{a}_i^{(l)}+b_k^{(l+1)}$ aur $a_i^{(l)}=\sigma (z_i^{(l)})$:

$\frac{\partial z_k^{(l+1)}}{\partial z_j^{(l)}}=W_{kj}^{(l+1)}{\sigma }^{\prime }(z_j^{(l)})$

Substitute karne par:

${\delta }_j^{(l)}=\left({\sum }_k{W}_{kj}^{(l+1)}{\delta }_k^{(l+1)}\right){\sigma }^{\prime }(z_j^{(l)})$

Matrix form mein:

$\boxed{{\delta }^{(l)}=((W^{(l+1)}{)}^{\top }{\delta }^{(l+1)})\odot {\sigma }^{\prime }(z^{(l)})}$

Step 3 — Weights ke w.r.t. Gradient

$z_j^{(l)}={\sum }_i{W}_{ji}^{(l)}{a}_i^{(l-1)}+b_j^{(l)}$, isliye $\frac{\partial z_j^{(l)}}{\partial W_{ji}^{(l)}}=a_i^{(l-1)}$:

$\frac{\partial L}{\partial W_{ji}^{(l)}}=\frac{\partial L}{\partial z_j^{(l)}}\cdot \frac{\partial z_j^{(l)}}{\partial W_{ji}^{(l)}}={\delta }_j^{(l)}{a}_i^{(l-1)}$

$\boxed{\frac{\partial L}{\partial W^{(l)}}={\delta }^{(l)}(a^{(l-1)}{)}^{\top }}$

Yeh outer product kyun? Har weight $W_{ji}$ input $i$ ko neuron $j$ se connect karta hai; uska gradient = ($j$ ka blame) × ($i$ ki activity).

Step 4 — Biases ke w.r.t. Gradient

$\frac{\partial z_j^{(l)}}{\partial b_j^{(l)}}=1$, isliye

$\boxed{\frac{\partial L}{\partial b^{(l)}}={\delta }^{(l)}}$

Worked example 1 — ek 2-2-1 network haath se

Input $x=\left[\begin{array}{c}1\\ 0\end{array}\right]$, target $y=1$. Sigmoid $\sigma$, MSE loss $L=\frac{1}{2}(a^{(2)}-y{)}^2$.

Weights: $W^{(1)}=\left[\begin{array}{cc}0.1& 0.2\\ 0.3& 0.4\end{array}\right]$, $b^{(1)}=0$, $W^{(2)}=[0.50.6]$, $b^{(2)}=0$.

Forward:

• $z^{(1)}=W^{(1)}x=\left[\begin{array}{c}0.1\\ 0.3\end{array}\right]$, $a^{(1)}=\sigma (z^{(1)})=\left[\begin{array}{c}0.525\\ 0.574\end{array}\right]$. Kyun? $\sigma (0.1)=0.525$, $\sigma (0.3)=0.574$.
• $z^{(2)}=0.5(0.525)+0.6(0.574)=0.607$, $a^{(2)}=\sigma (0.607)=0.647$.

Backward:

• ${\nabla }_{a}L=a^{(2)}-y=0.647-1=-0.353$. Kyun? $\frac{d}{da}\frac{1}{2}(a-y{)}^2=a-y$.
• ${\sigma }^{\prime }(z^{(2)})=a^{(2)}(1-a^{(2)})=0.647(0.353)=0.228$.
• ${\delta }^{(2)}=-0.353\times 0.228=-0.0805$ (BP1).
• $\frac{\partial L}{\partial W^{(2)}}={\delta }^{(2)}(a^{(1)}{)}^{\top }=[-0.0423,-0.0462]$ (BP3). Kyun? scalar $\delta$ ko har activation se multiply karo.
• ${\delta }^{(1)}=(W^{(2)}{)}^{\top }{\delta }^{(2)}\odot {\sigma }^{\prime }(z^{(1)})$. $(W^{(2)}{)}^{\top }{\delta }^{(2)}=\left[\begin{array}{c}-0.0403\\ -0.0483\end{array}\right]$; ${\sigma }^{\prime }(z^{(1)})=\left[\begin{array}{c}0.249\\ 0.244\end{array}\right]$; isliye ${\delta }^{(1)}=\left[\begin{array}{c}-0.0100\\ -0.0118\end{array}\right]$ (BP2).

Ab ek gradient-descent step in gradients ka $\eta \cdot$ subtract karta hai. Done — ek full pass.

Worked example 2 — sharing kaam kyun bachata hai

$W^{(1)}$ ke liye hume ${\delta }^{(1)}$ chahiye tha, jisne ${\delta }^{(2)}$ ko reuse kiya (jo already compute ho chuka tha). Agar hum instead $L$ se $W^{(1)}$ tak fully chain karte toh layer 2 se guzarne wale path ko dobara recompute karna padta. Ek 100-layer net mein woh reuse linear aur exponential cost ke beech ka farak hai.

Flashcards

Connections

• Chain rule (multivariable) — har step ka mathematical engine.
• Gradient descent — backprop ke produce kiye gradients consume karta hai.
• Activation functions and their derivatives — ${\sigma }^{\prime }(z)$ supply karta hai (sigmoid, ReLU, tanh).
• Computational graphs and autodiff — backprop reverse-mode automatic differentiation hai.
• Vanishing and exploding gradients — repeated $\odot {\sigma }^{\prime }$ aur $W^{\top }$ products ka seedha consequence.
• Loss functions (MSE, cross-entropy) — BP1 mein ${\nabla }_{a}L$ determine karte hain.

Concept Map