Chain rule for multivariate functions (backprop foundation)

1.2.14 · AI-ML › Calculus & Optimization Basics

What Is the Multivariate Chain Rule?

WHY does this work? Kyunki jab $x$ ek tiny amount $Δ x$ se change hota hai:

Isse $u$ mein $\approx \frac{\partial u}{\partial x} Δ x$ ka change aata hai
$u$ mein yeh change $z$ mein $\approx \frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x} Δ x$ ka change laata hai
Saath hi, $x$ $v$ ko bhi affect karta hai, jo doosre path se $z$ ko affect karta hai
$z$ mein total change dono pathway contributions ka sum hota hai

Derivation from First Principles

Chalo ise limits aur differentials ki ground truth se build karte hain.

Step 1: Single-variable chain rule se shuru karo $y = f (u)$ aur $u = g (x)$ ke liye, hum jaante hain: $\frac{d y}{d x} = \frac{d y}{d u} \cdot \frac{d u}{d x}$

WHY? Kyunki $Δ y \approx \frac{d y}{d u} Δ u$ aur $Δ u \approx \frac{d u}{d x} Δ x$ , isliye: $Δ y \approx \frac{d y}{d u} \cdot \frac{d u}{d x} \cdot Δ x$

$Δ x$ se divide karke aur limit lete hue chain rule milta hai.

Step 2: Multiple intermediate variables tak generalize karo Ab maano $z = f (u, v)$ jahan dono $u$ aur $v$ $x$ par depend karte hain.

Partial derivatives ki definition se: $d z = \frac{\partial f}{\partial u} d u + \frac{\partial f}{\partial v} d v$

Yeh $f$ ka total differential hai. Iska matlab hai: " $z$ mein ek tiny change $u$ ke through aur $v$ ke through contributions se aata hai."

Step 3: $d u$ aur $d v$ ko $d x$ ke terms mein express karo $d u = \frac{\partial u}{\partial x} d x + \frac{\partial u}{\partial y} d y$ $d v = \frac{\partial v}{\partial x} d x + \frac{\partial v}{\partial y} d y$

Step 4: Substitute karo aur terms collect karo $d z = \frac{\partial f}{\partial u} (\frac{\partial u}{\partial x} d x + \frac{\partial u}{\partial y} d y) + \frac{\partial f}{\partial v} (\frac{\partial v}{\partial x} d x + \frac{\partial v}{\partial y} d y)$

$d x$ ke coefficients collect karte hue: $d z = (\frac{\partial f}{\partial u} \frac{\partial u}{\partial x} + \frac{\partial f}{\partial v} \frac{\partial v}{\partial x}) d x + (\frac{\partial f}{\partial u} \frac{\partial u}{\partial y} + \frac{\partial f}{\partial v} \frac{\partial v}{\partial y}) d y$

Definition ke anusaar, $d x$ ka coefficient $\frac{\partial z}{\partial x}$ hai:

WHAT does this mean? Sum mein har term ek "pathway" hai input $x_{j}$ se intermediate $u_{i}$ ke through output $z$ tak. Hum har path ke along multiply karte hain aur sabhi paths ko sum karte hain.

Figure — Chain rule for multivariate functions (backprop foundation)

Worked Examples

Worked example Example 2: Neural Network Forward and Backward Pass

Given: Ek two-layer network compute karta hai: $h = σ (w_{1} x + b_{1}), y = w_{2} h + b_{2}, L = \frac{1}{2} (y - t)^{2}$ jahan $σ (a) = \frac{1}{1 + e ^{- a}}$ sigmoid hai, $t$ target hai. $\frac{\partial L}{\partial w _{1}}$ find karo.

Solution:

Step 1: Computational graph identify karo Path hai: $w_{1} \to a \to h \to y \to L$ jahan $a = w_{1} x + b_{1}$ . WHY? Hume loss se weight tak sabhi intermediates ke through backward trace karna hai.

Step 2: Chain mein har link compute karo $\frac{\partial L}{\partial y} = y - t$ WHY? $L = \frac{1}{2} (y - t)^{2}$ se, differentiate karne par $(y - t)$ milta hai.

$\frac{\partial y}{\partial h} = w_{2}$ WHY? $y = w_{2} h + b_{2}$ se, $h$ ka coefficient $w_{2}$ hai.

$\frac{\partial h}{\partial a} = σ (a) (1 - σ (a)) = h (1 - h)$ WHY? Standard sigmoid derivative: $σ^{'} (a) = σ (a) (1 - σ (a))$ . $\frac{\partial a}{\partial w _{1}} = x$ WHY? $a = w_{1} x + b_{1}$ se, $w_{1}$ ka coefficient $x$ hai.

Step 3: Path ke along multiply karo $\frac{\partial L}{\partial w _{1}} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial h} \cdot \frac{\partial h}{\partial a} \cdot \frac{\partial a}{\partial w _{1}}$ $= (y - t) \cdot w_{2} \cdot h (1 - h) \cdot x$ WHY? Yeh chain rule hai jo computation graph ke through sequentially apply ho raha hai.

Interpretation: Gradient tab large hota hai jab:

Prediction error $(y - t)$ bada ho
Second layer weight $w_{2}$ bada ho (strong connection)
Neuron "active learning zone" $h (1 - h)$ mein ho (saturated nahi)
Input $x$ bada ho

Worked example Example 3: Vector-to-Vector Chain Rule (Jacobian)

Given: $z = f (u)$ jahan $u = g (x)$ , dono vector functions hain. $\frac{\partial z _{i}}{\partial x _{j}}$ find karo.

Solution:

Step 1: Recognize karo ki yeh matrix multiplication hai $\frac{\partial z _{i}}{\partial x _{j}} = \sum_{k} \frac{\partial z _{i}}{\partial u _{k}} \frac{\partial u _{k}}{\partial x _{j}}$ WHY? Har input $x_{j}$ ke liye, hum un sabhi intermediate variables $u_{k}$ par sum karte hain jo $x_{j}$ ko $z_{i}$ se connect karte hain.

Step 2: Matrix form mein likho $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x}$

Yeh Jacobian matrix multiplication hai. Result ki har entry $(i, j)$ pehle Jacobian ki row $i$ aur doosre ki column $j$ ke dot product se compute hoti hai.

WHY does this matter? Deep learning mein, har layer ke through backprop exactly yahi Jacobian-vector product hai. Modern frameworks yeh efficiently compute karte hain bina kabhi full Jacobian matrix banaye.

Common mistake Common Mistake 2: Wrong Order in Chain Rule

Wrong thinking: $\frac{\partial z}{\partial u} \cdot \frac{\partial u}{\partial x}$ ki jagah $\frac{\partial u}{\partial x} \cdot \frac{\partial z}{\partial u}$ likhna.

Why it feels right: Regular multiplication mein order matter nahi karta. Notation confusing ho sakta hai.

The fix: Ise backpropagation order ki tarah socho: output se shuru karo aur backward kaam karo. Output (loss) ke sabse kareeb wala derivative pehle aata hai: $\frac{\partial Loss}{\partial input} = \frac{\partial Loss}{\partial intermediate} \cdot \frac{\partial intermediate}{\partial input}$ Mnemonic: "Outside-in"—sabse bahari function ka derivative pehle aata hai.

Steel-man: Scalar calculus mein Leibniz notation ke saath, tum $d u$ terms ko "cancel" kar sakte ho, jisse order irrelevant lagta hai. Lekin matrices (Jacobians) ke saath, order critically matter karta hai kyunki matrix multiplication non-commutative hai.

Connection to Backpropagation

Neural networks mein, ek computational graph hota hai jahan:

Nodes = intermediate values (activations, weighted sums)
Edges = operations (matrix multiply, activation functions)
Goal = har parameter $w$ ke liye $\frac{\partial L}{\partial w}$ compute karna

Backprop algorithm IS the chain rule:

Forward pass: Sabhi intermediates store karte hue $x$ se $z$ compute karo
Backward pass: Graph ke through gradients backward propagate karke $\frac{\partial L}{\partial x}$ compute karo

Har node par, agar $z$ $u_{1}, \dots, u_{n}$ par depend karta hai: $\frac{\partial L}{\partial u _{i}} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial u _{i}}$

Ise local gradient ( $\frac{\partial z}{\partial u _{i}}$ ) times upstream gradient ( $\frac{\partial L}{\partial z}$ ) kaha jaata hai.

WHY is this efficient? Har $\frac{\partial L}{\partial w _{i}}$ independently compute karne ki jagah (expensive), hum ek backward pass compute karte hain jo humein SABHI parameter gradients deta hai. Yeh reverse-mode automatic differentiation hai.

Recall Ek 12-Saal-Ke Bachche Ko Samjhao

Socho tum smoothie bana rahe ho. Tum daali strawberries aur kele, blend kiya, milk dala, phir blend kiya, phir shahad dala. Final taste har ingredient par depend karta hai.

Ab tumhara dost kehta hai "yeh bahut meetha hai!" Tum figure out karna chahte ho: mujhe shahad kitna kam karna chahiye? Lekin ruko—meethas bhi is baat par depend karti hai ki kele kitne pakke the. Aur kele thickness ko affect karte hain, jo affect karta hai ki tum shahad kitna taste karte ho.

Chain rule "taste detective" bonne jaisa hai. Tum backward kaam karte ho:

"Meethas 70% shahad se hai, 30% kele se"
"Kele ki ripeness sweetness ko affect karti hai is through ki unme kitna sugar hai"
"Toh meethas 10% kam karne ke liye, mujhe shahad 7% kam karna hai aur kam pakke kele use karne hain" Math mein, jab ek cheez doosri ko affect kare, jo teesri ko affect kare, hum chain ke along "effect sizes" multiply karte hain. Agar multiple chains hain (jaise shahad→meethas aur kele→meethas), hum unhe saara add kar lete hain. Yahi chain rule hai!

Why This Matters for AI/ML

Backpropagation sirf repeated chain rule application hai. Har neural network training isi par rely karti hai.
Computational graphs ise visual banate hain. Modern frameworks (PyTorch, TensorFlow) ek graph build karte hain aur automatically chain rule apply karte hain.
Efficiency: Chain rule ke bina, gradients compute karne ke liye $n$ parameters ke liye $O (n)$ forward passes chahiye honge. Iske saath, ek backward pass sabhi gradients deta hai.
Vanishing/exploding gradients tab hote hain jab chain rule products kai layers mein bahut chhote/bade ho jaate hain: $\frac{\partial L}{\partial w _{1}} = \frac{\partial L}{\partial z _{n}} \cdot \frac{\partial z _{n}}{\partial z _{n - 1}} \dots \frac{\partial z _{2}}{\partial z _{1}} \cdot \frac{\partial z _{1}}{\partial w _{1}}$ Agar har $\frac{\partial z _{i}}{\partial z _{i - 1}} < 1$ , toh product exponentially shrink karta hai (vanishing). Agar $> 1$ , toh explode karta hai.

Connections

Gradient Descent - In gradients ko parameters update karne ke liye use karta hai
Backpropagation Algorithm - Neural nets mein chain rule ki practical implementation
Computational Graphs - Visual representation jo automatic differentiation enable karti hai
Automatic Differentiation - Software jo chain rule automatically compute karta hai
Jacobian and Hessian Matrices - Multivariate derivatives ke matrix forms
Vanishing and Exploding Gradients - Long chains se aane wale problems
Activation Functions - Har ek ka ek derivative hota hai jo chain mein use hota hai
Loss Functions - Backprop ke liye starting point ( $\frac{\partial L}{\partial y}$ )

#flashcards/ai-ml

$z = f (u, v)$ ke liye jahan $u = g (x, y)$ aur $v = h (x, y)$ ho, $\frac{\partial z}{\partial x}$ ka multivariate chain rule formula kya hai? :: $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \frac{\partial u}{\partial x} + \frac{\partial z}{\partial v} \frac{\partial v}{\partial x}$ . $x$ se $z$ tak sabhi paths par sum karo, har path ke along derivatives multiply karte hue.

Multivariate chain rule mein hum terms KYU SUM karte hain? :: Kyunki input variable (e.g., $x$ ) output ko MULTIPLE intermediate variables ke through affect kar sakta hai (multiple paths). Har path independently total change mein contribute karta hai, isliye hum unhe add karte hain.

Backpropagation mein kisi node par "local gradient" kya hota hai?

Us node ke operation ka apne immediate inputs ke saath derivative:

\frac{\partial z}{\partial u _{i}}

. Upstream gradient

\frac{\partial L}{\partial z}

ke saath combine karke, yeh

\frac{\partial L}{\partial u _{i}}

deta hai.

$\frac{\partial z}{\partial x}$ aur $\frac{d z}{d x}$ mein kya difference hai? :: $\frac{\partial z}{\partial x}$ partial derivative hai jo doosre variables ko constant rakhta hai. $\frac{d z}{d x}$ total derivative hai jo un sabhi tareekon ko account karta hai jisse $x$ $z$ ko affect karta hai, including doosre variables ke through indirect paths.

Backpropagation ko "reverse mode" automatic differentiation KYU kaha jaata hai?

Kyunki yeh computational graph ko REVERSE order mein (output se input) traverse karta hai, chain rule ko backward apply karke gradients compute karta hai. Yeh ek pass mein sabhi parameter gradients deta hai.

Agar

z = u^{2} + v^{2}

u = 2 x

v = 3 x

ho, toh

\frac{\partial z}{\partial x}

kya hai?

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial u} \frac{\partial u}{\partial x} + \frac{\partial z}{\partial v} \frac{\partial v}{\partial x} = (2 u) (2) + (2 v) (3) = 4 u + 6 v = 4 (2 x) + 6 (3 x) = 8 x + 18 x = 26 x

Deep networks mein vanishing gradients kis cheez se hote hain?

Chain rule mein multiplication ki long chains se. Agar har layer ka gradient

\frac{\partial z _{i + 1}}{\partial z _{i}} < 1

ho, toh kai layers mein

\frac{\partial L}{\partial w _{1}}

ka product exponentially shrink karta hai.

Chain rule mein pehle kaun sa derivative aata hai: input ke kareeb wala ya output ke kareeb wala?

OUTPUT ke kareeb wala. Hum backward kaam karte hain:

\frac{\partial Loss}{\partial input} = \frac{\partial Loss}{\partial intermediate} \cdot \frac{\partial intermediate}{\partial input}

. "Outside-in" order.