Computational graphs and autograd — Hinglish notes

3.1.10 · AI-ML › Neural Network Fundamentals

Computational graph KYA hota hai?

Decompose kyun karein? Kyunki hum sirf simple operations ko easily differentiate karna jaante hain. Agar hum poori computation ko simple ops ki chain ke roop mein express karein, toh chain rule local derivatives ko aapas mein jod deta hai aur poori derivative ban jaati hai.

Example function: $f (x, y) = (x + y) \cdot sin (x)$

Ise primitives mein todein:

$a = x + y$
$b = sin (x)$
$f = a \cdot b$

Figure — Computational graphs and autograd

Autograd HOW kaam karta hai: forward + backward

Forward pass (WOH KYA compute karta hai)

Graph ko left-to-right evaluate karo, har intermediate value ko store karte hue. Store kyun karein? Local derivatives usually values par depend karti hain (jaise $\frac{d}{d x} sin x = cos x$ ko $x$ chahiye).

Backward pass (chain rule, systematized)

Backward rule ko first principles se derive karna

Lo $f = a \cdot b$ , $a = x + y$ , $b = sin x$ . Humein $\frac{\partial f}{\partial x}$ chahiye.

Multivariable chain rule se, $x$ do paths se $f$ ko influence karta hai ( $x \to a$ aur $x \to b$ ): $\frac{\partial f}{\partial x} = \frac{\partial f}{\partial a} \frac{\partial a}{\partial x} + \frac{\partial f}{\partial b} \frac{\partial b}{\partial x}$

Ab har local derivative compute karo (local kyun? har ek ek single primitive hai):

$\frac{\partial f}{\partial a} = b$ (kyunki $f = ab$ )
$\frac{\partial f}{\partial b} = a$
$\frac{\partial a}{\partial x} = 1$ (kyunki $a = x + y$ )
$\frac{\partial b}{\partial x} = cos x$

Toh: $\frac{\partial f}{\partial x} = b \cdot 1 + a \cdot cos x = sin x + (x + y) cos x$

Reverse mode kyun (forward mode kyun nahi)?

Worked Example 1 — poora trace

$f = ab$ compute karo upar di gayi values ke saath $x = 1, y = 2$ par.

Forward (Kyun? values store karo):

$a = 1 + 2 = 3$
$b = sin (1) \approx 0.841$
$f = 3 \times 0.841 = 2.524$

Backward:

Seed $\overset{ˉ}{f} = 1$ . Kyun? $\partial f / \partial f = 1$ .
$\overset{a}{ˉ} = \overset{ˉ}{f} \cdot b = 1 \cdot 0.841 = 0.841$ . Kyun? $ab$ ki local deriv $a$ ke respect mein $b$ hai.
$\overset{ˉ}{b} = \overset{ˉ}{f} \cdot a = 1 \cdot 3 = 3$ . Kyun? symmetric hai.
$\overset{x}{ˉ} + = \overset{a}{ˉ} \cdot \frac{\partial a}{\partial x} = 0.841 \cdot 1 = 0.841$ . Kyun? $a = x + y$ .
$\overset{x}{ˉ} + = \overset{ˉ}{b} \cdot cos x = 3 \cdot cos 1 = 3 \cdot 0.540 = 1.621$ . Kyun? $b = sin x$ ; accumulate karo kyunki $x$ $b$ se bhi flow karta hai. → $\overset{x}{ˉ} = 0.841 + 1.621 = 2.462$ .
$\overset{y}{ˉ} = \overset{a}{ˉ} \cdot 1 = 0.841$ .

Formula se check karo: $sin 1 + 3 cos 1 = 0.841 + 1.621 = 2.462$ . ✓

Worked Example 2 — ek chhota sa neuron

$L = \frac{1}{2} (\overset{y}{^} - t)^{2}$ , $\overset{y}{^} = σ (z)$ , $z = w x + b$ , jahaan $σ (z) = \frac{1}{1 + e ^{- z}}$ .

Goal: $\partial L / \partial w$ . Graph ke through chain karo: $\frac{\partial L}{\partial w} = \partial L / \partial \overset{y}{^} (\overset{y}{^} - t) \cdot \partial \overset{y}{^} / \partial z σ (z) (1 - σ (z)) \cdot \partial z / \partial w x$

Kyun $\partial L / \partial \overset{y}{^} = \overset{y}{^} - t$ ? $\frac{1}{2} (\overset{y}{^} - t)^{2}$ ki derivative.
Kyun $σ^{'} (z) = σ (1 - σ)$ ? Standard sigmoid identity (derive karo: $σ^{'} = \frac{e ^{- z}}{( 1 + e ^{- z} ) ^{2}} = σ (1 - σ)$ ).
Kyun $\partial z / \partial w = x$ ? $z = w x + b$ $w$ mein linear hai.

Autograd backward pass mein exactly yeh teen numbers compute karta hai aur unhe multiply karta hai. Tumhare taraf se koi calculus nahi.

Common mistake Common errors ko steel-man karna

1. "Shared nodes ke liye gradients accumulate karna bhool jaana." Kyun sahi lagta hai: tum sochte ho har node ki "ek" derivative hai, toh tum $\overset{x}{ˉ}$ overwrite kar dete ho. Fix: jab ek variable multiple operations ko feed karta hai, uska adjoint saare paths ka sum hota hai ( $+ =$ ), kyunki woh har path se output ko affect karta hai. Yeh literally multivariable chain rule hai.

2. "Autograd numerical/finite differences hai." Kyun sahi lagta hai: dono derivatives approximate karte hain. Lekin autograd exact hai (symbolic local rules numerically apply hote hain), $\frac{f ( x + h ) - f ( x )}{h}$ nahi. Koi step-size error nahi.

3. "Backprop aur autograd alag cheezein hain." Fix: backpropagation hai hi reverse-mode automatic differentiation neural nets par apply kiya hua. Same algorithm.

4. "Mujhe har step mein gradients zero karne yaad rakhne chahiye." Kyunki adjoints accumulate karte hain, real frameworks .grad mein add karte hain. Agar tum reset nahi karte, toh pichle batches ke gradients add hote rehenge. (optimizer.zero_grad()).

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho ek machine chhote chhote gears se bani hai (add, multiply, sin). Tum input knob ghoomao aur answer nikalte hai. Ab tum poochho: "Agar main yeh ek gear thoda sa wiggle karoon, toh final answer kitna wiggle karega?" Autograd iska jawab output se shuru hokar gear-by-gear peeche chalta hua deta hai, har gear par yeh multiply karta hai ki woh apne neighbour ke liye kitna sensitive hai. Agar ek gear do doosron se juda hai, tum dono wiggle-effects ko add karte ho. Yeh ek baar karo aur tum seekh lo ki har knob answer ko kaise affect karta hai — toh tum jaante ho ki har knob ko kis direction mein nudge karo taaki answer better ho.

Active recall

Computational graph kya hota hai?

Ek directed acyclic graph jiske nodes variables hain aur edges elementary operations hain jinki known local derivatives hain; yeh ek complex function ko differentiable primitives mein decompose karta hai.

Forward pass mein intermediate values kyun store karte hain?

Local derivatives usually un values par depend karti hain (jaise d/dx sin x = cos x ko x chahiye), isliye backprop unhe reuse karta hai.

Ek node ka adjoint (bar) kya hota hai?

Sensitivity ∂(final output)/∂(woh node) — output kitna change hoga agar node wiggle kare.

Reverse-mode update rule batao.

Har edge u→v ke liye:

\overset{u}{ˉ} + = \overset{v}{ˉ} \cdot \partial v / \partial u

\overset{ˉ}{f} = 1

se seed kiya hua.

Adjoints accumulate (+=) kyun karte hain?

Ek node jo multiple children ko feed karta hai woh output ko kai paths se affect karta hai; multivariable chain rule un contributions ko sum karta hai.

Neural nets ke liye reverse mode prefer kyun kiya jaata hai?

Cost outputs ki sankhya ke saath scale hoti hai; loss ka ek scalar output hota hai lekin millions of inputs hote hain, isliye ek backward pass sabhi gradients deta hai ≈ ek forward pass ki cost.

Kya autograd finite differences se same hai?

Nahi. Autograd exact local derivative rules apply karta hai (koi step size h nahi), isliye iska koi truncation error nahi; finite differences approximate karte hain.

Backprop aur autograd ka kya rishta hai?

Backpropagation reverse-mode automatic differentiation hai jo neural networks par apply ki gayi hai — same algorithm.

Sigmoid σ(z) ki derivative?

σ(z)(1−σ(z)).

Har training step mein gradients zero kyun karte hain?

Kyunki .grad accumulate karta hai (+=), pichle batches ke bache hue gradients warna galat tarike se add hote rahenge.

Connections

Chain rule (multivariable calculus)
Backpropagation
Gradient descent
Sigmoid and activation functions
Automatic differentiation (forward vs reverse mode)
Loss functions
PyTorch autograd / TensorFlow GradientTape