Autograd and computational graphs in PyTorch

3.3.2 · AI-ML › Deep Learning Frameworks

Overview

PyTorch ka autograd system woh engine hai jo automatic differentiation ko power karta hai—yeh tensors par operations ko track karta hai aur gradients automatically compute karta hai. Isse backpropagation ke dauran manual derivative calculations ki zaroorat nahi rehti, jis se deep learning research fast aur error-free ban jaati hai.

Figure — Autograd and computational graphs in PyTorch

Core Concepts

Autograd Kaise Kaam Karta Hai: Step-by-Step Derivation

Step 1: Forward Pass aur Graph Construction

Jab aap requires_grad=True wale tensors par operations perform karte hain, PyTorch unhe record karta hai:

import torch
 
x = torch.tensor([2.0], requires_grad=True)
y = torch.tensor([3.0], requires_grad=True)
 
z = x * y + x**2  # z = 3x + x²

Internally kya hota hai:

x * y ek node create karta hai grad_fn=<MulBackward> ke saath
x**2 ek node create karta hai grad_fn=<PowBackward> ke saath
+ inhe grad_fn=<AddBackward> ke saath combine karta hai
z ab z.grad_fn = <AddBackward> ke saath poore graph ki taraf point karta hai

Yeh kyun matter karta hai: Graph structure operations ki chain encode karta hai, jo exactly wahi hai jo humein chain rule ke liye chahiye.

Step 2: Backward Pass aur Chain Rule

Chain rule kehta hai: agar $z = f (g (x))$ , toh $\frac{d z}{d x} = \frac{d z}{d g} \cdot \frac{d g}{d x}$ .

Hamare example $z = x y + x^{2}$ ke liye:

$\frac{d z}{d x} = \frac{\partial}{\partial x} (x y) + \frac{\partial}{\partial x} (x^{2}) = y + 2 x$

$\frac{d z}{d y} = \frac{\partial}{\partial y} (x y) + \frac{\partial}{\partial y} (x^{2}) = x + 0 = x$

First principles se Derivation:

Derivative ki definition se shuru karte hain: $\frac{d z}{d x} = lim_{h \to 0} \frac{z ( x + h , y ) - z ( x , y )}{h}$

$= lim_{h \to 0} \frac{[( x + h ) y + ( x + h ) ^{2} ] - [ x y + x ^{2} ]}{h}$

$= lim_{h \to 0} \frac{x y + h y + x ^{2} + 2 x h + h ^{2} - x y - x ^{2}}{h}$

$= lim_{h \to 0} \frac{h y + 2 x h + h ^{2}}{h} = lim_{h \to 0} (y + 2 x + h) = y + 2 x$

Yeh step kyun? Hum function ko $x + h$ par expand karte hain, terms cancel karte hain, $h$ factor out karte hain, aur limit lete hain. Yahi derivative ki fundamental definition hai.

PyTorch ise automate karta hai:

z.backward()  # Compute dz/dx and dz/dy
print(x.grad)  # tensor([8.]) = 3+ 2*2
print(y.grad)  # tensor([2.]) = 2

Step 3: Gradient Accumulation

Default se, .backward() call karna gradients ko .grad mein accumulate karta hai:

$tensor.grad \leftarrow tensor.grad + \frac{\partial L}{\partial tensor}$

Accumulation kyun? Mini-batch training mein, weights update karne se pehle aap multiple samples ke gradients sum karte hain. Lekin iska matlab hai aapko iterations ke beech gradients zero karne padte hain:

x.grad.zero_()  # Clear previous gradients

Derivation ki accumulation kyun zaroori hai:

Batch gradient descent mein, loss hai: $L_{batch} = \frac{1}{N} \sum_{i = 1}^{N} L_{i}$

Gradient hai: $\frac{\partial L _{batch}}{\partial w} = \frac{1}{N} \sum_{i = 1}^{N} \frac{\partial L _{i}}{\partial w}$

Agar hum har sample ke liye $\frac{\partial L _{i}}{\partial w}$ alag compute karte hain, toh humein unhe sum karna hoga. Accumulation is sum ko automatically implement karta hai.

Detailed Examples

Worked example Example 1: Simple Neural Network Layer

Problem: Ek single linear layer $y = W x + b$ ke baad ReLU ke liye gradients compute karo.

import torch
# Input and parameters
x = torch.tensor([[1.0, 2.0]], requires_grad=True)
W = torch.tensor([[0.5, -0.3], [0.2, 0.4]], requires_grad=True)
b = torch.tensor([[0.1, -0.2]], requires_grad=True)
 
# Forward pass
z = torch.matmul(x, W) + b  # z = xW + b
y = torch.relu(z)            # y = max(0, z)
loss = y.sum()               # Scalar loss
 
# Backward pass
loss.backward()
print(x.grad)  # Gradient of loss w.r.t input
print(W.grad)  # Gradient of loss w.r.t weights

Step-by-step reasoning:

$z$ compute karo: $z = x W + b = [1, 2] [0.5 0.2 - 0.3 0.4] + [0.1, - 0.2]$

$z_{1} = 1 (0.5) + 2 (0.2) + 0.1 = 1.0$
$z_{2} = 1 (- 0.3) + 2 (0.4) - 0.2 = 0.3$
Toh $z = [1.0, 0.3]$

Yeh step kyun? Matrix multiplication aur addition—standard forward pass. 2. ReLU apply karo: $y = max (0, z) = [1.0, 0.3]$ (dono positive hain, toh unchanged) Yeh step kyun? ReLU negatives ko zero karta hai; yahan kuch zero nahi hota.

Loss compute karo: $L = y_{1} + y_{2} = 1.3$ Yeh step kyun? Backpropagate karne ke liye humein ek scalar chahiye.
Sum ke through Backward: $\frac{\partial L}{\partial y _{1}} = 1$ , $\frac{\partial L}{\partial y _{2}} = 1$ Kyun? $L = y_{1} + y_{2}$ , toh derivatives constants hain.
ReLU ke through Backward: $\frac{\partial L}{\partial z _{i}} = {1 if z_{i} > 00 if z_{i} \leq 0$ Yahan dono $z$ elements positive hain, toh $\frac{\partial L}{\partial z} = [1, 1]$ . Kyun? ReLU ka derivative 1 hota hai jahan input positive hai, warna 0.
Linear layer ke through Backward: $\frac{\partial L}{\partial W _{ij}} = \frac{\partial L}{\partial z _{j}} \cdot x_{i}$

For example, $\frac{\partial L}{\partial W _{11}} = \frac{\partial L}{\partial z _{1}} \cdot x_{1} = 1 \cdot 1 = 1$ Similarly: $W . grad = x^{T} \cdot \frac{\partial L}{\partial z} = [12] \cdot [1, 1] = [1212]$

Yeh step kyun? $z = x W$ se, chain rule se: $\frac{\partial L}{\partial W} = x^{T} \frac{\partial L}{\partial z}$ (standard matrix calculus).

Worked example Example 2:

retain_graph=True ko Samajhna Problem: Ek hi graph ke through do baar backpropagate karo.

x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
y.backward()  # First backward
print(x.grad)  # tensor([4.])
# Try second backward (will error!)
# y.backward()  # RuntimeError: graph has been freed
# Solution: retain graph
x.grad.zero_()
y = x ** 2
y.backward(retain_graph=True)
print(x.grad)  # tensor([4.])
y.backward()  # Works now!
print(x.grad)  # tensor([8.]) - accumulated!

Yeh kyun hota hai: PyTorch .backward() ke baad graph ko memory bachane ke liye free kar deta hai (dynamic graphs by default ek baar hi use hote hain). retain_graph=True set karne se woh alive rehta hai.

Iska kab use karein: Ek hi forward pass se multiple losses compute karte waqt (jaise adversarial training, shared encoder ke saath multi-task learning).

Accumulated gradient ki Derivation: Pehle .backward() ke baad: $x . grad = \frac{d y}{d x} = 2 x = 4$ Doosre .backward() ke baad: $x . grad = 4 + 2 x = 4 + 4 = 8$ Accumulation kyun? Har .backward() .grad mein add karta hai. Jab tak explicitly accumulation nahi chahiye, gradients hamesha zero karo.

Worked example Example 3: Gradient Flow Rokne ke liye Tensors Detach Karna

Problem: Ek loss compute karo jo kisi intermediate value par depend karta hai, lekin us se backpropagate mat karo.

x = torch.tensor([3.0], requires_grad=True)
# Compute intermediate
hidden = x ** 2  # hidden = 9
# Use hidden as a CONSTANT (stop gradient)
output = hidden.detach() * x  # output = 9 * 3 = 27
output.backward()
print(x.grad)  # tensor([9.])

9 kyun, 15 kyun nahi? Agar hum detach nahi karte: $output = x^{2} \cdot x = x^{3}$ $\frac{d ( output )}{d x} = 3 x^{2} = 3 (^{2}) = 27$ Lekin detach ke saath, hidden ko ek constant $c = 9$ ki tarah treat kiya jaata hai: $output = c \cdot x$ $\frac{d ( output )}{d x} = c = 9$

Iska kab use karein: Reinforcement learning (purani policy ko constant treat karte hue), GAN discriminator freezing, ya kisi bhi waqt jab aap ek boundary par gradients rokna chahte hain.

Common Mistakes

Common mistake Mistake 1: Gradients Zero Karna Bhool Jaana

Galat approach:

for epoch in range(3):
   loss = model(x)
   loss.backward()
   optimizer.step()  # Gradients keep accumulating!

Yeh sahi kyun lagta hai: Aap assume karte hain ki .backward() gradients ko ek clean slate ki tarah reset kar deta hai.

Sachai yeh hai: PyTorch gradients accumulate karta hai (mini-batches mein gradient accumulation ke liye useful). Zero kiye bina, aapke gradients unbounded badhte rehte hain: Iteration 1: $θ . grad = g_{1}$ Iteration 2: $θ . grad = g_{1} + g_{2}$ (galat!) Iteration 3: $θ . grad = g_{1} + g_{2} + g_{3}$ (bahut galat!)

Fix:

for epoch in range(3):
   optimizer.zero_grad()  # Clear gradients
   loss = model(x)
   loss.backward()
   optimizer.step()

Rule of thumb: optimizer.zero_grad() (ya tensor.grad.zero_()) ko har training iteration ki SHURUAT mein call karo.

Common mistake Mistake 2: Non-Scalar par

.backward() Call Karna Galat approach:

x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x ** 2
y.backward()  # RuntimeError!

Yeh sahi kyun lagta hai: Aap sochte hain "backpropagation kisi bhi tensor par kaam karta hai."

Sachai yeh hai: .backward() $\frac{\partial L}{\partial x}$ compute karta hai jahan $L$ ek scalar hai. Agar y ek vector hai, toh PyTorch nahi jaanta ki kaunse scalar ko differentiate karna hai (kya yeh sum hai? Mean? Koi specific element?). Mathematical reason: Gradient $\frac{\partial y}{\partial x}$ ek Jacobian matrix hota hai (vector nahi). Pehle scalar reduce karna zaroori hai.

Fixes:

# Option 1: Sum to scalar
y.sum().backward()
 
# Option 2: Provide gradient argument
y.backward(torch.ones_like(y))  # Equivalent to sum
# Option 3: Specific element
y[0].backward()  # Only backprop through first element

Har ek kab useful hai:

Sum: Loss functions ke liye standard
Gradient argument: Custom weighting (jaise importance sampling)
Specific element: Individual neurons debug karne ke liye

Common mistake Mistake 3: In-Place Operations se Autograd Toot Jaana

Galat approach:

x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
y += 1  # In-place operation
y.backward()  # RuntimeError: modified by inplace operation

Yeh sahi kyun lagta hai: += jaisi in-place ops memory-efficient hoti hain.

Sachai yeh hai: Autograd ko gradients sahi se compute karne ke liye y ki original value chahiye. In-place ops history destroy kar deti hain.

Problem ki Derivation:

Maano hum $\frac{d L}{d x}$ chahte hain jahan $L = (x^{2} + 1)$ . Humein chahiye: $\frac{d L}{d x} = \frac{d ( x ^{2} + 1 )}{d x} = 2 x$

Lekin y += 1 ke baad, graph sirf y (modified value) dekhta hai, original $x^{2}$ nahi. Link toot jaata hai.

Fix:

y = x ** 2
y = y + 1  # Out-of-place (creates new tensor)
y.backward()

Exceptions: Leaf tensors (user-created inputs) par in-place ops forbidden hain. Intermediate tensors par, agar carefully karein toh kaam karte hain, lekin generally training ke dauran inhe avoid karo.

Dynamic vs Static Graphs

Intuition Dynamic Graphs Kyun Matter Karte Hain

Static graphs (TensorFlow 1.x): Aap graph ek baar define karte hain, phir data us se guzarta hai. Jaise factory assembly line ko on karne se pehle banana.

Dynamic graphs (PyTorch): Graph code execute hone ke saath banta hai. Jaise ek flexible assembly line ho jo har product ke liye khud ko reconfigure kar le.

Dynamic ke advantages:

Debugging: Print statements, debuggers, standard Python control flow use kar sakte hain
Variable sequences: Padding ke bina alag input lengths handle kar sakte hain
Conditional logic: Forward pass mein if statements (jaise layer depth ke basis par skip connections) Trade-off: Thoda slow (har iteration mein graph overhead) lekin kaafi zyada flexible. Modern TensorFlow (2.x) ne bhi dynamic execution (Eager mode) adopt kiya kyunki research flexibility maangti hai.

The `.grad_fn` Chain

Kisi operation se bane har tensor mein us function ka reference store hota hai jisne use banaya:

x = torch.tensor([1.0], requires_grad=True)
y = x * 2
z = y + 3
 
print(y.grad_fn)  # <MulBackward0>
print(z.grad_fn)  # <AddBackward0>
print(z.grad_fn.next_functions)  # Shows MulBackward0

Yahi chain computational graph hai. grad_fn pointers ko backward follow karne se poora computation reconstruct ho jaata hai.

Leaf tensors (user ne banaye, operations se nahi) ka grad_fn=None aur is_leaf=True hota hai:

print(x.is_leaf)  # True
print(y.is_leaf)  # False (created by operation)

Yeh kyun matter karta hai: By default sirf leaf tensors mein gradients store hote hain. Intermediate gradients free ho jaate hain jab tak aap retain_grad() call nahi karte.

Advanced: `torch.no_grad()` aur Inference Mode

Inference ke dauran gradients ki zaroorat nahi hoti. Autograd disable karne se memory bachti hai aur computation fast hota hai:

with torch.no_grad():
    output = model(x)  # No graph built

Yeh kyun kaam karta hai: PyTorch grad_fn assignment skip karta hai, jis se overhead khatam ho jaata hai.

Mathematical insight: Inference mein sirf forward pass chahiye: $\overset{y}{^} = f (x; θ)$

$\frac{\partial L}{\partial θ}$ compute karne ki zaroorat nahi, toh koi graph nahi chahiye.

Inference mode (newer, faster):

with torch.inference_mode():
    output = model(x)

Difference: inference_mode() zyada aggressive hai—andar banaye gaye tensors baad mein gradient computation mein use nahi ho sakte (context se bahar bhi). Pure inference ke liye use karo; agar baad mein gradients chahiye ho toh no_grad() use karo.

Memory Management

Strategies:

Gradient checkpointing: Backward pass ke dauran intermediate activations recompute karo (memory ke liye compute trade karo)
Graphs use ke baad delete karo: PyTorch ka default (graphs .backward() ke baad free ho jaate hain)
Unnecessary tensors detach karo: Jahan zaroorat nahi, wahan gradients rok lo

Memory cost ka formula: $Memory \approx O (N \cdot S)$ jahan $N$ = number of layers, $S$ = activation size. Gradient checkpointing isse $O (N \cdot S)$ tak reduce karta hai $2 \times$ compute ki cost par.

Active Recall Exercises

Recall Ek 12-Saal Ke Bachche Ko Samjhao

Socho tum LEGO blocks se ek bahut unchi tower bana rahe ho. Har baar jab tum ek block rakhte ho, tum ek kagaz par likhte ho: "Maine block #5 ko block #3 ke upar rakha." Woh notes ki list tumhara computational graph hai—yeh record hai ki tumne tower kaise banaya.

Ab, end mein koi tumhe kehta hai, "Tower 2 inch zyada unchi hai." Tumhe pata karna hai ki kaunse blocks hatane hain. Tum upar se peeche kaam karte ho: "Agar main yeh block hatata hoon, tower 1 inch choti ho jaati hai. Agar neeche wala hatata hoon, ek aur inch chhoti hoti hai." Yahi backpropagation hai—apne notes mein backward jaana yeh dekhne ke liye ki har piece ne final height mein kitna contribute kiya.

PyTorch ka autograd ek robot ki tarah hai jo automatically woh notes likhta hai aur calculate karta hai ki har block change karne se tower ki height par kya asar padega. Tum bas banao (forward pass), aur robot changes figure out kar leta hai (backward pass) tumhare liye!

Connections

3.301-Introduction-to-PyTorch-tensors-and-operations - Tensors ki foundation jis par autograd operate karta hai
3.2.04-Backpropagation-algorithm - Woh mathematical algorithm jo autograd implement karta hai
3.4.01-Building-neural-networks-with-torch.nn.Module - Models train karne ke liye autograd use karta hai
3.5.02-Gradient-descent-variants - Optimizers jo autograd ke computed gradients consume karte hain
4.2.03-Batch-normalization - Layer jiske running stats ko carefully handle karna padta hai (graph se detach)
5.3.01-Transfer-learning - Fine-tuning mein control karna hota hai ki kaun se layers mein requires_grad ho
6.4.02-Gradient-checkpointing - Deep networks ke liye advanced memory optimization technique

#flashcards/ai-ml

PyTorch mein computational graph kya hota hai? :: Ek directed acyclic graph (DAG) jahan nodes tensors/operations hain aur edges data flow hain. PyTorch ise forward pass ke dauran dynamically build karta hai aur backward pass mein chain rule se gradients compute karne ke liye use karta hai.

requires_grad=True kya karta hai?

Ek tensor ko gradient tracking ke liye mark karta hai. PyTorch aise tensors par saare operations record karta hai, computational graph banata hai jo backpropagation ke dauran automatic differentiation enable karta hai.

Har training iteration se pehle optimizer.zero_grad() call kyun karna chahiye?

PyTorch by default gradients accumulate karta hai (naye gradients existing .grad values mein add karta hai). Zero kiye bina, pichli iterations ke gradients galat tarike se jud jaate hain, jis se parameter updates galat ho jaate hain.

torch.no_grad() aur torch.inference_mode() mein kya fark hai?

Dono gradient tracking disable karte hain. no_grad() tensors ko baad mein gradient computation mein use karne deta hai (context se bahar nikalne ke baad). inference_mode() zyada aggressive hai—andar banaye gaye tensors baad mein autograd mein participate nahi kar sakte, lekin pure inference ke liye faster hai.

.backward() ko scalar input kyun chahiye?

Backpropagation

\frac{\partial L}{\partial x}

compute karta hai jahan

L

scalar hai. Vector outputs ke liye, gradient ek Jacobian matrix hogi. Scalar mein reduce karna zaroori hai (sum, mean, ya indexing se) ya gradient argument provide karna padega.

.detach() kya karta hai?

Ek naya tensor create karta hai jo original ke saath data share karta hai lekin requires_grad=False aur grad_fn=None ke saath. Detached tensors ke through gradients flow nahi karte—unhe computational graph mein constants ki tarah treat kiya jaata hai.

grad_fn kya hai?

Us function ka reference jo tensor create karta hai (jaise <MulBackward>, <AddBackward>). Yeh computational graph banata hai—grad_fn pointers ko backward follow karne se backpropagation ke liye operations ka sequence reconstruct hota hai.

In-place operations (jaise +=) kabhi kabhi autograd kyun tod dete hain?

Autograd ko correct gradients compute karne ke liye original values chahiye. In-place ops history overwrite kar dete hain. Agar aap ek tensor modify karte hain jo backward pass mein baad mein zaroori hai, toh gradient computation fail ho jaata hai. Training ke dauran out-of-place operations use karo (y += 1 ki jagah y = y + 1).

Computational graph path

x \to a \to b \to z

ke liye chain rule formula kya hai?

\frac{\partial z}{\partial x} = \frac{\partial z}{\partial b} \cdot \frac{\partial b}{\partial a} \cdot \frac{\partial a}{\partial x}

. Autograd graph ko backward traverse karke, har node par local gradients multiply karke yeh compute karta hai.

retain_graph=True kab use karoge?

Jab aapko ek hi graph par .backward() multiple baar call karna ho (jaise multi-task learning ya adversarial training mein ek forward pass se multiple losses compute karne ke liye). By default, PyTorch memory bachane ke liye pehle .backward() ke baad graph free kar deta hai.

Leaf tensor kya hota hai?

User ne directly banaya tensor (kisi operation ka result nahi). Iska is_leaf=True aur grad_fn=None hota hai. By default sirf leaf tensors gradients store karte hain; intermediate tensors ko gradients rakhne ke liye retain_grad() chahiye.

$\frac{d}{d x} (x^{2})$ ko limit definition se derive karo :: $\frac{d}{d x} (x^{2}) = lim_{h \to 0} \frac{( x + h ) ^{2} - x ^{2}}{h} = lim_{h \to 0} \frac{x ^{2} + 2 x h + h ^{2} - x ^{2}}{h} = lim_{h \to 0} \frac{2 x h + h ^{2}}{h} = lim_{h \to 0} (2 x + h) = 2 x$

PyTorch mein by default gradient accumulation kyun hoti hai?

Mini-batch training support karne ke liye jahan aap multiple samples ke gradients sum karte hain:

\frac{\partial L _{batch}}{\partial w} = \frac{1}{N} \sum_{i = 1}^{N} \frac{\partial L _{i}}{\partial w}

. Accumulation is sum ko implement karta hai. Training iterations ke beech manually zero karna padta hai.

Computational graph store karne ki memory complexity kya hai?

O (N \cdot S)

jahan

N

operations/layers ki sankhya hai aur

S

activations ka size hai. Har operation backward pass ke liye inputs aur metadata store karta hai. Gradient checkpointing isse

O (N \cdot S)

tak reduce karta hai activations recompute karke.