GPU acceleration and device management — Hinglish notes

3.3.6 · AI-ML › Deep Learning Frameworks

Overview

GPU acceleration tensor computations ko CPU se specialized parallel processors (GPUs) par move karta hai jo ek saath hazaron operations perform kar sakte hain. Device management ek practice hai jisme explicitly control kiya jaata hai ki tensors aur models kahan rahenge (CPU vs GPU) aur devices ke beech data kaise move hoga.

Figure — GPU acceleration and device management

Core Concepts

GPUs kyun Jeetते Hain: Physics

Derivation: Speedup Factor

Matrix multiplication $C = A B$ compute karne ka time jahan $A \in R^{m \times k}$ , $B \in R^{k \times n}$ :

$T_{CPU} = \frac{2 mk n}{f _{CPU} \cdot c _{CPU}}$

jahan $f_{CPU}$ clock frequency (Hz) hai, $c_{CPU}$ cores (~16) hai.

GPU ke liye jisme $c_{GPU}$ cores (~10,000) hain:

$T_{GPU} = \frac{2 mk n}{f _{GPU} \cdot c _{GPU}} + T_{transfer}$

Speedup: $S = \frac{T _{CPU}}{T _{GPU}} \approx \frac{f _{GPU} \cdot c _{GPU}}{f _{CPU} \cdot c _{CPU}}$

Typical values daalo: $f_{GPU} \approx 1.5$ GHz, $c_{GPU} = 10000$ , $f_{CPU} = 3$ GHz, $c_{CPU} = 16$ :

$S \approx \frac{1.5 \times 10000}{3 \times 16} = \frac{15000}{48} \approx 312 \times$

Lekin: Yeh assume karta hai ki $T_{transfer} ≪ T_{GPU}$ . Agar tum har operation mein data transfer karo, toh speedup collapse ho jaata hai.

PyTorch Device Management: First Principles Se

Tensors Ko Apni Location Kaise Pata Hoti Hai

Har PyTorch tensor ka ek .device property hota hai. Under the hood, yeh memory mein ek pointer hai ya toh:

CPU DRAM (CPU cores se accessible)
GPU VRAM (GPU cores se accessible, CPU se isolated)

Worked example Example 1: Devices Par Tensors Banana

import torch
 
# Default: CPU
x = torch.randn(1000, 1000)
print(x.device)  # cpu
 
# Explicit GPU (if available)
if torch.cuda.is_available():
   device = torch.device('cuda')
   y = torch.randn(1000, 1000, device=device)
   print(y.device)  # cuda:0
   
   # Move existing tensor
   x_gpu = x.to(device)  # Copies data to GPU
   print(x_gpu.device)  # cuda:0

Yeh step kyun?

torch.device('cuda') ek device handle banata hai.
device=device GPU VRAM mein memory allocate karta hai.
.to(device) under the hood cudaMemcpy trigger karta hai (PCIe transfer).

Movement Tax

Worked example Example 2: Transfer Cost Quantify Karna

import time
x_cpu = torch.randn(1000, 1000)  # 400 MB
# CPU computation
start = time.time()
z_cpu = x_cpu @ x_
print(f"CPU time: {time.time() - start:.4f}s")
 
# GPU computation (with transfer)
start = time.time()
x_gpu = x_cpu.to('cuda')  # Transfer cost
z_gpu = x_gpu @ x_gpu
torch.cuda.synchronize()  # Wait for GPU to finish
print(f"GPU time (with transfer): {time.time() - start:.4f}s")
# GPU computation (no transfer)
start = time.time()
z_gpu2 = x_gpu @ x_gpu  # Data already on GPU
torch.cuda.synchronize()
print(f"GPU time (no transfer): {time.time() - start:.4f}s")

Typical output:

CPU: 2.3s
GPU with transfer: 0.8s (transfer ~0.7s laga, compute ~0.1s)
GPU no transfer: 0.1s (23× faster!)

Yeh step kyun? torch.cuda.synchronize() crucial hai—GPU ops asynchronous hote hain. Iske bina timing galat hogi.

Training Loop: Device-Aware Pattern

Standard Recipe

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
 
# 1. Move model to device (once)
model = MyModel().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())
 
# 2. In training loop, move batches to device
for epoch in range(num_epochs):
    for batch_idx, (data, target) in enumerate(train_loader):
        # Move batch to GPU
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)  # Computed on GPU
        loss = criterion(output, target)  # Loss on GPU
        loss.backward()  # Gradients computed on GPU
        optimizer.step()  # Parameter update on GPU
        
        # Only move to CPU for logging
        if batch_idx % 100 == 0:
            print(f'Loss: {loss.item():.4f}')  # .item() copies scalar to CPU

Har step kyun?

Model par .to(device) saare parameters/buffers ko GPU memory mein move karta hai (one-time cost)
Batches ko GPU par move karna: unavoidable hai (data loader disk se padhta hai → CPU RAM → GPU VRAM)
Logging ke liye .item(): scalars bahut chote hote hain (~4 bytes), transfer negligible hai
Baaki sab GPU par rehta hai: koi unnecessary round-trips nahi

Multi-GPU: Data Parallelism Derivation

Problem

Ek GPU ki limited memory hoti hai (~24 GB). Bade models ya batches fit nahi hote.

Solution: Batch Split Karo

Worked example Example 3: PyTorch DataParallel

if torch.cuda.device_count() > 1:
   print(f"Using {torch.cuda.device_count()} GPUs")
   model = nn.DataParallel(model)  # Wraps model
model.to(device)
# Training loop unchanged!
# DataParallel automatically:
# - Splits batch across GPUs
# - Gathers outputs
# - Averages gradients

Yeh kyun kaam karta hai? nn.DataParallel forward() ko intercept karta hai, input split karta hai, replicas run karta hai, output gather karta hai. Gradients .backward() mein average hote hain.

Modern alternative: DistributedDataParallel (DP) ek process per GPU use karta hai, NCCL ke through faster all-reduce hota hai.

Common Mistakes

Common mistake Mistake 1: CPU aur GPU Tensors Mix Karna

x = torch.randn(100, 100).to('cuda')
y = torch.randn(100, 100)  # CPU
z = x + y  # RuntimeError: Expected all tensors on same device

Kyun sahi lagta hai: "Kya PyTorch auto-transfer nahi kar sakta?" Reality yeh hai: Auto-transfer expensive ops ko hide kar deta. Explicit better hota hai. GPU kernels CPU memory access nahi kar sakte.

Fix: Operation se pehle y = y.to('cuda'). Ya match karne ke liye .to(x.device) use karo.

Common mistake Mistake 2: Tight Loop Mein Synchronous CPU Access

for i in range(1000):
   x_gpu = torch.randn(1000, 1000, device='cuda')
   result = some_gpu_op(x_gpu)
   print(result[0, 0])  # Transfers ONE element 1000 times!

Kyun sahi lagta hai: "Sirf ek number print kar raha hoon, kitna bura ho sakta hai?"

Reality yeh hai: Har access GPU→CPU sync force karta hai (cudaDeviceSynchronize). GPU PCIe ka wait karte hue idle baitha rehta hai.

Fix: GPU par accumulate karo, ek baar transfer karo: results.append(result) phir cpu_results = torch.stack(results).cpu().

Common mistake Mistake 3: Logging Ke Liye Loss Ko CPU Par Move Karna Bhool Jaana

loss_history = []
for batch in train_loader:
   # ... training code ...
   loss_history.append(loss)  # loss is CUDA tensor!

Kyun sahi lagta hai: "Lists kisi bhi Python object ke saath kaam karti hain."

Reality yeh hai: Har appended GPU tensor GPU memory allocated rakhta hai. 1000 iterations = 1000 tensors VRAM mein, OOM crash.

Fix: loss_history.append(loss.item()) ya loss.detach().cpu().

Mixed Precision Training: 2× Speedup Trick

FP16 kyun?

Modern GPUs (Tensor Cores) FP16 matrix multiplies FP32 se 2× FLOPS par kar sakte hain.

Problem yeh hai: FP16 range $\pm 65, 504$ hai. Gradients ko aksar FP32 precision chahiye hoti hai (tiny values).

Solution hai: Automatic Mixed Precision (AMP)

Worked example Example 4: PyTorch Mein AMP

from torch.cuda.amp import autocast, GradScaler
 
scaler = GradScaler()
 
for data, target in train_loader:
   data, target = data.to(device), target.to(device)
   optimizer.zero_grad()
   # Forward in FP16
   with autocast():
       output = model(data)
       loss = criterion(output, target)
   
   # Backward with scaled loss
   scaler.scale(loss).backward()
   # Unscale, check for infs/NaNs, step
   scaler.step(optimizer)
   scaler.update()

Yeh step kyun?

autocast(): Eligible ops ko FP16 (matmul, conv) mein cast karta hai lekin numerically sensitive ops (softmax, loss) ko FP32 mein rakhta hai.
scaler.scale(loss): .backward() se pehle scale factor se multiply karta hai.
scaler.step(): Gradients unscale karta hai, overflow check karta hai (agar mila, update skip karta hai aur scale factor reduce karta hai).

Memory Management: Hidden Enemy

GPU Memory Lifecycle

Allocate → Use → Free (maybe)

PyTorch ek caching allocator use karta hai:

Pehla allocation: cudaMalloc (slow, 1ms)
Baad mein: cached blocks reuse karta hai (fast, 1µs)
.empty_cache() memory CUDA ko return karta hai lekin OS ko free nahi karta

Worked example Example 5: OOM Debug Karna

import torch
 
print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
# Force garbage collection
import gc
gc.collect()
torch.cuda.empty_cache()
# Find large tensors
for obj in gc.get_objects():
   if torch.is_tensor(obj) and obj.is_cuda:
       print(type(obj), obj.size(), obj.element_size() * obj.nelement() / 1e6, "MB")

Yeh kyun help karta hai: memory_allocated live tensors dikhata hai. memory_reserved mein cache bhi shaamil hai. Gap = wasted space.

Bade Batches Ke Liye Gradient Accumulation

Jab batch size 512 fit nahi hoti, gradient accumulation use karo:

accumulation_steps = 4  # Effective batch size = 128× 4 = 512
 
for i, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)
    
    output = model(data)
    loss = criterion(output, target) / accumulation_steps  # Scale loss
    loss.backward()  # Accumulate gradients
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Derivation:

Normal: $\nabla_{θ} L = \frac{1}{B} \sum_{i = 1}^{B} \nabla_{θ} ℓ_{i}$
Accumulated: $\nabla_{θ} L = \frac{1}{4} \sum_{j = 1}^{4} [\frac{1}{B /4} \sum_{i \in batch_{j}} \nabla_{θ} ℓ_{i}] = \frac{1}{B} \sum_{i = 1}^{B} \nabla_{θ} ℓ_{i}$ ✓

Loss divide kyun karo? .backward() gradients sum karta hai. /accumulation_steps ke bina, final gradients 4× zyada bade ho jaate.

Recall 12 Saal Ke Bacche Ko Explain Karo

Socho tumhare paas math homework hai: 1000 multiplication problems. CPU (tumhara brain): Tum unhe ek-ek karke solve karte ho. Tum smart aur fast ho, lekin time lagta hai kyunki tum ek time mein ek kaam karte ho.

GPU (1000 bachchon ki class): Har baccha EK problem solve karta hai. Yeh geniuses nahi hain, lekin milke woh utne hi time mein finish karte hain jitna ek bacche ko ek problem solve karne mein lagta hai!

Neural networks isi homework jaisi hain—lakho simple multiplications. GPUs ke paas hazaron "bacche" (cores) hain jo parallel mein kaam karte hain.

Catch yeh hai: Homework ko tumhare desk (CPU memory) se classroom (GPU memory) mein le jaane mein time lagta hai. Isliye hum saare problems ek saath move karte hain (batches), ek-ek karke nahi.

Mixed precision: Kuch problems simpler math se solve ho sakti hain (FP16 = kam decimal places use karna). Yeh faster hai lekin kabhi kabhi galti karta hai, isliye hum answer ki ek "master copy" high precision mein rakhte hain (FP32).

Connections

3.3.05-Computational-graphsand-autograd: GPU computational graph ke through forward aur backward passes dono accelerate karta hai
3.2.03-Backpropagation: Gradient computation embarrassingly parallel hai, GPUs ke liye perfect
3.4.02-Batch-normalization: Multi-GPU mein all-reduce chahiye (sync mean/variance)
4.1.01-CNN-architectures: Convolutions GPU bottleneck hain—kernels optimize karna critical hai
3.3.07-Distributed-training-strategies: Multi-node device management ko machines ke across extend karta hai

#flashcards/ai-ml

Deep learning ke liye GPUs CPUs se better kyun perform karte hain? :: GPUs ke paas hazaron cores hain jo parallel arithmetic (low clock speed lekin massive parallelism) ke liye optimized hain, jabki CPUs ke paas ~16 complex cores hote hain sequential tasks ke liye. Neural network operations (matrix multiplies) embarrassingly parallel hain—har element independently compute hota hai—jo GPU architecture ke saath perfectly match karta hai.

GPU memory transfer bottleneck kya hai?

PCIe bandwidth (~16 GB/s) GPU memory bandwidth (~900 GB/s) se 50× slow hai. Frequent CPU↔GPU transfers speedup khatam kar dete hain. Solution: data ek baar move karo, computations GPU par rakho, sirf results transfer karo.

torch.cuda.synchronize() kya karta hai aur yeh kyun zaroori hai?

GPU operations asynchronous hoti hain—Python GPU compute karte waqt continue karta rehta hai. synchronize() tab tak block karta hai jab tak saare GPU kernels finish nahi ho jaate. Accurate timing ke liye aur jab CPU ko GPU results read karne hon tab zaroori hai. Iske bina, timing compute time nahi balki queue time measure karti hai.

Data parallelism ka all-reduce step explain karo :: Har GPU apne batch partition par independently gradients compute karta hai. All-reduce gradients synchronize karta hai: har GPU ko saare GPUs ke average gradient milte hain. Formula: $\nabla θ = \frac{1}{N} \sum_{i = 1}^{N} \nabla θ_{i}$ . Ring all-reduce (NCCL library) ke through efficiently implement hota hai, bandwidth cost $O (2 P / N)$ jahan $P$ parameter count hai, $N$ GPUs hain.

Mixed precision training loss scaling kyun use karta hai?

FP16 range

\pm 65, 504

hai with min

6 \times 1 0^{- 8}

. Deep networks mein gradients

1 0^{- 10}

ho sakte hain (FP16 mein zero par underflow). Backward pass se pehle loss ko

2^{16}

se scale karne par gradients FP16 range mein aa jaate hain. Backward ke baad, optimizer step se pehle gradients unscale karo. Chhote gradients overflow ke bina preserve hote hain.

Gradient accumulation kya hai aur kab use karna chahiye?

Limited GPU memory par large batch sizes simulate karna. Mini-batches par forward/backward compute karo, gradients accumulate karo (optimizer step mat lo), phir N accumulations ke baad step lo. Effective batch size = mini-batch × N. Critical: loss ko N se divide karo warna optimizer N× bade gradients dekhega. Tab use karo jab batch size B memory mein fit nahi hoti lekin B/N hoti hai.

GPU tensors ko list mein append karne se OOM kyun hota hai?

Har tensor reference GPU memory allocated rakhta hai. 1000 tensors ki list = VRAM mein 1000 separate allocations. PyTorch ka garbage collector Python object delete hone tak free nahi karta. Solution: scalars ke liye .item() (CPU par Python float ki tarah copy hota hai), ya bade tensors ke liye .detach().cpu(), GPU reference tod deta hai.

memory_allocated aur memory_reserved mein kya difference hai? :: memory_allocated(): currently live tensor memory. memory_reserved(): memory jo PyTorch CUDA se hold karta hai (cache bhi shaamil hai). Gap = cached free blocks. del tensor ke baad, memory_allocated girta hai lekin reserved high rehta hai (cache). .empty_cache() reserved memory CUDA ko return karta hai lekin OS ko nahi (CUDA driver ab bhi hold karta hai).