GPU vs CPU design philosophy — Hinglish notes

6.2.1 · Hardware › GPU Architecture

Overview

GPU (Graphics Processing Unit) aur CPU (Central Processing Unit) ki design philosophy mein jo fundamental difference hai, woh computation ke do bilkul alag approaches ko reflect karta hai: latency optimization versus throughput optimization. Is distinction ko samajhna modern computing architecture ke liye bahut zaroori hai.

CPU Design Philosophy: Latency Minimize Karo

CPUs Is Tarah Kyun Bane Hain

First principles se CPU design priorities ki derivation:

Starting assumption: General-purpose code mein data dependencies hoti hain
- Instruction $I_{n}$ aksar $I_{n - 1}$ ke result par depend karta hai
- Jo sequential hona chahiye usse parallelize nahi kar sakte
Implication: Sequential code ko FAST execute karna hoga
- Agar task time $T = n \times t_{in s t}$ hai (n instructions, har ek $t_{in s t}$ leta hai)
- $T$ reduce karne ka ek hi tarika hai: $t_{in s t}$ reduce karo
$t_{in s t}$ kaise reduce karein?
- Memory latency dominate karti hai: DRAM access ≈ 100ns vs compute ≈ 1ns
- Solution: Bade L1/L2/L3 caches add karo (modern CPUs par 64MB tak)
- Cache hit memory access ko 100ns se 1-4ns tak reduce kar deta hai
Control flow ki unpredictability
- Average par har 5-7 instructions mein branches aati hain
- Wrong branch = pipeline flush = wasted cycles
- Solution: Branch predictor (>95% accuracy) + speculative execution
Instruction-level parallelism (ILP)
- Sequential code mein bhi independent instructions hoti hain
- Solution: Out-of-order execution independent instructions ko dhundh kar simultaneously run karta hai

CPU Die Area Budget

Chaliye derive karte hain ki ek modern CPU apna transistor budget kaise use karta hai:

Total transistors ≈ 10-20 billion (e.g., Intel Core i9)

Compute cores: 8-16 cores × 0.5B transistors = 4-8B (compute + local logic)
- Har core mein: ALUs, FPUs, registers
Cache hierarchy (SRAM, ~6 transistors/bit):
- L1: 32KB × 16 cores = 512KB
- L2: 256KB × 16 cores = 4MB
- L3: 32MB (shared)
- Total SRAM ≈ 36.5MB → $36.5 \times 1 0^{6} \times 8 bits \times 6 \approx 1.8 - 2.0 B transistors$
Control logic: 3-4B transistors
- Branch predictors, instruction schedulers, memory controllers

Key observation: Die area ka ek bada hissa caches aur control logic par kharch hota hai taaki ek single thread ko jaldi serve kiya ja sake — actual arithmetic bahut kam portion mein hoti hai. (Note: transistor count aur die area alag hote hain, kyunki SRAM cells dense lekin bahut zyada hote hain; design intent clear hai — CPUs latency reduction ke liye bhaari "payment" karte hain.)

GPU Design Philosophy: Throughput Maximize Karo

GPUs Is Tarah Kyun Bane Hain

Graphics workload se derivation:

Starting assumption: Graphics mein millions of pixels/vertices par independently operate karna padta hai
- Pixel $(x_{1}, y_{1})$ ka color calculation pixel $(x_{2}, y_{2})$ par depend nahi karta
- Embarrassingly parallel workload
Implication: Hazaron tasks simultaneously run kar sakte hain
- Agar hamare paas $N$ independent tasks hain, total time: $T_{t o t a l} = \frac{N}{P} \times t_{t a s k}$
- Jahaan $P$ = parallel processors ki number
- $T_{t o t a l}$ minimize karne ke liye, $P$ maximize karo (chahe $t_{t a s k}$ badh jaaye!)
Design constraint: Fixed transistor budget $B$
- Simple core: $C_{s im pl e}$ transistors
- Complex core: $C_{co m pl e x} = 10 \times C_{s im pl e}$ (cache, control logic ki wajah se) Cores ki number: $P_{GP U} = \frac{B}{C _{s im pl e}} = 10 \times \frac{B}{C _{co m pl e x}} = 10 \times P_{C P U}$
Caches ke bina memory latency handle karna
- 1000s cores par per core cache bahut expensive hai
- Solution: Hardware threading (warp scheduling)
- Jab ek warp memory ka wait karta hai, turant doosre ready warp par switch karo
- DRAM latency ke saikdon cycles ko poori tarah hide karne ke liye — often dozens of warps (hundreds se ~2048 threads tak) per SM — bahut saare resident warps chahiye
Branch divergence problem
- Koi branch predictor nahi → wrong branches expensive hain
- Solution: SIMT (Single Instruction Multiple Threads)
- Ek warp ke saare threads (32 threads) same instruction execute karte hain
- Agar branches diverge karein, execution serialize ho jaati hai (performance loss)

GPU Die Area Budget

Total transistors ≈ 50-80 billion (e.g., NVIDIA A100)

Compute cores: hazaron CUDA cores → compute-oriented transistor budget ka majority
- CPU ke mukable bahut zyada transistors actual math kar rahe hain
Cache (SRAM, ~6 transistors/bit): pure chip mein total ~40MB
- $40 \times 1 0^{6} \times 8 \times 6 \approx 1.9 B transistors$ — poore chip ke relative chhota, aur per-core bilkul tiny
Control logic: warp schedulers, memory controllers
- Koi branch predictors NAHI, koi out-of-order logic NAHI

Key observation: CPU ke comparison mein, GPU ka bahut bada hissa arithmetic units ko dedicated hai, na ki caches aur speculative control logic ko.

Worked example Example 1: Image Blur Operation

Problem: Ek 4K image (8.3M pixels) blur karo, har pixel ko 9 neighbors average karne chahiye (= 9 reads + 1 write) CPU approach (16-core Xeon):

Har core 518K pixels sequentially process karta hai
Per pixel: 9 reads (assume L3 cache hit = 10ns each) + compute (5ns) + write (10ns) = 105ns
Time per core: $518, 000 \times 105 n s = 54 m s$
Total: 54ms (saare cores parallel)

Yeh step kyun? CPU large L3 cache use karta hai taaki neighbor pixels cache-resident rahein, har core ke sequential work ke liye memory latency minimize ho.

GPU approach (NVIDIA RTX 3080, 8704 cores):

8.3M threads launch karo (ek per pixel)
Saare threads same blur kernel execute karte hain (SIMT)
Per pixel: 9 reads (DRAM, 200ns lekin doosre threads ke saath overlapped) + compute (5ns) + write
Effective time: ~2ms Yeh step kyun? Jab ek warp memory ka wait karta hai, scheduler doosre resident warps run karta hai. Dozens of warps per SM hone se, memory latency poori tarah hide ho jaati hai.

Result: GPU 27× faster hai massive parallelism ki wajah se jo individual thread latency ko overwhelm kar deti hai.

Worked example Example 2: Binary Search (Sequential Task)

Problem: 1 billion elements ke sorted array mein binary search karo

CPU approach:

$lo g_{2} (1 0^{9}) = 30$ iterations
Har iteration: 1 comparison (pichle par depend karta hai)
Branch prediction (95% accuracy) aur cache ke saath: ~2ns per iteration
Total: $30 \times 2 n s = 60 n s$ Yeh step kyun? Branch predictor search pattern seekhta hai; speculative execution agla iteration start kar deta hai current complete hone se pehle.

GPU approach:

Same 30 iterations (inherently sequential hain!)
Koi branch prediction nahi, koi speculation nahi
Per iteration: 5ns (slower core, no prediction)
Total: $30 \times 5 n s = 150 n s$

Result: CPU 2.5× faster hai. GPU ke hazaron cores help nahi kar sakte — task sequential hai.

Key insight: Amdahl's Law sequential portions par parallel speedup ko limit karta hai!

Worked example Example 3: Matrix Multiplication

Problem: Do 1024×1024 matrices multiply karo ( $C = A \times B$ )

Operations: $N^{3} = 102 4^{3} \approx 1.07$ billion multiply-adds (≈ $2.15 \times 1 0^{9}$ FLOPs counting mul + add)

CPU approach (16 cores, AVX-512):

AVX-512 FMA: 16 FP32 lanes × 2 FMA units × 2 (mul+add) = 32 FP32 FLOPs/cycle/core
Total throughput: $16 cores \times 32 FLOPs/cycle \times 3 GHz \approx 1536 GFLOPS \approx 1.5 TFLOPS$
Time: $\frac{2.15 \times 1 0 ^{9} FLOPs}{1.5 \times 1 0 ^{12} FLOPs/s} \approx 1.4 m s$ Yeh step kyun? Tiling algorithm submatrices ko L1/L2 cache mein rakhta hai, peak SIMD throughput enable karta hai; FMA count (32/cycle) true AVX-512 peak hai.

GPU approach (RTX 3080: FP32 CUDA cores):

Peak FP32 throughput ≈ 29.8 TFLOPS (standard CUDA cores, Tensor Cores nahi)
Time: $\frac{2.15 \times 1 0 ^{9}}{29.8 \times 1 0 ^{12}} \approx 0.072 m s$

Yeh step kyun? Matrix multiplication compute-bound hai (arithmetic intensity high hai), GPU ke massive parallel FPUs ke liye perfect. Thread blocks cooperatively tiles ko shared memory mein load karte hain. (RTX 3080 ke 272 Tensor Cores mixed precision ke saath aur bhi fast hote, lekin woh FP32 units nahi hain — FP32-to-FP32 compare karna comparison ko honest rakhta hai.)

Result: FP32 par, GPU ≈ 20× faster hai — massive parallelism plus zyada FPUs.

Fundamental Tradeoffs

Characteristic	CPU	GPU	Kyun?
Core count	8-64	1000s-10000s	Transistor budget trade-off
Core complexity	Bahut high	Bahut low	Latency vs. throughput
Cache per core	MB-scale	KB-scale	Area constraint
Clock speed	3-5 GHz	1-2 GHz	Scale par power/thermal limits
Threading	1-2 per core	dozens of warps per SM	Latency hiding mechanism
Branch handling	Prediction + speculation	Divergence serialization	Control complexity cost
Best workload	Sequential, unpredictable	Parallel, regular	Design optimization target

Common mistake Mistake 1: "Zyada cores hamesha = faster"

Galat reasoning: GPUs mein 100× zyada cores hain, toh unhe sab cheez mein 100× faster hona chahiye. Steel-man (kyun sahi lagta hai): Zyada workers kaam jaldi khatam karte hain — yeh parallel tasks ke liye sach hai jaise fence paint karna.

Fix: Amdahl's Law! Agar aapke code mein KOI bhi serial portion hai, parallel speedup limited hai:

1% serial → max 100× speedup (chahe kitne bhi cores hon)
10% serial → max 10× speedup
50% serial → max 2× speedup Real example: 10 numbers add karna:

sum = 0
for i in range(10):
    sum += array[i]  # pichle sum par depend karta hai!

Yeh 100% serial hai. 10,000-core GPU ZERO speedup deta hai.

Common mistake Mistake 2: "GPUs math ke liye hamesha faster hain"

Galat reasoning: GPUs mein zyada FLOPs hain, isliye saari computation mein faster hain.

Steel-man: FLOPS (floating-point operations per second) standard benchmark hai, aur GPUs yahan 10-100× se jeette hain.

Fix: Arithmetic intensity matter karti hai: $I = \frac{operations}{bytes transferred}$ Example: Vector addition $C [i] = A [i] + B [i]$

Operations: 1 add per element
Memory: 2 floats read (8 bytes) + 1 float write (4 bytes) = 12 bytes
Intensity: $I = 1/12 = 0.083$ ops/byte

Yeh kyun matter karta hai: NVIDIA A100 mein:

Compute: 19.5 TFLOPS (FP32)
Memory bandwidth: 1.5 TB/s
Compute/bandwidth ratio: $19.5 \times 1 0^{12} /1.5 \times 1 0^{12} = 13$ ops/byte

Vector addition ko $I = 0.083$ chahiye lekin GPU $I = 13$ offer karta hai. Yeh memory-bound hai, compute-bound nahi! CPU jo per core similar memory bandwidth rakhta hai, comparably perform karta hai.

Common mistake Mistake 3: "Branch divergence sirf if-statements ko affect karta hai"

Galat reasoning: SIMT sirf explicit if (condition) {...} branches ko penalize karta hai.

Steel-man: "Branch" traditionally assembly mein conditional jumps ko kehte hain.

Fix: KOI bhi per-thread control flow divergence cause karta hai:

Thread-dependent iteration counts wala loop: for (i = 0; i < threadId; i++)
Early returns: if (x < 0) return;
Yahan tak ki function pointers jo per thread alag hain!

Example: Sirf positive values ke liye $x$ compute karna

if (x[i] > 0)  // Warp ka aadha hissa yeh branch leta hai
    result[i] = sqrt(x[i]);
else  // Doosra aadha yeh leta hai
    result[i] = 0;

Warp DONO branches sequentially execute karta hai, inactive threads ko mask karke. Effective throughput = 50%.

Kab Kya Choose Karein

CPU-optimal tasks:

Operating system kernels (unpredictable, I/O-heavy)
Database query planning (sequential decision tree)
Compilation (complex, data-dependent)
Web serving (latency-sensitive, irregular)

GPU-optimal tasks:

Graphics rendering (millions of independent pixels)
Deep learning training (matrix multiplications, highly parallel)
Scientific simulations (regular grids par PDEs)
Cryptography (millions of candidates independently hash karna)

Hybrid approach (modern reality):

Control logic, task scheduling, preprocessing ke liye CPU use karo
Parallel kernels GPU par offload karo
Example: Video encoding mein CPU scene analysis, GOP structure ke liye use hota hai; GPU motion estimation, DCT transforms ke liye

Recall 12-saal ke bachchon ko explain karo

Imagine karo ki homework ke liye 1000 math problems solve karne hain. Tumhare paas do choices hain:

Choice 1 (CPU): Tumhara super-smart tutor jo har trick jaanta hai. Woh shortcuts use karke, similar problems yaad karke, aur aage ka guess karke har problem 6 seconds mein solve kar sakta hai. Total time: 1000 × 6s = 100 minutes.

Choice 2 (GPU): 500 first-graders ka ek classroom. Har ek slower hai — problem mein 10 seconds lagte hain kyunki woh shortcuts nahi jaante. Lekin saath mein kaam karke, woh ek saath 500 problems solve karte hain! Total time: (1000 ÷ 500) × 10s = 20 seconds!

Catch kya hai? Problems similar aur independent honi chahiye. Agar har problem pichle answer par depend karti hai, toh sirf tutor help kar sakta hai — first-graders bas wait karte rahenge!

Connections

6.1.01-von-Neumann-architecture — CPUs isko extend karte hain; GPUs isse break karte hain (Harvard-style separate instruction/data paths)
6.2.02-CUDA-programming-model — Software GPU ki parallel philosophy ko kaise expose karta hai
6.2.03-memory-hierarchy-GPU — GPU ki caching strategy CPU se kaise alag hai
6.3.01-SIMD-vs-SIMT — Instruction-level parallel execution model
9.1.02-Amdahls-law — Parallel speedup ki mathematical limit
9.2.01-parallel-programming-models — Software patterns jo CPU vs GPU strengths se match karte hain

#flashcards/hardware

CPU aur GPU ke beech fundamental design difference kya hai? :: CPU latency optimize karta hai (single task ke liye time minimize) complex control logic, large caches, branch prediction use karke. GPU throughput optimize karta hai (tasks per second maximize) hazaron simple cores aur hardware threading use karke.

Transistor budget se derive karo ki GPUs mein CPUs se 10× zyada cores kyun hote hain :: Agar simple core ko $C_{s}$ transistors lagte hain aur complex core (cache, branch prediction ke saath) ko $C_{c} = 10 \times C_{s}$ , toh fixed budget $B$ ke saath: GPU ke paas $P_{G} = B / C_{s}$ cores hain jabki CPU ke paas $P_{C} = B / C_{c} = B / (10 C_{s}) = P_{G} /10$ cores hain.

Arithmetic intensity kya hai aur yeh kyun matter karta hai? :: $I = operations / bytes transferred$ . GPU mein high compute-to-bandwidth ratio hai (~10-100 ops/byte). Agar workload ka $I$ kam ho, toh woh memory-bound hai aur GPU advantage khatam ho jaata hai. Example: vector addition mein $I = 0.083$ hai (1 op, 12 bytes) = memory-bound.

Amdahl's Law aur uski implication batao :: $S (P) = 1/ (f + (1 - f) / P)$ jahaan $f$