GPU vs CPU design philosophy

Overview

The fundamental difference between GPU (Graphics Processing Unit) and CPU (Central Processing Unit) design philosophy reflects two radically different approaches to computation: latency optimization versus throughput optimization. Understanding this distinction is critical to modern computing architecture.

CPU Design Philosophy: Minimize Latency

Why CPUs Are Built This Way

Derivation of CPU design priorities from first principles:

Starting assumption: General-purpose code has data dependencies
- Instruction $I_n$ often depends on result from $I_{n-1}$
- Can't parallelize what must be sequential
Implication: Need to execute sequential code FAST
- If task time $T = n \times t_{inst}$ (n instructions, each taking $t_{inst}$ )
- Only way to reduce $T$ is reduce $t_{inst}$
How to reduce $t_{inst}$ ?
- Memory latency dominates: DRAM access ≈ 100ns vs compute ≈ 1ns
- Solution: Add large L1/L2/L3 caches (up to 64MB on modern CPUs)
- Cache hit reduces memory access from 100ns to 1-4ns
Control flow unpredictability
- Branches occur every 5-7 instructions on average
- Wrong branch = pipeline flush = wasted cycles
- Solution: Branch predictor (>95% accuracy) + speculative execution
Instruction-level parallelism (ILP)
- Even in sequential code, independent instructions exist
- Solution: Out-of-order execution finds and runs independent instructions simultaneously

CPU Die Area Budget

Let's derive how a modern CPU uses its transistor budget:

Total transistors≈ 10-20 billion (e.g., Intel Core i9)

Compute cores: 8-16 cores × 0.5B transistors = 4-8B (compute + local logic)
- Each core: ALUs, FPUs, registers
Cache hierarchy (SRAM, ~6 transistors/bit):
- L1: 32KB × 16 cores = 512KB
- L2: 256KB × 16 cores = 4MB
- L3: 32MB (shared)
- Total SRAM ≈ 36.5MB → $36.5 \times 10^6 \times 8 \text{ bits} \times 6 \approx 1.8\text{–}2.0\text{B transistors}$
Control logic: 3-4B transistors
- Branch predictors, instruction schedulers, memory controllers

Key observation: A large share of the die area is spent on caches and control logic to serve a single thread quickly — only a modest fraction does actual arithmetic. (Note: transistor count and die area differ, because SRAM cells are dense but numerous; the design intent is clear — CPUs "pay" heavily for latency reduction.)

GPU Design Philosophy: Maximize Throughput

Why GPUs Are Built This Way

Derivation from graphics workload:

Starting assumption: Graphics requires operating on millions of pixels/vertices independently
- Pixel $(x_1, y_1)$ color calculation doesn't depend on pixel $(x_2, y_2)$
- Embarrassingly parallel workload
Implication: Can run thousands of tasks simultaneously
- If we have $N$ independent tasks, total time: $T_{total} = \frac{N}{P} \times t_{task}$
- Where $P$ = number of parallel processors
- To minimize $T_{total}$ , maximize $P$ (even if $t_{task}$ increases!)
Design constraint: Fixed transistor budget $B$
- Simple core: $C_{simple}$ transistors
- Complex core: $C_{complex} = 10 \times C_{simple}$ (due to cache, control logic) Number of cores: $P_{GPU} = \frac{B}{C_{simple}} = 10 \times \frac{B}{C_{complex}} = 10 \times P_{CPU}$
Handling memory latency without caches
- Cache per core too expensive at 1000s of cores
- Solution: Hardware threading (warp scheduling)
- When a warp waits for memory, instantly switch to another ready warp
- Need many resident warps — often dozens of warps (hundreds to ~2048 threads) per SM — to fully hide hundreds of cycles of DRAM latency
Branch divergence problem
- No branch predictor → wrong branches are expensive
- Solution: SIMT (Single Instruction Multiple Threads)
- All threads in a warp (32 threads) execute same instruction
- If branches diverge, serialize execution (performance loss)

Formula GPU Performance Model

For throughput-oriented workload: $T_{GPU} = \frac{N_{tasks}}{N_{cores} \times Occupancy \times f_{clock}} \times t_{instruction}$ Where:

$N_{cores}$ = thousands of CUDA cores (e.g., 4096)
$Occupancy$ = fraction of cores actively computing (0.5-0.9)
$t_{instruction}$ is HIGHER than CPU but doesn't matter when $N_{tasks} \gg N_{cores}$ Derivation of speedup: If $N_{tasks} =10^6$ , $N_{cores}^{GPU} = 4096$ , $N_{cores}^{CPU} = 16$ : $\frac{T_{CPU}}{T_{GPU}} = \frac{N_{cores}^{GPU}}{N_{cores}^{CPU}} = \frac{4096}{16} = 256 \times \text{ speedup}$

GPU Die Area Budget

Total transistors ≈ 50-80 billion (e.g., NVIDIA A100)

Compute cores: thousands of CUDA cores → the majority of the compute-oriented transistor budget
- Vastly more transistors doing actual math than a CPU
Cache (SRAM, ~6 transistors/bit): ~40MB total across entire chip
- $40 \times 10^6 \times 8 \times 6 \approx 1.9\text{B transistors}$ — small relative to the whole chip, and tiny per-core
Control logic: warp schedulers, memory controllers
- NO branch predictors, NO out-of-order logic

Key observation: Compared to a CPU, a far larger share of the GPU is dedicated to arithmetic units rather than to caches and speculative control logic.

Worked example Example 1: Image Blur Operation

Problem: Blur a 4K image (8.3M pixels), each pixel requires averaging 9 neighbors (= 9 reads + 1 write) CPU approach (16-core Xeon):

Each core processes 518K pixels sequentially
Per pixel: 9 reads (assume L3 cache hit = 10ns each) + compute (5ns) + write (10ns) = 105ns
Time per core: $518,000 \times 105ns = 54ms$
Total: 54ms (all cores parallel)

Why this step? CPU uses large L3 cache to ensure neighbor pixels are cache-resident, minimizing memory latency for each core's sequential work.

GPU approach (NVIDIA RTX 3080, 8704 cores):

Launch 8.3M threads (one per pixel)
All threads execute same blur kernel (SIMT)
Per pixel: 9 reads (DRAM, 200ns but overlapped with other threads) + compute (5ns) + write
Effective time: ~2ms Why this step? While each warp waits for memory, the scheduler runs other resident warps. With dozens of warps per SM, memory latency is completely hidden.

Result: GPU is 27× faster due to massive parallelism overwhelming individual thread latency.

Worked example Example 2: Binary Search (Sequential Task)

Problem: Binary search in sorted array of 1 billion elements

CPU approach:

$\log_2(10^9) = 30$ iterations
Each iteration: 1 comparison (depends on previous)
With branch prediction (95% accuracy) and cache: ~2ns per iteration
Total: $30 \times 2ns = 60ns$ Why this step? Branch predictor learns the search pattern; speculative execution starts next iteration before current completes.

GPU approach:

Same 30 iterations (inherently sequential!)
No branch prediction, no speculation
Per iteration: 5ns (slower core, no prediction)
Total: $30 \times 5ns = 150ns$

Result: CPU is 2.5× faster. GPU's thousands of cores can't help—task is sequential.

Key insight: Amdahl's Law limits parallel speedup to sequential portions!

Worked example Example 3: Matrix Multiplication

Problem: Multiply two 1024×1024 matrices ( $C = A \times B$ )

Operations: $N^3 = 1024^3 \approx 1.07$ billion multiply-adds (≈ $2.15 \times 10^9$ FLOPs counting mul + add)

CPU approach (16 cores, AVX-512):

AVX-512 FMA: 16 FP32 lanes × 2 FMA units × 2 (mul+add) = 32 FP32 FLOPs/cycle/core
Total throughput: $16 \text{ cores} \times 32 \text{ FLOPs/cycle} \times 3 \text{ GHz} \approx 1536 \text{ GFLOPS} \approx 1.5 \text{ TFLOPS}$
Time: $\frac{2.15 \times 10^9 \text{ FLOPs}}{1.5 \times 10^{12} \text{ FLOPs/s}} \approx 1.4ms$ Why this step? Tiling algorithm keeps submatrices in L1/L2 cache, enabling peak SIMD throughput; the FMA count (32/cycle) is the true AVX-512 peak.

GPU approach (RTX 3080: FP32 CUDA cores):

Peak FP32 throughput ≈ 29.8 TFLOPS (standard CUDA cores, not Tensor Cores)
Time: $\frac{2.15 \times 10^9}{29.8 \times 10^{12}} \approx 0.072ms$

Why this step? Matrix multiplication is compute-bound (arithmetic intensity is high), perfect for GPU's massive parallel FPUs. Thread blocks cooperatively load tiles into shared memory. (Using the RTX 3080's 272 Tensor Cores with mixed precision would go even faster, but those are not FP32 units — comparing FP32-to-FP32 keeps the comparison honest.)

Result: On FP32, GPU is ≈ 20× faster—massive parallelism plus more FPUs.

The Fundamental Tradeoffs

Characteristic	CPU	GPU	Why?
Core count	8-64	1000s-10000s	Transistor budget trade-off
Core complexity	Very high	Very low	Latency vs. throughput
Cache per core	MB-scale	KB-scale	Area constraint
Clock speed	3-5 GHz	1-2 GHz	Power/thermal limits at scale
Threading	1-2 per core	dozens of warps per SM	Latency hiding mechanism
Branch handling	Prediction + speculation	Divergence serialization	Control complexity cost
Best workload	Sequential, unpredictable	Parallel, regular	Design optimization target

Common mistake Mistake 1: "More cores always = faster"

Wrong reasoning: GPUs have 100× more cores, so they should be 100× faster at everything. Steel-man (why it feels right): More workers should complete work faster—this is true for parallel tasks like painting a fence.

The fix: Amdahl's Law! If your code has ANY serial portion, parallel speedup is limited:

1% serial → max 100× speedup (no matter how many cores)
10% serial → max 10× speedup
50% serial → max 2× speedup Real example: Adding 10 numbers:

sum = 0
for i in range(10):
    sum += array[i]  # depends on previous sum!

This is 100% serial. A 10,000-core GPU offers ZERO speedup.

Common mistake Mistake 2: "GPUs are always faster for math"

Wrong reasoning: GPUs have more FLOPs, therefore faster at all computation.

Steel-man: FLOPS (floating-point operations per second) is the standard benchmark, and GPUs win here by 10-100×.

The fix: Arithmetic intensity matters: $I = \frac{\text{operations}}{\text{bytes transferred}}$ Example: Vector addition $C[i] = A[i] + B[i]$

Operations: 1 add per element
Memory: Read 2 floats (8 bytes) + write 1 float (4 bytes) = 12 bytes
Intensity: $I = 1/12 = 0.083$ ops/byte

Why this matters: NVIDIA A100 has:

Compute: 19.5 TFLOPS (FP32)
Memory bandwidth: 1.5 TB/s
Compute/bandwidth ratio: $19.5 \times 10^{12} / 1.5 \times 10^{12} = 13$ ops/byte

Vector addition needs $I=0.083$ but GPU offers $I=13$ . It's memory-bound, not compute-bound! CPU with similar memory bandwidth per core performs comparably.

Common mistake Mistake 3: "Branch divergence only affects if-statements"

Wrong reasoning: SIMT only penalizes explicit if (condition) {...} branches.

Steel-man: The term "branch" traditionally means conditional jumps in assembly.

The fix: ANY per-thread control flow causes divergence:

Loop with thread-dependent iteration counts: for (i = 0; i < threadId; i++)
Early returns: if (x < 0) return;
Even function pointers that differ per thread!

Example: Computing $\sqrt{x}$ only for positive values

if (x[i] > 0)  // Half the warp takes this branch
    result[i] = sqrt(x[i]);
else  // Other half takes this
    result[i] = 0;

Warp executes BOTH branches sequentially, masking inactive threads. Effective throughput = 50%.

When to Choose What

CPU-optimal tasks:

Operating system kernels (unpredictable, I/O-heavy)
Database query planning (sequential decision tree)
Compilation (complex, data-dependent)
Web serving (latency-sensitive, irregular)

GPU-optimal tasks:

Graphics rendering (millions of independent pixels)
Deep learning training (matrix multiplications, highly parallel)
Scientific simulations (PDEs on regular grids)
Cryptography (hash millions of candidates independently)

Hybrid approach (modern reality):

Use CPU for control logic, task scheduling, preprocessing
Offload parallel kernels to GPU
Example: Video encoding uses CPU for scene analysis, GOP structure; GPU for motion estimation, DCT transforms

Recall Explain to a 12-year-old

Imagine you need to solve 1000 math problems for homework. You have two choices:

Choice 1 (CPU): Your super-smart tutor who knows every trick. They can solve each problem in 6 seconds by using shortcuts, remembering similar problems, and guessing what's coming next. Total time: 1000 × 6s = 100 minutes.

Choice 2 (GPU): A classroom with 500 first-graders. Each one is slower—takes 10 seconds per problem because they don't know shortcuts. But working together, they solve 500 problems at once! Total time: (1000 ÷ 500) × 10s = 20 seconds!

The catch? The problems must be similar and independent. If each problem depends on the previous answer, only the tutor can help—the first-graders would just sit around waiting!

Connections

6.1.01-von-Neumann-architecture — CPUs extend this; GPUs break from it (Harvard-style separate instruction/data paths)
6.2.02-CUDA-programming-model — How software exposes GPU's parallel philosophy
6.2.03-memory-hierarchy-GPU — How GPU caching strategy differs from CPU
6.3.01-SIMD-vs-SIMT — The instruction-level parallel execution model
9.1.02-Amdahls-law — Mathematical limit on parallel speedup
9.2.01-parallel-programming-models — Software patterns matching CPU vs GPU strengths

#flashcards/hardware

What is the fundamental design difference between CPU and GPU? :: CPU optimizes for latency (minimize time for single task) using complex control logic, large caches, branch prediction. GPU optimizes for throughput (maximize tasks per second) using thousands of simple cores and hardware threading.

Derive why GPUs have 10× more cores than CPUs from transistor budget :: If simple core costs $C_s$ transistors and complex core (with cache, branch prediction) costs $C_c = 10 \times C_s$ , then with fixed budget $B$ : GPU has $P_G = B/C_s$ cores while CPU has $P_C = B/C_c = B/(10C_s) = P_G/10$ cores.

What is arithmetic intensity and why does it matter? :: $I = \text{operations}/\text{bytes transferred}$ . GPU has high compute-to-bandwidth ratio (~10-100 ops/byte). If workload's $I$ is lower, it's memory-bound and GPU advantage disappears. Example: vector addition has $I=0.083$ (1 op, 12 bytes) = memory-bound.

State Amdahl's Law and its implication :: $S(P) = 1/(f + (1-f)/P)$ where $f$

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo ek simple si baat samajhte hain. CPU aur GPU dono computation karte hain, lekin dono ka soch bilkul alag hai. CPU ek master chef jaisa hai jo har dish perfectly bana sakta hai, par ek time pe ek hi customer ko serve karta hai. Uska pura design isliye bana hai ki ek single task jaldi se jaldi khatam ho jaye. Isko hum bolte hain latency optimization — matlab ek kaam ka time kam karna. Isiliye CPU mein bade-bade caches, branch prediction, aur out-of-order execution hota hai, taaki jab tumhara code ek line ke baad doosri line pe depend karta hai (data dependency), tab bhi speed maintain rahe.

GPU ka approach ekdum ulta hai. Ye ek cafeteria jaisa hai jisme 100 chote-chote stations hain, har station ek simple cheez banata hai. Ek order thoda slow ho sakta hai, par ek saath hazaron log serve ho jaate hain. Yahi hai throughput optimization — matlab per second kitne kaam complete ho sakte hain. GPU mein hazaron simple cores hote hain, minimal caching, no branch prediction, in-order execution. Ye tab kaam aata hai jab tumhare paas bahut saare independent tasks ho jo saath-saath chal sakein, jaise image processing ya deep learning matrices.

Ab ye samajhna kyun zaroori hai? Kyunki modern computing — jaise AI, gaming, aur scientific simulations — mein tumhe dono cheezon ka combination chahiye. Agar tum jaan lo ki tumhara problem sequential hai (ek step doosre pe depend karta hai), toh CPU better hai. Aur agar problem parallel hai (bahut saare same-type kaam ek saath), toh GPU jeet jaata hai. Yeh design philosophy samajhne se tum sahi hardware choose kar paoge aur apne code ko sahi tarike se optimize kar paoge. Isi liye die area budget bhi dono mein alag hota hai — CPU apna zyada area cache aur control logic pe kharch karta hai, jabki GPU seedha arithmetic power pe.

Overview

CPU Design Philosophy: Minimize Latency

Why CPUs Are Built This Way

CPU Die Area Budget

GPU Design Philosophy: Maximize Throughput

Why GPUs Are Built This Way

GPU Die Area Budget

The Fundamental Tradeoffs

When to Choose What

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — GPU Architecture

Connections