GPU vs CPU design philosophy: Visual walkthrough — Hardware deep dive

This page rebuilds the central result of the parent note from absolute zero, one picture at a time. We start with a single question — "how long does a job take?" — and end up seeing why CPUs and GPUs make opposite bets. No symbol appears before we draw it.

If you have not met the machine that runs one instruction after another, peek at 6.1.01-von-Neumann-architecture first; everything below builds on that single idea.

Step 1 — What does "how long does a job take?" even mean?

WHAT. Before any chip, let's agree on one word. A task is one small piece of work — for us, "compute the colour of one pixel". Latency is the time from start to finish of one task. Throughput is how many tasks finish per second.

WHY. These two numbers pull in opposite directions, and the whole GPU-vs-CPU story is nothing but choosing which one to make small (latency) or big (throughput). If we don't separate them, we can't see the trade-off.

PICTURE. Look at the two timelines. On top, one worker finishes each task quickly but does them one after another. On the bottom, many slow workers each take longer, but they overlap — so more finish per second.

Here $t_{t a s k}$ is our symbol for the latency of one task. Every equation on this page is built from it.

Step 2 — The one formula everything rests on

WHAT. Suppose we have $N$ independent tasks and $P$ workers, each worker able to run one task at a time, each task costing $t_{t a s k}$ seconds. The total wall-clock time is:

$T_{t o t a l} = tasks per worker \frac{N}{P} \times seconds per task t_{t a s k}$

Term by term: $N$ is the pile of work, $P$ is how many hands we have, so $N / P$ is how many tasks each hand must do; multiply by $t_{t a s k}$ (seconds per task) and you get seconds.

WHY. This is the only equation we need. Every design choice — big cache, many cores, branch prediction — is just an attempt to push one of these three letters ( $N$ , $P$ , or $t_{t a s k}$ ) in a favourable direction. We will now watch CPU and GPU pull different levers.

PICTURE. The formula as a set of three dials. Turning down $t_{t a s k}$ (CPU's move) or turning up $P$ (GPU's move) both shrink $T_{t o t a l}$ — but they cost transistors in different places.

Step 3 — The CPU's bet: make $t_{t a s k}$ tiny

WHAT. The CPU assumes the worst input: a task where step $I_{n}$ needs the answer of step $I_{n - 1}$ . That is a data dependency — you literally cannot start the next step early. When work is a chain like this, $P$ is stuck at 1 (extra workers have nothing to do), so the only free lever left is $t_{t a s k}$ .

WHY these tools, not others? Two things make a single task slow, so the CPU spends transistors killing exactly those two:

Memory is far away. Reading DRAM costs ~ $100 ns$ ; doing arithmetic costs ~ $1 ns$ . So the CPU builds a big cache (fast nearby memory). A cache hit drops $100 ns$ to a few ns. We use a cache — not more cores — because more cores can't help a chain that must run in order.
Branches are surprises. Roughly every 5–7 instructions the code asks "if…?" and the chip doesn't yet know which way to go. Guessing wrong wastes work. So the CPU adds a branch predictor (>95% right) and speculative execution — it guesses and runs ahead. Again: this shrinks $t_{t a s k}$ for one chain, which is the only lever available.

PICTURE. The CPU floorplan: a small patch of green (the arithmetic units — the part that does math) surrounded by a big blue sea of cache and a fat orange block of control logic. Most silicon is not doing math; it exists to make the little math it does happen fast.

Step 4 — The GPU's bet: make $P$ enormous

WHAT. The GPU assumes the best input: a mountain of tasks that don't depend on each other. Colouring pixel $(x_{1}, y_{1})$ needs nothing from pixel $(x_{2}, y_{2})$ . That is an embarrassingly parallel workload — you could hand every pixel to a different worker. Now $P$ is a free lever, so the GPU pushes it as hard as the transistor budget allows.

WHY. Fix a transistor budget $B$ . A simple core costs $C_{s im pl e}$ transistors; a complex CPU-style core (with its cache and control) costs about ten times more, $C_{co m pl e x} = 10 C_{s im pl e}$ . So:

$P_{GP U} = \frac{B}{C _{s im pl e}} = 10 \times \frac{B}{C _{co m pl e x}} = 10 \times P_{C P U}$

Reading it: same budget $B$ , but by refusing to pay for cache/branch-prediction per core, each core is 10× cheaper, so you can afford ~10× more of them. The GPU accepts a bigger $t_{t a s k}$ per core in exchange for a much bigger $P$ — and the formula from Step 2 says that's a great trade when $N$ is huge.

PICTURE. The GPU floorplan next to Step 3's: now the green (math units) fills almost the whole chip; the blue cache and orange control shrink to thin strips.

This is the same lever-pulling logic later formalised in 9.2.01-parallel-programming-models.

Step 5 — But GPUs have no big cache. How do they survive slow memory?

WHAT. We just threw away the CPU's cache. So every core still waits ~ $100$ – $200 ns$ for DRAM. If a core just sat idle while waiting, all those extra cores would be useless. The GPU's trick: keep many groups of threads resident, and the instant one group stalls on memory, the scheduler swaps in another group that's ready to compute.

WHY this tool — hardware threading — and not caching? A cache big enough for thousands of cores would eat the whole transistor budget, undoing Step 4. Instead the GPU hides latency rather than reducing it: while group A waits, groups B, C, D… do math. You need enough waiting groups (dozens of warps — a warp is a bundle of 32 threads that march in lockstep) so that the compute of the others fills every gap of the stalled one.

PICTURE. A row of coloured bars (warps). Warp A hits a grey "memory wait" gap; underneath, warps B, C, D keep the math units busy through that gap, so the arithmetic pipeline is never idle.

The "same instruction, many threads" idea is SIMT, unpacked in 6.3.01-SIMD-vs-SIMT; the swapping machinery lives in the 6.2.03-memory-hierarchy-GPU.

Step 6 — The payoff: when $N ≫ P$ , only core-count matters

WHAT. Plug the huge- $N$ case into Step 2. When there are far more tasks than cores ( $N ≫ P$ ), the per-task time cancels out of the ratio of the two machines and the speedup collapses to a comparison of core counts:

$\frac{T _{C P U}}{T _{GP U}} = \frac{N / P _{C P U}}{N / P _{GP U}} = \frac{P _{GP U}}{P _{C P U}}$

The $N$ (same pile of work) cancels; the $t_{t a s k}$ (per-task cost) sits in both and cancels as long as it's comparable enough not to dominate — which is exactly what latency-hiding (Step 5) buys. With $P_{GP U} = 4096$ cores and $P_{C P U} = 16$ cores:

$\frac{T _{C P U}}{T _{GP U}} = \frac{4096}{16} = 256 \times$

WHY. This is the central result of the parent note, and now you can see it isn't magic — it's Step 2's formula with the two levers each machine chose. The GPU wins by pure headcount only when the work is embarrassingly parallel.

PICTURE. Two shrinking bars: as $N$ grows, the GPU's total time drops far below the CPU's, and the gap settles at the ratio of core counts.

Step 7 — The edge case: a job that cannot be split (the GPU loses)

WHAT. Now feed both machines a sequential job: binary search in a sorted array of $1 0^{9}$ elements. Each step needs the comparison result of the previous step — a pure chain, so $P$ is stuck at 1 on both machines. The pile has only $N = lo g_{2} (1 0^{9}) \approx 30$ tasks.

WHY. With $P = 1$ , Step 2 becomes $T = N \times t_{t a s k}$ — and now $t_{t a s k}$ is the only thing that matters. The GPU deliberately made $t_{t a s k}$ bigger (no branch predictor, no speculation). So here its bet backfires:

CPU: branch predictor learns the pattern, speculation starts the next step early → $t_{t a s k} \approx 2 ns$ → $30 \times 2 = 60 ns$ .
GPU: slow simple core, no prediction → $t_{t a s k} \approx 5 ns$ → $30 \times 5 = 150 ns$ .

$\frac{T _{GP U}}{T _{C P U}} = \frac{150}{60} = 2.5 \times (CPU wins)$

PICTURE. One narrow staircase of 30 steps, one worker on each machine. The CPU worker takes short strides; the GPU worker takes long ones and finishes last. All those thousands of extra GPU cores stand idle — nothing to give them.

Step 8 — The degenerate cases: $N$ small, or one branch that splits the warp

WHAT. Two more corners the reader must never trip on.

Tiny $N$ (fewer tasks than cores, $N < P$ ). With $P_{GP U} = 4096$ but only $N = 100$ tasks, most cores sit idle; you also pay a fixed cost to launch the GPU job. The 256× from Step 6 evaporates — a CPU may even win.
Branch divergence. Recall a warp = 32 threads running the same instruction. If an if sends 16 threads down path A and 16 down path B, the hardware runs A while B waits, then B while A waits — the two halves are serialized. In the worst case a warp runs at half speed (or worse). The GPU has no branch predictor to soften this; it's the price of the Step-4 bet.

WHY. These aren't exotic — they decide whether your real code speeds up. Small problems and branchy code are precisely where the throughput machine has nothing to chew on.

PICTURE. Left: a GPU with 4096 slots but only 100 filled (lots of grey idle cores). Right: one warp of 32 threads splitting at an if into two half-warps that must take turns, doubling the time.

Recall Check the corners

When does a GPU's throughput advantage vanish? ::: When tasks are dependent (chain, $P = 1$ ), when there are too few tasks ( $N < P$ , idle cores + launch overhead), or when a warp's threads diverge at a branch (serialized execution). On a sequential 30-step chain, which wins and why? ::: The CPU — with $P$ frozen at 1, only $t_{t a s k}$ matters, and the CPU spent its transistors making $t_{t a s k}$ small (branch prediction, speculation).

The one-picture summary

Everything above is one formula, $T_{t o t a l} = (N / P) \times t_{t a s k}$ , and two opposite bets on which lever to pull. This final figure lays the whole derivation on one canvas: the shared formula in the middle, the CPU shrinking $t_{t a s k}$ on the left (big cache, few cores), the GPU growing $P$ on the right (many cores, tiny cache), and the two verdicts — GPU wins the huge-independent pile, CPU wins the tiny dependent chain.

Recall Feynman retelling — say it to a 12-year-old

Every job is a pile of small tasks. The time it takes is how many tasks each worker gets (that's $N / P$ ) times how long one task takes ( $t_{t a s k}$ ). There are only two ways to finish faster: make each task quicker, or hire more workers. A CPU hires just a few very clever workers and spends most of its silicon making each one lightning-fast — perfect when the tasks form a chain where you can't start the next until the last is done. A GPU hires thousands of plain workers and barely trains any of them — perfect when there's a mountain of tasks nobody has to wait on, like a million pixels. Because a GPU refuses to buy expensive helpers per worker, each worker is about ten times cheaper, so it fits about ten times more of them. When the pile is gigantic, only the worker-count matters and the GPU crushes the CPU — in our numbers, 4096 versus 16 cores means 256× faster. But shrink the job to a single chain of 30 steps and the GPU's cheap slow workers lose to the CPU's clever fast ones, 60 ns to 150 ns. Same formula, opposite bets, opposite winners.