GPU vs CPU design philosophy: Worked examples — Hardware deep dive

This page is the drill ground for GPU vs CPU design philosophy. The parent note taught you why CPUs chase latency (finish one task fast) and GPUs chase throughput (finish the most tasks per second). Here we hammer that idea against every kind of workload you could meet — the ones where the GPU wins by a mile, the ones where it loses, and the tricky edge cases in between.

Before any numbers, let us lay out the full battlefield.

The scenario matrix

Think of each workload as landing in exactly one cell below. If you can classify a new problem into one of these cells, you already know who wins before doing arithmetic.

Cell	Case class	What makes it this class	Winner (expected)
A	Massively parallel, independent	Millions of tasks, no task depends on another	GPU (big)
B	Strictly sequential	Each step needs the previous step's result	CPU
C	Zero / degenerate size	Almost no work ( $N \approx 1$ )	CPU (launch cost matters)
D	Limiting case $N \to \infty$	Task count grows without bound	GPU (ratio → core ratio)
E	Branch divergence	Parallel but threads take different paths	GPU, but degraded
F	Memory-latency bound	Lots of slow DRAM reads, little compute	GPU (hides latency)
G	Partly parallel (Amdahl)	A serial fraction refuses to parallelize	Capped — see 9.1.02-Amdahls-law
H	Real-world word problem	A described situation, you must classify it	Decide from the story
I	Exam twist	Looks like one cell, is secretly another	Read carefully!

Below, every symbol is the one from the parent's two performance models. Let us re-anchor them so this page stands alone.

Recall The two formulas we will keep using

CPU (latency model): $T_{C P U} = \frac{N _{in s t}}{I P C \times f}$ — time for one thread, $N_{in s t}$ = number of instructions, $I P C$ = instructions finished per clock tick, $f$ = ticks per second. GPU (throughput model): $T_{GP U} = \frac{N _{t a s k s}}{N _{cor es} \times O cc \times f} \times C P I$ — total time for all tasks spread across $N_{cor es}$ cores, $O cc$ = fraction of cores busy, $C P I$ = cycles per instruction (a GPU core needs more clock ticks per instruction than a CPU core — this is why we multiply by it after dividing by $f$ ).

Definition The "ns-per-task-per-core" shorthand we use below

Handling $f$ (cycles/s) and $C P I$ (cycles/instruction) separately is fiddly. Notice that for a fixed core doing a fixed job, the product $t_{cor e} = \frac{C P I \times ( instructions per task )}{f}$ is just one measured time in seconds — "how long one core takes to finish one task." We call this ns-per-task-per-core and write it $t_{cor e}$ . It folds the parent's $f$ and $C P I$ (and the per-task instruction count) into a single number you can measure with a stopwatch. Then the parent's GPU model collapses to the friendly form $T_{GP U} = \frac{N _{t a s k s}}{N _{cor es} \times O cc} \times t_{cor e},$ and the CPU model to $T_{C P U} = \frac{N _{t a s k s}}{N _{cor es}^{C P U}} \times t_{cor e}^{C P U}$ . Every worked number below is a $t_{cor e}$ (e.g. " $1 ns$ ", " $2 ns$ "), so we never juggle $f$ and $C P I$ by hand — but you can always unfold them with the boxed formula.

Now the examples. Guess the winner before you read the steps.

Worked example Example 1 — Cell A: Independent, massively parallel (vector add)

Problem: Add two arrays of $N = 1 0^{7}$ numbers, element by element: $c_{i} = a_{i} + b_{i}$ . CPU has $16$ cores; GPU has $4096$ cores. Ignore memory for now — pure compute.

Forecast: Who wins, and by roughly what factor? Write your guess.

Classify. Each $c_{i}$ depends on nothing but $a_{i}, b_{i}$ → tasks are independent. → Cell A. Why this step? Independence is the permission slip for using all cores at once.
State the equal- $t_{cor e}$ assumption. For this one, tiny instruction (a single add), assume the CPU core and GPU core finish it in the same $t_{cor e}$ . That is, we set $t_{cor e}^{C P U} = t_{cor e}^{GP U}$ , which folds equal $C P I / f$ for both. Why this step? The parent's models differ only in $N_{cor es}$ , $O cc$ and $t_{cor e}$ . If we don't pin $t_{cor e}$ equal, the " $256 \times$ " would silently absorb a per-core speed difference — a hidden assumption. Stating it makes the clean ratio honest. (Examples 2 and 4 break this equality on purpose to show what changes.)
CPU time. With $O cc = 1$ : $T_{C P U} = \frac{1 0 ^{7}}{16} \times t_{cor e} = 625, 000 t_{cor e}$ ( $16$ cores, $625, 000$ tasks each). Why this step? A latency machine finishes its slice one at a time; $\frac{N}{N _{cor es}}$ is that slice.
GPU time. With $O cc = 1$ : $T_{GP U} = \frac{1 0 ^{7}}{4096} \times t_{cor e} \approx 2442 t_{cor e}$ . Why this step? A throughput machine spreads the same work over far more workers.
Speedup ratio. Because $t_{cor e}$ and $N_{t a s k s}$ are equal top and bottom, they cancel: $\frac{T _{C P U}}{T _{GP U}} = \frac{N _{cor es}^{GP U}}{N _{cor es}^{C P U}} = \frac{4096}{16} = 256 \times .$ Why this step? Only the count of hands survives — the whole point of Cell A.

Verify: $625000/2442 \approx 256$ . Same number two ways → consistent. Units: (tasks/core)÷(tasks/core) is dimensionless, a pure speedup. ✓

Read the figure: the left panel shows the CPU's $16$ magenta tiles, each stamped "625k" — a few cores swallowing huge slices. The right panel is a dense violet grid of $4096$ tiny cores, each with a sliver of work. The orange caption " $256 \times = 4096/16$ " is exactly step 5: count the tiles on each side, take the ratio.

Worked example Example 2 — Cell B: Strictly sequential (binary search)

Problem: Binary search a sorted array of $1 0^{9}$ elements. CPU: $2 ns$ per iteration (branch predictor + cache warm). GPU: $5 ns$ per iteration (simple core, no prediction). Here $t_{cor e}^{C P U} \neq = t_{cor e}^{GP U}$ — we deliberately break Example 1's equality.

Forecast: One task, thirty dependent steps. Who wins?

Count iterations. Binary search halves the range each step, so it needs $lo g_{2} (1 0^{9})$ steps. Why this step? Every comparison uses the previous comparison's result — this is the definition of sequential, cell B.
Evaluate the log. $lo g_{2} (1 0^{9}) = 9 lo g_{2} 10 \approx 9 \times 3.3219 \approx 29.9 \to 30$ iterations. Why this step? We need a concrete step count to multiply by per-step time.
CPU total. $30 \times 2 ns = 60 ns$ . Why this step? With only one thread of work, $N_{cor es}$ is irrelevant ( $N_{t a s k s} = 1$ chain); total time is just steps × per-step $t_{cor e}^{C P U}$ .
GPU total. $30 \times 5 ns = 150 ns$ . Why this step? The GPU's 4096 cores are useless — there is only one thread of work, so its slower $t_{cor e}^{GP U}$ is now a pure penalty with nothing to hide behind.
Ratio. $\frac{T _{GP U}}{T _{C P U}} = \frac{150}{60} = 2.5 \times$ — CPU wins by 2.5×.

Verify: $2^{30} = 1, 073, 741, 824 > 1 0^{9}$ , so $30$ halvings suffice. ✓ The GPU/CPU ratio equals the per-step ratio $5/2 = 2.5$ because both do the same number of steps. ✓

Worked example Example 3 — Cell C: Zero / degenerate input (add 4 numbers)

Problem: Add two arrays of only $N = 4$ elements. GPU kernel launch overhead is $\approx 5 μ s$ ; each core does the add in $\approx 2 ns$ . CPU does 4 adds at $1 ns$ each with ~ $0$ overhead.

Forecast: Tiny problem. Does the mighty GPU still win?

Classify. $N = 4$ is far below the number of cores → the machine can't be filled. → Cell C (degenerate). Why this step? Throughput only pays off when there are many tasks; here there are almost none.
CPU time. $4 \times 1 ns = 4 ns$ , no launch cost. Why this step? Four independent adds, no per-launch tax — the CPU just does them.
GPU time. Launch $5000 ns$ + compute $2 ns \approx 5002 ns$ . Why this step? Fixed startup cost is paid regardless of $N$ ; with tiny $N$ it dominates entirely.
Ratio. $\frac{T _{GP U}}{T _{C P U}} = \frac{5002}{4} \approx 1250 \times$ slower on GPU.

Verify: As $N \to 0$ , $T_{GP U} \to$ launch overhead (a constant), while $T_{C P U} \to 0$ . So the GPU ratio blows up — matching our $1250 \times$ . ✓ Units: ns/ns, dimensionless. ✓

Worked example Example 4 — Cell D: Limiting behaviour

$N \to \infty$ Problem: Take Example 1's vector add but include the GPU's $5 μ s$ launch overhead. CPU: $t_{cor e}^{C P U} = 1 ns$ , $16$ cores. GPU: $t_{cor e}^{GP U} = 2 ns$ (slower core — equality broken again), $4096$ cores. What is the speedup at $N = 1 0^{3}$ , $N = 1 0^{6}$ , and in the limit $N \to \infty$ ?

Forecast: Does the speedup grow forever, or hit a ceiling?

Write both times as functions of $N$ . $T_{C P U} (N) = \frac{N}{16} \times 1 ns, T_{GP U} (N) = 5000 ns + \frac{N}{4096} \times 2 ns .$ Why this step? To see a limit we must keep $N$ as a variable, not a fixed number.
Small $N = 1 0^{3}$ . $T_{C P U} = \frac{1000}{16} = 62.5 ns$ ; $T_{GP U} = 5000 + \frac{1000}{4096} \cdot 2 \approx 5000.5 ns$ → CPU wins ( $\sim 80 \times$ ). Why this step? At small $N$ the constant $5000 ns$ launch cost swamps the tiny compute term — this is the Cell-C regime bleeding into the curve.
Medium $N = 1 0^{6}$ . $T_{C P U} = \frac{1 0 ^{6}}{16} = 62, 500 ns$ ; $T_{GP U} = 5000 + \frac{1 0 ^{6}}{4096} \cdot 2 \approx 5000 + 488 = 5488 ns$ → GPU wins $\approx 11.4 \times$ . Why this step? Now compute has grown enough to dwarf the fixed launch cost, so the GPU's parallelism starts paying off — but overhead still drags the ratio below its ceiling.
Limit $N \to \infty$ . Overhead becomes negligible next to the $N$ -term: $lim_{N \to \infty} \frac{T _{C P U}}{T _{GP U}} = \frac{N /16}{N /4096 \cdot 2} = \frac{4096}{16 \times 2} = 128 \times .$ Why this step? Dividing top and bottom by $N$ kills the constant $5000$ — the classic limit trick. Note the ceiling is $128$ , not $256$ , because the GPU core is $2 \times$ slower here.

Verify: $\frac{4096}{32} = 128$ . ✓ At $N = 1 0^{6}$ we got $11.4 \times$ , well below the ceiling $128 \times$ — consistent, because overhead still matters at a million. The crossover (GPU catches up) is where $T_{C P U} = T_{GP U}$ : solving gives $N \approx 8.1 \times 1 0^{4}$ , between our small and medium points. ✓

Read the figure: the magenta curve is the actual speedup as $N$ grows along a log axis — it starts below the dotted "break-even" line (CPU winning, the small- $N$ region of step 2), crosses it at the orange dot near $N \approx 8 \times 1 0^{4}$ , then climbs and flattens toward the violet dashed " $128 \times$ " ceiling from step 4. The flattening is the limit made visible.

Worked example Example 5 — Cell E: Branch divergence (if/else in a warp)

Problem: A GPU warp = $32$ threads that must execute the same instruction each cycle. A kernel has if (x>0) doA(); else doB(); where $d o A$ and $d o B$ each take $10 ns$ . In one warp, $20$ threads take the if, $12$ take the else. How long does the warp take vs. an ideal no-divergence warp?

Forecast: All 32 threads run at once — so is it still $10 ns$ ?

Classify. Tasks are parallel but take different paths → Cell E. Why this step? SIMT (see 6.3.01-SIMD-vs-SIMT) forces one shared instruction, so divergence isn't free.
How SIMT handles divergence. The hardware runs both branches, masking off the threads not on the current path — it serializes the two branches. Why this step? With no per-thread instruction pointer, it cannot run doA and doB in the same cycle.
Divergent time. $d o A$ (10 ns, else-threads idle) then $d o B$ (10 ns, if-threads idle) $= 20 ns$ . Why this step? Because step 2 says the paths are serialized, we add their times rather than overlapping them — this is the concrete cost of masking.
Ideal (no divergence) time. If all $32$ took one branch: $10 ns$ . Why this step? We need a baseline to measure the penalty against — the "no divergence" case is the best the warp could ever do.
Slowdown. $\frac{20}{10} = 2 \times$ — divergence doubled the cost. Why this step? Dividing divergent by ideal isolates the penalty caused solely by divergence, stripping out the base work time.

Verify: With $k$ distinct branch paths of equal length $t$ , time $= k \cdot t$ . Here $k = 2$ , $t = 10 ns$ → $20 ns$ . ✓ Worst case $k = 32$ (every thread different) would give $32 \times$ — a useful sanity bound. ✓

Worked example Example 6 — Cell F: Memory-latency bound (image blur, latency hiding)

Problem: Blur $8.3 \times 1 0^{6}$ pixels; each needs $9$ DRAM reads at $200 ns$ each plus $5 ns$ compute. On the GPU, while one warp waits for memory, the scheduler runs others. CPU (16 cores, L3 hits $10 ns$ /read + $5 ns$ compute + $10 ns$ write) takes $105 ns$ /pixel.

Forecast: Memory is 20× slower on GPU per read — can it still win?

Classify. Pixels independent, many, dominated by memory waits → Cell F.
CPU total. Each core: $\frac{8.3 \times 1 0 ^{6}}{16} \approx 518, 750$ pixels $\times 105 ns \approx 54.5 ms$ . Why this step? CPU relies on its big L3 cache to make reads cheap ( $10 ns$ ), then grinds sequentially.
Per-pixel raw work on GPU. One pixel needs $9 \times 200 ns + 5 ns = 1805 ns$ of stall + compute if a single core did it alone. That would be catastrophic — so we do NOT let a core sit idle waiting. Why this step? We must first size the latency before we can talk about hiding it — you can't hide a number you haven't measured.
Derive the effective per-pixel cost from latency hiding + core count. The chip has $8704$ cores; while any one waits $1805 ns$ , the other cores retire pixels. If latency is fully hidden, the chip completes roughly one pixel every $\frac{1805 ns}{8704 cores} \approx 0.207 ns$ ; a modest occupancy/scheduling overhead rounds this up to $\approx 0.24 ns$ per pixel of wall-clock throughput. Why this step? This is the crux: latency ÷ number of overlapping workers = effective throughput cost. The $200 ns$ never shrank — it was divided across thousands of cores working during the wait.
GPU total. $8.3 \times 1 0^{6} \times 0.24 ns \approx 2.0 ms$ . Why this step? Multiply the hidden per-pixel throughput cost by the pixel count to get wall-clock time — the same "tasks × cost-per-task" shape as every other example, just with a hidden cost.
Speedup. $\frac{54.5 ms}{2 ms} \approx 27 \times$ . Why this step? Dividing the two totals answers the forecast: despite $20 \times$ slower reads, hiding wins by $27 \times$ .

Verify: $518750 \times 105 ns = 54.47 ms$ ✓. $\frac{1805}{8704} = 0.207 ns$ , and $0.24 ns$ (with overhead) $\times 8.3 \times 1 0^{6} \approx 1.99 ms$ ✓. $54.47/2 \approx 27$ ✓. Key lesson: the GPU never reduced the $200 ns$ latency — it hid it behind other work (6.2.03-memory-hierarchy-GPU).

Worked example Example 7 — Cell G: Partly parallel (Amdahl's ceiling) — and where

$O cc$ lives Problem: A program is $95%$ parallelizable, $5%$ strictly serial. You move it to a GPU with effectively "infinite" cores. What is the maximum speedup? Then: if only $60%$ of cores stay busy ( $O cc = 0.6$ ) instead of $100%$ , how does that enter the parent's GPU model, and what does a realistic combined speedup look like?

Forecast: Infinite cores → infinite speedup? Guess a number.

Classify. A fixed serial fraction refuses to parallelize → Cell G, governed by 9.1.02-Amdahls-law. Why this step? The presence of any non-parallel work changes the ceiling entirely, so we must name that fraction first.
State Amdahl's law. With serial fraction $s$ and $P$ processors: $Speedup (P) = \frac{1}{s + \frac{1 - s}{P}} .$ Why this step? Total time = serial part (never shrinks) + parallel part (shrinks by $P$ ); the reciprocal of that time, relative to $P = 1$ , is the speedup.
Take $P \to \infty$ . The parallel term $\frac{1 - s}{P} \to 0$ , leaving $Speedup_{m a x} = \frac{1}{s} = \frac{1}{0.05} = 20 \times .$ Why this step? No matter how many cores, the un-parallelizable $5%$ is a hard floor on time — dividing by zero cores of parallel work still can't erase the serial slice.
Where $O cc$ lives — plug it in. In $T_{GP U} = \frac{N _{t a s k s}}{N _{cor es} \times O cc} \times t_{cor e}$ , occupancy multiplies $N_{cor es}$ : only $O cc \times N_{cor es}$ cores actually retire work. So the effective processor count in Amdahl's law is $P_{e f f} = O cc \times N_{cor es}$ . Why this step? $O cc$ isn't a separate mystery — it simply shrinks the parallel machine. Substituting $P \to P_{e f f}$ lets one formula carry both effects.
Realistic combined number. With $N_{cor es} = 4096$ and $O cc = 0.6$ : $P_{e f f} = 0.6 \times 4096 \approx 2458$ . Then $Speedup = \frac{1}{0.05 + \frac{0.95}{2458}} \approx 19.85 \times .$ Why this step? This shows the honest outcome: even with 60% occupancy on 4096 cores, Amdahl's $20 \times$ ceiling — not occupancy — is what pins the result. Occupancy matters most before the serial fraction dominates.
Isolate the pure occupancy penalty. If a workload were fully parallel ( $s = 0$ ) and dominated by the $t_{cor e}$ term, dropping $O cc$ from $1.0$ to $0.6$ alone inflates $T_{GP U}$ by $\frac{1}{0.6} \approx 1.67 \times$ . Why this step? Separating the two effects shows when occupancy bites: it hurts throughput-bound code directly ( $1.67 \times$ here), but is masked once a serial fraction sets a lower ceiling.

Verify: At $s = 0.05, P = 4096$ (ideal $O cc = 1$ ): $\frac{1}{0.05 + 0.95/4096} \approx 19.9 \times$ ✓. At $P_{e f f} = 2458$ : $\approx 19.85 \times$ ✓ — barely different, proving the ceiling dominates. Pure occupancy check: $1/0.6 \approx 1.667$ ✓. This is why cell G is capped, and why real GPUs rarely hit their paper peak. ✓

Read the figure: three curves (magenta $s = 5%$ , violet $s = 10%$ , orange $s = 25%$ ) all climb as processors $P$ increase but each flattens onto its own dashed ceiling $1/ s$ = $20 \times$ , $10 \times$ , $4 \times$ . The smaller the serial slice, the higher the ceiling — but every curve stops rising. That plateau is step 3 made visible.

Worked example Example 8 — Cell H: Real-world word problem (payroll vs particles)

Problem: You must (a) generate one PDF pay-slip for the CEO, and (b) simulate the positions of $2 \times 1 0^{6}$ independent smoke particles for a movie frame. You own one CPU and one GPU (16 vs 4096 cores). Assign each job to the right device and justify — then estimate the particle speedup.

Forecast: Which job goes where, and by how much does (b) win?

Classify (a). One document, sequential formatting logic, lots of branches (fonts, tables). $N \approx 1$ → Cell C/B → CPU. Why this step? There is nothing to parallelize; the CPU's branch predictor and cache shine on branchy, single-thread work.
Classify (b). $2 \times 1 0^{6}$ particles, each updated by the same physics, independent of the others → Cell A/F → GPU. Why this step? A huge count of identical independent tasks is the GPU's home turf — exactly the "independent AND many" both-yes case.
Estimate the particle speedup. Assuming near-equal $t_{cor e}$ (as in Cell A) and $O cc \approx 0.7$ : $P_{e f f} = 0.7 \times 4096 \approx 2867$ , so $\frac{T _{C P U}}{T _{GP U}} \approx \frac{P _{e f f}}{N _{cor es}^{C P U}} = \frac{2867}{16} \approx 179 \times .$ Why this step? We reuse the Cell-A core-ratio result but discount by occupancy (from Example 7) to get a realistic — not paper-peak — figure.

Verify: Rule check — "independent AND many?" (a) No → CPU. (b) Yes → GPU ✓. Number: $\frac{0.7 \times 4096}{16} = 179.2$ ✓, sensibly below the ideal $256 \times$ ceiling because $O cc < 1$ . ✓

Worked example Example 9 — Cell I: The exam twist (looks parallel, is serial)

Problem: "Compute the running total (prefix sum) of an array: $o u t_{i} = a_{0} + a_{1} + \dots + a_{i}$ . There are $1 0^{6}$ elements and it's just addition — surely embarrassingly parallel? Compute the naive speedup, then find the catch."

Forecast: Is prefix sum cell A... or a trap?

Spot the dependency. $o u t_{i} = o u t_{i - 1} + a_{i}$ . Each output needs the previous output. Why this step? The naive definition is a chain → looks like Cell B, not A. This is the twist.
Naive serial cost. $1 0^{6}$ dependent additions, one after another → no core-count speedup at all ( $1 \times$ ) if done this way. Why this step? A dependency chain pins the whole job onto one core, no matter how many are idle — so $N_{cor es}$ drops out of the model entirely.
The real resolution. A parallel scan algorithm (Blelloch) restructures the work into a tree of independent partial sums, taking $lo g_{2} (1 0^{6}) \approx 20$ dependent stages instead of $1 0^{6}$ . Why this step? Changing the algorithm changes the cell — it converts B into a shallow-A. See 9.2.01-parallel-programming-models and 6.2.02-CUDA-programming-model.
Parallel-scan depth. $lo g_{2} (1 0^{6}) = 6 lo g_{2} 10 \approx 6 \times 3.3219 \approx 19.9 \to 20$ stages. Why this step? We need the depth (number of dependent stages) — that, not the element count, sets how sequential the job still is, and hence the achievable speedup.

Verify: $2^{20} = 1, 048, 576 \geq 1 0^{6}$ ✓, so $20$ tree stages cover a million elements. The lesson: the cell depends on the algorithm, not just the problem statement — the exam's "surely parallel?" is the bait. ✓

Recall Rapid self-test (cover the answers)

Independence is the permission to use many cores ::: Without it you're stuck on one core (cell B). The GPU wins big at large $N$ but its speedup ceiling is ::: the core ratio (Ex. 1: $256 \times$ ; Ex. 4 with slower cores: $128 \times$ ). A tiny $N = 4$ job runs faster on the ::: CPU, because GPU launch overhead ( $\sim 5 μ s$ ) dwarfs the work (cell C). Warp divergence with 2 equal paths costs ::: $2 \times$ (paths serialize; general: paths × path-length). With $5%$ serial code the max speedup ever is ::: $20 \times$ by Amdahl ( $1/ s$ ), no matter how many cores. The GPU beats the CPU on memory-bound blur by ::: hiding the $200 ns$ latency behind other warps (latency ÷ overlapping cores = effective cost). Occupancy enters the model by shrinking cores to ::: $P_{e f f} = O cc \times N_{cor es}$ (Ex. 7); pure penalty $= 1/ O cc$ , e.g. $1/0.6 \approx 1.67 \times$ . A word problem is solved by asking ::: "independent AND many?" — payroll → CPU, 2M particles → GPU (Ex. 8). Prefix sum looks parallel but is secretly ::: a dependency chain (cell I); a parallel scan cuts it to $lo g_{2} N \approx 20$ stages.

Next: revisit the parent GPU vs CPU design philosophy and the 6.2.01 GPU vs CPU design philosophy (Hinglish) version, then dig into 6.3.01-SIMD-vs-SIMT for the divergence machinery behind Example 5.