GPU vs CPU design philosophy: Exercises — Hardware deep dive

These problems climb from recognising the vocabulary to synthesising whole design decisions. Every solution is hidden inside a collapsible callout so you can test yourself first. Do the arithmetic on paper, then reveal. Parent note: GPU vs CPU design philosophy.

Before we start, one reminder about the two words that everything hangs on:

Figure 1 (below) is your compass for the whole page: the left half is the CPU's single fast lane (small latency), the right half is the GPU's stack of slower lanes (huge throughput). Every exercise is really asking "am I on the amber lane or the cyan lanes?" — glance back at it whenever you feel lost.

Figure 1. Latency vs throughput. Amber box = one fast CPU worker finishing a single task quickly. Cyan boxes = many slower GPU workers finishing a whole flood of tasks per second. Note that no single cyan arrow is as fast as the amber one — the win comes from having many of them.

We will reuse two formulae from the parent note. They are written out here so no symbol appears unexplained.

Level 1 — Recognition

Before Exercise 1.1 uses the word, one piece of vocabulary must exist first:

Exercise 1.1

For each phrase, say whether it belongs to a latency-oriented (CPU) or throughput-oriented (GPU) design: (a) branch predictor, (b) 8704 simple cores, (c) 64 MB L3 cache, (d) warp scheduler hiding memory stalls, (e) out-of-order execution.

Recall Solution

(a) branch predictor → CPU (speculates to keep one thread moving fast = latency).
(b) thousands of simple cores → GPU (many workers = throughput).
(c) huge L3 cache → CPU (keeps one thread's data 1 tick away = latency).
(d) warp scheduler → GPU (swaps stalled work for ready work = keeps throughput high).
(e) out-of-order execution → CPU (reorders to raise a single thread's $I P C$ = latency). Map each answer onto Figure 1: (a),(c),(e) sharpen the single amber lane; (b),(d) feed the cyan stack.

Exercise 1.2

Fill in the blank: a group of 32 threads on a GPU that all execute the same instruction at the same time is called a ______, and this execution model is called ______.

Recall Solution

A warp; the model is SIMT (Single Instruction, Multiple Threads). Why 32 matters: the scheduler issues one instruction to all 32 at once, so it only needs one decoder for 32 lanes — that is exactly how a GPU saves control-logic transistors.

Level 2 — Application

Exercise 2.1

A CPU runs a thread of $N_{instructions} = 6 \times 1 0^{9}$ instructions at $I P C = 3$ and $f_{clock} = 4 GHz$ . Compute $T_{C P U}$ .

Recall Solution

Plug into $T_{C P U} = \frac{N _{instructions}}{I P C \times f _{clock}}$ . Denominator: $3 \times 4 \times 1 0^{9} = 1.2 \times 1 0^{10}$ instructions/second. $T_{C P U} = \frac{6 \times 1 0 ^{9}}{1.2 \times 1 0 ^{10}} = 0.5 s .$ What it means: at 12 billion finished instructions per second, six billion take half a second.

Exercise 2.2

An image has $N = 8.3 \times 1 0^{6}$ pixels, each an independent task taking $t_{task} = 105 ns$ . Compute the batch time on (a) a CPU with $P = 16$ cores, (b) a GPU with $P = 8704$ cores (assume the same $t_{task}$ for simplicity).

Recall Solution

Use $T_{total} = \frac{N}{P} \times t_{task}$ , with $t_{task} = 105 \times 1 0^{- 9}$ s.

(a) CPU: $\frac{8.3 \times 1 0 ^{6}}{16} = 518, 750$ tasks per core. $T = 518, 750 \times 105 \times 1 0^{- 9} \approx 0.05447 s \approx 54.5 ms .$

(b) GPU: $\frac{8.3 \times 1 0 ^{6}}{8704} \approx 953.6$ tasks per core. $T = 953.6 \times 105 \times 1 0^{- 9} \approx 1.001 \times 1 0^{- 4} s \approx 0.10 ms .$

Reading it: the only thing that changed is $P$ . More workers → the same pile of work is split thinner → shorter batch time. This is the whole throughput story in one division — exactly the cyan stack in Figure 1 doing the amber lane's job many times over.

Exercise 2.3

Using the parent note's simplifying assumption that a batch's speedup equals the core-count ratio, find the speedup of the GPU over the CPU for $N_{cores}^{GP U} = 4096$ , $N_{cores}^{C P U} = 16$ .

Recall Solution

$\frac{T _{C P U}}{T _{GP U}} = \frac{N _{cores}^{GP U}}{N _{cores}^{C P U}} = \frac{4096}{16} = 256 \times .$ This holds only when $N_{tasks} ≫ N_{cores}$ so both machines stay fully busy.

Exercise 2.4 — The edge case: fewer tasks than cores

A GPU has $P = 4096$ cores. You launch a batch of only $N = 512$ independent tasks, each $t_{task} = 400$ ns. (a) How many cores actually do work? (b) What is the batch time $T_{total}$ ? (c) What speedup did you get over using a single core, and why is it far below 4096×?

Recall Solution

(a) One task per core, so only 512 of the 4096 cores light up; the other 3584 sit idle. (b) $T_{total} = \frac{N}{P} \times t_{task}$ . Here $N / P = 512/4096 = 0.125$ , which rounds up to 1 wave of work (you cannot run a fraction of a pass), so $T_{total} = 1 \times 400 ns = 400 ns .$ (c) Versus one core doing all 512 tasks ( $512 \times 400 = 204, 800$ ns), the speedup is $204, 800/400 = 512 \times$ — not 4096×. Why the ceiling is $N$ , not $P$ : you can never get more parallel speedup than you have independent pieces of work. When $N_{tasks} < P$ , the effective speedup is capped at $N_{tasks}$ , and $(P - N_{tasks})$ cores are wasted silicon. This is the throughput machine's worst regime — a giant cyan stack in Figure 1 with most lanes empty. Feeding a GPU too little work is the single most common way to get a "GPU is slower!" surprise.

Level 3 — Analysis

Before the exercise, we earn the GPU formula rather than quoting it. Start from the batch model $T_{total} = \frac{N}{P} \times t_{task}$ and refine each piece for a real GPU.

Formula Building

$T_{GP U}$ from $T_{total}$ — every unit tracked Step 1 — how many workers really run? Not all $N_{cores}$ compute every cycle; a fraction $O cc u p an cy$ (a pure number between 0 and 1) is busy. So effective workers $P_{eff} = N_{cores} \times O cc u p an cy$ (still just a count).

Step 2 — measure task cost in cycles, not seconds. A GPU core does its work in ticks. Let $t_{instruction}$ = how many clock cycles one task costs (units: cycles/task). To turn cycles into seconds we divide by the clock: $f_{clock}$ has units cycles/second, so $\frac{t _{instruction} ( cycles/task )}{f _{clock} ( cycles/second )} = seconds/task = t_{task} .$ That is the bridge back to the parent note's $t_{task}$ : it is just cycles-per-task divided by the clock.

Step 3 — substitute. Put $P_{eff}$ and $t_{task}$ into the batch model:

= \frac{N_{\text{tasks}}}{N_{\text{cores}}\times Occupancy}\times \frac{t_{\text{instruction}}}{f_{\text{clock}}}.$$ Rearranging the four factors gives the parent note's shape: $$T_{GPU} = \frac{N_{\text{tasks}}}{N_{\text{cores}} \times Occupancy \times f_{\text{clock}}}\times t_{\text{instruction}}.$$ **Why this shape?** The denominator $N_{\text{cores}} \times Occupancy \times f_{\text{clock}}$ has units (workers)$\times$(cycles/second) = **task-slots per second** — how fast the whole machine retires cycles of work. Multiply the leftover $t_{\text{instruction}}$ (cycles/task) at the end and the cycle-units cancel, leaving seconds. Nothing magic: it is $T_{\text{total}}$ with the seconds split into "cycles ÷ clock".

Exercise 3.1

A GPU kernel launches on a chip with $N_{cores} = 4096$ , $f_{clock} = 1.5 GHz$ , occupancy $= 0.75$ , and $N_{tasks} = 1 0^{7}$ . Each task costs $t_{instruction} = 4$ cycles. Compute $T_{GP U}$ . Then state what happens to $T_{GP U}$ if occupancy drops to $0.375$ (half).

Recall Solution

Effective working cores: $4096 \times 0.75 = 3072$ workers. Task-slot rate (denominator): $3072 \times 1.5 \times 1 0^{9} = 4.608 \times 1 0^{12}$ cycles/second. $T_{GP U} = \frac{1 0 ^{7}}{4.608 \times 1 0 ^{12}} \times 4 cycles = 2.170 \times 1 0^{- 6} \times 4 \approx 8.68 \times 1 0^{- 6} s \approx 8.68 μ s .$ Units check: $\frac{tasks}{cycles/s} \times \frac{cycles}{task} = seconds$ . ✓ Halving occupancy halves the denominator, so $T_{GP U}$ doubles to $\approx 17.36 μ s$ . Why occupancy is the hidden villain: the GPU hides memory latency by having many warps ready to swap in. Low occupancy = too few ready warps = cores sit idle waiting on DRAM. The arithmetic units exist but nothing feeds them.

Exercise 3.2

Explain, using $T_{C P U} = N_{instructions} / (I P C \times f_{clock})$ , why a CPU designer who cannot raise the clock $f_{clock}$ any further would spend transistors on branch prediction and out-of-order execution.

Recall Solution

$T_{C P U}$ has exactly three knobs. $N_{instructions}$ is fixed by the program. $f_{clock}$ hit a power/heat wall around a few GHz. The only remaining knob is $I P C$ — instructions finished per tick. Branch prediction stops the pipeline from stalling on unknown branches, and out-of-order execution finds independent instructions to run while others wait. Both raise $I P C$ , which shrinks $T_{C P U}$ without touching the clock. That is precisely why those transistors are worth it.

Level 4 — Synthesis

Exercise 4.1 — Amdahl's ceiling

A program is 95% parallelisable ( $p = 0.95$ ) and 5% strictly sequential. Using Amdahl's law $S (P) = \frac{1}{( 1 - p ) + \frac{p}{P}},$ where $P$ is the number of processors and $S$ the speedup, compute the speedup with $P = 4096$ GPU cores. Then compute the theoretical maximum speedup as $P \to \infty$ .

Recall Solution

Sequential fraction $1 - p = 0.05$ . With $P = 4096$ : $S (4096) = \frac{1}{0.05 + \frac{0.95}{4096}} = \frac{1}{0.05 + 0.0002319} = \frac{1}{0.0502319} \approx 19.91 \times .$ As $P \to \infty$ , the term $p / P \to 0$ : $S_{m a x} = \frac{1}{1 - p} = \frac{1}{0.05} = 20 \times .$ The punchline: 4096 cores buy you $19.9 \times$ , and infinite cores buy only $20 \times$ . The tiny serial 5% caps everything. This is why a GPU's raw core count does not translate into raw speedup unless the workload is nearly embarrassingly parallel.

Exercise 4.2 — Crossover point

Suppose a task has $N$ independent sub-tasks. CPU: $P_{C P U} = 16$ cores at $t_{task} = 100$ ns each. GPU: $P_{GP U} = 4096$ cores at $t_{task} = 400$ ns each (slower cores), plus a fixed launch overhead of $10 μ s$ . Find the smallest $N$ for which the GPU beats the CPU.

Recall Solution

CPU time: $T_{C P U} = \frac{N}{16} \times 100 ns = 6.25 N ns$ . GPU time: $T_{GP U} = 10, 000 ns + \frac{N}{4096} \times 400 ns = 10, 000 + 0.09766 N ns$ . Set $T_{C P U} = T_{GP U}$ and solve for $N$ : $6.25 N = 10, 000 + 0.09766 N \Rightarrow 6.15234 N = 10, 000 \Rightarrow N \approx 1625.4.$ So the GPU wins for $N \geq 1626$ sub-tasks. This is the crossover marked in Figure 2. Why the crossover exists: the GPU pays a fixed $10 μ s$ launch tax and uses slower cores. Only once the pile of work is big enough does its 4096-way parallelism pay back that tax. Small jobs belong on the CPU — this is the quantitative version of "don't launch a kernel to add two numbers."

Figure 2 plots both times against $N$ . Read it like a race: the amber CPU line starts at the origin (no launch tax) and climbs steeply; the cyan GPU line starts high (the $10 μ s$ tax) but climbs almost flat. Where they cross is the answer to 4.2 — left of it the CPU wins, right of it the GPU.

Figure 2. Crossover of CPU (amber) and GPU (cyan) batch time versus problem size $N$ . The white dot at $N \approx 1625$ is exactly the break-even point solved in Exercise 4.2. Notice the GPU's flat slope — this is throughput; the CPU's steep slope is what limited latency looks like at scale.

Level 5 — Mastery

Exercise 5.1 — Design your own budget

You have a transistor budget $B = 6 \times 1 0^{10}$ . A simple GPU-style core costs $C_{simple} = 5 \times 1 0^{6}$ transistors; a complex CPU-style core costs $C_{complex} = 5 \times 1 0^{7}$ transistors. (a) How many cores of each type can you build if you spend the entire budget on cores? (b) A latency-bound task uses only 1 core but needs 4× the per-core performance a simple core gives; a throughput-bound task has $1 0^{7}$ independent sub-tasks. For each task, which chip finishes first? Justify with numbers.

Recall Solution

(a) All-simple: $B / C_{simple} = 6 \times 1 0^{10} /5 \times 1 0^{6} = 12, 000$ cores. All-complex: $B / C_{complex} = 6 \times 1 0^{10} /5 \times 1 0^{7} = 1, 200$ cores. The ratio is $10 \times$ , exactly as the parent note predicts ( $C_{complex} = 10 C_{simple}$ ).

(b) Latency task uses 1 core. The complex core delivers the 4× single-core performance; the simple core cannot (it is a plain in-order lane). So the 1200-complex-core chip wins here — 11,999 of the 12,000 simple cores would sit idle, useless for a single dependent chain.

Throughput task has $1 0^{7}$ independent sub-tasks. Even at slower per-core speed, the 12,000-core chip splits the work $10 \times$ more finely: $1 0^{7} /12000 \approx 833$ tasks/core vs $1 0^{7} /1200 \approx 8333$ tasks/core. Assuming the simple core is even up to $9 \times$ slower per task it still wins, because $9 \times 833 = 7497 < 8333$ . The many-simple-core chip wins.

The mastery point: the same silicon budget produces two opposite machines, and the right choice is dictated entirely by whether your work is one dependent chain or a million independent pieces. That is the GPU-vs-CPU philosophy, reduced to a budgeting problem — the two halves of Figure 1.

Exercise 5.2 — Explain the choice in one sentence each

Match the mechanism to the goal, then state why the alternative would be wrong on that chip: (a) GPU uses hardware threading instead of big caches. (b) CPU uses speculative execution instead of more cores.

Recall Solution

(a) With 1000s of cores, a private cache per core would blow the transistor budget, so the GPU instead keeps many resident warps and swaps a stalled warp for a ready one — hiding DRAM latency with parallelism rather than with memory. A big cache would be the wrong buy: it optimises one thread's latency, which a throughput machine does not care about. See GPU memory hierarchy. (b) A CPU's job is one fast dependent thread, so extra cores it cannot use are wasted silicon; speculation instead raises that single thread's $I P C$ . More cores would be the wrong buy for code that cannot be parallelised — recall the von Neumann sequential fetch-decode-execute chain, and how SIMD vs SIMT handles width differently.

Recall Self-test recap

A GPU beats a CPU only when ::: the work is highly parallel, big enough to pay launch overhead, and has a small serial fraction. The only CPU knob left after the clock wall is ::: IPC (instructions per cycle), raised by branch prediction and out-of-order execution. Occupancy enters $T_{GP U}$ as ::: a multiplier on effective core count in the denominator (latency-hiding capacity). When N_tasks is less than P, the speedup is capped at ::: N_tasks (fewer tasks than cores means idle cores). Amdahl's ceiling for a program with serial fraction $1 - p$ is ::: $1/ (1 - p)$ , independent of processor count.