Heterogeneous computing concepts: Question bank — Hardware deep dive

Before we start, three words appear over and over — let us pin them down in plain language so no reveal below uses them cold:

Every symbol used in the reveals below is defined once here, so nothing appears cold:

Formula Symbol glossary and the two formulas the traps lean on

Symbols (with units):

$D$ ::: data size moved across the link, in gigabytes (GB).
$B$ ::: interconnect bandwidth (e.g. PCIe), in GB per second.
$W$ ::: total work of the task, in floating-point operations (FLOPs).
$F_{C P U}$ , $F_{GP U}$ ::: how fast each processor computes, in FLOPs per second (FLOPS).
$f_{s}$ ::: fraction of the program that is serial (CPU-only), $0 \leq f_{s} \leq 1$ .
$f_{p}$ ::: fraction that is parallel (GPU-able), with $f_{s} + f_{p} = 1$ .
$t_{x f er}$ ::: data-transfer overhead as a fraction of the original CPU-only time.
$t_{i}$ ::: time of pipeline stage $i$ (read, transfer, compute, write), in seconds.

Break-even work (from Heterogeneous computing concepts) — the minimum work that makes offloading pay off: $W > \frac{2 D F _{C P U} F _{GP U}}{B ( F _{GP U} - F _{C P U} )}$ The $2$ is because data crosses the link twice (CPU→GPU→CPU). Below this $W$ , transfer cost beats compute savings.

Heterogeneous Amdahl speedup (from Amdahl's Law and Scalability): $S_{o v er a l l} = \frac{1}{f _{s} + \frac{f _{p}}{S _{GP U}} + t _{x f er}}$

Linked prerequisites you may want open: Amdahl's Law and Scalability, Roofline Performance Model, Memory Hierarchy and Caching, Parallel Programming Models, SIMD and Vector Processing, DMA and I/O Controllers, Power and Energy Optimization.

True or false — justify

A GPU is simply "a faster CPU"

False — it is higher throughput but higher latency per task; a single GPU thread is slower than a CPU thread, and it only wins when thousands of threads run the same operation at once.

Adding a GPU always speeds up a program

False — if the parallel fraction is tiny or the data-transfer cost exceeds the compute saved, the heterogeneous version can be slower than the CPU alone (below the break-even work

W

defined in the glossary above).

Heterogeneous means "more than one core"

False — that is just multicore. Heterogeneous means cores of different types (different instruction sets, memory models, or strengths), e.g. CPU + GPU + NPU.

Eight identical CPU cores form a heterogeneous system

False — identical cores running the same instruction set are homogeneous; heterogeneity requires asymmetric strengths, not just multiple units.

S_{GP U} = \infty

, overall speedup is infinite

False — Amdahl caps it at

\frac{1}{f _{s} + t _{x f er}}

; the serial fraction and transfer tax are fixed costs the GPU can never touch.

Matrix–matrix multiply is a great GPU workload

True — it does

O (n^{3})

work on

O (n^{2})

data, so arithmetic intensity grows with

n

; plenty of math per byte keeps the cores fed.

Vector addition is a great GPU workload

False —

O (n)

work on

O (n)

data means ~1 FLOP/byte; it is memory-bound, so transfer and bandwidth dominate and the GPU sits idle waiting for data.

Unified memory removes the data-movement cost entirely

False — it removes the explicit copy step in code, but the bytes still physically travel across the interconnect; it hides the cost, it does not delete it.

FPGAs are slower than CPUs because they run at lower clock speeds

False — an FPGA can be far faster per watt and per operation for its target datapath, because it builds a custom pipeline instead of fetching/decoding general instructions; clock speed alone is misleading.

A pipelined model's total time is the sum of all stage times per item

False — after the pipeline fills, throughput is set by the slowest stage, so total time

\approx (\sum t_{i}) + (n - 1) max (t_{i})

, not

n \sum t_{i}

Spot the error

"CPUs have thousands of cores so they win at parallel work."

Swapped. GPUs have the thousands of simple cores; CPUs have few powerful cores tuned for latency and control flow.

"We should send tiny tasks to the GPU to save the CPU's time."

Wrong direction — tiny tasks have low work

W

, so transfer overhead dominates and you lose. Offload only when work is large relative to data moved.

T_{t o t a l} = T_{co m p u t e_GP U}

; ignore the copies."

Ignores the two transfers.

T_{t o t a l} = T_{C P U \to GP U} + T_{co m p u t e_GP U} + T_{GP U \to C P U}

, and those copies often dominate for low-intensity kernels.

"Arithmetic intensity is measured in FLOPS."

No — FLOPS is a rate (ops per second). Intensity is FLOP per byte, a ratio of work to data moved; it decides if you are compute- or memory-bound.

"Amdahl says raise the serial fraction to go faster."

Backwards — speedup improves as

f_{s}

shrinks. A large serial fraction is the ceiling that caps everything.

"Bigger caches make the GPU the throughput king."

The GPU wins through many cores + huge bandwidth, not big caches. Its caches per core are small; the CPU is the one with large caches for latency.

"A pipeline with a slow write stage is fixed by adding a faster GPU."

No — the bottleneck is the max-time stage (the write). Speeding a non-bottleneck stage barely helps; balance or overlap the slow stage.

Why questions

Why do CPUs use branch prediction and out-of-order execution while GPUs mostly don't?

CPUs chase low latency for irregular control flow, so they spend transistors hiding stalls; GPUs hide latency instead by switching among thousands of threads, so they spend those transistors on more ALUs.

Why is explicit data movement called "the bottleneck" of heterogeneous systems?

Each processor has its own memory, so data must be copied across a limited-bandwidth link; when compute per byte is low, this copy time swamps the compute time.

Why does the transfer term have a factor of 2 in

T_{h e t er o} = \frac{2 D}{B} + \frac{W}{F _{GP U}}

Data goes CPU→GPU and back GPU→CPU, so the size

D

crosses the link twice.

Why are specialized accelerators (TPU, crypto units) so energy-efficient?

They hardwire one operation so there is no instruction fetch/decode/schedule overhead — every transistor does useful work for that one task, giving 10–100× better energy per op (Power and Energy Optimization).

Why can a pipelined model beat an offload model for streaming many items?

Pipelining overlaps read, transfer, compute, and write across different items, so you pay full latency only once and then run at the slowest-stage rate; offload does each item's stages serially.

Why does high memory bandwidth matter more for GPUs than large caches?

GPUs stream through huge, regular datasets once, so feeding all cores fast beats reusing a small working set; caches help reuse, bandwidth helps streaming.

Why does the Roofline model care about arithmetic intensity?

Because intensity places a kernel on the roof: low intensity hits the bandwidth-limited slope (memory-bound), high intensity hits the flat peak-FLOPS ceiling (compute-bound) — see Roofline Performance Model.

Edge cases

What happens at arithmetic intensity ≈ 0 (no compute, pure copy)?

The kernel is entirely transfer; heterogeneous execution is strictly slower because you copy data for no computational benefit. Keep it on the CPU.

What if

F_{GP U} = F_{C P U}

(GPU no faster) in the break-even formula

W > \frac{2 D F _{C P U} F _{GP U}}{B ( F _{GP U} - F _{C P U} )}

The denominator

B (F_{GP U} - F_{C P U}) \to 0

, so required work

W \to \infty

— no amount of work justifies the transfer when the GPU gives no speed advantage.

What if the parallel fraction

f_{p} = 0

All work is serial;

S_{o v er a l l} = \frac{1}{f _{s} + t _{x f er}} \leq 1

, so the GPU only adds transfer overhead and cannot help.

What if

f_{s} = 0

but

t_{x f er} > 0

Even a perfectly parallel program is capped at

\frac{1}{t _{x f er}}

— the transfer tax alone sets the ceiling, motivating unified memory or overlapping copies with compute.

In a pipeline, what limits speedup as

n \to \infty

It approaches

\frac{\sum t _{i}}{m a x t _{i}}

— the slowest stage caps throughput no matter how many items you stream.

What does data size

D = 0

(data already resident on the GPU) imply?

The transfer term vanishes, so any parallel speedup helps; this is why keeping data on-device across kernels (avoiding round trips) is a key optimization.

Recall One-line summary to carry away

Heterogeneous wins only when parallel work is large, arithmetic intensity is high, serial fraction is small, and transfers are hidden — miss any one and the accelerator becomes dead weight.