GPU acceleration and device management: Visual walkthrough

This page rebuilds the single most important number in GPU acceleration — how much faster is a GPU, really? — from the ground up. We will not assume you know what a "core", a "clock", or a "matrix multiply" costs. We build every symbol from a picture first, then assemble the speedup formula, then break it on purpose to see where it fails.

Step 1 — What does "one multiply-add" cost?

WHAT. Before we count a whole neural network, we count the smallest unit of work: taking two numbers $a$ and $b$ , multiplying them, and adding the result to a running total $s$ . In symbols:

$s \leftarrow s + a \times b$

$a, b$ — two plain numbers (weights and inputs in a network).
$\times$ — one multiplication.
$+$ — one addition.
$s$ — the accumulator, the box we keep adding into.
$\leftarrow$ — "becomes"; the right side overwrites the left.

This bundle (one multiply and one add) is called a MAC — a Multiply-ACcumulate. We count MACs because neural networks are almost nothing but MACs.

WHY count MACs and not "operations" vaguely? Because a MAC is the exact thing hardware is built to do fast, and because it is independent — computing one MAC never needs the answer of another. That independence is the whole reason GPUs help. Hold onto it.

PICTURE. One tiny gear turning: two numbers go in, one number comes out.

Figure — GPU acceleration and device management

Step 2 — How many MACs is a matrix multiply?

WHAT. The heaviest operation in a layer is a matrix multiply $C = A B$ . Picture $A$ as a grid with $m$ rows and $k$ columns, and $B$ as a grid with $k$ rows and $n$ columns. The result $C$ has $m$ rows and $n$ columns.

$A \in R^{m \times k}, B \in R^{k \times n}, C \in R^{m \times n}$

$R$ — "the real numbers" (ordinary decimals).
$m \times k$ — a grid with $m$ rows and $k$ columns.
Each entry $C_{ij}$ is one row of $A$ slid across one column of $B$ , multiplying pairs and summing.

To fill one entry $C_{ij}$ you slide along $k$ pairs — that is $k$ MACs. There are $m \times n$ entries. So the total work, which we name $W$ right here so we can reuse it, is:

$W = m \cdot k \cdot n MACs$

$W$ — total multiply-accumulates in the whole matrix multiply.
$m \cdot n$ — number of output cells; $k$ — MACs per cell.

People often write $2 mk n$ operations (counting the multiply and the add separately). We will keep it as work $W \propto mk n$ and not worry about the factor of 2 — it cancels in the ratio we care about.

WHY does this matter for GPUs? Look at the picture: every entry $C_{ij}$ is computed from data that does not overlap the computation of any other entry. All $m \cdot n$ entries could be filled at the same time if we had enough workers. This is the "embarrassingly parallel" property in one image.

PICTURE. Row of $A$ (magenta) meets column of $B$ (violet) to produce one cell of $C$ ; every cell independent.

This is the same $C = A B$ shape you meet in 3.2.03-Backpropagation (both forward and backward passes are matrix multiplies) and inside every layer of 4.1.01-CNN-architectures.

Step 3 — Turning "work" into "time" on one worker

WHAT. A processor does a fixed number of things per second. Its clock frequency $f$ (in hertz, Hz = ticks per second) is how many ticks it beats each second. Suppose for now one worker finishes one MAC per tick. Then the time for $W$ MACs on a single worker is:

$t_{one worker} = \frac{W}{f}$

$W$ — total MACs (from Step 2).
$f$ — ticks per second (Hz).
Dividing work by rate gives time — same as "distance ÷ speed = time".

WHY frequency and not something fancier? Because time is literally work divided by how fast you chew through it. Frequency is the "how fast" of a single lane. We need this before we can talk about many lanes.

PICTURE. A conveyor belt: MACs go by, one per clock tick; a faster belt (higher $f$ ) clears them sooner.

Step 4 — Many workers at once: adding cores

WHAT. A processor has several independent workers called cores. If it has $c$ cores and the work splits cleanly (Step 2 proved it does), then $c$ cores share the $W$ MACs:

$T = \frac{W}{f \cdot c} = \frac{m \cdot k \cdot n}{f \cdot c}$

$c$ — number of cores (parallel lanes).
$f \cdot c$ — total MACs per second for the whole chip.
Bigger $c$ → smaller $T$ . Bigger $f$ → smaller $T$ .

A crucial caveat: a CPU core and a GPU core are not the same animal. A CPU has a few heavyweight cores, each with wide SIMD vector units and large caches, built to race through sequential logic. A GPU packs its arithmetic into many SMs, each holding dozens of lightweight ALUs built only to do simple math in bulk. So "cores" below is a throughput bookkeeping device, not a claim that one CPU core equals one GPU ALU. What matters for our formula is the product $f \cdot c$ — the whole chip's MAC rate.

Now the CPU-vs-GPU difference is just two different $(f, c)$ pairs (throughput here is an order-of-magnitude figure, in MACs per second, using the fold-in of SIMD/FMA width from Step 3):

	frequency $f$ (Hz)	lanes $c$ (count)	throughput $f \cdot c$ (MACs/s)
CPU	~3 GHz $= 3 \times 1 0^{9}$	few (~16 cores)	~ $4.8 \times 1 0^{10}$
GPU	~1.5 GHz $= 1.5 \times 1 0^{9}$	many (~10 000 ALUs)	~ $1.5 \times 1 0^{13}$

(The bare "48" and "15000" we quote later are just these throughputs with the shared $1 0^{9}$ scale factored out — they carry units of $1 0^{9}$ MACs/s.)

WHY does the GPU accept a lower clock? Because a slower belt with 10 000 lanes still clears far more per second than a fast belt with 16 lanes. GPUs trade single-lane speed for a flood of lanes — exactly the trade that pays off when work is independent.

PICTURE. Left: 16 fast belts. Right: 10 000 slow belts. The right side is a wall of throughput.

Step 5 — The theoretical speedup

WHAT. Speedup $S$ is just "how long the slow way takes" divided by "how long the fast way takes". Using Step 4 for both, and noticing the work $mk n$ is identical on both machines (same matrix!):

$S = \frac{T _{CPU}}{T _{GPU}} = \frac{\frac{mk n}{f _{CPU} c _{CPU}}}{\frac{mk n}{f _{GPU} c _{GPU}}} = \frac{f _{GPU} \cdot c _{GPU}}{f _{CPU} \cdot c _{CPU}}$

The $mk n$ on top and bottom cancels — the answer does not depend on matrix size (in this idealized world).
What survives is the ratio of throughputs (units cancel too — MACs/s over MACs/s is dimensionless).

Plug in the table (the shared $1 0^{9}$ scale cancels, leaving the clean ratio):

$S \approx \frac{1.5 \times 10000}{3 \times 16} = \frac{15000}{48} \approx 312 \times$

WHY does the matrix size cancel? Because both machines pay the same $mk n$ ; only the rate differs. That cancellation is why we can quote "~300×" as one universal number instead of one per problem.

PICTURE. Two bars — a short GPU bar and a tall CPU bar — with the numerator/denominator labelled right on the arrow between them.

Step 6 — The tax we ignored: data transfer

WHAT. The GPU cannot compute on data it cannot see. Data starts in CPU memory and must be copied across a PCIe bus into GPU memory before compute begins. That copy time is not a mystery constant — it is bytes to move divided by bytes per second the bus delivers. For our matrices the bytes are:

$bytes = m k n_{elems} \cdot sizeof(float)$

For example the input matrices hold $(mk + k n)$ elements; at 4 bytes each (32-bit float) that is $4 (mk + k n)$ bytes. So:

$T_{transfer} = \frac{bytes to move}{β _{PCIe}}$

$β_{PCIe}$ — the PCIe bus bandwidth in bytes/s.
More elements → more bytes → longer transfer. This is how the tax is grounded in problem size.

The honest GPU time is:

$T_{GPU}^{real} = T_{compute} \frac{mk n}{f _{GPU} c _{GPU}} + T_{transfer}$

Divide top and bottom of the real speedup by $T_{compute}$ to expose the tax:

$S_{real} = \frac{T _{CPU}}{T _{compute} + T _{transfer}} = \frac{S}{1 + \frac{T _{transfer}}{T _{compute}}}$

$S$ — the ideal 312× from Step 5.
$\frac{T _{transfer}}{T _{compute}}$ — the tax ratio: transfer time measured in units of compute time.
When the ratio is $0$ , $S_{real} = S$ (transfer is free). As the ratio grows, the fraction shrinks.

WHY write it as a divided-down version of $S$ ? Because it isolates the one knob you actually control: keep data on the GPU so $T_{transfer} \to 0$ and you recover the full 312×. Transfer every iteration and the tax ratio blows up and eats the win.

PICTURE. The 312× bar, then a narrow orange "transfer pipe" choking it down to a short real bar.

Step 7 — Degenerate cases: when the GPU loses

WHAT. The formula predicts its own failure modes. Let us push it to extremes.

Case A — tiny matrix ( $mk n$ small). Then $T_{compute} \to 0$ , so the tax ratio $T_{transfer} / T_{compute} \to \infty$ , so $S_{real} \to 0$ . The GPU is slower. A $2 \times 2$ matrix is not worth shipping across PCIe.

Case B — repeated transfer (Mistake 2 in the parent). If you copy one scalar back to the CPU inside a tight loop of $L$ iterations, transfer happens $L$ times: $T_{transfer}$ scales with $L$ while useful compute barely grows. The tax ratio explodes.

Case C — data already on GPU (the ideal). $T_{transfer} = 0$ , tax ratio $= 0$ , $S_{real} = S$ . This is the "GPU no transfer: 0.1s, 23× faster" line in the parent's Example 2.

Case D — no GPU at all. $c_{GPU}$ and $f_{GPU}$ do not exist; you fall back to $T_{CPU}$ and $S = 1$ . This is exactly why the code writes 'cuda' if torch.cuda.is_available() else 'cpu'.

WHY enumerate these? Because the same formula covers "GPU is a rocket" and "GPU is a paperweight". Nothing surprising can happen that the tax ratio (plus the fixed overhead above) did not already warn you about.

PICTURE. A curve of $S_{real}$ versus matrix size: near-zero for tiny problems, rising, flattening toward 312× for large ones — with break-even marked where $S_{real} = 1$ .

The one-picture summary

Everything above is one chain: MAC → matrix work $mk n$ → time $mk n / (f c)$ → throughput-ratio speedup $S$ → tax-adjusted $S_{real}$ (with fixed overhead lurking for small problems).

The same principle scales up: keeping data resident and minimising cross-device chatter is exactly what makes multi-GPU distributed training worthwhile, and why layers like 3.4.02-Batch-normalization are cheap to keep on-device once the batch is already there.

Recall Feynman retelling — say it back in plain words

A neural network is a mountain of tiny "multiply two numbers and add" jobs, and — this is the magic — none of those jobs needs another one's answer. A CPU is 16 brilliant workers (each secretly a small vector engine doing several at once) doing them in a fast line. A GPU is ten thousand ordinary workers doing them all at once. Ten thousand slow workers beat sixteen fast ones by roughly 312 to 1 on this kind of work. But the GPU lives in a separate room, and the only door into that room (PCIe) is narrow — and how narrow depends on your bus version and whether you pinned the memory. If you keep shoving data through the door every second, the waiting-at-the-door time swallows your 312×. Worse, every trip to the room has a fixed sign-in cost (launch + sync) that dwarfs tiny jobs. So the whole art is: get the data into the room once, do everything inside, and only carry small answers back out. Tiny jobs aren't worth the trip at all — sometimes it's faster to just do them in the CPU's line. One formula, $S_{real} = S / (1 + tax)$ , tells you every one of these stories.

Recall Quick self-checks

Why does matrix size cancel in the ideal speedup? ::: Both CPU and GPU do the same $mk n$ MACs; only the rate $f \cdot c$ differs, so $mk n$ divides out. A $2 \times 2$ matrix on GPU is slower — which terms dominate? ::: $T_{transfer}$ and the fixed $T_{overhead}$ (launch/sync); compute is nearly zero so both dwarf it. Ideal speedup with $f_{GPU} c_{GPU} = 15000$ , $f_{CPU} c_{CPU} = 48$ ? ::: $15000/48 \approx 312 \times$ . How do you drive the transfer tax to zero? ::: Keep tensors resident on the GPU; never round-trip per iteration. Name two things that change $β_{PCIe}$ by 2×–4×. ::: PCIe version (3.0 vs 4.0 vs 5.0) and pinned vs pageable memory.