GPU acceleration and device management: Worked examples

This page is a companion to the parent note on GPU acceleration and device management. There we built the ideas: why GPUs win (thousands of small cores doing multiply-adds in parallel), the movement tax (copying data between CPU and GPU is slow), and the device-aware training loop. Here we do the arithmetic — every case class the topic can throw at you, worked end-to-end.

Before any symbol appears, here is the vocabulary, in plain words:

The three time-quantities from the parent note, restated so this page stands alone. For multiplying an $m \times k$ matrix by a $k \times n$ matrix, the number of multiply-add operations is $2 mk n$ (each output entry is a sum of $k$ products, and there are $m \times n$ outputs, and each product-then-add is two flops). Dividing that work by "how fast can we do flops" gives time:

$T_{CPU} = \frac{2 mk n}{f _{CPU} \cdot c _{CPU}}, T_{GPU} = \frac{2 mk n}{f _{GPU} \cdot c _{GPU}} + T_{transfer}$

Here $f$ = clock frequency (cycles per second, in Hz), $c$ = number of cores. The GPU line has the extra $T_{transfer}$ term because the numbers have to get there first.

Two more formulas we will lean on, restated inline so this page needs no other tab open:

The scenario matrix

Every GPU-timing / device-management problem you meet is one of these cells. Each worked example below is tagged with the cell it fills, and all eight examples appear on this page.

#	Case class	The question it answers	Example
A	Compute-bound, no transfer	Best-case speedup, data already on GPU	Ex 1
B	Transfer-dominated	Speedup collapses when we copy every op	Ex 2
C	Break-even point	How much work makes the GPU trip worth it?	Ex 3
D	Degenerate: tiny tensor	GPU slower than CPU — the zero-benefit case	Ex 4
E	Limiting case: transfer → 0	What is the theoretical ceiling?	Ex 5
F	Multi-GPU split	Batch of $B$ across $N$ GPUs, communication cost	Ex 6
G	Real-world word problem	Training-loop transfer budget over an epoch	Ex 7
H	Exam twist: async timing bug	Why a "0.0001 s GPU" measurement is a lie	Ex 8

The two axes of this matrix: how big is the work (rows A→D→E move along "amount of compute") and where does the time actually go (compute vs transfer vs communication). Together the eight examples touch every corner.

The figure below is the map you should hold in your head for the whole page. Read it like this: the blue curve is GPU compute time, which grows like $n^{3}$ (steeper); the orange curve is transfer time, which grows only like $n^{2}$ (flatter). Because the steeper curve starts below the flatter one for small $n$ and overtakes it later, there is exactly one crossing — the red dashed line at $n \approx 12$ (Cell C, Ex 3). Left of the crossing (gray region) transfer dominates and the GPU can lose (Cell D, Ex 4); far right (green region) compute dominates and you approach the full 312× ceiling (Cells A & E, Ex 1 and Ex 5). Every numeric example on this page is a single point somewhere on this picture.

Figure — GPU acceleration and device management

Example 1 — Cell A: compute-bound, best case

Worked example Multiply two

$4096 \times 4096$ matrices, data already on GPU $f_{CPU} = 3 \times 1 0^{9}$ Hz, $c_{CPU} = 16$ ; $f_{GPU} = 1.5 \times 1 0^{9}$ Hz, $c_{GPU} = 10000$ . Both matrices already live in VRAM, so $T_{transfer} = 0$ . Find $T_{CPU}$ , $T_{GPU}$ , and the speedup $S$ .

Forecast: guess the speedup before reading — is it 10×, 100×, or 300×?

Count the work. $m = k = n = 4096$ , so operations $= 2 mk n = 2 \cdot 409 6^{3} = 1.374 \times 1 0^{11}$ flops. Why this step? Both machines do the same arithmetic; the only difference is how fast. So we compute the work once.
CPU time. $T_{CPU} = \frac{1.374 \times 1 0 ^{11}}{3 \times 1 0 ^{9} \cdot 16} = \frac{1.374 \times 1 0 ^{11}}{4.8 \times 1 0 ^{10}} \approx 2.863 s$ . Why this step? Divide total work by CPU throughput ( $f \cdot c$ = flops per second).
GPU time. $T_{GPU} = \frac{1.374 \times 1 0 ^{11}}{1.5 \times 1 0 ^{9} \cdot 10000} = \frac{1.374 \times 1 0 ^{11}}{1.5 \times 1 0 ^{13}} \approx 9.16 \times 1 0^{- 3} s$ (transfer is zero). Why this step? Same work, but GPU throughput is $1.5 \times 1 0^{13}$ flops/s — 312× higher.
Speedup. $S = \frac{2.863}{0.00916} \approx 312 \times$ . Why this step? Speedup is just the ratio of the two times; the $2 mk n$ cancels, leaving $\frac{f _{GPU} c _{GPU}}{f _{CPU} c _{CPU}}$ .

Verify: the shortcut $\frac{1.5 \times 1 0 ^{9} \cdot 10000}{3 \times 1 0 ^{9} \cdot 16} = \frac{1.5 \times 1 0 ^{13}}{4.8 \times 1 0 ^{10}} = 312.5$ . Matches. Units: seconds/seconds = dimensionless ✓. This is the ceiling — no problem beats it.

Example 2 — Cell B: transfer-dominated, the collapse

Worked example Same matrices, but you copy them to the GPU every single call

Each $4096 \times 4096$ float32 matrix is $409 6^{2} \times 4 bytes = 6.71 \times 1 0^{7}$ bytes $\approx 67.1$ MB. We must send two of them: $1.34 \times 1 0^{8}$ bytes. PCIe bandwidth $= 16 GB/s = 1.6 \times 1 0^{10}$ bytes/s. Compute time on GPU is $9.16 \times 1 0^{- 3}$ s (from Ex 1). Find real GPU time and real speedup vs CPU.

Forecast: will we still get ~300×, or has the pipe killed it?

Transfer time. $T_{transfer} = \frac{1.34 \times 1 0 ^{8}}{1.6 \times 1 0 ^{10}} \approx 8.39 \times 1 0^{- 3} s$ . Why this step? Bytes ÷ (bytes per second) = seconds. This is the cost of feeding the pipe.
Real GPU time. $T_{GPU} = T_{compute} + T_{transfer} = 9.16 \times 1 0^{- 3} + 8.39 \times 1 0^{- 3} \approx 1.755 \times 1 0^{- 2} s$ . Why this step? The GPU can't start until the numbers arrive; the two times add.
Real speedup. $S_{real} = \frac{2.863}{0.01755} \approx 163 \times$ . Why this step? Ratio again — but now the denominator carries the transfer tax.
Sanity via the Golden Rule. Using the inline formula above, $S_{real} = \frac{S _{th}}{1 + T _{transfer} / T _{compute}} = \frac{312.5}{1 + 0.00839/0.00916} = \frac{312.5}{1.916} \approx 163 \times$ . Why this step? Confirms our step-3 arithmetic through the Golden Rule, restated at the top of this page.

Verify: both routes give ≈163×. We halved the speedup by transferring once. Transferring every op in a loop is what truly collapses it — see Cell D. ✓

Example 3 — Cell C: the break-even point

Worked example For what matrix size

$n$ (square, $m = k = n$ ) does the GPU finally beat the CPU including one transfer? Break-even means $T_{CPU} = T_{GPU}$ , i.e. $\frac{2 n ^{3}}{f _{CPU} c _{CPU}} = \frac{2 n ^{3}}{f _{GPU} c _{GPU}} + T_{transfer}$ , where $T_{transfer} = \frac{2 \cdot 4 n ^{2}}{1.6 \times 1 0 ^{10}}$ (two float32 matrices of $n^{2}$ entries).

Forecast: dozens? hundreds? thousands of rows?

Write both sides in $n$ . LHS $= \frac{2 n ^{3}}{4.8 \times 1 0 ^{10}}$ . RHS $= \frac{2 n ^{3}}{1.5 \times 1 0 ^{13}} + \frac{8 n ^{2}}{1.6 \times 1 0 ^{10}}$ . Why this step? Everything now depends only on $n$ , so we can solve.
Subtract the GPU-compute term from both sides: $2 n^{3} (\frac{1}{4.8 \times 1 0 ^{10}} - \frac{1}{1.5 \times 1 0 ^{13}}) = \frac{8 n ^{2}}{1.6 \times 1 0 ^{10}}$ . Why this step? Group the two " $\propto n^{3}$ " terms so only transfer ( $\propto n^{2}$ ) stays on the right.
Divide by $n^{2}$ (valid since $n > 0$ ): $2 n \cdot (2.077 \times 1 0^{- 11}) = 5 \times 1 0^{- 10}$ . The bracket $= \frac{1}{4.8 \times 1 0 ^{10}} - \frac{1}{1.5 \times 1 0 ^{13}} = 2.083 \times 1 0^{- 11} - 6.67 \times 1 0^{- 14} = 2.077 \times 1 0^{- 11}$ . Why this step? Cubic vs quadratic — dividing by $n^{2}$ turns a scary cubic into a linear equation.
Solve for $n$ . $n = \frac{5 \times 1 0 ^{- 10}}{2 \cdot 2.077 \times 1 0 ^{- 11}} = \frac{5 \times 1 0 ^{- 10}}{4.153 \times 1 0 ^{- 11}} \approx 12.0$ . Why this step? Below $n \approx 12$ the transfer dwarfs the tiny compute; above it, compute dominates and the GPU wins.

Verify: at $n = 12$ , $T_{CPU} = \frac{2 \cdot 1728}{4.8 \times 1 0 ^{10}} = 7.2 \times 1 0^{- 8}$ s and $T_{transfer} = \frac{8 \cdot 144}{1.6 \times 1 0 ^{10}} = 7.2 \times 1 0^{- 8}$ s — the transfer alone already equals the entire CPU time, so break-even at $n \approx 12$ is exactly where the two curves cross. ✓ (Real-world break-even is higher because of kernel-launch overhead we ignored — but the algebra is sound.)

Example 4 — Cell D: degenerate tiny tensor, GPU loses

Worked example Add two

$10 \times 10$ tensors, copying both to GPU each call Compute: $2 mk n$ ? No — element-wise add is just $n^{2} = 100$ operations. CPU throughput $4.8 \times 1 0^{10}$ flops/s, GPU $1.5 \times 1 0^{13}$ . Transfer: two $10 \times 10$ float32 tensors $= 2 \cdot 100 \cdot 4 = 800$ bytes over $1.6 \times 1 0^{10}$ B/s. Which device is faster?

Forecast: surely the mighty GPU wins... or does it?

CPU time. $T_{CPU} = \frac{100}{4.8 \times 1 0 ^{10}} \approx 2.08 \times 1 0^{- 9} s$ . Why this step? Baseline: 100 adds on the fast sequential cores — essentially instant.
GPU compute. $T_{GPU,compute} = \frac{100}{1.5 \times 1 0 ^{13}} \approx 6.67 \times 1 0^{- 12} s$ . Why this step? The GPU's arithmetic is faster — but look how tiny the work is.
GPU transfer. $T_{transfer} = \frac{800}{1.6 \times 1 0 ^{10}} = 5 \times 1 0^{- 8} s$ . Why this step? Even 800 bytes takes 50 ns on the pipe — 24× longer than the whole CPU job.
Compare. $T_{GPU} = 6.67 \times 1 0^{- 12} + 5 \times 1 0^{- 8} \approx 5 \times 1 0^{- 8} s$ versus $T_{CPU} = 2.08 \times 1 0^{- 9} s$ . CPU wins by ≈24×. Why this step? Ratio $\frac{5 \times 1 0 ^{- 8}}{2.08 \times 1 0 ^{- 9}} \approx 24$ : the transfer alone buries the GPU.

Verify: GPU is $24.0 \times$ slower. This is the "don't ship tiny tensors to the GPU" mistake made quantitative — never send a scalar or tiny tensor to the GPU just because it's "the GPU device." ✓

Example 5 — Cell E: limiting case, transfer → 0

Worked example As batch size

$\to \infty$ , what speedup do we approach? Real speedup is $S_{real} = \frac{S _{th}}{1 + r}$ where $r = T_{transfer} / T_{compute}$ . For square matrix multiply, $T_{compute} \propto n^{3}$ and $T_{transfer} \propto n^{2}$ , so $r \propto 1/ n$ . What is $lim_{n \to \infty} S_{real}$ ?

Forecast: it converges to a number — which one?

Express $r$ in $n$ . $r = \frac{8 n ^{2} / ( 1.6 \times 1 0 ^{10} )}{2 n ^{3} / ( 1.5 \times 1 0 ^{13} )} = \frac{5 \times 1 0 ^{- 10} n ^{2}}{1.333 \times 1 0 ^{- 13} n ^{3}} = \frac{3750}{n}$ . Why this step? This is why big models love GPUs: doubling $n$ halves the relative transfer cost.
Take the limit. As $n \to \infty$ , $r \to 0$ , so $S_{real} \to \frac{S _{th}}{1 + 0} = S_{th} = 312.5$ . Why this step? The $\propto n^{3}$ compute grows faster than the $\propto n^{2}$ transfer, so transfer becomes negligible.
Check at a finite $n = 4096$ . $r = 3750/4096 = 0.9155$ , giving $S = 312.5/1.9155 \approx 163$ — exactly Ex 2's answer. Why this step? Ties the limit back to a case we already trust.

Verify: limit $= 312.5$ (the theoretical ceiling from Ex 1), and the finite check reproduces $163 \times$ . ✓ Large matrices asymptote to the ceiling; that is the whole point of big batches.

Example 6 — Cell F: multi-GPU batch split

Worked example Batch of

$B = 512$ split across $N = 4$ GPUs; model has $P = 25 \times 1 0^{6}$ float32 parameters Using the inline all-reduce formula above, each GPU gets $B / N$ samples and after backward the gradients are all-reduced. Find (a) per-GPU batch, (b) bytes exchanged in one all-reduce, treating it as each GPU sending its full gradient once.

Forecast: does splitting the batch reduce or increase communication?

Per-GPU batch. $B / N = 512/4 = 128$ samples per GPU. Why this step? Scatter step — this is what lets a batch too big for one GPU fit.
Gradient size. One gradient vector = one number per parameter = $P \times 4 = 25 \times 1 0^{6} \times 4 = 1 \times 1 0^{8}$ bytes $= 100$ MB. Why this step? All-reduce moves gradients, one per weight; sizeof(float32)=4 bytes.
All-reduce volume. A ring all-reduce moves about $2 P \cdot sizeof(float)$ per GPU $= 2 \times 1 0^{8}$ bytes $= 200$ MB per GPU per iteration. Why this step? The inline cost formula: each GPU both sends and receives ≈ the full gradient, hence the factor 2.
Interpretation. Splitting the batch does not change $P$ , so communication per iteration is fixed at ≈200 MB/GPU regardless of $B$ — but each iteration now processes $4 \times$ the samples. Communication per sample drops. Why this step? This is why data-parallel scaling helps: fixed comm cost amortized over more samples.

Verify: $B / N = 128$ ✓; gradient $= 100$ MB, all-reduce $= 200$ MB ✓. If you double $N$ to 8, per-GPU batch halves to 64 but per-GPU comm stays 200 MB — the known scaling wall that motivates 3.3.07-Distributed-training-strategies. ✓

Example 7 — Cell G: real-world epoch transfer budget

Worked example A

CNN trains on 50 000 images of shape $3 \times 224 \times 224$ (float32), batch size 128, for 1 epoch on one GPU. How much data crosses PCIe, and how long does that transfer take at 16 GB/s?

Forecast: minutes? seconds? sub-second?

Bytes per image. $3 \times 224 \times 224 \times 4 = 602, 112$ bytes $\approx 0.574$ MB. Why this step? Every pixel-channel is a float32; this is what a data loader hands the GPU.
Bytes per epoch. Every image is sent once per epoch: $50, 000 \times 602, 112 = 3.011 \times 1 0^{10}$ bytes $\approx 28.0$ GB. Why this step? Batching doesn't reduce total bytes — it just groups them; 50 000 images cross the pipe regardless.
Transfer time. $\frac{3.011 \times 1 0 ^{10}}{1.6 \times 1 0 ^{10}} \approx 1.88 s per epoch$ of pure PCIe time. Why this step? Bytes ÷ bandwidth. This is the floor on your data-feeding time.
Design takeaway. ≈1.88 s/epoch of pure transfer is usually hidden behind compute if you overlap it with the previous batch's forward pass (pinned memory + non_blocking=True copies). If you do not overlap, this 1.88 s is dead time added to every single epoch — over 100 epochs that is ≈188 s (≈3.1 min) wasted just feeding the pipe. Why this step? Turns the raw number into an engineering decision, and connects it to the parent note's "keep data on GPU / overlap transfers" rule.

Verify: 28.0 GB/epoch, 1.88 s at 16 GB/s ✓; 100 epochs → 188 s ✓. Note that 3.4.02-Batch-normalization and every other layer then run entirely on-GPU, so this 1.88 s is only the entry toll, not the compute cost. ✓

Example 8 — Cell H: exam twist, the async timing trap

Worked example A student times a GPU matmul and reports "0.0002 s" for the same

$4096 \times 4096$ job that Ex 1 says takes 9.16 ms. What went wrong, and what is the true time?

Forecast: did the GPU really get 45× faster than physics allows?

Recall async launch. In PyTorch, z = x @ x on CUDA returns immediately — it only queues the kernel; the CPU races ahead to time.time(). Why this step? The 0.2 ms measured is how long it took the CPU to launch the work, not to finish it.
The fix. Insert torch.cuda.synchronize() before reading the clock, forcing the CPU to wait for the GPU. Why this step? Only after synchronize is the answer actually computed, so the elapsed time is real.
True time. From Ex 1, compute alone is $9.16 \times 1 0^{- 3}$ s. So the honest measurement is ≈9.16 ms, not 0.2 ms. Why this step? The physics ceiling ( $1.5 \times 1 0^{13}$ flops/s) forbids doing $1.374 \times 1 0^{11}$ flops in 0.2 ms — that would imply $6.9 \times 1 0^{14}$ flops/s, 46× over the hardware limit.
The lie detector. $\frac{1.374 \times 1 0 ^{11}}{2 \times 1 0 ^{- 4}} = 6.87 \times 1 0^{14}$ flops/s vs the true peak $1.5 \times 1 0^{13}$ — impossible, so the 0.2 ms is a measurement artifact. Why this step? Always sanity-check a timing against the flop ceiling before believing it.

Verify: implied throughput $6.87 \times 1 0^{14}$ exceeds hardware $1.5 \times 1 0^{13}$ by ≈45.8× — physically impossible, confirming the missing synchronize(). True time ≈9.16 ms. ✓

Recall Self-test: does each cell click?

Which cell is "GPU slower than CPU"? ::: Cell D — tiny tensor, transfer buries the trivial compute. As matrix size $n \to \infty$ , real speedup approaches what? ::: The theoretical ceiling $S_{th} \approx 312 \times$ (transfer $\propto n^{2}$ loses to compute $\propto n^{3}$ ). Why can a "0.0002 s" GPU timing be a lie? ::: GPU launches are asynchronous; without torch.cuda.synchronize() you time the launch, not the finish. Splitting a batch across more GPUs changes per-GPU communication how? ::: Not at all — all-reduce moves gradients ( $\propto$ parameter count $P$ ), which is independent of batch size.

See also: 3.2.03-Backpropagation (the .backward() whose gradients Ex 6 all-reduces) and 3.3.07-Distributed-training-strategies (when Cell F's communication wall forces DistributedDataParallel).