GPU acceleration and device management: Exercises

These exercises build up from recognizing what lives where, to computing transfer costs, to designing a device-aware training strategy. Each has a full worked solution hidden in a collapsible callout so you can test yourself first, then reveal.

Prerequisite tools you should have from the parent note the parent topic:

Device: an address for where numbers live — cpu (in your computer's normal RAM) or cuda:0 (in the GPU's private memory, called VRAM).
Transfer: physically copying numbers across the PCIe cable between CPU RAM and GPU VRAM. Slow lane.
Speedup $S$ : how many times faster the GPU finishes a job than the CPU.

Everything else we define exactly when it appears.

A quick picture of the whole board

Before the problems, fix this map in your head — every exercise is really a question about one arrow in this diagram.

Figure — GPU acceleration and device management

The thin arrow (PCIe, ~16 GB/s) is the bottleneck. The fat arrows (compute inside CPU, compute inside VRAM ~900 GB/s) are fast. Most mistakes are secretly "I sent data over the thin arrow too many times."

Level 1 — Recognition

Exercise 1.1

Which device does each tensor end up on?

a = torch.randn(4, 4)                       # (i)
b = torch.randn(4, 4, device='cuda')        # (ii)
c = a.to('cuda')                            # (iii)
d = b.cpu()                                 # (iv)

Recall Solution 1.1

(i) cpu. No device given, so PyTorch defaults to CPU RAM.
(ii) cuda:0. Allocated directly in VRAM.
(iii) cuda:0. .to('cuda') copies a across PCIe; a itself stays on CPU, but c is on GPU.
(iv) cpu. .cpu() copies b back to CPU RAM.

Key idea: .to(...) and .cpu() return a new tensor on the target device; they do not move the original in place (tensors are not mutated by these).

Exercise 1.2

True or false: after x_gpu = x.to('cuda'), the original x is now on the GPU.

Recall Solution 1.2

False. x is unchanged and still on CPU. Only x_gpu is on the GPU. To reuse the name you must write x = x.to('cuda').

Level 2 — Application

Exercise 2.1

A tensor of shape $(1000, 1000)$ holds 32-bit floats. Each float is $4$ bytes. How many megabytes ( $1 MB = 1 0^{6}$ bytes) does it occupy?

Recall Solution 2.1

Number of elements: $1000 \times 1000 = 1 0^{6}$ . Bytes: $1 0^{6} \times 4 = 4 \times 1 0^{6}$ bytes. $\frac{4 \times 1 0 ^{6}}{1 0 ^{6}} = 4 MB .$ Answer: 4 MB. (Not 400 MB — the parent note's "400 MB" comment was a typo; a $1000 \times 1000$ float32 tensor is exactly 4 MB.)

Exercise 2.2

That 4 MB tensor is copied CPU→GPU over a PCIe link running at $16 GB/s$ (take $1 GB = 1 0^{9}$ bytes). Roughly how long is the transfer?

Recall Solution 2.2

What tool and why: time $=$ data $\div$ rate. We divide because rate has units bytes-per-second, so bytes divided by (bytes/second) leaves seconds. $t = \frac{4 \times 1 0 ^{6} bytes}{16 \times 1 0 ^{9} bytes/s} = 2.5 \times 1 0^{- 4} s = 0.25 ms .$ Answer: 0.25 ms. Tiny for one tensor — the pain only appears when you do this thousands of times.

Exercise 2.3

Using the parent's speedup formula $S \approx \frac{f _{GPU} c _{GPU}}{f _{CPU} c _{CPU}},$ compute $S$ for $f_{GPU} = 1.5 GHz$ , $c_{GPU} = 8000$ , $f_{CPU} = 3 GHz$ , $c_{CPU} = 16$ .

Recall Solution 2.3

$S \approx \frac{1.5 \times 8000}{3 \times 16} = \frac{12000}{48} = 250.$ Answer: $250 \times$ . (The GHz units cancel top and bottom, so we can plug in the numbers 1.5 and 3 directly.)

Level 3 — Analysis

Exercise 3.1

The parent's "Golden Rule" says $S_{real} = \frac{S _{theoretical}}{1 + \frac{T _{transfer}}{T _{compute}}} .$ Take $S_{theoretical} = 250$ . In a loop the GPU compute per step is $T_{compute} = 0.1 s$ but you transfer data each step at $T_{transfer} = 0.7 s$ . What is $S_{real}$ ?

Recall Solution 3.1

Ratio: $\frac{T _{transfer}}{T _{compute}} = \frac{0.7}{0.1} = 7$ . $S_{real} = \frac{250}{1 + 7} = \frac{250}{8} = 31.25.$ Answer: $\approx 31.25 \times$ . Transfer overhead ate almost 90% of the potential speedup.

Exercise 3.2

Now keep the data resident on the GPU so $T_{transfer} = 0$ . What is $S_{real}$ , and by what factor did fixing the transfer improve things versus Exercise 3.1?

Recall Solution 3.2

With $T_{transfer} = 0$ : $\frac{0}{0.1} = 0$ , so $S_{real} = \frac{250}{1 + 0} = 250.$ Improvement factor: $\frac{250}{31.25} = 8$ . Answer: $S_{real} = 250 \times$ ; that is $8 \times$ better than the transfer-every-step version. This is exactly why the training-loop recipe moves the model to GPU once and keeps activations there.

Level 4 — Synthesis

Exercise 4.1 (Multi-GPU all-reduce)

You train with data parallelism across $N = 4$ GPUs. The model has $P = 25, 000, 000$ parameters (25 million), each a 4-byte float. The all-reduce step must exchange every gradient so all GPUs agree on the average. Using the parent's estimate "bandwidth needed $\approx 2 P \cdot sizeof(float)$ per iteration," how many bytes move per iteration?

Recall Solution 4.1

Why the factor of 2: a ring all-reduce sends each parameter's gradient around once to sum, and once to share the result — roughly two passes over the data. $2 P \cdot sizeof(float) = 2 \times 25, 000, 000 \times 4 = 2 \times 1 0^{8} bytes = 200 MB .$ Answer: 200 MB per iteration. (This links to 3.3.07-Distributed-training-strategies, where NCCL makes this exchange fast.)

Exercise 4.2 (Batch splitting)

A global batch of $B = 256$ samples is split evenly across the $N = 4$ GPUs (data parallelism). (a) How many samples per GPU? (b) After the 4 GPUs each compute a per-GPU gradient, how are they combined into the update?

Recall Solution 4.2

(a) Per-GPU batch $= B / N = 256/4 = 64$ samples. (b) Average the gradients: $\nabla_{θ} L = \frac{1}{N} \sum_{i = 0}^{N - 1} \nabla_{θ_{i}} L_{i},$ then every GPU applies the same update $θ \leftarrow θ - α \nabla_{θ} L$ , so the replicas stay identical. This mirrors 3.2.03-Backpropagation run 4 times in parallel, then merged. Answer: 64 per GPU; gradients averaged, then one synchronized step.

Exercise 4.3 (Debug the device error)

This code crashes. Explain why and give a one-line fix.

x = torch.randn(100, 100).to('cuda')
y = torch.randn(100, 100)
z = x + y

Recall Solution 4.3

Why it crashes: x lives in GPU VRAM, y lives in CPU RAM. A GPU kernel can only read GPU memory, so x + y raises "Expected all tensors to be on the same device." Fix: bring y to the same device before the op:

y = y.to(x.device)   # now y is on cuda:0
z = x + y

Using x.device (instead of hard-coding 'cuda') makes the code work whether or not a GPU is present — the same portable pattern used when moving batches inside a training loop (see also 3.4.02-Batch-normalization layers, whose running stats must ride to the GPU with the model).

Level 5 — Mastery

Exercise 5.1 (End-to-end cost model)

You benchmark a step of your loop. Per step:

CPU→GPU transfer of one batch: $T_{transfer} = 5 ms$
GPU forward+backward compute: $T_{compute} = 20 ms$
One print(loss.item()) that forces a GPU→CPU sync of a single scalar: $T_{log} = 1 ms$ , but you call it every step.

You have $1000$ steps. (a) Total wall-clock time with logging every step. (b) Total time if you log only every $100$ th step instead. (c) What percentage of time did per-step logging waste?

Recall Solution 5.1

Per-step base (transfer + compute) $= 5 + 20 = 25 ms$ .

(a) Every step also pays $1 ms$ log: $(25 + 1) \times 1000 = 26, 000 ms = 26.0 s$ .

(b) Log every 100th step $\Rightarrow 1000/100 = 10$ logs: $25 \times 1000 + 1 \times 10 = 25, 000 + 10 = 25, 010 ms = 25.01 s .$

(c) Time wasted by per-step logging $= 26, 000 - 25, 010 = 990 ms$ . As a fraction of the full-logging run: $\frac{990}{26 , 000} \approx 0.0381 = 3.81%$ .

Answers: (a) 26.0 s, (b) 25.01 s, (c) ≈ 3.81%. Small here, but if T_log forced a full cudaDeviceSynchronize that idled the GPU for longer, this fraction balloons — exactly the parent's "synchronous CPU access in a tight loop" warning.

Exercise 5.2 (Design decision)

You must train a CNN (see 4.1.01-CNN-architectures) whose model fits in one GPU, but the batch you want ( $B = 512$ ) does not fit in one GPU's memory. You have 4 GPUs. Describe the device-management plan and the one arithmetic constraint that must hold.

Recall Solution 5.2

Plan (data parallelism):

Move the model to each GPU once (nn.DataParallel or DistributedDataParallel replicates it).
Scatter the global batch: each GPU gets $B / N = 512/4 = 128$ samples — small enough to fit.
Forward + backward run in parallel on each GPU (activations stay resident on their GPU — no per-op transfer).
All-reduce averages the 4 gradients (~200 MB-scale exchange from Ex 4.1 if 25M params).
Synchronized update keeps the 4 replicas identical.

The constraint: $B$ must be divisible by $N$ , i.e. $B / N$ is a whole number. Here $512/4 = 128$ ✓. If it were not divisible, the last GPU gets a smaller batch and the per-GPU gradients are weighted unequally, biasing the average. Answer: split 512 into 4×128; require $N ∣ B$ .

Self-test clozes

Per-GPU batch size for global batch

B

across

N

GPUs

B / N

How gradients are combined in data-parallel training

averaged:

\frac{1}{N} \sum_{i} \nabla_{θ_{i}} L_{i}

Bytes in a

1000 \times 1000

float32 tensor

4 MB

(

4 \times 1 0^{6}

bytes)

The Golden-Rule denominator that throttles speedup

1 + T_{transfer} / T_{compute}

Why x + y crashes when they're on different devices

a GPU kernel can only read GPU memory