GPU acceleration and device management: Question bank

Before we start, three plain-word anchors so no term is used before it's defined:

True or false — justify

The model is on cuda, so my input tensors are automatically on cuda too.

False. Moving a model with .to(device) moves only its parameters and buffers. Input batches are separate tensors and stay wherever they were created (usually CPU) until you move them yourself.

A GPU makes every program faster.

False. It only helps work that is massively parallel (millions of independent multiply-adds). Sequential logic, tiny tensors, or heavy branching can run slower on a GPU because you pay transfer + launch cost for little parallel gain.

x.to('cuda') changes x into a GPU tensor.

False. .to() returns a new tensor on the GPU and leaves the original on the CPU. You must rebind: x = x.to('cuda'), otherwise the CPU copy is what your later code still uses.

Calling loss.item() every iteration is free because the loss is one number.

Mostly true for the value, but it forces a sync. The 4 bytes are negligible, but reading them requires the GPU to finish and hand the scalar back, stalling the pipeline. Fine once per 100 steps; harmful inside a tight inner loop.

nn.DataParallel with 4 GPUs gives a clean 4× training speedup.

False. Gradients must be all-reduced (summed/averaged) across GPUs every step, and DataParallel also scatters/gathers through a single main process. Communication overhead makes real speedup less than 4×, often much less than DistributedDataParallel.

GPU memory (VRAM) and CPU memory (RAM) are the same pool the OS shares.

False. They are physically separate memories. GPU cores literally cannot dereference a CPU pointer, which is exactly why mixing devices raises an error — there is no shared address space to fall back on.

Bigger batch size always trains faster on a GPU.

False, past a point. Larger batches use the GPU's parallel cores better until VRAM fills up; then you get out-of-memory errors, and even before that, fewer parameter updates per epoch can slow convergence even if each step is efficient.

Tensor Cores speed up any matrix multiply on a modern GPU.

False. Tensor Cores accelerate mixed/low-precision matmuls (e.g. float16/bfloat16) with shapes that fit their tile size. Plain float32 work often bypasses them entirely, so you need the right dtype to see the gain.

Spot the error

y = torch.randn(100,100); z = x_gpu + y — what breaks and why?

y is on the CPU, x_gpu on the GPU. The add would need one kernel touching two separate memories, which is impossible, so PyTorch raises a device mismatch error. Fix: y = y.to(x_gpu.device).

A benchmark reports the GPU matmul took 0.0001 s — suspiciously fast. What's wrong?

The GPU call returned before finishing (it's asynchronous). The timer stopped while the work was still queued. You must call torch.cuda.synchronize() before reading the end time.

Inside the loop: for x in loader: x.to(device); out = model(x). Bug?

x.to(device) returns a new tensor that is thrown away; x is still on CPU, so model(x) mismatches the GPU model. Must write x = x.to(device).

model = MyModel(); model.cuda(); opt = Adam(model.parameters()) then later model = nn.DataParallel(model). Ordering problem?

The optimizer was built on the old parameter objects. Wrapping in DataParallel afterward can leave the optimizer tracking the wrong parameters. Wrap (and finalize the model) before constructing the optimizer.

Logging: losses.append(loss) for 10k steps, then plotting. What silently goes wrong?

Each loss is a GPU tensor that still holds its whole autograd graph. Appending them keeps thousands of graphs alive → VRAM leak and eventual OOM. Store loss.item() (a plain Python float) instead.

data.to('cuda'); output = model(data); print(output.mean()) inside a 100k loop. The hidden cost?

print(output.mean()) transfers and synchronizes every iteration, so the GPU idles waiting on PCIe. Accumulate on-device and move to CPU once at the end.

Why questions

Why doesn't PyTorch just auto-move CPU tensors onto the GPU when they meet a GPU tensor?

Because an implicit transfer would hide the single most expensive thing in GPU code. Making it explicit forces you to notice — and avoid — silent PCIe round-trips.

Why can keeping data on the GPU turn a "312× theoretical" speedup into only 2–3× in practice?

The real speedup is

S_{theory} / (1 + T_{transfer} / T_{compute})

. If you copy data across PCIe for every op, transfer time dominates the denominator and swamps the parallel win.

Why does .item() implicitly synchronize but a plain .to('cpu') on a big tensor also does?

Both must read actual computed values off the GPU. The GPU can only hand back finished results, so any read forces it to complete all pending work first — that's the sync.

Why is DistributedDataParallel usually faster than DataParallel?

DataParallel uses one Python process that scatters/gathers through a bottleneck (and hits Python's global interpreter lock). DistributedDataParallel runs one process per GPU and overlaps gradient all-reduce (via NCCL) with the backward pass, so communication hides behind computation.

Why do we average gradients across GPUs in data parallelism instead of summing them?

Each GPU computed the gradient over

B / N

samples. Averaging makes the combined gradient equal to what one big batch of

B

samples would give, keeping the learning rate's meaning consistent. (This is the same averaged gradient 3.2.03-Backpropagation would produce on the full batch.)

Why must batch-normalization get special care under data parallelism?

Each GPU only sees its shard, so per-GPU 3.4.02-Batch-normalization statistics are computed over

B / N

samples, not the full batch. For small per-GPU batches this biases the mean/variance, which is why synchronized BatchNorm exists.

Why do convolution-heavy models like 4.1.01-CNN-architectures benefit so dramatically from GPUs?

A convolution is thousands of independent multiply-add windows sliding over the image — exactly the "embarrassingly parallel" pattern GPU cores devour, so the parallel cores stay busy and transfer cost is amortized over lots of compute.

Edge cases

What happens on torch.device('cuda') when there is no GPU installed?

Constructing the device object may succeed, but the first allocation or .to('cuda') raises a runtime error. Guard with torch.cuda.is_available() and fall back to 'cpu'.

cuda:0 vs cuda — are they the same device?

cuda means "the current default CUDA device," which is cuda:0 unless you've changed it. On a multi-GPU box they can diverge, so being explicit avoids surprises.

You put the model on GPU but a custom buffer you registered as a plain Python attribute stays on CPU. Why, and what fixes it?

.to(device) only moves things registered via register_buffer or nn.Parameter. A raw attribute isn't tracked, so it's left behind. Register it properly (or move it manually).

Batch size is not divisible by the number of GPUs — what does DataParallel do?

It splits as evenly as it can, giving the last GPU a smaller shard. Uneven shards can slightly skew batch statistics and waste a few cores, so divisible batch sizes are preferred.

Zero-size or single-element tensor sent to the GPU — is it worth it?

No. The kernel-launch and transfer overhead dwarfs the trivial compute, so tiny operations are often faster left on the CPU. GPUs win only when the parallel work is large.

You call torch.cuda.empty_cache() and free VRAM in nvidia-smi, but Python still holds the tensor. Contradiction?

No — empty_cache() only returns PyTorch's cached, unused blocks to the driver. Memory still referenced by live tensors is never freed until those tensors go out of scope.

Recall One-line summary to lock in

The GPU only wins when compute dwarfs transfer, and PyTorch stays explicit about device precisely so you can't hide a slow copy from yourself. The single rule ::: keep data on the GPU, move it across PCIe as rarely as possible, and synchronize only when you truly need a value back on the CPU.