Foundations — GPU acceleration and device management

This page builds every symbol, word, and picture the parent note GPU acceleration and device management leans on. Read it top to bottom: nothing later uses a thing that wasn't drawn earlier.

---

1. What is a "number in a computer" and where does it live?

Before we can talk about moving numbers between a CPU and a GPU, we need to know that a number occupies physical space — it lives in a chip called memory.

Look at the two coloured mailbox rows in the figure. A piece of data drawn in one room cannot be read by the worker in the other room — it must first be copied across the bridge in the middle. That bridge is the slow part we will meet later.

---

2. Bytes and float — how much space one number takes

Why we care: memory is finite. A tensor of a million numbers costs $10^6\times 4=4\times 10^6$ bytes $=4$ MB. When the parent says "one GPU has ~24 GB", it means 24 billion bytes of mailboxes — enough for about 6 billion floats. This is why big models don't fit and why multi-GPU exists.

---

3. CPU vs GPU — the two kinds of worker

The picture shows why the GPU wins on the right kind of job: 16 fast workers finishing tasks one-by-one, versus thousands of slower workers all finishing together. The GPU only wins when the job splits into thousands of independent pieces — which brings us to the next idea.

---

4. Why a neural network is the "right kind" of job

A neural network's heaviest work is matrix multiplication. We need to see exactly why that is "thousands of independent tiny sums".

The reason this matters: every entry of $C$ is computed independently of every other entry. Entry $C_{1,1}$ doesn't need to know $C_{1,2}$. So if $C$ has, say, a million entries, that is a million independent multiply-add jobs — exactly what thousands of GPU cores devour in parallel.

In the figure, the highlighted red row of $A$ and blue column of $B$ produce the one green entry of $C$. Notice no other cell of $C$ needs that computation — this is the "embarrassingly parallel" property the parent note names.

The backward pass and the normalization steps are also built from these same matrix multiplies, which is why the whole pipeline benefits from the GPU.

---

5. The bridge between rooms — PCIe transfer

Because RAM and VRAM are separate rows of mailboxes (Section 1), moving a number from one to the other means copying it across a physical cable called PCIe.

The figure shows the bandwidth gap as pipe widths: a narrow straw across the bridge, a fire-hose inside the GPU. This single mismatch is the whole reason for the parent's "Golden Rule": keep data on the GPU, cross the bridge as rarely as possible.

---

6. Symbols you'll meet in the formulas

---

7. How it all connects

Read it as: memory + the bridge + the CPU/GPU contrast + the matrix-multiply insight all feed into the topic. Nothing enters the parent note that wasn't drawn on this page.

---

Equipment checklist

Cover the right side of each line and try to answer before revealing.

• Why can't a CPU read a number sitting in GPU memory directly? ::: Because RAM and VRAM are physically separate rows of mailboxes; the CPU has no address into VRAM — the data must be copied across the PCIe bridge first.
• How many bytes does a standard float take, and how much memory do a million floats need? ::: 4 bytes each, so $10^6\times 4=4$ MB (using decimal MB = $10^6$ bytes).
• Why do MB and MiB give slightly different numbers? ::: MB is decimal ($10^6$ bytes) while MiB is binary ($2^{20}=1,048,576$ bytes), so the same byte count is a smaller number of MiB.
• In one sentence, why does a GPU beat a CPU on neural networks but not on all jobs? ::: A GPU has thousands of slow cores that only help when a job splits into thousands of independent pieces — matrix multiplication does exactly that, most everyday sequential code does not.
• What does the notation $A\in {\mathbb{R}}^{m\times k}$ mean? ::: $A$ is a matrix of real numbers with $m$ rows and $k$ columns.
• Why is each entry of $C=AB$ an independent job? ::: Each entry is a single dot product of one row of $A$ with one column of $B$, and needs no other entry of $C$ to compute.
• Why is the exact operation count $mn(2k-1)$, and why do we usually write $2mkn$? ::: Each entry needs $k$ multiplications plus $k-1$ additions = $2k-1$ operations; rounding $2k-1$ up to $2k$ (negligible for large $k$) gives the tidy $2mkn$.
• What is bandwidth, and how does PCIe compare to GPU internal memory? ::: Bandwidth is data moved per second; PCIe is ~16 GB/s while GPU internal memory is ~900 GB/s, over 50× faster inside.
• Why must you call synchronize before timing a GPU operation? ::: GPU work is asynchronous — the CPU fires the order and continues, so without a barrier the timer stops before the GPU actually finishes.
• What do $\theta$ and ${\nabla }_{\theta }L$ stand for? ::: $\theta$ is the bundle of all the model's adjustable parameters; ${\nabla }_{\theta }L$ is the gradient, the direction to nudge $\theta$ to reduce the loss.
• What is $S$, in what units are its frequencies, and roughly what value did the parent's numbers give? ::: $S$ is the speedup $T_{\text{CPU}}/T_{\text{GPU}}$; both frequencies are in GHz so the units cancel, and the plug-in gave about 312× (ignoring transfer).