GPU memory bandwidth optimization: Visual walkthrough

This page builds ONE central idea from nothing: why arranging threads so they read neighbouring addresses turns a slow GPU kernel into a fast one. We derive the efficiency formula the parent note states, but as a sequence of pictures. By the end you will be able to look at any access pattern and predict how much bandwidth it wastes.

We assume you have never seen a "warp", a "transaction", or the ceiling symbol $⌈ ⌉$ . Everything is earned as we go.

Step 1 — What is memory, and what is a "thread"?

WHAT. Picture GPU memory (called DRAM) as one enormously long shelf of tiny boxes. Each box holds a few bytes. A byte is just 8 bits — think of it as one small parcel of data. A float (a decimal number the GPU works with) takes exactly 4 bytes, so it occupies 4 boxes in a row.

Each box has an address — its position number on the shelf. Box 0, box 1, box 2, ... The address is just "how far along the shelf".

A thread is one worker. The GPU runs thousands, but the hardware herds them into fixed groups of 32 called a warp. A warp is important because all 32 threads in it are marched in lockstep: when the warp reads memory, all 32 read at the same moment.

WHY start here. Coalescing is entirely about where on the shelf the 32 threads of a warp reach at that shared moment. So we must first see the shelf and the 32 hands reaching into it.

PICTURE. The shelf of addressed boxes, one warp of 32 threads above it, each thread an arrow pointing at the box it wants.

Figure — GPU memory bandwidth optimization

Step 2 — Why the hardware moves data in fixed chunks

WHAT. The memory system cannot hand you a single 4-byte box on its own. It only moves data in fixed-size blocks. Call this block size $T$ bytes — a transaction. On real GPUs $T$ is usually 128 bytes, i.e. 32 floats wide.

If you ask for even one byte inside a 128-byte block, the whole 128-byte block travels down the bus.

WHY this tool. You might ask: why not fetch exactly the bytes requested? Because the wires (the "bus") and the memory chips are built to burst a wide block per request — it is far cheaper in hardware to move one fat block than 32 skinny ones. This single fact is the entire reason coalescing exists.

PICTURE. The shelf is now painted in stripes, each stripe one 128-byte transaction. Touch any box in a stripe → the whole stripe moves.

Step 3 — The best case: consecutive addresses (fully coalesced)

WHAT. Suppose thread $i$ reads the float at address $A + 4 i$ . So:

thread 0 → address $A$
thread 1 → address $A + 4$
thread 2 → address $A + 8$
... thread 31 → address $A + 124$

The 32 requested floats fill exactly $32 \times 4 = 128$ bytes — one single transaction.

WHY it matters. Every byte the truck carries is wanted by some thread. Nothing is wasted. This is the target we optimise toward.

PICTURE. All 32 arrows land inside one stripe; the stripe lights up "100% useful".

Let us count formally. For a warp of $w$ threads each requesting $b$ bytes, spread across an address span that fits in transactions of size $T$ :

$Transactions = ⌈ \frac{w \cdot b}{T} ⌉$

$w = 32$ — number of threads in the warp (how many hands reach in).
$b = 4$ — bytes each thread wants (one float).
$w \cdot b = 128$ — total useful bytes the warp wants.
$T = 128$ — bytes one transaction moves.
$⌈ ⌉$ — the ceiling: round up to the next whole number, because you cannot order half a truck.

$⌈ \frac{32 \cdot 4}{128} ⌉ = ⌈ 1 ⌉ = 1 transaction$

$η$ (the Greek letter "eta") is our score from 0 to 1: what fraction of the delivered pallet was actually wanted.

Step 4 — The worst case: strided access

WHAT. Now suppose thread $i$ reads address $A + 128 i$ — each thread jumps a whole transaction ahead (a stride of 32 floats). Thread 0 sits in stripe 0, thread 1 in stripe 1, ..., thread 31 in stripe 31.

Every thread lands in a different stripe. Each stripe is a separate truck. So we need 32 transactions to serve 32 threads.

WHY show this. This is the disaster case that makes kernels 32× slow. Seeing exactly why — one thread per stripe — is what lets you recognise it in code (a stride equal to the transaction width).

PICTURE. 32 arrows, each into its own stripe, 31/32 of every pallet greyed out as "wasted".

Counting: here the useful bytes are still $w \cdot b = 128$ , but the address span is huge, so:

$Transactions = 32, η = \frac{128}{32 \times 128} = \frac{128}{4096} = 0.03125 = 3.125%$

Numerator $128$ — still only 128 useful bytes (we asked for 32 floats).
Denominator $4096$ — 32 pallets $\times$ 128 bytes each actually rolled down the bus.

So we paid for 4096 bytes to use 128. That is a $32 \times$ waste.

Step 5 — The in-between case: a struct stride (partial coalescing)

WHAT. Real code often lands between best and worst. Say each element is a 24-byte struct (Array-of-Structures), and every thread reads the first float of its own struct: thread $i$ → address $A + 24 i$ .

The 32 requested floats now span $32 \times 24 = 768$ bytes. That many bytes crosses several stripes:

$Transactions = ⌈ \frac{768}{128} ⌉ = ⌈ 6 ⌉ = 6$

$768$ — address span the warp touches (not the useful bytes!).
$128$ — transaction size.
Result 6 — six pallets travel.

Useful bytes are still $32 \times 4 = 128$ :

$η = \frac{128}{6 \times 128} = \frac{1}{6} \approx 16.7%$

WHY this case matters. It disproves the myth that access is either "perfect" or "fully serial". Strides between 1 and 32 floats give a graded penalty. Switching to Structure-of-Arrays (thread $i$ → $A + 4 i$ ) collapses this back to Step 3: 1 transaction, 100%.

PICTURE. Arrows landing every 24 bytes, sweeping across ~6 stripes, each stripe only partly used.

Step 6 — Degenerate cases: all threads hit ONE address, and misalignment

WHAT — the broadcast case. Suppose every thread reads the same address $A$ . Naively that is 32 requests for one box. But hardware handles this specially: it fetches the box once and broadcasts the value to all 32 threads — 1 transaction, $η$ counted as full for that datum. No penalty.

WHAT — the misalignment case. Suppose access is perfectly consecutive (Step 3, stride 4) but starts at address $A + 4$ instead of a stripe boundary. The 128 useful bytes now straddle two stripes: bytes $4..128$ in stripe 0 and byte $128..131$ in stripe 1. So:

$Transactions = 2, η = \frac{128}{2 \times 128} = 50%$

Even perfectly consecutive reads waste half the bandwidth if they don't begin on a 128-byte boundary. Aligning your arrays fixes this.

WHY cover these. These are the traps that surprise people: identical addresses are free (good news), but a one-float misalignment halves bandwidth (bad news). A reader who only saw Steps 3–5 would mispredict both.

PICTURE. Left panel: 32 arrows into one box → single broadcast. Right panel: consecutive block shoved off the boundary, spilling into a second stripe.

Recall

Same address for all 32 threads costs how many transactions? ::: One — the hardware broadcasts. Consecutive floats starting one float off a 128-byte boundary — efficiency? ::: 50%, because they straddle two transactions.

Step 7 — From efficiency to bandwidth (the payoff)

WHAT. Physical bandwidth $B$ is how many bytes the bus can move per second (an A100 is ~2 TB/s). But your kernel only makes useful progress at:

$M = B \times η$

$B$ — peak bytes/second the wires deliver (fixed by hardware).
$η$ — the efficiency we just derived (fixed by your access pattern).
$M$ — the useful bytes/second your program actually gets.

WHY this is the whole point. You cannot buy more $B$ ; but you can raise $η$ from 3% to 100% by re-laying your data — a potential 32× speedup with zero new hardware. That is why memory coalescing is the first optimisation any GPU programmer reaches for. (When compute, not memory, becomes the limit, you graduate to the Roofline-Model.)

PICTURE. A bar of full bandwidth $B$ , with $η$ slicing off the useful portion for each of our four cases.

The one-picture summary

Every access pattern is a story about span vs useful bytes. The 32 threads always want the same 128 useful bytes; what changes is how many 128-byte pallets those bytes are smeared across. Fewer pallets = higher $η$ = more of your real bandwidth $B$ .

Recall Feynman retelling — the whole walkthrough in plain words

The GPU marches its workers 32 at a time (a warp). Memory only arrives in full 128-byte truckloads. If the 32 workers reach for 32 neighbouring floats, all 32 fit on one truck — one trip, nothing wasted, 100% efficient. If instead each worker reaches a whole truckload apart, you need 32 trucks for 32 floats and throw away 97% of what you hauled. In between — say a 24-byte struct stride — you smear the wanted floats across 6 trucks and keep only 1/6. Two special cases: if all 32 workers want the same box the truck delivers it once and copies it to everyone (free); but if your neat consecutive block starts one float off a truck boundary it spills into a second truck and you lose half. Your useful speed is just the truck capacity times how full the trucks were: $M = B η$ . Re-laying data (Structure-of-Arrays, alignment) is how you keep the trucks full.

Related: CUDA-Shared-Memory · Memory-Latency-Hiding · GPU-Warp-Scheduling · Cache-Hierarchy · Parallel-Algorithm-Design · 6.2.14 GPU memory bandwidth optimization (Hinglish)