Foundations — GPU memory bandwidth optimization

This page assumes you have seen nothing. Before we can talk about coalescing, banks, tiling, or the roofline, we must earn every letter the parent note uses. Read top to bottom — each symbol is built on the one above it.

0. What is a byte, an address, and a "consecutive" access?

Everything in the parent note is about where data lives. So we start there.

Figure s01 — the address street. Memory is drawn as one long row of numbered byte-boxes. Each box is one byte; its number is its address. The blue block covers bytes A through A+3 — that is one 4-byte float. The orange block right beside it covers A+4 through A+7 — the next float. The green arrow underneath marks that these two floats are neighbours on the street: no gap between them. This picture is the definition of "consecutive" that the whole page leans on.

1. Threads, warps, and why they move in lockstep

For more on how warps are chosen and scheduled, see GPU-Warp-Scheduling.

2. The memory transaction — the truck's fixed cargo size

Figure s02 — one full truck vs many empty ones. Two warps are shown, each with 32 threads asking for one float apiece. In the top (green) row the 32 floats sit side-by-side and fit inside a single 128-byte transaction — one full truck. In the bottom (red) row the same 32 floats are scattered far apart, so the hardware must send many separate trucks, each carrying only one useful float and lots of empty air. Same amount of wanted data, wildly different number of trips.

Now let us name the pieces the formula needs.

Best case, worked from zero. 32 threads reading consecutive floats, so $b=4$, lowest address $A$, highest address $A+31\times 4$: $\text{span}=(A+124)-A+4=128\text{ bytes}\Rightarrow \lceil \frac{128}{128}\rceil =1\text{ transaction.}$

Worst case (stride 32). Thread $i$ reads $A+32i\times 4$, so highest is $A+31\times 128$: $\text{span}=(A+3968)-A+4=3972\text{ bytes}\Rightarrow \lceil \frac{3972}{128}\rceil =32\text{ transactions.}$

3. Efficiency $\eta$ — how full each truck is

Consecutive case: $\eta =\frac{128}{1\times 128}=100\%$. Stride-32 case: $\eta =\frac{128}{32\times 128}=3.125\%$.

4. Where $B$ itself comes from (raw bandwidth)

You will not usually compute $B$ — the datasheet gives it (e.g. an A100 ≈ 2 TB/s). The formula matters only so the symbol $B$ isn't a mystery.

The parent also multiplies several efficiencies together: ${\eta }_{\text{total}}={\eta }_{\text{coalesce}}\times {\eta }_{\text{cache}}\times {\eta }_{\text{occupancy}}.$ Each is a fraction ≤ 1 from a different loss. ${\eta }_{\text{cache}}$ (fewer trips because data was already nearby) is the business of Cache-Hierarchy; ${\eta }_{\text{occupancy}}$ (keeping the road never idle) ties into Memory-Latency-Hiding.

5. Shared memory and banks — the fast local shelf

For how you actually program it, see CUDA-Shared-Memory.

Before the bank formula, we need one small piece of arithmetic notation.

Figure s03 — banks as a clock face. The 32 banks are drawn around a circle like the numbers on a clock. Slot-indices are assigned going clockwise: 0, 1, 2, … 31, and then slot-index 32 wraps straight back onto bank 0. The orange dot highlights byte-address 70: first floor-divide by the slot size, $\lfloor 70/4\rfloor =17$, giving slot-index 17; then $17\mathrm{mod}32=17$, so it lands on bank 17. The clock shape is the modulo made visible: counting past 31 loops around; here 17 is still on the first lap, so bank = slot-index.

6. Counting work: FLOPs and arithmetic intensity

Figure s04 — the roofline. The horizontal axis is arithmetic intensity $I$ (math per byte); the vertical axis is the speed you can actually attain. The black line has two parts: a slanted ramp on the left (where the memory road limits you) and a flat roof on the right (where raw compute limits you). They meet at the dotted vertical line, the machine balance $I^{\star }$. The red dot (naive, $I=0.25$) sits far left and low — starved. The orange dot (tiled, $I=8$) has climbed the ramp but is still left of the corner. The green dot ($I=40$) sits up on the flat roof — compute-bound. The whole message: raising $I$ walks you rightward up the ramp until you hit the roof.

The parent's tiling trick (parameter $T_{\text{tile}}$ = tile edge length) raises $I$ from $0.25$ (no tiling) toward and past $I^{\star }$. Why tiling reuses data is a story of good Parallel-Algorithm-Design — you compute a value while its neighbours are still hot in shared memory.

7. How it all connects

The diagram below is a prerequisite map: read an arrow "$X\to Y$" as "you must understand $X$ before $Y$ makes sense." Each box is one concept we just built, in the same order as the sections above; follow the arrows and you re-walk the whole page. Everything funnels into the parent topic at the bottom.

Equipment checklist

Cover the right side and answer aloud. If any stalls, re-read that section.