GPU memory bandwidth optimization: Question bank — Hardware deep dive

This page is a question bank of misconceptions. Each item is a one-line reveal: read the prompt, answer it in your head, then check. The goal is not arithmetic (that lives elsewhere) — it is to stress-test what you think you understand about how a GPU moves bytes.

Before we start, let us make this page self-contained: every term the traps use is defined here, with a picture.

The picture below shows a warp's 32 requests hitting DRAM chunks. Look at the top row (green): all 32 addresses fall inside one 128-byte chunk — one trip. Bottom row (red): the same 32 requests scattered across 32 chunks — 32 trips, each mostly wasted.

Figure — GPU memory bandwidth optimization

Two more anchors the traps lean on

The efficiency you actually get is a product of three separate fractions, each between 0 and 1. If any one is small, the whole product is small.

$η_{total} = η_{c} \times η_{h} \times η_{o}, M = B \times η_{total}$

Here $B$ is the raw bus bandwidth (bytes per second) and $M$ is the throughput you really see. The picture: three gates in series — each gate only lets a fraction through, so the water reaching the end is the product.

Arithmetic intensity and machine balance — derived here, not borrowed

Two symbols appear in the traps below. Let us earn them from scratch using tiled matrix multiply.

Derivation of $I = T /4$ (step by step):

Count the math. Multiplying two $N \times N$ matrices does $N^{3}$ multiply-adds; each is 2 FLOPs → total $2 N^{3}$ FLOPs.
Count the DRAM bytes with tiling. Without tiling each output element reads $2 N$ elements from DRAM. Tiling loads a tile once per block and reuses it, cutting global reads by a factor of $T$ , so each output element now triggers $\frac{2 N}{T}$ global reads. With $N^{2}$ outputs and 4 bytes per float: $Bytes = N^{2} \times \frac{2 N}{T} \times 4 = \frac{8 N ^{3}}{T}$
Divide. Intensity is FLOPs over bytes: $I = \frac{2 N ^{3}}{8 N ^{3} / T} = \frac{T}{4} FLOPs/byte$ The $N^{3}$ cancels — intensity depends only on the tile you chose. For $T = 32$ : $I = 8$ .

Machine balance example: a GPU with 20 TFLOP/s peak compute and 1 TB/s bandwidth has $I_{bal} = \frac{20 \times 1 0 ^{12}}{1 \times 1 0 ^{12}} = 20 FLOPs/byte$

The rule this sets up: if $I < I_{bal}$ you are bandwidth-bound (starved for data); if $I > I_{bal}$ you are compute-bound (ALUs are the limit). The picture below plots this — the roofline: a sloped bandwidth ceiling meeting a flat compute ceiling exactly at $I_{bal}$ .

Everything below is a trap built around one of these ideas, or around caches, shared memory banks, or the roofline.

True or false — justify

More threads always means more effective bandwidth

False. Bandwidth is capped by the physical bus; extra threads only help by hiding latency (Memory-Latency-Hiding) and raising occupancy — once the bus is saturated, more threads just wait.

A fully coalesced kernel is automatically fast

False. Coalescing gives you high efficiency per transaction, but if the kernel is compute-bound or bottlenecked on occupancy, memory was never the limit — coalescing fixes a problem you may not have.

Reading the same address from all 32 threads is a 32-way bank conflict

False. A shared-memory broadcast serves an identical address to every lane in one cycle — the conflict rule only bites when threads hit the same bank at different addresses.

Structure-of-Arrays is always better than Array-of-Structures

Mostly true for GPUs, but not universal. SoA coalesces field-wise access; however if a thread genuinely needs all fields of one element together (locality of use), AoS can win on cache reuse — the "best" layout follows the access pattern, not a slogan.

A strided access with stride 32 still coalesces "a little"

False for stride-32 floats. Each of the 32 lanes lands in a different 128-byte chunk, so you pay 32 transactions and use one thirty-second of each — effectively fully un-coalesced (about three percent efficiency).

Padding a 32-by-32 shared array to 32-by-33 wastes memory for no reason

False. The extra column shifts each row's bank mapping by one, breaking the column-access bank conflict during transpose — a tiny memory cost buys a 32-times shared-throughput win.

Caches on a GPU work the same way as on a CPU

False in emphasis. GPU caches are small per-thread but face huge aggregate demand; they exist mainly to catch reuse and smooth transactions, not to hold a single thread's working set like a CPU's L1.

If arithmetic intensity is below the machine balance, you are bandwidth-bound

True. On the roofline, intensity below

I_{bal}

means the sloped bandwidth ceiling caps you — no faster ALUs help until you raise

I

(e.g. bigger tiles).

Coalescing and bank conflicts are the same phenomenon

False. Coalescing is about global DRAM transactions across the warp; bank conflicts are about shared-memory parallel ports. Different memories, different fixes.

Spot the error

"AoS with a 24-byte struct fully serializes into 32 separate reads."

The error: it partially coalesces. A warp touches

32 \times 24 = 768

bytes, needing

⌈ 768/128 ⌉ = 6

transactions, not 32. It wastes about five-sixths of bandwidth but is not the worst case.

"Tiling reduces the total number of arithmetic operations."

The error: tiling changes memory traffic, not FLOP count. You still do

2 N^{3}

FLOPs; you just fetch the operands far fewer times, raising arithmetic intensity

I

"Shared memory is fast, so copy the whole matrix into it."

The error: shared memory is tiny (tens of KB per block). You tile precisely because it cannot hold the whole matrix — you stream

T \times T

tiles through it.

"Bank = address mod 32, so consecutive floats always conflict."

The error: consecutive 4-byte words map to consecutive banks

0, 1, 2, \dots

, so a warp reading consecutive floats hits 32 different banks — the conflict-free ideal, not a conflict.

"To get 100 percent bandwidth just make every access coalesced."

The error: total efficiency is

η_{c} \times η_{h} \times η_{o}

. Perfect coalescing (

η_{c} = 1

) with low occupancy (

η_{o}

small, controllers idle) still leaves throughput far below peak.

"Since the A100 has 2 TB/s bandwidth, my kernel gets 2 TB/s."

The error: 2 TB/s is peak

B

. Real throughput is

M = B \times η_{total}

; with a realistic

η_{total}

around 0.5 to 0.8, you see a fraction of peak even when well-optimized.

Why questions

Why does the hardware read 128-byte chunks instead of the exact bytes requested?

DRAM is optimized for burst transfers; the fixed transaction amortizes addressing and row-activation overhead. Fine-grained single-byte reads would drown in overhead, so the bus is designed wide.

Why do shared-memory banks exist instead of a full crossbar?

A true 32-by-32 crossbar (any thread to any address in one cycle) is hardware-expensive. Banks partition memory so each address maps to one port, giving cheap 32-way parallelism — at the cost of conflicts when the mapping collides. See CUDA-Shared-Memory.

Why does SoA turn 6 transactions per field into 1?

SoA stores each field contiguously, so the 32 lanes reading one field ask for 32 consecutive floats, exactly one 128-byte chunk. AoS interleaves fields, scattering the wanted field across chunks.

Why can raising the tile size flip a kernel from bandwidth-bound to compute-bound?

Arithmetic intensity scales with tile size (

I = T /4

, derived above). Bigger tiles reuse each fetched byte for more FLOPs, pushing

I

past the machine balance

I_{bal}

on the roofline.

Why does a broadcast avoid a bank conflict but scattered same-bank reads do not?

The bank hardware can fan one fetched word out to all lanes for free (broadcast), but it cannot fetch different words from one bank simultaneously — those must serialize, one per cycle.

Why is memory bandwidth, not compute, so often the GPU bottleneck?

Thousands of cores can consume data far faster than the bus can supply it. Peak FLOP/s vastly exceeds peak bytes/s times typical intensity, so most kernels starve for data — the roofline's sloped region.

Edge cases

What is the coalescing efficiency when a warp reads a single 4-byte float broadcast to all lanes from global memory?

One 128-byte transaction serves the warp, and although only 4 bytes are unique, every lane genuinely uses that value, so it is fully coalesced (1 transaction) with useful reuse, not waste.

What happens with a warp where only lane 0 is active (the other 31 predicated off)?

You still pay a full transaction (minimum chunk) for lane 0's data alone — efficiency

η_{c}

collapses to

4/128 \approx 0.031

. Divergence and masking hurt bandwidth as badly as bad strides.

What is the arithmetic intensity of a pure memory copy (out[i]=in[i])?

Zero FLOPs per byte moved, so

I = 0

— the extreme left of the roofline, purely bandwidth-bound. Its only optimization is perfect coalescing; there is nothing to compute-hide behind.

For a tile size

T = 1

in the matmul derivation, what does the "tiled" formula collapse to?

I = T /4 = 0.25

FLOPs/byte and traffic reduction

= T = 1

times — i.e. exactly the naive un-tiled case, confirming the formula degenerates correctly.

If a warp's 32 threads all access the same shared-memory bank at 32 different addresses, how long does it take?

32 cycles — fully serialized, the worst-case bank conflict, the opposite of the 1-cycle broadcast.

Recall Quick self-check

Coalescing lives in which memory? ::: Global/DRAM transactions across a warp. Bank conflicts live in which memory? ::: Shared memory. Total efficiency is the product of which three factors? ::: Coalescing $η_{c}$ , cache hit rate $η_{h}$ , occupancy $η_{o}$ . What is $I_{bal}$ ? ::: Machine balance — peak FLOP/s divided by peak bytes/s; the intensity where compute and bandwidth match. Below $I_{bal}$ you are ::: bandwidth-bound.