Occupancy and latency hiding: Worked examples — Hardware deep dive

This page is one thing only: practice. We take the ideas from the parent note and grind them through every kind of situation the topic can hand you — every resource limit, every degenerate input, every "trick" twist an exam loves. Before we start, one promise: nothing here uses a symbol you have not met. If you need a refresher on where warps come from, that lives in 6.2.1-SM-Architecture and 6.2.8-Warp-Scheduling.

Throughout, our reference chip is the A100, whose per-SM budgets are:

Before we touch a single example, here is how to compute each of the three resource-limited warp counts. Every example just plugs numbers into these three definitions.

The scenario matrix

Occupancy problems all reduce to: "which resource runs out first?" Here is every case-class that can decide the answer. Each row is a cell; the examples that follow are tagged with the cell they cover.

Cell	What runs out (or the twist)	Degenerate/limit angle	Covered by
A	Registers are the bottleneck	very high regs/thread	Ex 1, Ex 6
B	Shared memory is the bottleneck	block granularity waste	Ex 2
C	Block-count cap is the bottleneck	tiny blocks, hit $B_{m a x}$	Ex 3
D	Nothing limits → 100% possible	the "free" case	Ex 4
E	Zero / degenerate input	0 shared mem, 1 thread/block	Ex 4, Ex 5
F	Two limits tie	pick the min, both equal	Ex 6
G	Real-world word problem	latency-hiding decision	Ex 7
H	Exam twist: high occupancy ≠ faster	spills vs. occupancy	Ex 8

The workflow is always the same, so learn it once (chart nodes are labelled P1–P5 so they never clash with the Cell letters above):

Example 1 — Cell A: registers bite first

Worked example Statement

Kernel uses 64 registers/thread, no shared memory, 256 threads/block. What is the occupancy on A100?

Forecast: Guess now — is it 25%, 50%, or 100%? Write it down before reading on.

Step 1 — Registers per warp. $64 \frac{regs}{thread} \times 32 \frac{threads}{warp} = 2048 \frac{regs}{warp}$ Why this step? Registers are handed out per thread, but occupancy counts warps. Multiply by 32 to convert the budget into "warps' worth."

Step 2 — How many warps fit in the register file. $W_{regs} = ⌊ \frac{65536}{2048} ⌋ = 32 warps$ Why the floor? You cannot host half a warp. Any leftover registers that can't complete a full warp are simply wasted.

Step 3 — Other limits. No shared memory ⇒ $W_{smem} = \infty$ . Blocks: 256 threads/block $= 8$ warps/block, so $W_{blocks} = B_{m a x} \times 8 = 32 \times 8 = 256$ warps — far above the roster, so blocks aren't tight either.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 32 , \infty , 256 , 64 )}{64} = \frac{32}{64} = 50%$

Verify: $32 \times 2048 = 65536$ — the register file is exactly full, zero waste. That's the signature of a clean register-bound kernel: budget divides evenly. If forecast ≠ 50%, re-check that you multiplied regs by 32.

Example 2 — Cell B: shared memory bites first

Worked example Statement

Kernel uses 32 registers/thread, 48 KB shared memory per block, 256 threads/block (= 8 warps/block). Occupancy on A100?

Forecast: Registers look tame here. Does that mean 100%? Guess.

Step 1 — Register limit (should be generous). $W_{regs} = ⌊ \frac{65536}{32 \times 32} ⌋ = ⌊ \frac{65536}{1024} ⌋ = 64 warps$ Why? At only 32 regs/thread we can fit the full 64-warp roster — registers are not the villain here.

Step 2 — Shared-memory limit, in blocks. $⌊ \frac{164 KB}{48 KB} ⌋ = 3 blocks$ Why floor to blocks? Shared memory is allocated per block, all-or-nothing. $3 \times 48 = 144$ KB used, 20 KB left over — not enough for a 4th block, so it's wasted.

Step 3 — Convert blocks to warps. $W_{smem} = 3 blocks \times 8 \frac{warps}{block} = 24 warps$ Also $W_{blocks} = B_{m a x} \times 8 = 32 \times 8 = 256$ warps — not binding.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 64 , 24 , 256 , 64 )}{64} = \frac{24}{64} = 37.5%$

Verify: Shared mem is scarcest (24 < 64), so it decides — consistent with $min$ . Sanity on the leftover: $164 - 144 = 20 KB < 48$ , confirming no 4th block fits. Answer is below Example 1's 50% because the block-granular waste of shared memory is harsher than the exact register fit.

Example 3 — Cell C: the block-count cap bites

Worked example Statement

A "wide grid" kernel launches tiny blocks of 32 threads (= 1 warp/block), 16 registers/thread, no shared memory. What limits occupancy?

Forecast: With only 16 regs/thread and no shared mem, surely 100%? Careful.

Step 1 — Register limit. $W_{regs} = ⌊ \frac{65536}{16 \times 32} ⌋ = ⌊ \frac{65536}{512} ⌋ = 128 warps$ Why the floor? You cannot host a fraction of a warp — leftover registers that don't complete a whole warp are wasted. Here $128 > W_{m a x} = 64$ , so registers alone would allow the full roster.

Step 2 — The block-count cap. Each block is 1 warp. To reach 64 warps we would need 64 blocks on the SM. But the hardware caps blocks at $B_{m a x} = 32$ . $W_{blocks} = B_{m a x} \times 1 \frac{warp}{block} = 32 \times 1 = 32 warps$ Why this step? Blocks are hardware slots with a hard ceiling. Tiny blocks "spend" block slots faster than warp slots, so you run out of slots before warps.

Step 3 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 32 , 64 )}{64} = \frac{32}{64} = 50%$

Verify: The bottleneck is $B_{m a x}$ , not registers or shared memory — the classic "blocks too small" trap. Fix: use ≥64 threads/block (2 warps) so 32 blocks give the full 64 warps. Sanity: $32 \times 2 = 64 = W_{m a x}$ . ✓

Example 4 — Cells D & E: the "free" case and the zero input

Worked example Statement

Kernel uses 16 registers/thread, 0 KB shared memory, 128 threads/block (= 4 warps). Occupancy?

Forecast: After three limited cases, is anything even in the way here?

Step 1 — Registers. $W_{regs} = ⌊ \frac{65536}{16 \times 32} ⌋ = ⌊ \frac{65536}{512} ⌋ = 128 warps$ Why the floor? Same reason as always: a partial warp cannot be hosted, so we round down. $128 > 64$ , so registers give no limit.

Step 2 — Shared memory (the zero input, Cell E). Shared mem per block $= 0$ . Dividing the budget by zero would be "infinity blocks," which really means shared memory never constrains you. Treat $W_{smem} = \infty$ . Why? A resource you don't request can't run out.

Step 3 — Blocks. 4 warps/block; $W_{blocks} = B_{m a x} \times 4 = 32 \times 4 = 128$ warps. To reach the roster of 64 warps needs only $64/4 = 16$ blocks $\leq B_{m a x} = 32$ , so blocks are fine.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 128 , 64 )}{64} = \frac{64}{64} = 100%$

Verify: Every candidate is $\geq 64$ , so the $W_{m a x}$ cap itself is the binding "limit" — the healthy sign of a resource-light kernel. Degenerate check: 0 shared mem correctly dropped out of the $min$ rather than crashing it.

Example 5 — Cell E extreme: one thread per block

Worked example Statement

Someone (by mistake!) launches 1 thread per block, 16 regs/thread, no shared memory on A100. What's the ceiling on occupancy?

Forecast: 1 thread is much less than 32. Does the warp "round up"?

Step 1 — A block still costs a whole warp. Even 1 thread occupies a full warp slot (the other 31 lanes are masked off — idle but reserved). So 1 thread/block = 1 warp/block, with 31 lanes wasted. Why? Warps are the smallest schedulable unit; you cannot have a fraction of a warp. See 6.2.8-Warp-Scheduling.

Step 2 — Compute each limit. With $r = 16$ , $n = 1$ :

Registers: $W_{regs} = ⌊ 65536/ (16 \times 32)⌋ = 128$ warps $> 64$ — not binding.
Shared mem: none requested $\Rightarrow W_{smem} = \infty$ — not binding.
Blocks: $W_{blocks} = B_{m a x} \times n = 32 \times 1 = 32$ warps — this binds. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 32 , 64 )}{64}$

Step 3 — Occupancy of warps. $\frac{32}{64} = 50%$

Step 4 — Effective useful work. Warp occupancy is 50%, but each active warp uses only $1/32$ lanes. Useful-thread fraction: $50% \times \frac{1}{32} \approx 1.56%$

Verify: $0.50/32 = 0.015625 = 1.5625%$ . This is why "occupancy" alone can lie — high warp occupancy with masked lanes still wastes the machine. Always launch blocks that are multiples of 32.

Example 6 — Cell A + F: a register limit that ties

Worked example Statement

Kernel uses 128 registers/thread, no shared memory, 128 threads/block (4 warps). Show the register limit and find whether the block cap ties it.

Forecast: High registers → low occupancy. But does anything else match that number?

Step 1 — Register limit. $W_{regs} = ⌊ \frac{65536}{128 \times 32} ⌋ = ⌊ \frac{65536}{4096} ⌋ = 16 warps$ Why? At 128 regs/thread each warp gobbles 4096 registers — only 16 fit (floor: a partial 17th warp can't be hosted).

Step 2 — In blocks. 16 warps at 4 warps/block $= 4$ blocks. Block cap gives $W_{blocks} = B_{m a x} \times 4 = 32 \times 4 = 128$ warps — plenty. So registers, not blocks, bind.

Step 3 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 16 , \infty , 128 , 64 )}{64} = \frac{16}{64} = 25%$

Step 4 — Where would a tie happen (Cell F)? If instead each block had exactly 4 warps and the block cap were 4, both $W_{regs}$ and $W_{blocks}$ would equal 16 — a true tie. $min$ handles it gracefully: equal values give the same answer, so no ambiguity.

Verify: $16 \times 4096 = 65536$ — register file exactly full again. Halving registers to 64/thread would double $W_{regs}$ to 32 → 50% (matches Example 1). ✓

The register–occupancy relationship is worth a picture. Figure s01 plots occupancy against registers/thread on the A100, with our worked example points marked:

Look at the stepped pink line: occupancy jumps down only when a new warp can no longer fit — it's a staircase, not a smooth slope, because of the floor function. The yellow dots mark Examples 1 (64 regs → 50%), 4 (16 regs → 100%), and 6 (128 regs → 25%).

Example 7 — Cell G: the real-world latency-hiding decision

Worked example Statement

A memory-bound kernel: each warp issues loads that take $L = 400$ cycles. Your best tuning gives 50% occupancy (32 warps). A colleague insists you must reach 100% (64 warps). Using Little's Law thinking, decide whether it's worth it.

Forecast: More warps → more latency hiding → always faster? Guess yes/no.

Step 1 — Define "throughput" precisely. 5.3.4-Littles-Law says $outstanding work = L \times throughput,$ where here throughput = memory instructions issued per cycle, SM-wide. An A100 SM has 4 warp schedulers, each able to issue 1 instruction/cycle, so at best throughput $= 4$ memory-issues/cycle. Why this step? Little's Law converts a time (latency) into a count (how many things must be in flight) — that count is what occupancy must supply.

Step 2 — How many outstanding loads are needed. $outstanding loads = L \times throughput = 400 cycles \times 4 \frac{issues}{cycle} = 1600 loads in flight.$ But a single warp keeps only 1 memory instruction outstanding at a time (per dependent load). So warps needed $\approx 1600/ (loads per warp in flight)$ . With modest ILP each warp keeps roughly 2–4 loads outstanding across an unrolled loop, giving $warps \approx \frac{1600}{4 schedulers \times a few loads/warp} \approx 16 - 32 warps.$ Why? This is where the band "16–32 warps" comes from — it is $1600$ outstanding loads divided among 4 schedulers and the few loads each warp overlaps.

Step 3 — Compare against your 32 warps. 32 warps sits at the top of that band, so at 50% occupancy the pipeline is essentially never idle for lack of warps. Doubling to 64 hides only the small residual — a ≤5% speedup at best, and it may cost register spills (Example 8) that make things worse.

Step 4 — Decision. Stay at 50%. Spend effort on 6.2.9-Memory-Coalescing or the 7.1.3-Roofline-Model instead, which move the true bottleneck.

Verify: $400 \times 4 = 1600$ outstanding loads; divided by 4 schedulers $= 400$ per scheduler; with ~a dozen-plus loads overlapped per scheduler across warps, ~ $16$ – $32$ warps suffice, and $32 \geq$ that. ✓ Answer: do not chase 100%.

Example 8 — Cell H: the exam twist (higher occupancy, slower kernel)

Worked example Statement

Compute-bound kernel currently uses 128 regs/thread → 25% occupancy and runs at time $T$ . To "fix" occupancy you force the compiler to 64 regs/thread → 50% occupancy, but this spills values to local memory, adding 2 spill loads per inner iteration at ~300 cycles each. Does performance improve?

Forecast: Occupancy doubled from 25% to 50%. Faster or slower?

Step 1 — Occupancy before/after. $⌊ 65536/ (128 \cdot 32)⌋ = 16$ warps $= 25%$ ; $⌊ 65536/ (64 \cdot 32)⌋ = 32$ warps $= 50%$ . The number looks better.

Step 2 — But this kernel is compute-bound. Its bottleneck is ALU throughput, which high occupancy does not raise once scheduling bubbles are already gone (they are, via ILP/unrolling). So the 25→50 jump buys ~nothing on the compute side.

Step 3 — Cost of spills. Each inner iteration now pays $2 \times 300 = 600$ extra cycles moving spilled data. If the iteration was ~256 FMA cycles of useful work, the new per-iteration time is $\frac{256 + 600}{256} = \frac{856}{256} \approx 3.34 \times the original.$

Step 4 — Verdict. The "50% occupancy" version is ~3.34× slower per iteration. Occupancy is a means (hide latency), not an end. Keep 25% and the registers. Compare to 6.2.11-Tensor-Cores kernels, which deliberately run low occupancy for exactly this reason.

Verify: $2 \times 300 = 600$ ; $(256 + 600) /256 = 856/256 = 3.34375 > 1$ ⇒ the change hurts. ✓ This is the canonical "100% occupancy ≠ best performance" trap from the parent note.

Recall

Recall Which resource decides occupancy?

The scarcest one — occupancy $= min (W_{regs}, W_{smem}, W_{blocks}, W_{m a x}) / W_{m a x}$ . ::: The minimum, because a warp needs every resource simultaneously.

Recall Why must block sizes be multiples of 32?

A block always costs whole warps; extra lanes in a partial warp are masked and wasted (Example 5). ::: 1 thread/block gives 50% warp occupancy but ~1.56% useful work.

Recall Does 100% occupancy always mean fastest?

No — Examples 7 and 8: past "enough to hide latency," extra warps give diminishing returns and can trigger register spills that slow you down. ::: Occupancy is a means to hide latency, not the goal itself.

Related: parent topic · 6.2.1-SM-Architecture · 6.2.8-Warp-Scheduling · 6.2.9-Memory-Coalescing · 6.2.11-Tensor-Cores · 7.1.3-Roofline-Model · 5.3.4-Littles-Law