Occupancy and latency hiding: Question bank — Hardware deep dive

Before we start, three plain-word anchors so no term is used before it is earned:

A warp is a group of 32 threads that march together, executing the same instruction in lockstep.
Occupancy is a ratio: how many warps are actually resident on a Streaming Multiprocessor divided by how many that SM could ever hold at once. See 6.2.1-SM-Architecture.
Latency hiding means the SM never sits idle during a slow operation, because it switches to another ready warp. The scheduler that picks that warp is covered in 6.2.8-Warp-Scheduling.

True or false — justify

True or false: Doubling occupancy always doubles throughput.

False — throughput vs. occupancy is a saturating curve. Once enough warps exist to cover the longest latency, extra warps sit in the pool doing nothing new, so the curve flattens.

True or false: Switching between warps on an SM costs many idle cycles, like a CPU context switch.

False — each warp's registers stay resident on-chip, so the scheduler can pick a different warp on the very next cycle at essentially zero cost. That zero-cost switch is the whole reason latency hiding works.

True or false: A single warp can, by itself, keep 400 cycles of memory latency fully hidden.

False — one warp issues one memory instruction at a time and then must wait for its result. Hiding latency needs many warps in flight so the SM always has something else ready.

True or false: 100% occupancy is the design target for every kernel.

False — the real target is zero scheduling stalls. If 32 warps already hide all latency, forcing 64 only adds register and cache pressure for no gain.

True or false: Occupancy is a property of the hardware, fixed for a given GPU.

False — it is a property of the kernel running on the hardware. The same GPU gives different occupancy depending on registers/thread, shared memory/block, and block size.

True or false: A compute-bound kernel benefits from maxing out occupancy.

Usually false — compute-bound code is limited by ALU throughput, not by finding a ready warp. It hides its short latencies with a handful of warps and instruction-level parallelism, so extra warps waste registers.

True or false: Increasing the number of active warps reduces the latency of any single memory load.

False — the intrinsic latency

L

of one load is unchanged. More warps let you overlap many such latencies so the SM stays busy; the individual load is just as slow.

True or false: If a kernel uses no shared memory, shared memory can never limit its occupancy.

True — with zero shared memory per block, the shared-memory term is not a binding constraint, so occupancy is set by registers or the block/warp hardware caps instead.

True or false: Coalesced memory access changes a kernel's occupancy.

False — coalescing changes how efficiently the memory system serves requests, not how many warps are resident. It affects the latency you must hide, not the number of warps available to hide it.

Spot the error

"My kernel uses 128 registers/thread and I set occupancy to 100% in the launch config — done."

You can't set occupancy; the launch config plus per-thread resource use determines it. At 128 regs/thread on a 65,536-reg SM only 16 warps fit, capping occupancy at 25% regardless of intent.

"Occupancy = active threads ÷ max threads, so I count threads directly."

The unit that matters is the warp, because the scheduler issues one warp-instruction per cycle. Occupancy is active warps over max warps; counting threads only agrees when everything divides evenly into 32.

"Store latency is ~400 cycles, so I must add another 400 cycles to my critical path per thread."

Global stores retire asynchronously — they are buffered and drain in the background. The store's latency normally does not sit on the critical path; only the load whose result you actually need does.

"Higher occupancy caused register spills, so occupancy is bad."

Backwards — you raised occupancy by lowering registers/thread, which forced live data to spill to local memory. The spilling hurts, not the occupancy itself; you traded register room for warp count too aggressively.

"I have 4 schedulers, so I need 4× more warps than a single-scheduler estimate."

The opposite direction — 4 schedulers supply concurrency by issuing from different warps in consecutive cycles, so the naïve single-scheduler warp count overshoots. Real kernels hide latency with far fewer warps than the single-scheduler formula suggests.

"My occupancy is 50% but performance is fine, so the profiler is wrong."

50% is often enough. If those warps already cover the load latency, you are at the plateau of the curve; the profiler is right and 50% is simply sufficient here.

Why questions

Why is a warp exactly 32 threads and not 16 or 64?

32 lanes balances hardware cost against parallelism, and it matches memory hardware — a 128-byte cache line holds exactly 32 four-byte floats, so one coalesced warp access maps to one line.

Why does the GPU chase throughput rather than minimizing single-operation latency like a CPU?

A CPU spends transistors on caches and out-of-order logic to make one thread fast. A GPU spends them on thousands of resident threads, so it tolerates high latency by always having other work — the strategy Little's Law quantifies.

Why do we take the minimum of the register, shared-memory, and block limits when computing occupancy?

Whichever resource runs out first caps how many warps fit. It's a bottleneck: the scarcest resource decides, so the smallest limit wins.

Why can a memory-bound kernel tolerate lower occupancy than you'd fear?

Its warps issue loads that the memory system services in parallel, so even 16–32 warps put enough requests in flight to keep the SM busy while each load's hundreds of cycles elapse.

Why do register spills hurt more than simply having fewer warps?

Spilling serializes a thread — it must load/store its own data from local memory before continuing, adding latency inside every warp. Low occupancy just means fewer warps, but each surviving warp still runs at full speed.

Why doesn't 100% occupancy help a kernel that already has no scheduling bubbles?

If the scheduler always finds a ready warp, there is no idle cycle left to fill. Extra warps add nothing to hide and only consume registers and cache, sometimes making things worse.

Why does increasing arithmetic per byte (arithmetic intensity) reduce the pressure on occupancy?

More compute per memory access means each load's latency is naturally overlapped by real work, moving the kernel toward the compute-bound side of the roofline where fewer warps suffice.

Edge cases

What is the occupancy of a kernel launched with exactly one block of 32 threads on an SM that holds 64 warps?

One warp resident out of 64 → about 1.6% occupancy. There is essentially no latency hiding: when that lone warp stalls, the SM idles.

If two kernels each need 50% occupancy of an SM's shared memory, can they share an SM simultaneously?

Only if their combined resource demand (shared memory, registers, warp slots) fits the SM's totals. Concurrent-kernel residency is bounded by the same min-of-resources rule, not by occupancy percentages added up loosely.

A block requests more shared memory than the SM has in total — what happens to occupancy?

The kernel cannot launch at all; zero blocks fit, so occupancy is undefined/zero. The resource ceiling is a hard wall, not just a throttle.

A kernel is perfectly latency-hidden at 25% occupancy — is raising it to 50% ever harmful?

It can be. Halving registers/thread to double warps may force spills, and more resident data can thrash the L1/cache — so 50% may run slower than the already-sufficient 25%.

What happens to latency hiding when every warp in the pool is waiting on the same dependent load at the same instant?

Nothing is ready to issue, so the SM stalls despite high occupancy. High warp count only helps if the warps become ready at different times — occupancy is necessary but not sufficient.

If a Tensor-Core-heavy kernel is compute-bound, does occupancy still matter?

Much less — Tensor Cores saturate on ALU throughput, so a modest warp count keeps them fed. Extra warps rarely add speed once the matrix pipelines are full.

Recall Fast self-check

One-warp SM ⇒ ?occupancy and ?hiding ::: ~1.6% occupancy and effectively no latency hiding — the SM idles on every stall. Store latency on the critical path? ::: No — stores retire asynchronously via buffers; only the needed load counts. The occupancy formula uses which combining rule across resources? ::: The minimum — the scarcest resource caps resident warps.