Visual walkthrough — Occupancy and latency hiding

---

Step 1 — What is a warp, and what does "issue" mean?

WHAT. Look at the figure. Each cyan row is one warp. A tiny box on the timeline is one cycle — the smallest tick of the GPU's clock. When a warp is chosen, the SM issues its next instruction: it hands that instruction to an execution unit. That takes exactly one cycle to issue.

WHY start here. Every later count is "how many warps" and "how many cycles". If we don't nail down that a warp issues one instruction per cycle when it is picked, the arithmetic later is meaningless.

PICTURE. The amber marker shows the single warp that gets to issue on this cycle. Everything else waits its turn.

$\text{issue rate of one warp}=1\frac{\text{instruction}}{\text{cycle}}(\text{only when it is the chosen warp})$

• $\text{instruction}$ — one operation for all 32 threads at once
• $\text{cycle}$ — one tick of the clock, the unit we measure latency in

---

Step 2 — What is latency? Draw the gap.

WHAT. A warp issues a ld.global (read from DRAM). The number comes back only after $L\approx 400$ cycles. During those 400 cycles that warp can do nothing with the loaded value — the value isn't there yet.

WHY this tool — a "gap" not a "cost". People confuse latency with throughput. Latency is a delay on one operation; throughput is how many finish per cycle. We picture latency as an empty horizontal gap so you can literally see there is idle room to fill.

PICTURE. The white bracket is the 400-cycle gap. The warp that started it is greyed out — asleep — for the whole bracket.

$L=400\text{cycles}\text{(the length of the empty bracket)}$

• $L$ — the wait; the horizontal length of the gap in the picture
• $400$ — typical DRAM read latency, from the parent note

---

Step 3 — One warp alone wastes the gap

WHAT. With a single warp, the timeline after a load is one instruction, then 400 empty cycles, then the next instruction. The SM does useful work on only $1$ of every $401$ cycles.

WHY. This is the problem statement. We want the scheduler (see 6.2.8-Warp-Scheduling) to issue something useful every cycle. Right now it can't, because the only warp it has is asleep.

PICTURE. Count the empty boxes — they are the wasted cycles. Utilisation is the tiny amber sliver over the whole bar.

${\text{utilisation}}_{1\text{ warp}}=\frac{1\text{busy cycle}}{1+L\text{cycles}}=\frac{1}{401}\approx 0.25\%$

• numerator $1$ — the single cycle the warp was busy issuing
• denominator $1+L$ — that busy cycle plus the whole empty gap

---

Step 4 — Fill the gap with other warps

WHAT. Bring in more warps. While Warp A sleeps through its 400-cycle gap, the scheduler issues from Warp B, then Warp C, then Warp D… one per cycle. Each of these can also fire off its own load and go to sleep — but by then someone else is awake.

WHY this is the whole trick. The gap has $L$ empty cycles. If we have $L$ other warp-instructions to issue during that gap, every empty box gets filled. The stall vanishes — not because the load got faster (it didn't), but because the SM always found other work.

PICTURE. Stack the warps as rows. The diagonal amber staircase is the scheduler hopping to a fresh warp each cycle, painting the gap solid.

$\underset{\text{how many rows}}{\underbrace{N_{\text{warps}}}}\ge \underset{\text{empty boxes to fill}}{\underbrace{L}}\times \underset{\text{instr issued per cycle}}{\underbrace{1}}$

• $N_{\text{warps}}$ — number of warp-rows we stacked
• $L$ — the count of empty cycles that must be covered
• the $\times 1$ — one scheduler issues one instruction per cycle

This is exactly Little's Law: items in flight = arrival rate × time in system. Here "items in flight" are warps issuing, "time in system" is the latency $L$.

---

Step 5 — Generalise: the throughput factor

WHAT. In Step 4 we assumed one scheduler issuing 1 instruction/cycle, so $\Theta =1$ and $N=L$. A real SM has 4 schedulers (see 6.2.1-SM-Architecture), each issuing per cycle, so $\Theta =4$.

WHY a product. Every cycle you want $\Theta$ instructions to appear. The wait lasts $L$ cycles. Over the whole wait you therefore need $L\times \Theta$ instructions queued and ready — that is the area of the rectangle in the figure.

PICTURE. A rectangle: width $L$ (cycles of waiting), height $\Theta$ (instructions per cycle). Its area is the number of in-flight operations you must supply.

$N_{\text{required}}=\underset{\text{width: wait in cycles}}{\underbrace{L}}\times \underset{\text{height: issue/cycle}}{\underbrace{\Theta }}=\text{area of the rectangle}$

---

Step 6 — Why the "400 warps" number overshoots (the crucial edge case)

WHAT. The naïve reading says "need 400 warps". Real kernels hide the same latency with 16–32 warps. Nothing was cheated — three effects each multiply the instructions per warp per cycle you actually get.

WHY it matters. If you believed you needed 400 warps you'd panic — the hardware only allows 64. The resolution is that a warp supplies more than one outstanding operation over that window.

PICTURE. Three stacked bars show the three multipliers shrinking the required warp count from 400 down to ~16.

$N_{\text{warps}}=\frac{L\times \Theta }{\underset{\text{schedulers}}{\underbrace{S}}\times \underset{\text{ILP per warp}}{\underbrace{I}}}$

• $L\times \Theta$ — total in-flight ops needed (Step 5)
• $S=4$ — schedulers issuing in parallel, splitting the burden
• $I$ — instruction-level parallelism: a warp may have several independent loads/FMAs in flight before it stalls (see 6.2.9-Memory-Coalescing for why one load can be several transactions but is issued as one instruction)

With $L=400,\Theta =1$ effective per-scheduler, $S=4$, $I\approx 3$–$6$: $N\approx \frac{400}{4\times 6}\approx 16$ warps. This is why the parent note says 16–32 active warps usually suffice.

---

Step 7 — Degenerate cases: when more warps do NOTHING

WHAT. Two boundary situations where the rectangle logic tells you to stop adding warps.

1. Already hidden (memory-bound, gap full). Once $N\ge N_{\text{required}}$ every cycle is busy. Adding warps past this point adds zero throughput but steals registers — pushing you toward spills. This is the parent's "50% is enough" observation.
2. No gap at all (compute-bound). If a warp does 256 back-to-back FMAs, the pipeline is already full from ILP — there is barely any gap to fill, so $N_{\text{required}}$ is tiny (~4–8 warps).

WHY. The requirement is $L\times \Theta$ — a fixed target, not "bigger is better". Once you meet it, the marginal warp is idle capacity you paid registers for.

PICTURE. Left: gap already fully painted, extra warps hover unused. Right: a nearly gapless compute bar — almost nothing to hide.

---

The one-picture summary

WHAT. Everything at once: a warp issues a load (Step 2), opens a gap (Step 3); other warps march diagonally to fill it (Step 4); the amount you need is the rectangle $L\times \Theta$ (Step 5); schedulers and ILP shrink the warp count (Step 6); and past "full", extra warps are wasted (Step 7).

$\boxed{N_{\text{warps required}}=\frac{L\times \Theta }{S\times I}}\text{stop as soon as the gap is painted solid.}$

Recall Feynman retelling — say it back in plain words

A warp asks memory for a number and then has to nap for about 400 clock ticks until the number arrives. If that's your only warp, the whole SM naps with it — a terrible waste. So you keep a crowd of warps around: while one naps, the scheduler wakes another, and another, painting every idle tick with real work. How many nappers do you need in the crowd? Exactly enough to cover the nap: the length of the nap (400 ticks) times how many instructions per tick you want issued — that's a rectangle, and its area is your answer. But each warp is cleverer than one instruction: with 4 schedulers and several independent operations already in flight per warp, a crowd of just 16–32 warps covers a 400-tick nap. And crucially — once every tick is painted, adding more warps does nothing but hog registers. Aim for a full gap, not a full stadium.

Recall

What physically is the "gap" being hidden? ::: The latency $L$ — the ~400 idle cycles between issuing a global load and the result being ready. Why is the required count a product $L\times \Theta$? ::: Because you need $\Theta$ instructions every cycle, for $L$ cycles — that's the rectangle's area. Why don't you literally need 400 warps? ::: 4 schedulers issue in parallel and each warp has several independent ops in flight (ILP), dividing the requirement down to ~16. When do extra warps give zero benefit? ::: Once the gap is fully covered (memory-bound) or when the pipeline is already full from ILP (compute-bound).