Visual walkthrough — SIMT (single instruction multiple thread)

The parent note stated one deep result almost in passing:

$W\ge L\cdot T$

"To hide a latency $L$, you need at least $L\cdot T$ warps." That single line is the whole reason GPUs look the way they do. This page builds it from nothing — no formula assumed, every symbol earned before use — using pictures at every step.

We will discover why a GPU keeps thousands of threads in flight, and why it does not need the clever speculation tricks a CPU uses. See GPU vs CPU Architecture for the contrast, and GPU Occupancy for the tuning knob this derivation produces.

---

Step 1 — What is a warp, as a moving object?

WHAT. Before any formula, we need one clean picture of the thing we are counting. A warp is a bundle of 32 threads that march in lockstep: the Streaming Multiprocessor issues one instruction and all 32 lanes obey it in the same cycle.

WHY. The whole derivation counts warps, not threads. If "warp" stays fuzzy, every later number is fuzzy. So we pin it down as a single object that moves through time one instruction at a time.

PICTURE. Below, one warp is one horizontal track. A filled block = "this cycle, this warp did work". Right now there is only one warp, and it does exactly one instruction per cycle.

Nothing to compute yet — we just named time ($\text{cycle}$) and the goal ($T$).

---

Step 2 — The stall: what "latency" actually costs

WHAT. Some instructions do not finish in one cycle. A memory read must travel off-chip and back — call this waiting time the latency $L$, measured in cycles.

WHY. We introduce $L$ now because it is the villain of the whole story. Everything after this exists to hide $L$.

PICTURE. The warp issues a load at cycle $0$, then the track goes blank for $L$ cycles — the warp is frozen, waiting for data. Only after $L$ cycles can it move again.

$\underset{\text{cycles the warp is frozen}}{\underbrace{L}}\approx 400\text{ (typical global-memory read)}$

---

Step 3 — The fix: don't wait, switch

WHAT. Instead of staring at the frozen Warp 0, the warp scheduler simply picks a different warp that is ready, and issues its instruction. Warp 0's stall becomes background — free time we spend on someone else.

WHY. This is the key move. We do not make memory faster; we make the wait invisible by always having other work on hand. This is why GPUs run so many warps: not to be "parallel for its own sake," but to paper over latency.

PICTURE. Four tracks now. Warp 0 is frozen (blank), but on each cycle the scheduler drops down to a different ready warp and fires its instruction (colored blocks). The core stays busy even though Warp 0 is asleep.

---

Step 4 — Counting: how many warps hide the stall exactly?

WHAT. Now we count. During the $L$ cycles that Warp 0 sleeps, we want to issue $T$ instructions every cycle from other warps so the core never idles.

WHY. This turns the picture into arithmetic. We ask the precise question: how many warps must exist so the gap is always covered?

PICTURE. Stack the tracks. Warp 0 sleeps for a window of width $L$. In that window we need enough other filled blocks to keep every column busy. Count them.

Each cycle in the window needs $T$ instructions. The window is $L$ cycles wide. So the total number of instruction-slots to fill is:

$$\underset{\begin{array}{c}\text{width of}\\ \text{the gap}\end{array}}{\underbrace{L}}\times \underset{\begin{array}{c}\text{slots we must}\\ \text{fill per cycle}\end{array}}{\underbrace{T}}=\underset{\begin{array}{c}\text{total slots}\\ \text{to cover the stall}\end{array}}{\underbrace{L\cdot T}}$$

Now, how many warps supply those slots? A warp, once it fires an instruction, is itself busy/unavailable for a bit before it can fire again. In the cleanest case each warp can contribute one instruction per cycle-slot, so we need one warp per slot:

$$\boxed{W\ge L\cdot T}$$

---

Step 5 — The refinement: warps that can't fire every cycle

WHAT. Real warps cannot issue on every cycle — after issuing, a warp is busy in its own pipeline and can only fire again every $c$ cycles on average. So one warp does not cover one slot per cycle; it covers one slot every $c$ cycles.

WHY. Without this correction the count is too optimistic. We must divide by how often a single warp is available.

PICTURE. A single warp's track now has blocks spaced $c$ cycles apart (issue, then gap, then issue). To keep a column filled every cycle you clearly need more warps to interleave into those gaps.

$$W\ge \frac{L\cdot T}{c}\underset{\begin{array}{c}\text{cycles between}\\ \text{one warp’s issues}\end{array}}{\underbrace{c}}$$

---

Step 6 — The reality wall: SMs cap out at 32–64 warps

WHAT. Here is the twist. A real SM can hold at most 32–64 warps — nowhere near the 100 we just computed. So a single SM often cannot fully hide the latency by itself.

WHY. This edge case is not a footnote — it is the reason GPUs are built the way they are. When one SM can't hold enough warps, you don't build a deeper SM; you build many SMs and let the whole chip's occupancy do the hiding.

PICTURE. Left: what we need (100 warps). Right: what an SM has (say 64). The shortfall bar is the gap that forces the architecture to replicate SMs across the die.

---

Step 7 — The degenerate cases (never leave a scenario unshown)

WHAT. We check the boundary values of $L$, $T$, and $W$ so no reader hits a case we skipped.

WHY. A formula you trust is one whose extremes make sense. Let's test them.

PICTURE. Three mini-tracks: (a) $L=0$ no latency; (b) $W=1$ only one warp; (c) $W$ huge, more than enough.

Case	Meaning	What happens
$L=0$	instruction finishes instantly	$W\ge 0$ — even one warp keeps the core busy; no hiding needed.
$W=1$, $L$ large	single warp, long stall	core idles for the whole $L$ — the $400\times$ waste of Step 2 returns.
$W\ge L\cdot T/c$	enough (or more) warps	stall fully hidden; extra warps sit ready, no harm (until registers run out).
$T=0$	we want no work done	trivially satisfied — but pointless; we always want $T\ge 1$.

---

The one-picture summary

Everything above collapses into one image: a stall of width $L$, filled column-by-column by $L\cdot T/c$ interleaved warps, capped by what the SM can physically hold.

Recall Feynman retelling — say it back in plain words

Picture one worker who fires off a package order and then has to wait 400 seconds for it to arrive. If she just waits, she wastes 400 seconds. So a boss (the scheduler) says: "Don't wait — while your package is coming, do someone else's next step." To keep the boss handing out work every single second, we need enough workers so that at every moment at least one has a step ready. How many? Multiply the wait time (400) by how much work you want per second (1) → 400 slots to fill. If each worker can only act once every 4 seconds, divide → you need about 100 workers. But one room (an SM) only fits 64. So instead of building one giant room, the chip builds many rooms (many SMs). The lesson: a GPU hides slowness not by being clever, but by always having a huge crowd of ready work — that crowd is called occupancy, and this is why we keep it high.

Recall

What quantity does $L\cdot T$ count? ::: The total number of instruction-slots that must be filled by other warps to cover a stall of $L$ cycles at throughput $T$. Why divide by $c$? ::: Because a single warp can only issue once every $c$ cycles, so each warp covers one slot per $c$ cycles, not one per cycle. With $L=400$, $T=1$, $c=4$, how many warps? ::: $400/4=100$ warps. Why do real GPUs use many SMs instead of deeper pipelines? ::: One SM caps at ~32–64 warps, far below the ~100 needed, so the whole chip's warp count (across many SMs) supplies the latency hiding. How does SIMT hide latency differently from a CPU? ::: By massive multithreading (switch to a ready warp), not by speculation or out-of-order execution.

See also: SIMT (single instruction multiple thread) · Warp Divergence · GPU Memory Coalescing · CUDA Thread Hierarchy · Streaming Multiprocessor · Instruction-Level Parallelism · SIMD vs SIMT Comparison