Streaming multiprocessors (SM): Visual walkthrough

This page builds the central result of the SM topic — why an SM needs many warps at once, and how "occupancy" measures whether it has enough — entirely from pictures. We assume you know nothing: not what a warp is, not what a cycle is, not why a GPU stalls. Every symbol is earned before it is used.

Step 1 — What is a "cycle"? The heartbeat of the SM

WHAT. Before any formula, we need the unit of time the SM lives in. A cycle is one tick of the SM's clock — the smallest chunk of time in which the hardware can do one small step of work.

WHY this idea first. Every cost on this page ("memory takes 400 cycles", "the scheduler issues 1 instruction per cycle") is measured in cycles. If we don't nail down the tick, no later number means anything. We measure everything in cycles because that is the only clock the SM obeys.

PICTURE. Below, time runs left to right in equal ticks. On each tick the SM can start one instruction. That is our ruler.

Step 2 — A thread, and why a single thread wastes the machine

WHAT. A thread is one running copy of your program — one worker following one recipe. It computes, and sometimes it needs a value from global memory (the big, far-away DRAM off to the side of the chip).

WHY. We start with one thread to see the problem in its purest form. When a thread asks global memory for a number, the answer does not come back this cycle — it comes back after a long delay we call latency, written $L$ .

PICTURE. One thread issues a load, then stalls — it sits frozen doing nothing for $L$ cycles because it literally cannot continue without the number it asked for. Every red tick is a wasted heartbeat.

Look at the red stripe: for a load of $L = 8$ cycles, the SM does 1 cycle of useful work and 8 cycles of nothing. That is the disaster we must fix.

Step 3 — A warp: 32 threads glued to one instruction

WHAT. Threads are not scheduled one at a time. The hardware bundles 32 threads into one unit called a warp. The warp scheduler issues one instruction and all 32 threads in the warp run it together — this is SIMT (Single Instruction, Multiple Threads).

WHY 32, and why bundle at all? Bundling lets one instruction-fetch drive 32 lanes of arithmetic (the CUDA cores), so you pay the control cost once for 32 workers. The number 32 is fixed hardware. For timing, the key fact is: a warp stalls as a unit — if the warp does a load, all 32 threads wait the same $L$ cycles together. So for latency-hiding math, one warp behaves exactly like the single stalled worker of Step 2.

PICTURE. 32 lanes, one instruction pointer feeding all of them. When the warp loads, the whole block of 32 goes red together.

Since a warp stalls as one unit, from now on our unit of "job" is the warp, not the thread.

Step 4 — Thread-blocks: how warps are grouped and delivered to the SM

WHAT. You don't hand the GPU loose warps — you hand it thread-blocks. A thread-block is a batch of threads (up to 1024) that the hardware assigns to one SM and runs there from start to finish (no migration). Because a warp is 32 threads (Step 3), a block of $T$ threads is automatically chopped into

warps in one block w_{block} = \frac{T threads per block}{threads per warp 32}

$w_{block}$ — how many warps one block becomes.
$T$ — the block size you choose when launching the kernel.
$32$ — fixed warp width from Step 3.

WHY introduce the block now. The counting in the next steps talks about "how many warps are resident on the SM." Warps arrive only in block-sized bundles, so before we count warps we must know the bundle. A 256-thread block, for example, is $256/32 = 8$ warps — and the SM stacks several such blocks side by side to build up its crowd of warps.

PICTURE. One block = a stack of 8 warps; the SM holds several blocks at once, forming the warp pool the scheduler draws from.

Step 5 — The trick: switch to another warp while one waits

WHAT. Here is the whole idea of a GPU. When warp 0 stalls on its load, the scheduler does not sit idle — it switches instantly to warp 1, issues its instruction, then warp 2, and so on. Warp switching is free (no saving/restoring — every warp's state lives in the register file permanently while resident).

WHY. This is called latency hiding. We hide the wait of warp 0 behind the useful work of other warps.

We must be precise about the switching rate, because two different "per cycle" quantities lurk here. Let

schedulers per SM S \times instructions each scheduler issues per cycle i = warp-issues advanced per cycle r

$S$ — number of independent warp schedulers in the SM (real SMs have 2–4).
$i$ — instructions one scheduler issues per cycle (usually 1; "dual-issue" makes it 2).
$r$ — the product: how many warps get advanced each cycle across the whole SM. $r$ counts warp-issues per cycle, not instructions inside one warp. With one scheduler issuing once, $r = 1$ : exactly one warp advances per cycle. Note the letter $r$ (lower-case, "rate") is our per-cycle warp rate; do not confuse it with $r_{thread}$ introduced in Step 7, which is registers per thread — a completely different quantity that just happens to share the letter.

PICTURE. Watch the timeline: while warp 0's red wait runs across the top, warps 1, 2, 3, … each get an orange tick underneath. The waiting is no longer wasted — it is covered by neighbours.

Step 6 — Counting exactly how many warps we need

WHAT. Now the derivation. We have:

a stall of length $L$ cycles (Step 2),
a scheduler machinery that advances $r$ warps per cycle (Step 5).

WHY this arithmetic. We want warp 0's data to be back by the time we've cycled through all the other ready warps once, so we never run out of work. If each cycle advances $r$ warps and the hole is $L$ cycles wide, the number of distinct warps we can keep in flight during that hole — and therefore the number we need to fill it — is:

ready warps to fill the hole W_{needed} = cycles the load stalls L \times warp-issues per cycle (Step 5) r

$W_{needed}$ — the number of resident, ready warps we must keep on the SM.
$L$ — the stall length in cycles (the hole to fill), from Step 2.
$r$ — warp-issues advanced per cycle, from Step 5. With one scheduler, $r = 1$ and $W_{needed} = L$ .
$⌈ \dots ⌉$ — the ceiling (round up to the next whole number). Why round up, not down? Because you cannot have a fraction of a warp, and if you had, say, $L \times r = 12.3$ , then $12$ warps would leave a $0.3$ -cycle sliver of the hole uncovered — the SM would still stall. Rounding up to $13$ guarantees the hole is fully tiled. Always ceiling for a "how many do I need to be safe" count.

PICTURE. A bar of length $L$ that we must tile with warp slots. Too few warps → a gap of idle cycles at the end (the SM stalls). Just enough → the tiles reach warp 0's return with no gap.

Step 7 — The limit: the SM caps how many warps can be resident

WHAT. We would love $W_{have} = 300$ . But an SM can only hold so many warps at once — call this the hardware ceiling == $W_{m a x}$ == (typically 48–64 on real GPUs). You physically cannot exceed it, and three separate resources can each pull the actual limit below $W_{m a x}$ .

WHY three limits. Every resident warp consumes registers, its block consumes shared memory and a block slot. Whichever runs out first decides how many warps fit. So the real resident count is the minimum over three ceilings:

W_{have} = min (W_{m a x}, W_{reg}, W_{smem}, W_{slots})

PICTURE. The register file sliced into warp-sized wedges — fat threads (many registers each) → few warps fit; thin threads → many warps fit.

Formula The three resource ceilings

First, define the per-thread register count we are about to use:

$r_{thread}$ — registers per thread (a whole different quantity from the rate $r$ of Step 5; the subscript "thread" is there precisely to keep them apart). The compiler tells you this number for your kernel.

(1) Registers — the register file $R_{SM}$ split among warps, each warp needing $32 \times r_{thread}$ registers (32 threads, each wanting $r_{thread}$ registers): $W_{reg} = ⌊ \frac{R _{SM}}{r _{thread} \times 32} ⌋$ $R_{SM}$ = total 32-bit registers (e.g. $65, 536$ ); $r_{thread}$ = registers per thread; $\times 32$ turns per-thread into per-warp. Here we round down (floor $⌊ \dots ⌋$ ): a warp that only partly fits cannot run, so we keep only whole warps that fully fit.

(2) Shared memory — total shared memory $M_{SM}$ split among blocks (each block needs $m_{block}$ bytes), then converted to warps: $W_{smem} = ⌊ \frac{M _{SM}}{m _{block}} ⌋ \times w_{block}$ $M_{SM}$ = shared memory per SM (≈48–100 KB, see GPU-memory-hierarchy); $m_{block}$ = bytes one block requests; $w_{block}$ = warps per block (Step 4).

(3) Block slots — the SM holds at most $B_{m a x}$ blocks (typically 16–32): $W_{slots} = B_{m a x} \times w_{block}$ $B_{m a x}$ = max resident blocks; multiply by warps per block to get warps.

The figure above shows the three ceilings as three bars; the shortest bar is the one that actually limits you.

Step 8 — Warp divergence: not every lane does useful work

WHAT. So far we assumed all 32 threads in a warp always do useful work together. But a warp runs one instruction for all 32 lanes (Step 3). If the threads hit an if/else and take different branches, the hardware must run both paths one after the other, switching off (masking) the lanes that don't belong to the current path.

WHY this matters here. During each masked path, some lanes sit idle — they occupy the warp's issue slot but produce nothing. This is warp divergence. It does not change how many warps are resident, but it lowers the effective work per issue: a warp that is only half-active still costs a full issue slot, so your real throughput — and the benefit you get from occupancy — drops.

PICTURE. A warp split by an if (threadIdx < 16): first the low 16 lanes run path A (top 16 masked, greyed out), then the high 16 lanes run path B (low 16 masked). Two issue slots spent, only 16 lanes live in each.

Step 9 — Occupancy: the score that ties it all together

WHAT. We now define the single number the whole topic revolves around.

WHY this is the metric. Steps 6–7 showed the tension: latency hiding wants many warps ( $W_{needed} = ⌈ L \times r / p ⌉$ ), but hardware caps them at $W_{m a x}$ , and your kernel's register/shared-memory/block appetite may keep $W_{active}$ below even that cap. Occupancy is exactly "what fraction of the SM's latency-hiding capacity are you actually using?"

PICTURE. A gauge: the SM's $W_{m a x}$ slots, some filled (active), some empty (wasted). The needle is occupancy.

The one-picture summary

This final figure compresses Steps 1→9 into one diagram: the cycle ruler (Step 1), a warp stalling for $L$ (Steps 2–3), blocks delivering warps (Step 4), the scheduler filling the hole at rate $r$ (Step 5), the count $W_{needed} = ⌈ L r / p ⌉$ (Step 6), the three ceilings capping $W_{have}$ (Step 7), divergence wasting lanes (Step 8), and the occupancy gauge that scores it all (Step 9).

Recall Feynman retelling — say it back in plain words

The SM's clock ticks; each tick is a cycle. A thread does work, then asks slow far-away memory for a number and freezes for hundreds of cycles waiting. Threads come glued in bundles of 32 called warps, and warps arrive on the SM inside thread-blocks — a block of 256 threads is 8 warps, and the SM stacks several blocks to build a crowd of ready warps. The scheduler's trick: while one warp is frozen, advance a different ready warp — $r$ of them per cycle across all schedulers. To fill a hole $L$ cycles wide you need about $⌈ L \times r ⌉$ ready warps, rounded up so no sliver of the hole is left uncovered (fewer if each warp keeps several loads in flight, which is MLP). But the SM can only hold so many warps — registers, shared memory, and block slots each cap it, and the tightest one wins; that surviving count is $W_{have}$ , the very same number the occupancy formula calls $W_{active}$ . Occupancy is the fraction "warps you actually have ÷ warps the SM could hold." Too small a crowd and it runs out before the memory answer arrives, so the SM sits idle. And even a full crowd can waste work: if threads in one warp branch differently (divergence), the warp runs both paths with half its lanes switched off, so occupancy alone never guarantees full throughput. But if your kernel rarely waits on memory, you don't need a big crowd — so occupancy is a tool, not a trophy.

Recall Quick self-check

A cycle is ::: one tick of the SM clock — the unit all latencies are measured in. A warp is ::: 32 threads that execute one instruction together and stall together (SIMT). A thread-block is ::: a batch of up to 1024 threads assigned to one SM; it splits into $T /32$ warps. $r$ in $W_{needed} = ⌈ L r ⌉$ means ::: warp-issues advanced per cycle across all schedulers ( $S \times i$ ), not instructions inside one warp, and not registers per thread. $r_{thread}$ means ::: registers per thread (used in the register ceiling) — a different quantity from the rate $r$ . Why ceiling (round up) in $W_{needed}$ ? ::: you cannot have a fraction of a warp, and rounding down would leave a sliver of the stall uncovered, so the SM would still stall. $W_{have}$ denotes ::: the warps actually made resident on the SM right now — the same number the occupancy formula calls $W_{active}$ . To hide an $L$ -cycle stall with one scheduler and one load per warp you need about ::: $L$ resident warps. Warp divergence is ::: threads in one warp taking different branches, so the warp runs both paths serially with some lanes masked off — wasting issue slots. Lane efficiency is ::: average active lanes per issue ÷ 32; below 1 means occupancy overstates real throughput. Occupancy equals ::: active warps ÷ max warps the SM can hold. The three resource ceilings on resident warps are ::: registers, shared memory, and block slots — the tightest wins. Keeping several loads in flight per warp (MLP) means ::: fewer warps needed, $W_{needed} = ⌈ L r / p ⌉$ .