Streaming multiprocessors (SM): Worked examples — Hardware deep dive

This page is the "grind through every case" companion to Streaming multiprocessors (SM). The parent note gave you the formulas; here we torture them with every kind of input: normal numbers, zero, degenerate limits, a word problem, and an exam trap. Read the scenario matrix first, then guess before each solution.

Before any symbol, here are the only quantities we use on this page. Each is a plain count you could tally by hand.

Definition The vocabulary (defined once, used everywhere)

Thread — one runner executing your kernel. The smallest unit.
Warp — a fixed bundle of 32 threads that move in lockstep. The hardware never schedules fewer than a whole warp.
Block — a group of threads you request (up to 1024). A whole block lives on one SM, start to finish. See Thread-blocks.
Register — a tiny per-thread scratch slot inside the SM. Each thread grabs a fixed number; they come from one shared pool (~65,536 32-bit registers per SM).
Max warps/SM — a hardware ceiling on resident warps (we use 64 = 2048 threads on modern SMs).
Max blocks/SM — a separate hardware ceiling on how many resident blocks an SM will hold at once (commonly 16 or 32). Even with plenty of registers and warp slots, you cannot exceed this block count.
$N$ (resident warps) — the count of warps actually living on one SM right now; this is the number we plug into "active warps." We reuse the symbol $N$ in Example 6.
Occupancy — $\frac{active warps}{max warps/SM}$ , a number between 0 and 1.

The scenario matrix

Every SM sizing question lands in one of these cells. The examples below are labelled [Cell N] so you can see the whole space is covered.

Cell	Case class	The limiting resource	Trap it hides
1	Register-bound, clean divide	Registers per thread	none — warm-up
2	Register-bound, non-integer divide	Registers (floor kicks in)	you must round down, wasting slots
3	Shared-memory-bound	KB of Shared-memory per block	shared mem, not registers, is the cap
4	Warp-count-ceiling-bound	Max warps/SM (64)	resources are plenty — hardware caps you
5	Zero / degenerate input	occupancy $\to 0$	a block that cannot fit at all
6	Limiting behaviour	latency vs. warps ( $N \to \infty$ )	more warps stops helping past a point
7	Word problem (real workload)	blocks across SMs	the "leftover wave" of blocks
8	Exam twist	[[Warp-scheduling	divergence]] + occupancy interact

The single figure below is the mental model every cell shares: the SM is a bucket with several lids, and the lowest lid sets the water level. Keep this picture in mind — in every worked example below, we compute the height of each lid and read off the lowest one.

Why the floor and why the min? The floor (⌊ ⌋) exists because a partially-resident block is useless — it either fully fits or it does not. The min exists because each ceiling is a lid on the bucket in the figure above: you are blocked by whichever lid sits lowest, i.e. whichever resource runs out first. The fourth lid, $B_{m a x}$ , is a distinct hardware limit: even with registers and warp slots to spare, an SM refuses to host more than $B_{m a x}$ blocks at once.

Example 1 — Register-bound, clean divide `[Cell 1]`

Forecast: Guess now — is this SM full, half, or starved?

Registers per block $= 256 \times 32 = 8192$ . Why this step? A block must reserve registers for all its threads at once; that reservation is what competes for the pool.
Register-limited blocks $= ⌊ 65536/8192 ⌋ = 8$ . (This is the register lid in the s01 figure.) Why this step? Each block needs 8192 registers; the pool of 65536 divides evenly, so 8 blocks fit with nothing left over.
Warp-ceiling blocks $= ⌊ 64 \times 32/256 ⌋ = ⌊ 2048/256 ⌋ = 8$ . (The warp lid.) Why this step? Independently, the hardware caps resident warps at 64; each block is $256/32 = 8$ warps, so at most 8 blocks by that rule too.
Blocks/SM $= min (8, 8) = 8$ . Active warps $= 8 \times 8 = 64$ . Why this step? Both lids sit at 8 — the water rises to 8, no waste. Multiply blocks by warps-per-block to count resident warps.
Occupancy $= 64/64 = 1.0 = 100%$ .

Verify: 64 active warps is exactly $W_{m a x}$ , and 8 blocks × 256 = 2048 threads = the SM's max. Two independent ceilings agreeing at 100% is the sanity check.

Example 2 — Register-bound, non-integer divide `[Cell 2]`

Forecast: The pool is the same; each thread grabs 4× more registers. Does occupancy drop to exactly 25%, or does rounding make it worse?

Registers per block $= 256 \times 128 = 32768$ . Why this step? Same reservation logic — more registers per thread multiplies the block's demand.
Register-limited blocks $= ⌊ 65536/32768 ⌋ = 2$ . Why this step? $65536/32768 = 2.0$ exactly here, so the floor changes nothing — but always apply it.
Active warps $= 2 blocks \times 8 warps = 16$ .
Occupancy $= 16/64 = 0.25 = 25%$ .

The rounding trap (why this is Cell 2): now bump to 130 registers/thread. Registers/block $= 256 \times 130 = 33280$ , and $⌊ 65536/33280 ⌋ = ⌊ 1.969 ⌋ = 1$ . A tiny increase drops you from 2 blocks to 1 block → 8 warps → $8/64 = 12.5%$ . Half your occupancy vanished over 2 extra registers, because the floor threw away the fractional block — the register lid dropped just below 2.

Verify: $2 \times 32768 = 65536 \leq 65536$ ✓ (fits), but $2 \times 33280 = 66560 > 65536$ ✗ (does not), confirming the second case can host only 1 block.

Example 3 — Shared-memory-bound `[Cell 3]`

Forecast: Registers say "tons of blocks fit." Will registers win, or does shared memory clamp you?

Register-limited blocks $= ⌊ 65536/ (256 \times 2)⌋ = ⌊ 65536/512 ⌋ = 128$ . Why this step? Establish the register lid so we can prove it is not the binding one.
Shared-memory-limited blocks $= ⌊ \frac{98304 bytes}{49152 bytes/block} ⌋ = 2 blocks$ . Why this step? Convert to bytes first for honest unit consistency: $96 KB = 96 \times 1024 = 98304$ bytes and $48 KB = 49152$ bytes. Then bytes ÷ bytes/block = blocks (the KB would cancel too, but converting removes any doubt). Only two 48-KB chunks fit in 96 KB. This is the shared-memory lid.
Warp-ceiling blocks $= ⌊ 2048/256 ⌋ = 8$ .
Blocks/SM $= min (128, 2, 8) = 2$ . Active warps $= 2 \times 8 = 16$ . Why this step? Shared memory is the lowest lid in the s01 figure — it decides, even though registers could host 128 blocks.
Occupancy $= 16/64 = 25%$ .

Verify: $2 \times 49152 bytes = 98304 bytes \leq 98304$ ✓ fits; a 3rd block would need $147456 bytes > 98304$ ✗. Shared memory, not registers, is confirmed as the ceiling — the whole point of this cell.

Example 4 — Warp-ceiling-bound `[Cell 4]`

Forecast: Everything is cheap. So occupancy is 100%… right?

Register-limited blocks $= ⌊ 65536/ (64 \times 16)⌋ = ⌊ 65536/1024 ⌋ = 64$ . Why this step? Compute the register lid first so we can prove registers are not what limits this kernel — each block needs only 1024 registers, so 64 blocks would fit register-wise.
Warp-ceiling blocks $= ⌊ 2048/64 ⌋ = 32$ . Why this step? Each block is only $64/32 = 2$ warps, so the 64-warp ceiling allows 32 blocks — fewer than registers allow. The warp lid sits lower than the register lid.
Blocks/SM $= min (64, 32) = 32$ . Active warps $= 32 \times 2 = 64$ . Why this step? Take the lowest lid (32, the warp ceiling), then multiply resident blocks by warps-per-block ( $32 \times 2$ ) to count how many warps actually live on the SM.
Occupancy $= 64/64 = 100%$ . Why this step? Divide active warps by $W_{m a x}$ ; here the warp ceiling and the occupancy definition share the same 64, so a warp-bound kernel still reaches 100%.

The subtle point: you reached 100%, but you are pinned by the warp/block ceilings, not resources. If the SM also capped resident blocks at $B_{m a x} = 16$ , you'd get $16 \times 2 = 32$ warps $= 50%$ despite mountains of free registers. Small blocks can waste the warp budget through the $B_{m a x}$ block-count lid.

Verify: $32 \times 2 = 64$ warps $= W_{m a x}$ ✓, and $32 \times 1024 = 32768 \leq 65536$ registers ✓ (registers to spare, confirming they are not the limit). With $B_{m a x} = 16$ the min would be $min (64, 32, 16) = 16$ blocks → 32 warps → 50%.

Example 5 — Zero / degenerate input `[Cell 5]`

Forecast: Guess whether occupancy is small-but-positive, or literally zero.

Registers per block $= 512 \times 256 = 131072$ . Why this step? Compute demand exactly as always — no shortcut for extreme inputs.
Register-limited blocks $= ⌊ 65536/131072 ⌋ = ⌊ 0.5 ⌋ = 0$ . Why this step? The block needs more registers than the whole SM owns. The floor of 0.5 is 0 — you cannot host half a block. The register lid has dropped to the floor of the bucket.
Blocks/SM $= min (0, \dots) = 0$ . Active warps $= 0$ . Why this step? Any min that includes 0 is 0; with zero resident blocks there are zero resident warps.
Occupancy $= 0/64 = 0%$ — a launch failure (too many resources requested), the degenerate case.

Why this matters: occupancy 0 is a real, catchable outcome, not an abstraction. The fix is the same as always: cut registers per thread, or shrink the block so $T \cdot r \leq R_{total}$ .

Verify: the largest block that fits at 256 regs/thread satisfies $T \times 256 \leq 65536 \Rightarrow T \leq 256$ . So 512 threads is over the edge → 0 blocks. Boundary confirmed.

Example 6 — Limiting behaviour: warps vs. latency `[Cell 6]`

Forecast: Does throughput keep climbing forever, or flatten?

The key ratio is how many independent warps you need to keep the pipeline busy during a 300-cycle stall. While one warp waits, the scheduler runs others.

Warps needed to fully hide the stall $\approx$ latency in cycles $= 300$ (one warp issued per cycle covers one cycle of waiting). Why this step? Each cycle you feed one warp; to never idle across 300 cycles you need ~300 ready warps.
Efficiency model $η (N) = min (N, 300) /300$ — useful fraction of a cycle actually filled. Why this step? Below 300 warps you cover $N$ of the 300 cycles; above 300 you already fill every cycle.
Limiting behaviour: for $N \geq 300$ , $η = 1$ (saturated). Extra warps add nothing — this is the plateau. Why this step? Once every cycle is busy, more warps cannot make a cycle "more than full."
Reality check for one SM: $W_{m a x} = 64 ≪ 300$ , so a single SM with one scheduler reaches $η = 64/300 \approx 21%$ — it cannot fully hide a 300-cycle stall alone. Multiple schedulers per SM (typically 4) and short average stalls close the gap.

Verify: At $N = 64$ : $η = 64/300 \approx 0.213$ . At $N = 300$ : $η = 1.0$ . At $N = 600$ : still $1.0$ (plateau). The curve is a rising line that flattens at $N = 300$ .

Example 7 — Word problem: a real launch across the GPU `[Cell 7]`

Forecast: With more SMs (80) than blocks (40), guess whether the GPU is over- or under-utilised.

First, a symbol we now need. Alongside the floor $⌊ ⌋$ (round down), we use its mirror:

Total block-slots the GPU offers at once $= 80 SMs \times 8 blocks/SM = 640$ slots. Why this step? This is the GPU's combined capacity for one wave; we compare our 40 blocks against it to see if they all fit simultaneously.
SMs that receive a block $= min (40, 80) = 40$ . Why this step? The grid distribution unit hands each block to one SM, one block per SM first; with only 40 blocks, just 40 SMs light up.
Idle SMs $= 80 - 40 = 40$ — half the GPU sits idle this launch. Why this step? Blocks are the unit of distribution; too few blocks starves SMs no matter how good each block's occupancy is.
Waves $= ⌈ 40/640 ⌉ = ⌈ 0.0625 ⌉ = 1$ wave. Why this step? A "wave" is one round of concurrently resident blocks. All 40 blocks fit in the 640 available slots, so a single wave clears the whole grid — and the ceiling turns the fraction 0.0625 into 1 whole wave.

Lesson: occupancy per SM was 100% in Example 1, yet the whole-GPU utilisation here is only $40/80 = 50%$ because the grid was too small. Launch many more blocks than SMs.

Corner case — a big grid that needs multiple waves: relaunch the same kernel as 2000 blocks. Now block-slots (640) are exceeded, so waves $= ⌈ 2000/640 ⌉ = ⌈ 3.125 ⌉ = 4$ waves. The first 3 waves each fill all 640 slots ( $3 \times 640 = 1920$ blocks); the 4th wave holds only $2000 - 1920 = 80$ blocks, lighting just $⌈ 80/8 ⌉ = 10$ SMs while the other 70 idle during that final wave. Full waves are efficient; the ragged tail wave wastes SMs — a reason to size grids so the tail is small or absent.

Verify (small grid): total threads $= 40 \times 256 = 10240$ ; idle SM fraction $= 40/80 = 0.5$ ; one wave since $40 \leq 640$ block-slots. Verify (big grid): $⌈ 2000/640 ⌉ = 4$ waves, tail holds $2000 - 3 \times 640 = 80$ blocks, using $⌈ 80/8 ⌉ = 10$ SMs. ✓

Example 8 — Exam twist: divergence eats your occupancy win `[Cell 8]`

Forecast: Occupancy is equal — so surely they tie?

Kernel X cost: each instruction runs once per warp, all 32 lanes active. Baseline $= 1 \times$ work. Why this step? No divergence means one pass covers all 32 threads at once — the ideal case.
Kernel Y, first path: the warp executes the if branch with only 16 lanes active (threads 0–15), the other 16 lanes masked off and idle. Why this step? Threads in a warp share one instruction pointer (the SIMT rule from the parent note), so lanes taking the other branch must wait — they cannot run a different instruction in the same cycle.
Kernel Y, second path: the warp then executes the else branch with threads 16–31 active and threads 0–15 masked off. Why this step? The two branches cannot overlap; the hardware serialises them into two passes over the branched region.
Slowdown factor for the branched region $= 2 \times$ : the region is traversed twice, and in each pass half the lanes do nothing useful. Why this step? Two serial passes over code that a non-divergent warp would cover in one pass is exactly double the instruction issues.
Conclusion: Kernel X finishes first. Kernel Y is ~2× slower on the divergent region despite identical 100% occupancy.

The trap: occupancy measures how many warps are resident, not how efficiently each warp's lanes are used. High occupancy + high divergence can lose to lower occupancy + no divergence.

Verify: effective lane-utilisation of Y in the branch $= (16 + 16) / (32 + 32) = 32/64 = 0.5$ , i.e. 50% of issued lane-slots are useful → runtime ratio $1/0.5 = 2.0 \times$ . Occupancy of both $= 1.0$ . ✓

Recall Self-check: which ceiling binds?

Kernel: 100 regs/thread, 256 threads/block, 32 KB shared/block. SM: 65536 regs, 64 KB shared, $W_{m a x} = 64$ . Which resource limits occupancy? ::: Registers: $⌊ 65536/ (256 \cdot 100)⌋ = 2$ ; shared $⌊ 64/32 ⌋ = 2$ ; warps $⌊ 2048/256 ⌋ = 8$ . Min is 2 (a tie between registers and shared memory) → 16 warps → 25%. Why can two kernels at equal occupancy run at different speeds? ::: Occupancy counts resident warps, not lane efficiency; warp divergence idles lanes without changing occupancy. In Example 7, why is the GPU only 50% utilised despite 100% per-SM occupancy? ::: Too few blocks (40) for the number of SMs (80); blocks are the distribution unit, so 40 SMs stay idle.

See also: CUDA-cores · Tensor-cores · Hinglish version