Exercises — Streaming multiprocessors (SM)

A warp is always exactly $32$ threads (never a setting you change). So we divide: $\frac{256}{32}=8\text{ warps.}$ What we did: grouped threads into the hardware's fixed 32-wide bundles. Why: the warp scheduler never sees individual threads — it only ever issues to whole warps, so warps are the real unit of scheduling.

Using the SM latency ladder from the parent note: $\text{Registers }(1)<\text{Shared }(20\text{–}30)<\text{L2 }(\sim 200)<\text{Global }(400\text{–}800)\text{ cycles.}$ Why this order: each step moves the data physically farther from the CUDA cores. Registers sit inside the core; shared memory is on-chip SRAM with no tag lookup; L2 is shared across the chip; DRAM is off-chip entirely. Farther = more cycles. See GPU-memory-hierarchy.

The Special Function Unit (SFU). CUDA cores do plain integer/float add and multiply; Load/Store units move memory; SFUs are the dedicated hardware for transcendentals (sin, cos, exp, sqrt). Why it exists: these functions cost 20–100 cycles if emulated with basic arithmetic, but ~1–2 cycles on the SFU.

Step 1 — registers per block. Every thread needs its own copy: $256\times 128=32,768\text{ registers per block.}$ Step 2 — how many fit. Divide the SM's supply, then floor (you can't have a partial block resident): $\lfloor \frac{65,536}{32,768}\rfloor =2\text{ blocks.}$ Why floor: a block is all-or-nothing on an SM. If only enough registers remain for half a block, that block simply waits. See Occupancy-optimization.

Warps resident: $2\text{ blocks}\times \frac{256}{32}=2\times 8=16$ warps. Occupancy: $\frac{16}{64}=0.25=25\%.$ What this means: only a quarter of the scheduler's warp slots hold work, so latency-hiding is weak — see L3.

$\frac{65,536}{64}=1024\text{ threads.}$ But watch the second cap: if the SM also allows at most $64$ warps, that is $64\times 32=2048$ threads. Here $1024<2048$, so registers are the binding limit. Rule: actual occupancy is set by whichever resource runs out first.

The model: while the stalled warp waits, the scheduler must find a different ready warp to issue every cycle for $300$ cycles. Why one instruction per warp, then it stalls again: in a memory-bound kernel the next thing most warps need to do is itself another dependent global load (or an op that consumes the value still in flight). So after the scheduler issues one instruction to a warp, that warp is immediately waiting again on its own long-latency access. It therefore contributes only about one useful cycle of cover before dropping out. That is why we need many distinct warps rather than one warp used repeatedly: $\approx 300\text{ independent warps (one per cycle of latency).}$ Do 16 warps cover it? No — $16$ warps cover only about $16$ cycles of the $300$-cycle hole per scheduler pass. The remaining $\sim 284$ cycles the scheduler sits idle. Lesson: low occupancy directly starves latency-hiding. This is why the parent note screams "maximise resident warps."

No. Occupancy helps only by hiding latency. A compute-bound kernel is already keeping the CUDA cores busy; its bottleneck is arithmetic throughput, not waiting on memory. Extra warps just queue for the same busy ALUs. What changes vs. what doesn't: for a memory-bound kernel, going $16\to 32$ warps roughly doubles the latency you can hide and can noticeably help. For a compute-bound one, you may see near-zero improvement — you might even lose speed if the extra warps needed a register cut that spilled to memory. The analytical point: occupancy is a ceiling raiser for latency hiding, not a throughput multiplier.

Under SIMT, a warp issues one instruction for all $32$ lanes. When lanes split, the hardware runs both paths in sequence, masking the inactive half each time: $\underset{\text{path A, 16 active}}{\underbrace{10}}+\underset{\text{path B, 16 active}}{\underbrace{10}}=20\text{ issues.}$ Non-diverging cost is $10$. So this is a $2\times$ slowdown, and half the lanes are idle during each path. Key subtlety: cost scales with the number of distinct paths taken, not the number of threads. If all 32 took branch A, cost stays $10$ — no divergence penalty at all.

For each block size, compute the resident-block cap from every limit, take the minimum, then convert to warps.

Registers. Per thread cost $32$, so per SM the register file supports $65,536/32=2048$ threads regardless of block size → not the binding limit here (it exactly matches the $2048$-thread warp cap).

Shared memory. Each block wants $4$ KB and the SM has $96$ KB, so $\lfloor \frac{96\text{ KB}}{4\text{ KB}}\rfloor =24$ blocks max (the KB units cancel, leaving a pure count).

Block/thread caps, per block size:

• 128 threads (4 warps): thread cap $2048/128=16$ blocks; block cap $32$; SMEM $24$. Min $=16$ blocks $\Rightarrow 16\times 4=64$ warps → 100%.
• 256 threads (8 warps): thread cap $2048/256=8$ blocks; SMEM $24$; block cap $32$. Min $=8\Rightarrow 8\times 8=64$ warps → 100%.
• 512 threads (16 warps): thread cap $2048/512=4$ blocks; SMEM $24$. Min $=4\Rightarrow 4\times 16=64$ warps → 100%.

Answer: all three reach $64$ warps = $100\%$ occupancy, because this kernel is light on registers and shared memory. Insight: when no resource is scarce, occupancy is decided by the raw warp cap, and block size becomes a secondary choice (favour a size divisible into the caps and good for your shared-memory tiling — here 256 is a safe default).

Shared memory cap: $\lfloor \frac{96\text{ KB}}{32\text{ KB}}\rfloor =3$ blocks. Register cap: $2048/256=8$ blocks. Block cap: $32$. Min $=3$ blocks. Warps: $3\times 8=24$. Occupancy: $24/64=37.5\%$. Binding limit: shared memory. What to do: shrink the tile (smaller SMEM per block) or split the buffer, trading a bit of data reuse for more resident blocks. This is the classic Occupancy-optimization tug-of-war between reuse and parallelism.

(a) Diagnosis. From L2.1/L2.2: $128$ regs/thread → $\lfloor 65536/(256\times 128)\rfloor =2$ blocks → $16$ warps → $25\%$. A $400$-cycle stall needs $\sim 400$ warps to fully hide (L3.1 logic); $16$ warps cover $\sim 16$ cycles per scheduler pass, so the SM idles most of every stall. Root cause: register pressure caps resident warps far below what latency-hiding needs.

(b) After refactor to $64$ regs/thread. Registers per block: $256\times 64=16,384$. Blocks: $\lfloor 65,536/16,384\rfloor =4$. Warps: $4\times 8=32$. Occupancy: $32/64=50\%$.

(c) Latency hidden. Warps doubled $16\to 32$, so the hideable slice per scheduler pass roughly doubles ($\sim 16\to \sim 32$ cycles). Still far short of $400$, so occupancy alone won't fully cure this — you'd also pursue memory-coalescing and reuse (shared-memory tiling) to reduce the number of stalls. Mastery point: occupancy raises the ceiling for how much latency you can hide, but the complementary — and often bigger — lever is to reduce the latency itself, by coalescing accesses and reusing data in shared memory so there are fewer long stalls to hide in the first place.

To hold $8$ blocks resident: Registers. Total across all resident threads $\le 65,536$. Threads resident $=8\times 256=2048$. So per-thread budget: $\frac{65,536}{2048}=32\text{ registers/thread (max).}$ Shared memory. $8$ blocks share $96$ KB: $\frac{96\text{ KB}}{8}=12\text{ KB/block (max).}$ Check: $8$ blocks $\le$ block cap ($32$) ✓; warps $=64=$ cap ✓. Answer: stay at or below $32$ regs/thread and $12$ KB SMEM/block to hit $100\%$. Exceed either and resident blocks drop below $8$, and occupancy falls. This is the full budget-balancing act of Occupancy-optimization — and why Tensor Cores and thread block sizing are tuned hand-in-hand.

A warp is always 32 threads. Occupancy = active warps divided by ::: max warps per SM. Resident block count is the ::: minimum over all resource limits (registers, shared memory, warp cap, block cap). Extra warps hide ::: latency — they do not add ALU throughput. Divergence cost scales with the number of ::: distinct branch paths taken, not the number of threads.