Visual walkthrough — CUDA cores and execution model

Before line one, the only thing you need to accept from the parent is this: a GPU chops work into warps (bundles of 32 threads that march in lockstep), and a chip part called an SM (Streaming Multiprocessor) runs those warps. Everything else we build here.

---

Step 1 — What is a "cycle", and what does a warp do in one?

WHAT. A cycle is one tick of the GPU's clock — the smallest slice of time in which the hardware can do one thing. Picture a metronome: tick... tick... tick. On each tick, the SM can hand out one instruction.

A warp spends its life doing two kinds of things:

• Compute ticks: add, multiply — cheap, one tick each.
• A memory request: "go fetch a number from far-away DRAM." This is not one tick. The number lives far away, so the warp must wait.

WHY start here. Every symbol later (the letters $C$, $L$, $S$, $T$) is just a name for a count of ticks or a count of warps. If you don't feel what one tick buys you, the formula is gibberish.

PICTURE. One warp's timeline: a short green run of compute ticks, then a long empty stretch where it is stalled, sitting on its hands, waiting for memory.

---

Step 2 — The problem: an idle SM is wasted silicon

WHAT. During those $L$ waiting ticks, the compute hardware (the CUDA cores) has nothing to do for that one warp. The metronome keeps ticking; the cores keep being offered no work. That is pure waste.

WHY. The whole reason a GPU beats a CPU is that it never lets its arithmetic units go hungry (compare 6.2.01-GPU-vs-CPU-architecture). One warp alone cannot keep them fed — it stalls for $L$ ticks out of every $C+L$. So the SM's trick is: keep other warps around and switch to one of them the instant the first warp stalls.

PICTURE. Two warps stacked. When warp A stalls (empty stretch), warp B is already computing (green) right underneath — the SM slides its attention down to B and the cores stay busy.

---

Step 3 — Name the machine's issue power: the schedulers $S$

WHAT. How much fresh work can the SM start each tick? That is set by the warp schedulers. A scheduler is the traffic cop that, each tick, picks a ready warp and issues its next instruction. An SM has $S$ of them.

$\underset{\text{schedulers}}{\underbrace{S}}⟶\underset{\begin{array}{c}\text{warp-instructions}\\ \text{issued per tick}\end{array}}{\underbrace{S}}$

Read this literally: $S$ schedulers issue $S$ warp-instructions every single tick. If $S=4$, the SM launches 4 warps' worth of work per tick.

WHY name it. Because the amount of work the SM can start during the wait is what fills the empty stretch. More schedulers → more work started per tick → the gap fills faster.

PICTURE. A row of $S=4$ traffic cops, each waving one warp forward on the same tick.

---

Step 4 — Count the hole we must fill: $S\times L$ instruction-slots

WHAT. Here is the key act of counting. While warp A is stuck for $L$ ticks, how many warp-instructions could the SM have issued in that window?

$\text{issue slots during the stall}=\underset{\text{slots per tick}}{\underbrace{S}}\times \underset{\text{ticks of stalling}}{\underbrace{L}}=S\cdot L$

Term by term: each tick offers $S$ slots; the stall lasts $L$ ticks; multiply them. With $S=4$, $L=200$ that is $4\times 200=800$ empty slots screaming to be filled.

WHY multiply and not add? Because "slots per tick" and "number of ticks" are two independent dimensions of the same rectangle. This is the classic rate × time = amount move: cars-per-minute times minutes gives cars. Here, slots-per-tick times ticks gives slots.

PICTURE. A rectangle: width $L$ ticks across, height $S$ slots tall. Its area — the shaded grid of little boxes — is the number of instructions the SM has room to run while warp A waits.

---

Step 5 — Each warp only supplies $C$ instructions before it stalls

WHAT. We have $S\cdot L$ empty slots. Who fills them? Other warps. But each warp is not a bottomless supply: it hands over only $C$ compute instructions before it too hits a memory wall and stalls.

So one warp ≈ a bucket that pours exactly $C$ instructions into the hole.

$\text{instructions one warp gives}=C$

WHY this is the limiting piece. If warps were infinitely long streams of compute, you'd need very few of them. But real kernels touch memory often — small $C$. A stingy $C$ means each warp empties fast, so you need many warps.

PICTURE. The big $S\times L$ hole from Step 4, now being poured into by several warp-buckets, each bucket labelled "$C$ instructions" — you can literally count how many buckets it takes to fill the rectangle.

---

Step 6 — Divide: the number of warps we need

WHAT. Total hole ÷ what one warp supplies = number of warps.

$\boxed{T_{\text{required}}=\frac{\stackrel{\text{slots to fill}}{\overbrace{S\cdot L}}}{\underset{\begin{array}{c}\text{slots one}\\ \text{warp fills}\end{array}}{\underbrace{C}}}}$

• $T_{\text{required}}$ = how many warps must be resident so the cores never go hungry.
• Numerator $S\cdot L$ = the whole idle window (Step 4).
• Denominator $C$ = one warp's contribution (Step 5).

WHY division? "How many buckets of size $C$ fit inside a hole of size $S\cdot L$?" is exactly a division — the same question as "how many \5$notesmake$$800$?"($800/5 = 160$).

Plug in the parent's numbers ($S=4$, $L=200$, $C=10$):

$T_{\text{required}}=\frac{4\times 200}{10}=\frac{800}{10}=80\text{ warps.}$

PICTURE. The division shown as tiling: the $S\times L$ rectangle sliced into stacked $C$-length strips; count the strips → 80.

---

Step 7 — The reality check (why 80 means "impossible")

WHAT. The formula demands 80 warps. But an SM can only physically hold so many warps at once — the parent quoted a cap of about 64 resident warps ($2048$ threads $÷$ $32$ threads/warp $=64$).

$T_{\text{required}}=80>64=T_{\text{max resident}}$

So we cannot supply enough warps. The hole is bigger than everything we can pour into it. The cores will sit idle part of the time — latency is not fully hidden.

WHY this matters. This inequality is the birth certificate of every optimization in 6.2.04-occupancy-and-performance and 6.2.02-GPU-memory-hierarchy. You cannot raise the 64 (it's fixed hardware). You cannot lower $S$ (you want schedulers). $L$ is DRAM physics. The one knob you own is $C$ — do more math per memory trip. Coalescing (from the parent) and caching shrink effective $L$; higher arithmetic intensity raises $C$. Both push $T_{\text{required}}$ back under 64.

PICTURE. A bar chart: a tall "needed = 80" bar next to a shorter "available = 64" ceiling line, with the uncovered gap hatched — that hatched gap is wasted cycles.

---

Step 8 — Edge and degenerate cases

Every scenario the reader might meet, so nothing surprises them.

---

The one-picture summary

The whole story on one board: a warp stalls for $L$; the SM has $S$ slots/tick, so a hole of area $S\cdot L$ opens; each spare warp fills a strip of size $C$; you need $S\cdot L/C=80$ strips; but the ceiling is 64, so a gap of wasted cycles remains — and the only escape is to grow $C$.

Recall Feynman retelling — say it to a 12-year-old

Imagine a kitchen with a few cooks (the schedulers, $S$ of them). Each cook can start one dish per second (one tick). A dish takes a short bit of chopping ($C$ seconds of work) and then goes into a slow oven for a long bake ($L$ seconds). While one dish bakes, its cook is bored — unless there's another dish ready to chop. To never have a bored cook while $S$ cooks each wait through an $L$-second bake, you need enough dishes queued up: exactly $S\times L$ seconds of chopping to cover, and each dish only gives $C$ seconds of chopping — so you need $S\cdot L/C$ dishes. Plug in 4 cooks, 200-second bakes, 10 seconds chopping each: $4\times 200/10=80$ dishes. But the kitchen only has counter space for 64 dishes. So the cooks will stand around some of the time. The fix isn't more counters (that's fixed) — it's cooking dishes that need more chopping and less baking: raise $C$, and suddenly you need far fewer dishes. That's the entire art of writing fast GPU kernels.

Recall Quick self-test

What does $S$ count, in plain words? ::: The number of warp schedulers — how many warp-instructions the SM can issue per clock tick. Why is the hole to fill $S\times L$ and not $S+L$? ::: It's a rate times a time (slots-per-tick × ticks) = area of a rectangle; multiplying gives the total number of issue slots during the stall. With $S=4$, $L=200$, $C=10$, how many warps are required? ::: $4\times 200/10=80$ warps. Why is 80 a problem? ::: The SM caps at ~64 resident warps, and $80>64$, so latency can't be fully hidden. Which single variable can a programmer most directly raise to help, and what does raising it do? ::: $C$ (compute per warp before a stall); raising it lowers $T_{\text{required}}$ because it sits in the denominator.

---

See also: 7.3.01-parallel-programming-patterns · 6.1.03-SIMD-vs-SIMT · 6.2.04-occupancy-and-performance