Foundations — Streaming multiprocessors (SM)
This page assumes you know nothing. Before you read the parent note Streaming Multiprocessors, you must be able to read every symbol it throws at you. So we build each one from a picture. Nothing is used before it is drawn.
0. SM — the worker-factory itself
The picture: think of a GPU as a giant building filled with many identical small factories. Each factory (an SM) receives a pile of work and processes it on its own. This whole page is really an inventory list of what sits inside one such factory.
Why the topic needs it: the parent note uses the letters "SM" on almost every line. From here on, "SM" always means one worker-factory.
1. Thread — the smallest worker
The picture: imagine 32 identical workers standing in a row, each holding one number card. They all read the same recipe card ("add 5 to your number"), but each applies it to their own card.
Why the topic needs it: the whole point of a GPU is to run thousands of these workers at once. "Thread" is the atom — everything is counted in threads.
2. Warp — a bundle of 32 threads that move together
Why 32, and why bundle them? Because it is cheaper to build hardware that reads one instruction and fires it at 32 workers than to give every worker its own instruction-reader. One instruction, thirty-two workers.
The picture: the 32 workers from figure 1 are chained together. When the foreman shouts one command, all 32 obey on the same clock tick. This is the meaning of the parent note's word SIMT — Single Instruction, Multiple Thread: one instruction, many threads.
The tail: partial warps
Here means round up to the next whole number (the "ceiling").
When threads in a warp disagree: divergence
The picture: the foreman shouts "do path A". Threads that wanted path A work; the rest freeze. Then the foreman shouts "do path B" and the roles flip. The warp did A then B — twice the work. If all 32 threads happen to agree on the same path, there is no penalty.
Why the topic needs it: this is why the parent note warns that branching "costs performance". Divergence is the price of the flexibility that separates SIMT from rigid vector hardware. See Warp-scheduling for how these bundles take turns.
3. Block — a team of warps assigned to one SM
Why the 1024 cap? A block must fit its shared resources (registers, shared memory, warp slots) inside one SM, and the hardware reserves a fixed number of index bits for numbering threads. That budget currently tops out at 1024 threads = 32 warps per block. This number is an architectural limit — it has been 1024 for many recent generations, but such caps can differ across GPU architectures, so always check your specific hardware.
The picture: a block is a team. If a team has 256 threads, that team is warps. All 8 warps sit inside one factory (one SM).
Why the topic needs it: the block is the unit of assignment. The GPU hands out work one block at a time to SMs. See Thread-blocks.
4. Grid — all the blocks in one launch
The picture — the whole hierarchy at once: thread ⊂ warp ⊂ block ⊂ grid. Small to big.
Recall Read the hierarchy back
A single running instance of your code ::: a thread 32 threads locked to the same instruction ::: a warp Up to 1024 threads pinned to one SM ::: a block Every block in the launch ::: a grid
5. Register — a thread's private scratch pad
The picture: give each worker a small pocket. Numbers in the pocket are grabbed instantly (1 cycle — one clock tick). But there are only so many pockets total on the factory floor, and they must be split among everyone.
Why this creates a limit: if each thread demands many pockets, fewer threads fit. This is the parent note's whole "register pressure" story:
Here means round down to the next whole number (the "floor"): you cannot run a fraction of a thread or a fraction of a block, so any leftover capacity is simply unused.
6. Cycle — the GPU's heartbeat (the unit of time)
The picture: a metronome. Each tick, the factory can do one small step. Slow operations cost many ticks.
Why the topic needs it: every speed in the parent note is measured in cycles. Registers = 1 tick (instant). Global memory = 400–800 ticks (agonizingly slow). This gap is the single reason SMs are designed the way they are.
7. Global memory — the big, slow, shared warehouse
The picture: the registers are pockets on each worker; global memory is a giant warehouse across the yard. Fetching a box from the warehouse means a long walk. See GPU-memory-hierarchy for the full ladder of memories between them.
Why the topic needs it: global memory is the source of the long waits that the entire latency-hiding machinery (next section) exists to cover up.
8. Latency and latency-hiding — waiting, and hiding the wait
Why the topic needs it: almost every design choice in the parent note ("maximize resident warps") exists to hide latency.
9. The architectural limits — the factory's fixed capacities
Before occupancy makes sense, you need the SM's built-in ceilings. These are numbers baked into the silicon; you cannot exceed them.
Where do these come from? Each is a physical amount of hardware built into the SM: a fixed number of warp-slots (scheduler bookkeeping), a fixed number of block-slots, a fixed register file. Whichever cap you hit first decides how many warps actually fit. Because they are silicon amounts, they change between GPU generations — always look up your exact chip. The examples below use "max warps per SM = 64" as a representative figure.
10. Occupancy — how full the factory is
The picture: a factory with 64 warp-slots. If only 16 are filled, occupancy = . More filled slots → more warps ready to hide latency.
11. The physical parts of an SM (named boxes)
These are the hardware boxes the parent note lists. Now that you have the vocabulary, each is a one-liner. Warning: the counts below are rough, per-SM examples — every number here differs by GPU generation, so treat them as "typical order of magnitude", not fixed truth.
Prerequisite map
Equipment checklist
- What is an SM, in one sentence? ::: One self-contained worker-factory inside a GPU; a GPU is dozens of them tiled together.
- What is a thread, in one sentence? ::: One running copy of the program working on its own data.
- How many threads are in a warp, and why bundled? ::: 32 threads; they share one instruction so the hardware only needs one instruction-reader for the whole group.
- Convert a 256-thread block to warps. ::: warps.
- What happens if a block has 100 threads? ::: The hardware rounds up to warps; the last warp has 28 lanes switched off (masked, wasted).
- What is warp divergence? ::: When threads in one warp take different branch paths, the warp runs both paths one after the other — extra work.
- Where does a block run, and can it move? ::: On exactly one SM, from start to finish — it never migrates.
- Why is a block capped at 1024 threads? ::: A hardware limit: the block's resources must fit one SM and the index bits are fixed; it can vary across GPU architectures.
- What is global memory? ::: The GPU's large, slow (400–800 cycle) DRAM shared by all SMs and threads.
- How fast is a register vs global memory? ::: Register ≈ 1 cycle; global memory ≈ 400–800 cycles.
- What is a cycle? ::: One tick of the hardware clock — the GPU's unit of time.
- Define latency hiding in one sentence. ::: While one warp waits for data, the SM runs other ready warps so the wait costs no idle time.
- Name three architectural caps of an SM. ::: Max threads/block (~1024), max warps/SM (~48–64), max blocks/SM (~16–32) — all vary by architecture.
- Why do we floor the register/block ratios? ::: You cannot run a fraction of a thread or block, so leftover capacity is discarded.
- Give the occupancy formula. ::: Occupancy = active warps on SM ÷ max warps per SM.
- If 16 warps are resident and the max is 64, what is occupancy? ::: .
- Why do GPUs want thousands of threads? ::: To have enough ready warps to hide the long (hundreds of cycles) memory latency.