Streaming multiprocessors (SM): Worked examples — Hinglish deep dive

6.2.2 · D3 · Hardware › GPU Architecture › Streaming multiprocessors (SM)

Ye page Streaming multiprocessors (SM) ki "har case grind karo" companion hai. Parent note ne tumhe formulas diye; yahan hum unhe har tarah ke input se torture karte hain: normal numbers, zero, degenerate limits, ek word problem, aur ek exam trap. Pehle scenario matrix padho, phir har solution se pehle guess karo.

Kisi bhi symbol se pehle, yahan sirf wahi quantities hain jo hum is page par use karte hain. Har ek ek plain count hai jo tum haath se bhi ginn sakte ho.

Definition Vocabulary (ek baar define ki, har jagah use hogi)

Thread — ek runner jo tumhara kernel execute karta hai. Sabse chhoti unit.
Warp — 32 threads ka ek fixed bundle jo lockstep mein chalta hai. Hardware kabhi bhi ek poore warp se kam schedule nahi karta.
Block — threads ka ek group jo tum request karte ho (1024 tak). Ek poora block ek SM par rehta hai, shuru se ant tak. Dekho Thread-blocks.
Register — SM ke andar ek tiny per-thread scratch slot. Har thread ek fixed number leta hai; ye sab ek shared pool se aate hain (~65,536 32-bit registers per SM).
Max warps/SM — resident warps par ek hardware ceiling (hum modern SMs par 64 = 2048 threads use karte hain).
Max blocks/SM — ek alag hardware ceiling jo batata hai ek SM ek saath kitne resident blocks hold karega (commonly 16 ya 32). Plenty of registers aur warp slots hone ke bawajood, tum is block count ko exceed nahi kar sakte.
$N$ (resident warps) — warps ki count jo abhi ek SM par actually live hain; yahi number hai jo hum "active warps" mein plug karte hain. Hum symbol $N$ ko Example 6 mein reuse karte hain.
Occupancy — $\frac{active warps}{max warps/SM}$ , ek number 0 aur 1 ke beech.

Scenario matrix

Har SM sizing question in cells mein se ek mein aata hai. Neeche ke examples [Cell N] label kiye hain taaki tum dekh sako ki poora space cover kiya gaya hai.

Cell	Case class	Limiting resource	Chhupa hua trap
1	Register-bound, clean divide	Registers per thread	koi nahi — warm-up
2	Register-bound, non-integer divide	Registers (floor kick karta hai)	tumhe neeche round karna hai, slots waste hote hain
3	Shared-memory-bound	KB of Shared-memory per block	shared mem, registers nahi, cap hai
4	Warp-count-ceiling-bound	Max warps/SM (64)	resources plenty hain — hardware cap lagata hai
5	Zero / degenerate input	occupancy $\to 0$	ek block jo fit hi nahi ho sakta
6	Limiting behaviour	latency vs. warps ( $N \to \infty$ )	ek point ke baad zyada warps help karna band kar dete hain
7	Word problem (real workload)	blocks across SMs	blocks ki "leftover wave"
8	Exam twist	[[Warp-scheduling	divergence]] + occupancy interact karte hain

Neeche ki single figure woh mental model hai jo har cell share karta hai: SM ek bucket hai jiske kaafi lids hain, aur sabse neechi lid water level set karti hai. Yeh picture dimag mein rakho — har worked example mein hum har lid ki height compute karte hain aur sabse neechi padhte hain.

Floor kyun aur min kyun? Floor (⌊ ⌋) isliye hai kyunki partially-resident block bekar hai — ya to poora fit hota hai ya nahi. min isliye hai kyunki har ceiling upar ki figure mein bucket ki ek lid hai: tum usi lid se block hote ho jo sabse neechi hai, yani jo resource pehle khatam hoti hai. Chawthi lid, $B_{m a x}$ , ek alag hardware limit hai: registers aur warp slots spare mein hone ke bawajood, ek SM $B_{m a x}$ se zyada blocks ek saath host karne se mana kar deta hai.

Example 1 — Register-bound, clean divide `[Cell 1]`

Forecast: Abhi guess karo — kya yeh SM full hai, half hai, ya starved hai?

Registers per block $= 256 \times 32 = 8192$ . Yeh step kyun? Ek block ko apne saare threads ke liye ek saath registers reserve karne padte hain; wahi reservation pool ke liye compete karti hai.
Register-limited blocks $= ⌊ 65536/8192 ⌋ = 8$ . (Yeh s01 figure mein register lid hai.) Yeh step kyun? Har block ko 8192 registers chahiye; 65536 ka pool evenly divide hota hai, toh 8 blocks fit hote hain kuch leftover nahi.
Warp-ceiling blocks $= ⌊ 64 \times 32/256 ⌋ = ⌊ 2048/256 ⌋ = 8$ . (Warp lid.) Yeh step kyun? Independently, hardware resident warps ko 64 par cap karta hai; har block $256/32 = 8$ warps hai, toh us rule se bhi zyada se zyada 8 blocks.
Blocks/SM $= min (8, 8) = 8$ . Active warps $= 8 \times 8 = 64$ . Yeh step kyun? Dono lids 8 par hain — paani 8 tak uthta hai, koi waste nahi. Blocks ko warps-per-block se multiply karo resident warps count karne ke liye.
Occupancy $= 64/64 = 1.0 = 100%$ .

Verify: 64 active warps exactly $W_{m a x}$ hai, aur 8 blocks × 256 = 2048 threads = SM ka max. Do independent ceilings ka 100% par agree karna sanity check hai.

Example 2 — Register-bound, non-integer divide `[Cell 2]`

Forecast: Pool same hai; har thread 4× zyada registers leta hai. Kya occupancy exactly 25% tak girti hai, ya rounding use aur bura banata hai?

Registers per block $= 256 \times 128 = 32768$ . Yeh step kyun? Same reservation logic — zyada registers per thread block ki demand multiply karta hai.
Register-limited blocks $= ⌊ 65536/32768 ⌋ = 2$ . Yeh step kyun? $65536/32768 = 2.0$ yahan exactly hai, toh floor kuch nahi badalta — lekin hamesha apply karo.
Active warps $= 2 blocks \times 8 warps = 16$ .
Occupancy $= 16/64 = 0.25 = 25%$ .

Rounding trap (yeh Cell 2 kyun hai): ab 130 registers/thread kar do. Registers/block $= 256 \times 130 = 33280$ , aur $⌊ 65536/33280 ⌋ = ⌊ 1.969 ⌋ = 1$ . Thodi si badhot tumhe 2 blocks se 1 block par gira deti hai → 8 warps → $8/64 = 12.5%$ . Sirf 2 extra registers par aadhi occupancy gaayab ho gayi, kyunki floor ne fractional block phek diya — register lid 2 se thodi neeche aa gayi.

Verify: $2 \times 32768 = 65536 \leq 65536$ ✓ (fits), lekin $2 \times 33280 = 66560 > 65536$ ✗ (nahi fits), confirming karta hai ki doosra case sirf 1 block host kar sakta hai.

Example 3 — Shared-memory-bound `[Cell 3]`

Forecast: Registers kehte hain "tons of blocks fit honge." Kya registers jeetenge, ya shared memory tumhe clamp karega?

Register-limited blocks $= ⌊ 65536/ (256 \times 2)⌋ = ⌊ 65536/512 ⌋ = 128$ . Yeh step kyun? Register lid establish karo taaki hum prove kar sakein ki yeh binding wali nahi hai.
Shared-memory-limited blocks $= ⌊ \frac{98304 bytes}{49152 bytes/block} ⌋ = 2 blocks$ . Yeh step kyun? Pehle bytes mein convert karo honest unit consistency ke liye: $96 KB = 96 \times 1024 = 98304$ bytes aur $48 KB = 49152$ bytes. Phir bytes ÷ bytes/block = blocks (KB bhi cancel ho jaata, lekin convert karne se koi doubt nahi rehta). 96 KB mein sirf do 48-KB chunks fit hote hain. Yeh shared-memory lid hai.
Warp-ceiling blocks $= ⌊ 2048/256 ⌋ = 8$ .
Blocks/SM $= min (128, 2, 8) = 2$ . Active warps $= 2 \times 8 = 16$ . Yeh step kyun? Shared memory s01 figure mein sabse neechi lid hai — wahi decide karti hai, chahe registers 128 blocks host kar sakti.
Occupancy $= 16/64 = 25%$ .

Verify: $2 \times 49152 bytes = 98304 bytes \leq 98304$ ✓ fits; teesre block ko $147456 bytes > 98304$ chahiye honge ✗. Shared memory, registers nahi, ceiling confirmed hai — is cell ka poora point yahi hai.

Example 4 — Warp-ceiling-bound `[Cell 4]`

Forecast: Sab kuch sasta hai. Toh occupancy 100% hai… right?

Register-limited blocks $= ⌊ 65536/ (64 \times 16)⌋ = ⌊ 65536/1024 ⌋ = 64$ . Yeh step kyun? Pehle register lid compute karo taaki hum prove kar sakein registers is kernel ko limit nahi kar rahe — har block ko sirf 1024 registers chahiye, toh 64 blocks register-wise fit honge.
Warp-ceiling blocks $= ⌊ 2048/64 ⌋ = 32$ . Yeh step kyun? Har block sirf $64/32 = 2$ warps hai, toh 64-warp ceiling 32 blocks allow karta hai — registers se kam. Warp lid register lid se neeche baith jaati hai.
Blocks/SM $= min (64, 32) = 32$ . Active warps $= 32 \times 2 = 64$ . Yeh step kyun? Sabse neechi lid lo (32, warp ceiling), phir resident blocks ko warps-per-block se multiply karo ( $32 \times 2$ ) count karne ke liye ki SM par actually kitne warps rehte hain.
Occupancy $= 64/64 = 100%$ . Yeh step kyun? Active warps ko $W_{m a x}$ se divide karo; yahan warp ceiling aur occupancy definition dono 64 share karte hain, toh ek warp-bound kernel phir bhi 100% reach karta hai.

Subtle point: tumne 100% reach kiya, lekin tum warp/block ceilings se pinned ho, resources se nahi. Agar SM resident blocks ko $B_{m a x} = 16$ par bhi cap karta, toh tumhe $16 \times 2 = 32$ warps $= 50%$ milte despite mountains of free registers. Chhote blocks warp budget ko $B_{m a x}$ block-count lid ke through waste kar sakte hain.

Verify: $32 \times 2 = 64$ warps $= W_{m a x}$ ✓, aur $32 \times 1024 = 32768 \leq 65536$ registers ✓ (registers spare mein hain, confirm karta hai ki woh limit nahi hain). $B_{m a x} = 16$ ke saath min hoga $min (64, 32, 16) = 16$ blocks → 32 warps → 50%.

Example 5 — Zero / degenerate input `[Cell 5]`

Forecast: Guess karo ki occupancy small-but-positive hai, ya literally zero.

Registers per block $= 512 \times 256 = 131072$ . Yeh step kyun? Demand exactly compute karo jaise hamesha — extreme inputs ke liye koi shortcut nahi.
Register-limited blocks $= ⌊ 65536/131072 ⌋ = ⌊ 0.5 ⌋ = 0$ . Yeh step kyun? Block ko pure SM se zyada registers chahiye. 0.5 ka floor 0 hai — tum aadha block host nahi kar sakte. Register lid bucket ke floor tak aa gayi.
Blocks/SM $= min (0, \dots) = 0$ . Active warps $= 0$ . Yeh step kyun? Jis bhi min mein 0 ho woh 0 hai; zero resident blocks ke saath zero resident warps hain.
Occupancy $= 0/64 = 0%$ — ek launch failure (too many resources requested), degenerate case.

Yeh kyun matter karta hai: occupancy 0 ek real, catchable outcome hai, abstraction nahi. Fix wahi hai jo hamesha hai: registers per thread kam karo, ya block chhota karo taaki $T \cdot r \leq R_{total}$ ho.

Verify: 256 regs/thread par sabse bada block jo fit hoga woh satisfy karta hai $T \times 256 \leq 65536 \Rightarrow T \leq 256$ . Toh 512 threads edge ke bahar hai → 0 blocks. Boundary confirmed.

Example 6 — Limiting behaviour: warps vs. latency `[Cell 6]`

Forecast: Kya throughput hamesha badhta rehta hai, ya flatten ho jaata hai?

Key ratio yeh hai ki kitne independent warps chahiye pipeline ko 300-cycle stall ke dauran busy rakhne ke liye. Jab ek warp wait karta hai, scheduler doosre chalata hai.

Warps needed to fully hide the stall $\approx$ latency in cycles $= 300$ (ek warp per cycle issue karna waiting ke ek cycle ko cover karta hai). Yeh step kyun? Har cycle tum ek warp dete ho; 300 cycles mein kabhi idle na ho iske liye ~300 ready warps chahiye.
Efficiency model $η (N) = min (N, 300) /300$ — useful fraction of a cycle jo actually fill hoti hai. Yeh step kyun? 300 warps se neeche tum 300 cycles mein se $N$ cover karte ho; 300 se upar tum already har cycle fill karte ho.
Limiting behaviour: $N \geq 300$ ke liye, $η = 1$ (saturated). Extra warps kuch nahi add karte — yeh plateau hai. Yeh step kyun? Jab har cycle busy ho, zyada warps ek cycle ko "ek se zyada full" nahi bana sakte.
Reality check for one SM: $W_{m a x} = 64 ≪ 300$ , toh ek scheduler wala ek single SM $η = 64/300 \approx 21%$ reach karta hai — yeh akele 300-cycle stall fully hide nahi kar sakta. Multiple schedulers per SM (typically 4) aur chhote average stalls gap ko close karte hain.

Verify: $N = 64$ par: $η = 64/300 \approx 0.213$ . $N = 300$ par: $η = 1.0$ . $N = 600$ par: phir bhi $1.0$ (plateau). Curve ek rising line hai jo $N = 300$ par flatten ho jaati hai.

Example 7 — Word problem: ek real launch across the GPU `[Cell 7]`

Forecast: Blocks (40) se zyada SMs (80) hain — guess karo GPU over- ya under-utilised hai.

Pehle, ek symbol jis ki humein ab zaroorat hai. Floor $⌊ ⌋$ (round down) ke saath, hum iska mirror use karte hain:

Total block-slots jo GPU ek saath offer karta hai $= 80 SMs \times 8 blocks/SM = 640$ slots. Yeh step kyun? Yeh GPU ki combined capacity hai ek wave ke liye; hum apne 40 blocks ko iske against compare karte hain dekhne ke liye ki kya sab ek saath fit hote hain.
SMs jo block receive karte hain $= min (40, 80) = 40$ . Yeh step kyun? Grid distribution unit har block ko ek SM ko deta hai, pehle ek block per SM; sirf 40 blocks ke saath, sirf 40 SMs light up karte hain.
Idle SMs $= 80 - 40 = 40$ — GPU ka aadha hissa is launch mein idle baitha hai. Yeh step kyun? Blocks distribution ki unit hain; bahut kam blocks SMs ko starve karte hain chahe har block ki occupancy kitni bhi achi ho.
Waves $= ⌈ 40/640 ⌉ = ⌈ 0.0625 ⌉ = 1$ wave. Yeh step kyun? "Wave" ek round of concurrently resident blocks hai. Saare 40 blocks 640 available slots mein fit ho jaate hain, toh ek single wave poori grid clear kar deti hai — aur ceiling fraction 0.0625 ko 1 whole wave mein turn kar deta hai.

Lesson: Example 1 mein occupancy per SM 100% thi, phir bhi whole-GPU utilisation yahan sirf $40/80 = 50%$ hai kyunki grid bahut chhota tha. Bahut zyada blocks launch karo SMs se.

Corner case — ek badi grid jise multiple waves chahiye: same kernel ko 2000 blocks ke saath relaunch karo. Ab block-slots (640) exceed ho jaate hain, toh waves $= ⌈ 2000/640 ⌉ = ⌈ 3.125 ⌉ = 4$ waves. Pehli 3 waves mein se har ek saare 640 slots fill karti hai ( $3 \times 640 = 1920$ blocks); 4th wave mein sirf $2000 - 1920 = 80$ blocks hain, sirf $⌈ 80/8 ⌉ = 10$ SMs light up karte hain jabki doosre 70 us final wave ke dauran idle rehte hain. Full waves efficient hain; ragged tail wave SMs waste karta hai — ek wajah grids is size ki karo ki tail chhoti ya absent ho.

Verify (small grid): total threads $= 40 \times 256 = 10240$ ; idle SM fraction $= 40/80 = 0.5$ ; ek wave kyunki $40 \leq 640$ block-slots. Verify (big grid): $⌈ 2000/640 ⌉ = 4$ waves, tail mein $2000 - 3 \times 640 = 80$ blocks hain, $⌈ 80/8 ⌉ = 10$ SMs use ho rahe hain. ✓

Example 8 — Exam twist: divergence tumhari occupancy win kha jaata hai `[Cell 8]`

Forecast: Occupancy equal hai — toh zaroor tie honge?

Kernel X cost: har instruction warp par ek baar chalta hai, saare 32 lanes active. Baseline $= 1 \times$ kaam. Yeh step kyun? Koi divergence nahi matlab ek pass saare 32 threads ko ek saath cover karta hai — ideal case.
Kernel Y, pehla path: warp if branch sirf 16 lanes active (threads 0–15) ke saath execute karta hai, doosre 16 lanes masked off aur idle hain. Yeh step kyun? Ek warp mein threads ek instruction pointer share karte hain (parent note ka SIMT rule), toh doosra branch lene wale lanes wait karte hain — woh same cycle mein alag instruction nahi chala sakte.
Kernel Y, doosra path: warp phir else branch execute karta hai threads 16–31 active ke saath aur threads 0–15 masked off. Yeh step kyun? Do branches overlap nahi kar sakti; hardware unhe branched region ke upar do passes mein serialize karta hai.
Slowdown factor branched region ke liye $= 2 \times$ : region do baar traverse hoti hai, aur har pass mein aadhe lanes kuch useful nahi karte. Yeh step kyun? Code par do serial passes jo ek non-divergent warp ek pass mein cover karta, exactly double instruction issues hai.
Conclusion: Kernel X pehle khatam hota hai. Kernel Y divergent region par ~2× slower hai despite identical 100% occupancy.

Trap: occupancy measure karta hai kitne warps resident hain, na ki har warp ke lanes kitni efficiently use ho rahe hain. High occupancy + high divergence lower occupancy + no divergence se haar sakta hai.

Verify: Y ki lane-utilisation branch mein $= (16 + 16) / (32 + 32) = 32/64 = 0.5$ , yaani 50% issued lane-slots useful hain → runtime ratio $1/0.5 = 2.0 \times$ . Dono ki occupancy $= 1.0$ . ✓

Recall Self-check: kaun si ceiling bind karti hai?

Kernel: 100 regs/thread, 256 threads/block, 32 KB shared/block. SM: 65536 regs, 64 KB shared, $W_{m a x} = 64$ . Kaun sa resource occupancy limit karta hai? ::: Registers: $⌊ 65536/ (256 \cdot 100)⌋ = 2$ ; shared $⌊ 64/32 ⌋ = 2$ ; warps $⌊ 2048/256 ⌋ = 8$ . Min hai 2 (registers aur shared memory ke beech tie) → 16 warps → 25%. Do kernels equal occupancy par alag speeds par kyun chal sakte hain? ::: Occupancy resident warps count karta hai, lane efficiency nahi; warp divergence lanes idle karta hai occupancy change kiye bina. Example 7 mein, GPU sirf 50% utilised kyun hai despite 100% per-SM occupancy? ::: Bahut kam blocks (40) SMs ki sankhya (80) ke liye; blocks distribution ki unit hain, toh 40 SMs idle rehte hain.

See also: CUDA-cores · Tensor-cores · Hinglish version