Streaming multiprocessors (SM): Visual walkthrough

6.2.2 · D2 · Hardware › GPU Architecture › Streaming multiprocessors (SM)

Yeh page SM topic ka central result build karti hai — ek SM ko ek saath itne saare warps ki zaroorat kyun hai, aur "occupancy" kaise measure karti hai ki uske paas enough hain ya nahi — bilkul pictures se. Hum assume karte hain ki aap kuch nahi jaante: na warp kya hota hai, na cycle kya hota hai, na GPU kyun stall karta hai. Har symbol ko use karne se pehle earn kiya gaya hai.

Step 1 — "Cycle" kya hota hai? SM ki dhadkan

KYA HAI. Kisi bhi formula se pehle, humein woh time unit chahiye jisme SM jeeta hai. Cycle ek tick hai SM ki clock ka — time ka sabse chhota hissa jisme hardware kaam ka ek chhota step kar sakta hai.

YEH IDEA PEHLE KYUN. Is page par har cost ("memory 400 cycles leta hai", "scheduler 1 instruction per cycle issue karta hai") cycles mein measure hoti hai. Agar hum tick ko clearly define nahi karte, baad ka koi bhi number kuch matlab nahi rakhta. Hum sab kuch cycles mein measure karte hain kyunki yahi woh ek clock hai jo SM maanta hai.

PICTURE. Neeche, time left se right ki taraf equal ticks mein chalta hai. Har tick par SM ek instruction start kar sakta hai. Yeh hamara ruler hai.

Step 2 — Ek thread, aur kyun ek akela thread machine ko barbaad karta hai

KYA HAI. Ek thread aapke program ki ek running copy hai — ek worker jo ek recipe follow karta hai. Yeh compute karta hai, aur kabhi kabhi ise global memory (chip ke baaju mein waali badi, door waali DRAM) se ek value chahiye hoti hai.

KYUN. Hum ek thread se shuru karte hain taaki problem ko uske purest form mein dekh sakein. Jab ek thread global memory se koi number maangta hai, toh jawab is cycle mein nahi aata — woh ek lambi delay ke baad aata hai jise hum latency kehte hain, jise $L$ likhte hain.

PICTURE. Ek thread ek load issue karta hai, phir stall karta hai — woh $L$ cycles tak bilkul frozen hokar kuch nahi karta kyunki woh literally us number ke bina aage nahi badh sakta jo usne maanga tha. Har red tick ek barbaad dhadkan hai.

Red stripe dekho: $L = 8$ cycles ke load ke liye, SM 1 cycle kaam karta hai aur 8 cycles kuch nahi karta. Yahi woh disaster hai jo hume fix karna hai.

Step 3 — Ek warp: 32 threads ek instruction se jode hue

KYA HAI. Threads ko ek ek karke schedule nahi kiya jaata. Hardware 32 threads ko ek unit mein bandh karta hai jise warp kehte hain. Warp scheduler ek instruction issue karta hai aur warp ke saare 32 threads use saath mein run karte hain — yeh SIMT hai (Single Instruction, Multiple Threads).

32 KYUN, AUR BUNDLE KYUN. Bundle karne se ek instruction-fetch 32 lanes of arithmetic (CUDA cores) ko drive kar sakti hai, toh aap ek baar control cost dete hain 32 workers ke liye. 32 ki sankhya fixed hardware hai. Timing ke liye, key fact yeh hai: ek warp ek unit ke roop mein stall karta hai — agar warp ek load karta hai, toh saare 32 threads saath mein wahi $L$ cycles wait karte hain. Toh latency-hiding math ke liye, ek warp bilkul Step 2 ke ek stalled worker ki tarah behave karta hai.

PICTURE. 32 lanes, inhe saari feed karne wala ek instruction pointer. Jab warp load karta hai, 32 ka poora block saath mein red ho jaata hai.

Kyunki ek warp ek unit ke roop mein stall karta hai, ab se hamari "job" ki unit warp hai, thread nahi.

Step 4 — Thread-blocks: warps ko kaise group kiya jaata hai aur SM ko deliver kiya jaata hai

KYA HAI. Aap GPU ko loose warps nahi dete — aap use thread-blocks dete hain. Ek thread-block threads ka ek batch hai (1024 tak) jise hardware ek SM ko assign karta hai aur wahan se shuru se ant tak run karta hai (koi migration nahi). Kyunki ek warp 32 threads ka hota hai (Step 3), $T$ threads ka ek block automatically kaata jaata hai

ek block mein warps w_{block} = \frac{T threads per block}{threads per warp 32}

$w_{block}$ — ek block kitne warps banta hai.
$T$ — block size jo aap kernel launch karte waqt choose karte ho.
$32$ — Step 3 se fixed warp width.

BLOCK KO ABHI INTRODUCE KYUN KAREIN. Agle steps mein counting "SM par kitne warps resident hain" ke baare mein baat karti hai. Warps sirf block-sized bundles mein aate hain, toh warps count karne se pehle humein bundle jaanna chahiye. 256-thread block, for example, $256/32 = 8$ warps hai — aur SM apne warp crowd build karne ke liye kai aise blocks ek saath stack karta hai.

PICTURE. Ek block = 8 warps ka ek stack; SM kai blocks ek saath rakhta hai, jo warp pool banata hai jisse scheduler draw karta hai.

Step 5 — Trick: jab ek wait kare, doosre warp par switch karo

KYA HAI. Yahan GPU ka poora idea hai. Jab warp 0 apne load par stall karta hai, scheduler idle nahi baithta — woh instantly warp 1 par switch kar leta hai, uski instruction issue karta hai, phir warp 2, aur aise aage. Warp switching free hai (koi saving/restoring nahi — har warp ki state register file mein permanently rehti hai jab tak woh resident hai).

KYUN. Ise latency hiding kehte hain. Hum warp 0 ka wait doosre warps ke useful work ke peeche hide karte hain.

Hume switching rate ke baare mein precise rehna hai, kyunki yahan do alag "per cycle" quantities hain. Maano

schedulers per SM S \times instructions each scheduler issues per cycle i = warp-issues advanced per cycle r

$S$ — SM mein independent warp schedulers ki sankhya (real SMs mein 2–4 hote hain).
$i$ — ek scheduler ek cycle mein kitni instructions issue karta hai (usually 1; "dual-issue" mein 2 hoti hai).
$r$ — product: poore SM mein har cycle kitne warps aage badhte hain. $r$ warp-issues per cycle count karta hai, ek warp ke andar instructions nahi. Ek scheduler ek baar issue kare toh $r = 1$ : exactly ek warp har cycle aage badhta hai. Note karein letter $r$ (lower-case, "rate") hamari per-cycle warp rate hai; ise $r_{thread}$ se confuse mat karein jo Step 7 mein introduce hoga, jo registers per thread hai — ek bilkul alag quantity jo bas ittefaq se ek hi letter share karti hai.

PICTURE. Timeline dekho: jab warp 0 ki red wait upar chalta hai, warp 1, 2, 3, … har ek ko neeche ek orange tick milta hai. Wait ab barbaad nahi ho rahi — yeh paros waale warps se cover ho rahi hai.

Step 6 — Exactly kitne warps chahiye, yeh count karna

KYA HAI. Ab derivation. Hamare paas hai:

$L$ cycles ki stall (Step 2),
ek scheduler machinery jo $r$ warps per cycle advance karti hai (Step 5).

KYUN YEH ARITHMETIC. Hum chahte hain ki warp 0 ka data tab tak waapas aa jaaye jab tak hum baaki saare ready warps mein ek baar cycle kar lein, taaki hum kaam se baahar na ho jaayein. Agar har cycle $r$ warps advance karti hai aur hole $L$ cycles wide hai, toh us hole ke dauran flight mein rakhne waale distinct warps ki sankhya — aur isliye use fill karne ke liye jo sankhya chahiye — hai:

hole fill karne ke liye ready warps W_{needed} = cycles load stall karta hai L \times warp-issues per cycle (Step 5) r

$W_{needed}$ — resident, ready warps ki sankhya jo humein SM par rakhni hai.
$L$ — stall ki lambaai cycles mein (fill karne wala hole), Step 2 se.
$r$ — warp-issues advanced per cycle, Step 5 se. Ek scheduler ke saath, $r = 1$ aur $W_{needed} = L$ .
$⌈ \dots ⌉$ — ceiling (agle poore number tak upar round karo). Round up kyun, down kyun nahi? Kyunki aapke paas warp ka fraction nahi ho sakta, aur agar, say, $L \times r = 12.3$ hota, toh $12$ warps hole ka $0.3$ -cycle sliver uncovered chod dete — SM phir bhi stall karta. Upar $13$ tak round karna guarantee karta hai ki hole puri tarah tile ho jaaye. "Mujhe safe rehne ke liye kitne chahiye" count ke liye hamesha ceiling.

PICTURE. $L$ lambaai ka ek bar jo hume warp slots se tile karna hai. Kam warps → ant mein idle cycles ka ek gap (SM stalls). Bilkul enough → tiles warp 0 ke return tak bina gap ke pahunch jaate hain.

Intuition Edge case — har warp ke liye several loads in flight (memory-level parallelism)

Haari count ne assume kiya ek warp = ek outstanding load. Lekin ek warp stall hone se pehle kai independent loads fire kar sakta hai (woh tab tak freeze hota hai jab tak woh actually ek not-yet-arrived value use karta hai). Agar har warp $p$ loads flight mein rakhta hai (jise memory-level parallelism, MLP kehte hain), toh ek warp ab $p$ loads ki latency hide karta hai, toh warp requirement kam ho jaati hai: $W_{needed} = ⌈ \frac{L \times r}{p} ⌉, p \geq 1.$ $p = 1$ hamaara pehle wala formula recover karta hai; bada $p$ matlab kam warps chahiye. Isliye register-heavy kernels jo apne loads batch karte hain modest occupancy par bhi achhe chal sakte hain. Yeh ek lower bound helper hai, free lunch nahi — $p$ is baat se limited hai ki aapka algorithm aur load/store units kitne independent loads allow karte hain.

Step 7 — Limit: SM cap karta hai ki kitne warps resident ho sakte hain

KYA HAI. Hum chahenge ki $W_{have} = 300$ ho. Lekin ek SM sirf utne warps hold kar sakta hai ek saath — ise hardware ceiling == $W_{m a x}$ == kahein (real GPUs par typically 48–64). Aap physically isse exceed nahi kar sakte, aur teen alag resources mein se har ek actual limit ko $W_{m a x}$ se neeche kheench sakta hai.

TEEN LIMITS KYUN. Har resident warp registers consume karta hai, uska block shared memory aur ek block slot consume karta hai. Jo pehle khatam ho woh decide karta hai ki kitne warps fit hote hain. Toh real resident count teen ceilings ka minimum hai:

W_{have} = min (W_{m a x}, W_{reg}, W_{smem}, W_{slots})

PICTURE. Register file warp-sized wedges mein kaata gaya — mote threads (har ek ke paas bahut registers) → kam warps fit; patale threads → bahut warps fit.

Formula Teen resource ceilings

Pehle, per-thread register count define karo jo hum aage use karne waale hain:

$r_{thread}$ — registers per thread (Step 5 ke rate $r$ se bilkul alag quantity; subscript "thread" wahan exactly unhe alag rakhne ke liye hai). Compiler aapko aapke kernel ke liye yeh number batata hai.

(1) Registers — register file $R_{SM}$ warps mein baanta gaya, har warp ko $32 \times r_{thread}$ registers chahiye (32 threads, har ek $r_{thread}$ registers chahta hai): $W_{reg} = ⌊ \frac{R _{SM}}{r _{thread} \times 32} ⌋$ $R_{SM}$ = total 32-bit registers (e.g. $65, 536$ ); $r_{thread}$ = registers per thread; $\times 32$ per-thread ko per-warp mein badalta hai. Yahan hum neeche round karte hain (floor $⌊ \dots ⌋$ ): ek warp jo sirf partially fit hota hai nahi chal sakta, toh hum sirf woh poore warps rakhte hain jo puri tarah fit hote hain.

(2) Shared memory — total shared memory $M_{SM}$ blocks mein baanta gaya (har block ko $m_{block}$ bytes chahiye), phir warps mein convert kiya gaya: $W_{smem} = ⌊ \frac{M _{SM}}{m _{block}} ⌋ \times w_{block}$ $M_{SM}$ = shared memory per SM (≈48–100 KB, GPU-memory-hierarchy dekho); $m_{block}$ = bytes jo ek block request karta hai; $w_{block}$ = warps per block (Step 4).

(3) Block slots — SM zyada se zyada $B_{m a x}$ blocks hold kar sakta hai (typically 16–32): $W_{slots} = B_{m a x} \times w_{block}$ $B_{m a x}$ = max resident blocks; warps lene ke liye warps per block se multiply karo.

Upar ki figure teen ceilings ko teen bars ke roop mein dikhati hai; sabse chhota bar woh hai jo actually aapko limit karta hai.

Step 8 — Warp divergence: har lane useful work nahi karta

KYA HAI. Ab tak hum assume kar rahe the ki warp ke saare 32 threads hamesha saath useful work karte hain. Lekin ek warp saare 32 lanes ke liye ek instruction run karta hai (Step 3). Agar threads ek if/else par alag branches lete hain, toh hardware ko dono paths ek ke baad ek run karne honge, un lanes ko switch off (mask) karte hue jo current path se belong nahi karte.

KYUN YAHAN MATTER KARTA HAI. Har masked path ke dauran, kuch lanes idle baithte hain — woh warp ka issue slot occupy karte hain lekin kuch produce nahi karte. Yeh warp divergence hai. Yeh kitne warps resident hain yeh nahi badalta, lekin yeh per issue effective work ko lower karta hai: ek warp jo sirf half-active hai phir bhi ek poora issue slot cost karta hai, toh aapka real throughput — aur occupancy se milne wala faayda — girata hai.

PICTURE. Ek warp if (threadIdx < 16) se split: pehle low 16 lanes path A run karte hain (top 16 masked, greyed out), phir high 16 lanes path B run karte hain (low 16 masked). Do issue slots spend, har ek mein sirf 16 lanes live.

Step 9 — Occupancy: woh score jo sab kuch jodata hai

KYA HAI. Ab hum woh single number define karte hain jiske around poora topic revolve karta hai.

KYUN YEH THE METRIC HAI. Steps 6–7 ne tension dikhaya: latency hiding chahti hai bahut warps ( $W_{needed} = ⌈ L \times r / p ⌉$ ), lekin hardware unhe $W_{m a x}$ par cap karta hai, aur aapke kernel ki register/shared-memory/block appetite $W_{active}$ ko us cap se bhi neeche rakh sakti hai. Occupancy exactly hai "SM ki latency-hiding capacity ka aap actually kitna fraction use kar rahe ho?"

PICTURE. Ek gauge: SM ke $W_{m a x}$ slots, kuch filled (active), kuch empty (wasted). Needle occupancy hai.

Worked example Register-pressure walkthrough, ab fully derived

Kernel $r_{thread} = 128$ registers per thread use karta hai; $R_{SM} = 65, 536$ ; block = 256 threads = $w_{block} = 8$ warps; $W_{m a x} = 64$ ; assume karein registers binding limit hain.

Registers per block: $256 \times 128 = 32, 768$ .
Blocks jo fit hote hain: $⌊ 65, 536/32, 768 ⌋ = 2$ .
$W_{active} = W_{have} = 2 \times 8 = 16$ warps.
$Occupancy = 16/64 = 0.25 = 25%$ .

Takeaway: sirf 16 resident warps ek 400–800 cycle stall hide karne ke liye, lekin Step 6 ne kaha humein saikdon chahiye — toh yahan ek memory-bound kernel wait karte karte mar jaata hai. High register use ( $r_{thread} = 128$ ) ne SM ko parallelism se bhookha rakh diya. Ilaaj yeh hai ki registers per thread kaato → zyaada blocks fit → $W_{have}$ badhta hai → occupancy badhti hai → warp crowd finally stall cover kar leta hai (Occupancy-optimization dekho).

Ek-picture summary

Yeh final figure Steps 1→9 ko ek diagram mein compress karta hai: cycle ruler (Step 1), $L$ ke liye ek warp stalling (Steps 2–3), blocks warps deliver karte hue (Step 4), scheduler rate $r$ par hole fill karta hua (Step 5), count $W_{needed} = ⌈ L r / p ⌉$ (Step 6), teen ceilings $W_{have}$ cap karte hue (Step 7), divergence lanes waste karta hua (Step 8), aur occupancy gauge jo sab kuch score karta hai (Step 9).

Recall Feynman retelling — plain words mein wapas bolein

SM ki clock tick karti hai; har tick ek cycle hai. Ek thread kaam karta hai, phir slow door waali memory se ek number maangta hai aur saikdon cycles tak wait karte hue freeze ho jaata hai. Threads 32 ke bundles mein glue hote hain jise warps kehte hain, aur warps SM par thread-blocks ke andar aate hain — 256 threads ka ek block 8 warps hai, aur SM ready warps ki crowd build karne ke liye kai blocks stack karta hai. Scheduler ki trick: jab ek warp frozen hai, ek alag ready warp advance karo — saare schedulers mein har cycle mein $r$ unhe. $L$ cycles wide hole fill karne ke liye aapko lagbhag $⌈ L \times r ⌉$ ready warps chahiye, upar round kiya taaki hole ka koi bhi sliver uncovered na rahe (kam agar har warp kai loads flight mein rakhta hai, jo MLP hai). Lekin SM sirf itne warps hold kar sakta hai — registers, shared memory, aur block slots sab isse cap karte hain, aur sabse tight wala jeetta hai; woh surviving count $W_{have}$ hai, bilkul wahi number jo occupancy formula $W_{active}$ kehta hai. Occupancy fraction hai "warps jo actually paas hain ÷ warps jo SM hold kar sakta tha." Bahut chhota crowd aur woh memory answer aane se pehle khatam ho jaata hai, toh SM idle baithta hai. Aur ek full crowd bhi kaam barbaad kar sakta hai: agar ek warp mein threads alag branch karte hain (divergence), toh warp dono paths half lanes switched off ke saath run karta hai, toh occupancy akele kabhi full throughput guarantee nahi karta. Lekin agar aapka kernel rarely memory par wait karta hai, toh aapko badi crowd ki zaroorat nahi — toh occupancy ek tool hai, trophy nahi.

Recall Quick self-check

Ek cycle hai ::: SM clock ki ek tick — woh unit jisme saari latencies measure hoti hain. Ek warp hai ::: 32 threads jo ek instruction saath execute karte hain aur saath stall karte hain (SIMT). Ek thread-block hai ::: ek SM ko assign kiye gaye 1024 threads tak ka ek batch; yeh $T /32$ warps mein split hota hai. $W_{needed} = ⌈ L r ⌉$ mein $r$ ka matlab hai ::: saare schedulers mein har cycle warp-issues advanced ( $S \times i$ ), ek warp ke andar instructions nahi, aur registers per thread bhi nahi. $r_{thread}$ ka matlab hai ::: registers per thread (register ceiling mein use kiya gaya) — rate $r$ se ek alag quantity. $W_{needed}$ mein ceiling (round up) kyun? ::: aapke paas warp ka fraction nahi ho sakta, aur round down karne se stall ka ek sliver uncovered rehta, toh SM phir bhi stall karta. $W_{have}$ denote karta hai ::: warps jo SM par abhi actually resident banaye hain — wahi number jo occupancy formula $W_{active}$ kehta hai. Ek scheduler aur ek load per warp ke saath ek $L$ -cycle stall hide karne ke liye aapko lagbhag chahiye ::: $L$ resident warps. Warp divergence hai ::: ek warp mein threads alag branches lete hain, toh warp dono paths serially kuch lanes masked off ke saath run karta hai — issue slots waste karta hai. Lane efficiency hai ::: average active lanes per issue ÷ 32; 1 se neeche matlab occupancy real throughput overstate karti hai. Occupancy equals ::: active warps ÷ max warps jo SM hold kar sakta hai. Resident warps par teen resource ceilings hain ::: registers, shared memory, aur block slots — sabse tight wala jeetta hai. Har warp ke liye kai loads flight mein rakhna (MLP) matlab hai ::: kam warps chahiye, $W_{needed} = ⌈ L r / p ⌉$ .

Yeh bhi dekho: Thread-blocks · GPU-memory-hierarchy · Tensor-cores · CUDA-cores · Warp-scheduling