CUDA cores and execution model: Visual walkthrough

6.2.3 · D2 · Hardware › GPU Architecture › CUDA cores and execution model

Pehli line se pehle, sirf ek cheez parent se accept karni hai: ek GPU kaam ko warps mein chop karta hai (32 threads ke bundles jo lockstep mein chalte hain), aur chip ka ek hissa jise SM (Streaming Multiprocessor) kehte hain woh warps run karta hai. Baaki sab yahan build karenge.

Step 1 — Ek "cycle" kya hota hai, aur ek warp usme kya karta hai?

KYA. Ek cycle GPU ki clock ki ek tick hai — time ka sabse chhota slice jisme hardware ek kaam kar sakta hai. Ek metronome socho: tick... tick... tick. Har tick par, SM ek instruction hand out kar sakta hai.

Ek warp apni zindagi mein do tarah ki cheezein karta hai:

Compute ticks: add, multiply — saste, ek-ek tick.
Ek memory request: "door wale DRAM se ek number fetch karo." Yeh ek tick nahi hai. Number door rehta hai, isliye warp ko wait karna padta hai.

YEH KYUN pehle. Baad mein har symbol ( $C$ , $L$ , $S$ , $T$ letters) bas ticks ki count ya warps ki count ka naam hai. Agar aapko feel nahi hota ki ek tick kya deta hai, toh formula gibberish lagega.

PICTURE. Ek warp ki timeline: compute ticks ka ek chhota sa green run, phir ek lamba khali stretch jahan woh stalled hai, haath pe haath dhare, memory ka wait kar raha hai.

Step 2 — Problem: ek idle SM barbaad silicon hai

KYA. Un $L$ waiting ticks ke dauran, compute hardware (CUDA cores) ke paas us ek warp ke liye kuch nahi hai karne ko. Metronome tikta rehta hai; cores ko koi kaam offer nahi hota. Yeh pure waste hai.

KYUN. GPU ka CPU se jeettne ka poora reason yahi hai ki woh apne arithmetic units ko kabhi bhukha nahi rehne deta (compare 6.2.01-GPU-vs-CPU-architecture). Akela ek warp unhe fed nahi rakh sakta — woh har $C + L$ mein se $L$ ticks stall karta hai. Toh SM ki trick hai: doosre warps ko paas rakhna aur pehle warp ke stall hote hi unme se kisi par switch karna.

PICTURE. Do warps stacked. Jab warp A stall karta hai (khali stretch), warp B already compute kar raha hai (green) bilkul neeche — SM apna dhyan B par slide karta hai aur cores busy rehte hain.

Step 3 — Machine ki issue power ko naam do: schedulers $S$

KYA. SM har tick mein kitna fresh kaam start kar sakta hai? Yeh warp schedulers se set hota hai. Ek scheduler woh traffic cop hai jo, har tick par, ek ready warp pick karta hai aur uski next instruction issue karta hai. Ek SM mein $S$ hote hain.

$schedulers S ⟶ warp-instructions issued per tick S$

Ise literally padho: $S$ schedulers har ek tick par $S$ warp-instructions issue karte hain. Agar $S = 4$ hai, toh SM har tick par 4 warps ka kaam launch karta hai.

KYUN naam dena. Kyunki jitna kaam SM wait ke dauran start kar sakta hai woh khali stretch ko fill karta hai. Zyada schedulers → har tick par zyada kaam start → gap jaldi bharta hai.

PICTURE. $S = 4$ traffic cops ki ek row, har ek ek warp ko same tick par aage wave kar raha hai.

Step 4 — Jo hole fill karna hai use count karo: $S \times L$ instruction-slots

KYA. Yeh counting ka key act hai. Jab warp A $L$ ticks ke liye stuck hai, us window mein SM kitne warp-instructions issue kar sakta tha?

$issue slots during the stall = slots per tick S \times ticks of stalling L = S \cdot L$

Term by term: har tick $S$ slots offer karta hai; stall $L$ ticks tak rehta hai; inhe multiply karo. $S = 4$ , $L = 200$ ke saath yeh $4 \times 200 = 800$ empty slots hain jo fill hone ke liye chilla rahe hain.

KYUN multiply karo, add nahi? Kyunki "slots per tick" aur "ticks ki number" same rectangle ki do independent dimensions hain. Yeh classic rate × time = amount move hai: cars-per-minute times minutes gives cars. Yahan, slots-per-tick times ticks gives slots.

PICTURE. Ek rectangle: width $L$ ticks across, height $S$ slots tall. Iska area — chhote boxes ka shaded grid — woh instructions ki number hai jo SM warp A ke wait karte waqt run karne ki jagah rakhta hai.

Step 5 — Har warp sirf $C$ instructions supply karta hai khud stall hone se pehle

KYA. Hamare paas $S \cdot L$ empty slots hain. Unhe kaun fill karega? Doosre warps. Lekin har warp bottomless supply nahi hai: woh sirf $C$ compute instructions deta hai phir woh bhi memory wall hit karke stall ho jaata hai.

Toh ek warp ≈ ek bucket jo exactly $C$ instructions hole mein dalti hai.

$instructions one warp gives = C$

KYUN yeh limiting piece hai. Agar warps infinitely lamba compute streams hote, toh bahut kam ki zaroorat padti. Lekin real kernels memory zyada touch karte hain — chhota $C$ . Ek kanjoosi wala $C$ matlab hai har warp jaldi khali ho jaata hai, isliye bahut zyada warps chahiye.

PICTURE. Step 4 ka bada $S \times L$ hole, ab kai warp-buckets se pour ho raha hai, har bucket par " $C$ instructions" label — aap literally count kar sakte ho ki rectangle fill karne mein kitni buckets lagti hain.

Step 6 — Divide karo: humein kitne warps chahiye

KYA. Total hole ÷ ek warp kya supply karta hai = warps ki number.

$T_{required} = \frac{S \cdot L slots to fill}{slots one warp fills C}$

$T_{required}$ = kitne warps resident hone chahiye taaki cores kabhi bhukhe na rahen.
Numerator $S \cdot L$ = poori idle window (Step 4).
Denominator $C$ = ek warp ka contribution (Step 5).

KYUN division? "Size $C$ ki kitni buckets size $S \cdot L$ ke hole mein fit hongi?" exactly ek division hai — same sawaal jaise " $\$ 800 $banan e k e l i y e k i t n e$ $5 $k e n o t ec hahi y e ? " ($ 800/5 = 160$).

Parent ke numbers plug karo ( $S = 4$ , $L = 200$ , $C = 10$ ):

$T_{required} = \frac{4 \times 200}{10} = \frac{800}{10} = 80 warps.$

PICTURE. Division ko tiling ke roop mein dikhaaya: $S \times L$ rectangle ko stacked $C$ -length strips mein kata gaya; strips gino → 80.

Step 7 — Reality check (kyun 80 ka matlab hai "impossible")

KYA. Formula demand karta hai 80 warps. Lekin ek SM physically itne warps ek saath hold kar sakta hai — parent ne approximately 64 resident warps ka cap quote kiya ( $2048$ threads $\div$ $32$ threads/warp $= 64$ ).

$T_{required} = 80 > 64 = T_{max resident}$

Toh hum enough warps supply nahi kar sakte. Hole usse bada hai jo hum usme dal sakte hain. Cores kuch waqt idle zaroor baitheinge — latency fully hidden nahi hoti.

KYUN yeh matters karta hai. Yeh inequality 6.2.04-occupancy-and-performance aur 6.2.02-GPU-memory-hierarchy ke har optimization ka birth certificate hai. Aap 64 raise nahi kar sakte (yeh fixed hardware hai). $S$ lower nahi kar sakte (aap schedulers chahte ho). $L$ DRAM physics hai. Aapke paas ek knob hai — $C$ — har memory trip mein zyada math karo. Coalescing (parent se) aur caching effective $L$ ghataate hain; higher arithmetic intensity $C$ badhata hai. Dono $T_{required}$ ko 64 ke neeche push karte hain.

PICTURE. Ek bar chart: ek lamba "needed = 80" bar ek chhote "available = 64" ceiling line ke saath, uncovered gap hatched — woh hatched gap wasted cycles hain.

Step 8 — Edge aur degenerate cases

Har woh scenario jo reader meet kar sakta hai, taaki kuch surprise na kare.

Worked example Formula ke corners

$C \to \infty$ (ek warp jo kabhi memory touch nahi karta). $T_{required} = \frac{S \cdot L}{\infty} \to 0$ . Ek ya do warps cores ko hamesha busy rakhte hain — ek pure-compute kernel trivially latency-hidden hoti hai. Matrix-heavy training kernels ke liye yeh dream case hai.
$C \to 0$ (sab memory, koi math nahi). $T_{required} = \frac{S \cdot L}{0} \to \infty$ . Koi finite number of warps help nahi karta; SM chahe kuch bhi karo starved rehta hai. Yeh ek memory-bound kernel hai.
$L = 0$ (data already registers mein, ~1 tick). $T_{required} = \frac{S \cdot 0}{C} = 0$ . Kuch hide karna nahi — koi waiting nahi matlab spare warps ki koi zaroorat nahi.
$S = 1$ (single scheduler). Har tick mein fill karne ke liye kam slots, toh $T_{required}$ $S = 2$ ke mukable half ho jaata hai. Kam warps chahiye — lekin SM ki peak throughput bhi girti hai. Ek trade-off, free win nahi.

Ek-picture summary

Poori kahaani ek board par: ek warp $L$ ke liye stall karta hai; SM ke paas $S$ slots/tick hain, toh $S \cdot L$ area ka ek hole khulta hai; har spare warp $C$ size ki ek strip fill karta hai; aapko $S \cdot L / C = 80$ strips chahiye; lekin ceiling 64 hai, toh wasted cycles ka ek gap rehta hai — aur iska ek hi escape hai ki $C$ badhao.

Recall Feynman retelling — ek 12-saal ke bacche ko batao

Ek kitchen socho jisme kuch cooks hain (schedulers, jinki number $S$ hai). Har cook ek second (ek tick) mein ek dish start kar sakta hai. Ek dish mein thodi chopping ( $C$ seconds kaam) lagti hai aur phir woh slow oven mein jaati hai lambi baking ke liye ( $L$ seconds). Jab ek dish bake ho rahi hoti hai, us cook ke paas kuch nahi karna — jab tak koi aur dish ready na ho choppable. $S$ cooks ko kabhi bored na karne ke liye jab woh $L$ -second bake ke through wait kar rahe hain, aapke paas enough dishes queue mein hone chahiye: exactly $S \times L$ seconds ki chopping cover karne ke liye, aur har dish sirf $C$ seconds ki chopping deti hai — toh aapko $S \cdot L / C$ dishes chahiye. 4 cooks, 200-second bakes, 10 seconds chopping each plug karo: $4 \times 200/10 = 80$ dishes. Lekin kitchen mein sirf 64 dishes ke liye counter space hai. Toh cooks zaroor kuch waqt khade rahenge. Fix zyada counters nahi hai (woh fixed hai) — yeh hai aisi dishes banana jo zyada chopping aur kam baking maange: $C$ badhao, aur achanak bahut kam dishes chahiye. Yahi fast GPU kernels likhne ki poori kala hai.

Recall Quick self-test

$S$ kya count karta hai, plain words mein? ::: Warp schedulers ki number — SM har clock tick mein kitne warp-instructions issue kar sakta hai. Hole to fill $S \times L$ kyun hai, $S + L$ kyun nahi? ::: Yeh ek rate times time hai (slots-per-tick × ticks) = ek rectangle ka area; multiply karne se stall ke dauran total issue slots milti hain. $S = 4$ , $L = 200$ , $C = 10$ ke saath, kitne warps required hain? ::: $4 \times 200/10 = 80$ warps. 80 problem kyun hai? ::: SM ~64 resident warps par cap karta hai, aur $80 > 64$ , toh latency fully hide nahi ho sakti. Kaunsa ek variable ek programmer sabse directly raise kar sakta hai madad ke liye, aur use raise karne se kya hota hai? ::: $C$ (warp ke stall se pehle compute); ise raise karne se $T_{required}$ ghatta hai kyunki yeh denominator mein hai.

See also: 7.3.01-parallel-programming-patterns · 6.1.03-SIMD-vs-SIMT · 6.2.04-occupancy-and-performance