Streaming multiprocessors (SM): Foundations — Hinglish deep dive

6.2.2 · D1 · Hardware › GPU Architecture › Streaming multiprocessors (SM)

Is page mein assume kiya gaya hai ki tum kuch nahi jaante. Parent note Streaming Multiprocessors padhne se pehle, tumhe har wo symbol padhna aana chahiye jo wahan use hote hain. Isliye hum har ek ko ek picture se banate hain. Koi bhi cheez draw kiye bina use nahi ki jaati.

0. SM — worker-factory khud

Picture: GPU ko ek badi building samjho jisme bahut saari identical choti factories bhari hain. Har factory (ek SM) ek pile of work leti hai aur use apne dam par process karti hai. Ye poora page aslmein ek inventory list hai ek aisi factory ke andar kya hota hai.

Topic ko ye kyun chahiye: parent note mein "SM" letters lagbhag har line par hain. Ab se, "SM" ka matlab hamesha ek worker-factory hoga.

1. Thread — sabse chhota worker

Picture: 32 identical workers ki ek row imagine karo, har koi ek number card pakde hue. Wo sab ek hi recipe card padhte hain ("apne number mein 5 add karo"), lekin har koi use apne khud ke card par apply karta hai.

Topic ko ye kyun chahiye: GPU ka poora point hi hai ek saath hazaron aise workers chalana. "Thread" atom hai — har cheez threads mein count hoti hai.

2. Warp — 32 threads ka ek bundle jo saath chalte hain

32 kyun, aur bundle kyun? Kyunki hardware banana sasta hota hai jo ek instruction padhta hai aur use 32 workers par fire karta hai, bajaye ki har worker ko apna instruction-reader dene ke. Ek instruction, baattees workers.

Picture: figure 1 ke 32 workers ek saath chain ho jaate hain. Jab foreman ek command chillata hai, saare 32 usi clock tick par maan lete hain. Ye parent note ke word SIMT ka matlab hai — Single Instruction, Multiple Thread: ek instruction, bahut saare threads.

Tail: partial warps

warps per block = ⌈ \frac{threads per block}{32} ⌉

Yahan $⌈ ⌉$ ka matlab hai agale poore number tak oopar round karna (yaani "ceiling").

Jab ek warp ke threads agree nahi karte: divergence

Picture: foreman chillata hai "path A karo". Jo threads path A chahte the wo kaam karte hain; baaki freeze ho jaate hain. Phir foreman chillata hai "path B karo" aur roles ulat jaate hain. Warp ne A phir B kiya — double kaam. Agar 32 threads ittefaq se same path par agree kar lein, toh koi penalty nahi hoti.

Topic ko ye kyun chahiye: isliye parent note warn karta hai ki branching "performance cost" karti hai. Divergence us flexibility ki kimat hai jo SIMT ko rigid vector hardware se alag karti hai. Dekho Warp-scheduling ye jaanne ke liye ki ye bundles kaise baari baari kaam karte hain.

3. Block — ek SM par assign warps ki ek team

1024 cap kyun? Ek block ko apne shared resources (registers, shared memory, warp slots) ek SM ke andar fit karne chahiye, aur hardware threads ki numbering ke liye fixed index bits reserve karta hai. Wo budget abhi 1024 threads = 32 warps per block par top out hoti hai. Ye number ek architectural limit hai — ye kaafi recent generations mein 1024 raha hai, lekin aisi caps GPU architectures ke paas alag ho sakti hain, isliye apna specific hardware hamesha check karo.

Picture: block ek team hai. Agar ek team mein 256 threads hain, toh wo team $256/32 = 8$ warps hai. Saare 8 warps ek factory (ek SM) ke andar baithte hain.

Topic ko ye kyun chahiye: block assignment ka unit hai. GPU ek waqt mein ek block SMs ko deta hai. Dekho Thread-blocks.

warps per block = ⌈ \frac{threads per block}{32} ⌉

4. Grid — ek launch ke saare blocks

Picture — poori hierarchy ek saath: thread ⊂ warp ⊂ block ⊂ grid. Chhote se bade tak.

Recall Hierarchy ko wapas padho

Tumhare code ka ek running instance ::: ek thread 32 threads ek hi instruction se locked ::: ek warp Ek SM par pinned 1024 tak threads ::: ek block Launch ke saare blocks ::: ek grid

5. Register — ek thread ka private scratch pad

Picture: har worker ko ek chhoti pocket do. Pocket mein numbers instantly (1 cycle — ek clock tick) mein grab kiye jaate hain. Lekin factory floor par kul itni hi pockets hain, aur unhe sabke beech divide karna padta hai.

Ye limit kyun banata hai: agar har thread bahut saari pockets maange, toh kam threads fit honge. Ye parent note ki poori "register pressure" story hai:

max threads on SM = ⌊ \frac{total registers}{registers per thread} ⌋

Yahan $⌊ ⌋$ ka matlab hai agale poore number tak neeche round karna (yaani "floor"): tum ek thread ya ek block ka fraction nahi chala sakte, isliye koi bhi bacha hua capacity simply unused chali jaati hai.

6. Cycle — GPU ki heartbeat (time ki unit)

Picture: ek metronome. Har tick par, factory ek chhota step le sakti hai. Slow operations kitni hi ticks karti hain.

Topic ko ye kyun chahiye: parent note mein har speed cycles mein measured hai. Registers = 1 tick (instant). Global memory = 400–800 ticks (bahut hi slow). Ye gap hi ek wajah hai ki SMs ko jaise banaya gaya hai waise banaya gaya hai.

7. Global memory — bada, slow, shared warehouse

Picture: registers har worker ki pocket hain; global memory yard ke paar ek bada warehouse hai. Warehouse se ek box lana ek lamba safar hai. Unke beech memories ki poori ladder ke liye dekho GPU-memory-hierarchy.

Topic ko ye kyun chahiye: global memory wahi hai jahan se lambi waits aati hain jinhe chipane ke liye poori latency-hiding machinery (agla section) exist karti hai.

8. Latency aur latency-hiding — wait karna, aur wait chipana

Topic ko ye kyun chahiye: parent note ka lagbhag har design choice ("maximize resident warps") latency chipane ke liye exist karta hai.

9. Architectural limits — factory ki fixed capacities

Occupancy samajhne se pehle, tumhe SM ki built-in ceilings chahiye. Ye numbers silicon mein baked in hain; tum inhe exceed nahi kar sakte.

Ye aate kahan se hain? Har ek SM mein built hardware ki physical amount hai: ek fixed number of warp-slots (scheduler bookkeeping), ek fixed number of block-slots, ek fixed register file. Jis bhi cap ko tum pehle hit karo wo decide karta hai ki kitne warps actually fit honge. Kyunki ye silicon amounts hain, ye GPU generations ke beech change hote hain — hamesha apna exact chip lookup karo. Neeche ke examples "max warps per SM = 64" ko ek representative figure ki tarah use karte hain.

10. Occupancy — factory kitni bhari hai

Occupancy = \frac{Active Warps on SM}{Max Warps per SM}

Picture: ek factory jisme 64 warp-slots hain. Agar sirf 16 bhari hain, toh occupancy = $16/64 = 0.25 = 25%$ . Zyada bhari slots → latency chipane ke liye zyada warps ready.

11. SM ke physical parts (named boxes)

Ye wo hardware boxes hain jo parent note list karta hai. Ab jab tumhare paas vocabulary hai, har ek ek liner hai. Warning: neeche ki counts rough, per-SM examples hain — yahan har number GPU generation ke hisaab se alag hota hai, isliye inhe "typical order of magnitude" samjho, fixed truth nahi.

Formula SM ke parts (typical current-generation counts, architecture ke hisaab se alag)

CUDA cores (a.k.a. SP, Streaming Processors): chote calculators jo $+, -, \times$ karte hain; roughly 64–128 per SM. — CUDA-cores
SFU (Special Function Unit): mushkil math (sin, cos, sqrt, exp) ke liye fast hardware; thodi si, often ~4 per SM partition.
LD/ST (Load/Store Unit): wo darwaze jo memory se data fetch/store karte hain; typically 16–32 per SM.
Register File: saari private pockets (section 5) ek pool mein; ~65,536 × 32-bit registers per SM.
Shared Memory: ek scratchpad jise block ke saare threads read/write kar sakte hain; ~48–100 KB per SM — Shared-memory.
Warp Scheduler: wo foreman jo har cycle mein ek ready warp chunata hai aur uski instruction chillata hai; usually 4 per SM — Warp-scheduling.
Tensor Cores: bonus hardware jo poore matrices ek saath multiply karta hai; newer chips par per SM kuch hote hain — Tensor-cores.

Prerequisite map

Equipment checklist

SM kya hai, ek sentence mein? ::: GPU ke andar ek self-contained worker-factory; ek GPU aise dozens factories ko saath tile karke banta hai.
Thread kya hai, ek sentence mein? ::: Program ka ek running copy jo apne data par kaam karta hai.
Ek warp mein kitne threads hote hain, aur bundle kyun karte hain? ::: 32 threads; wo ek instruction share karte hain isliye hardware ko poore group ke liye sirf ek instruction-reader chahiye hota hai.
Ek 256-thread block ko warps mein convert karo. ::: $256/32 = 8$ warps.
Agar block mein 100 threads hain toh kya hoga? ::: Hardware $⌈ 100/32 ⌉ = 4$ warps tak round up karta hai; aakhri warp mein 28 lanes switched off (masked, wasted) hoti hain.
Warp divergence kya hai? ::: Jab ek warp ke threads alag branch paths lete hain, toh warp dono paths ek ke baad ek run karta hai — extra kaam.
Block kahan run hota hai, aur kya ye move ho sakta hai? ::: Exactly ek SM par, shuru se ant tak — ye kabhi migrate nahi karta.
Block 1024 threads par cap kyun hai? ::: Ek hardware limit: block ke resources ek SM mein fit hone chahiye aur index bits fixed hain; ye GPU architectures ke paas alag ho sakta hai.
Global memory kya hai? ::: GPU ki badi, slow (400–800 cycle) DRAM jo saare SMs aur threads ke beech shared hoti hai.
Register vs global memory kitna fast hai? ::: Register ≈ 1 cycle; global memory ≈ 400–800 cycles.
Cycle kya hai? ::: Hardware clock ki ek tick — GPU ka time ka unit.
Ek sentence mein latency hiding define karo. ::: Jab ek warp data ka wait karta hai, SM doosre ready warps chalata hai taaki wait mein koi idle time na jaye.
SM ke teen architectural caps batao. ::: Max threads/block (~1024), max warps/SM (~48–64), max blocks/SM (~16–32) — sab architecture ke hisaab se alag hote hain.
Register/block ratios ko floor kyun karte hain? ::: Tum ek thread ya block ka fraction nahi chala sakte, isliye bacha hua capacity discard ho jaata hai.
Occupancy formula do. ::: Occupancy = SM par active warps ÷ max warps per SM.
Agar 16 warps resident hain aur max 64 hai, toh occupancy kya hai? ::: $16/64 = 25%$ .
GPUs hazaron threads kyun chahte hain? ::: Taaki itne saare ready warps hon jo lambi (saikdon cycles) memory latency chipaa sakein.