Exercises — Streaming multiprocessors (SM)

Ek warp hamesha exactly $32$ threads hota hai (yeh koi aisi setting nahi jise tum change kar sako). Toh hum divide karte hain: $\frac{256}{32}=8\text{ warps.}$ Hamne kya kiya: threads ko hardware ke fixed 32-wide bundles mein group kiya. Kyun: warp scheduler kabhi individual threads nahi dekhta — yeh hamesha sirf poore warps ko issue karta hai, isliye warps scheduling ki asli unit hain.

Parent note ki SM latency ladder use karke: $\text{Registers }(1)<\text{Shared }(20\text{–}30)<\text{L2 }(\sim 200)<\text{Global }(400\text{–}800)\text{ cycles.}$ Yeh order kyun hai: har step data ko CUDA cores se physically door le jaata hai. Registers core ke andar hi hote hain; shared memory on-chip SRAM hai jisme koi tag lookup nahi; L2 poore chip pe shared hai; DRAM bilkul off-chip hai. Jitna door = utne zyada cycles. Dekho GPU-memory-hierarchy.

Special Function Unit (SFU). CUDA cores plain integer/float add aur multiply karte hain; Load/Store units memory move karte hain; SFUs dedicated hardware hain transcendentals ke liye (sin, cos, exp, sqrt). Yeh kyun exist karta hai: yeh functions basic arithmetic se emulate karne par 20–100 cycles lagte hain, lekin SFU par sirf ~1–2 cycles.

Step 1 — registers per block. Har thread ko apni copy chahiye: $256\times 128=32,768\text{ registers per block.}$ Step 2 — kitne fit hote hain. SM ki supply ko divide karo, phir floor karo (aadha block resident nahi ho sakta): $\lfloor \frac{65,536}{32,768}\rfloor =2\text{ blocks.}$ Floor kyun: ek block SM par ya toh poora hai ya nahi. Agar sirf aadhe block ke liye registers bache hain, toh woh block wait karta hai. Dekho Occupancy-optimization.

Resident warps: $2\text{ blocks}\times \frac{256}{32}=2\times 8=16$ warps. Occupancy: $\frac{16}{64}=0.25=25\%.$ Iska matlab: scheduler ke sirf ek quarter warp slots mein kaam hai, isliye latency-hiding kamzor hai — L3 dekho.

$\frac{65,536}{64}=1024\text{ threads.}$ Lekin doosra cap bhi dekho: agar SM mein zyada se zyada $64$ warps bhi allowed hain, toh yeh $64\times 32=2048$ threads hain. Yahan $1024<2048$, isliye registers binding limit hain. Rule: actual occupancy us resource se set hoti hai jo pehle khatam hoti hai.

Model: jab stalled warp wait karta hai, scheduler ko $300$ cycles tak har cycle mein ek alag ready warp issue karna hoga. Ek warp se ek instruction, phir woh phir stall kyun ho jaata hai: memory-bound kernel mein zyaatar warps ko aage jo karna hota hai woh khud ek aur dependent global load (ya aisi op jo in-flight value consume karegi) hota hai. Toh scheduler kisi warp ko ek instruction issue karne ke baad, woh warp turant apne long-latency access par phir wait karne lagta hai. Isliye yeh cover ke sirf roughly ek useful cycle contribute karta hai phir drop out ho jaata hai. Isliye humein ek warp ko baar baar use karne ki jagah bahut saare alag-alag warps chahiye: $\approx 300\text{ independent warps (latency ke har cycle ke liye ek).}$ Kya 16 warps kaafi hain? Nahi — $16$ warps sirf $300$-cycle hole ke $16$ cycles cover karte hain per scheduler pass. Baaki $\sim 284$ cycles scheduler idle baitha rehta hai. Lesson: kam occupancy directly latency-hiding ko bhookha rakhti hai. Yahi wajah hai ki parent note "resident warps maximize karo" par itna zor deta hai.

Nahi. Occupancy sirf latency hide karne mein help karti hai. Ek compute-bound kernel already CUDA cores ko busy rakhta hai; uski bottleneck arithmetic throughput hai, memory ka wait karna nahi. Extra warps sirf same busy ALUs ke liye queue karte hain. Kya badalta hai vs. kya nahi badhta: ek memory-bound kernel ke liye, $16\to 32$ warps jaana roughly double karta hai jo latency tum chhupa sakte ho aur noticeable help kar sakta hai. Ek compute-bound ke liye, tum near-zero improvement dekh sakte ho — agar extra warps ke liye register cut ki zaroorat padi toh tum speed bhi kho sakte ho, kyunki spills memory mein jaayenge. Analytical point: occupancy ek ceiling raiser for latency hiding hai, throughput multiplier nahi.

SIMT ke under, ek warp saare $32$ lanes ke liye ek instruction issue karta hai. Jab lanes split hote hain, hardware dono paths ko sequence mein run karta hai, har baar inactive half ko mask karke: $\underset{\text{path A, 16 active}}{\underbrace{10}}+\underset{\text{path B, 16 active}}{\underbrace{10}}=20\text{ issues.}$ Non-diverging cost $10$ hai. Toh yeh $2\times$ slowdown hai, aur half lanes har path ke dauran idle rehte hain. Key subtlety: cost liye gaye distinct paths ki sankhya ke saath scale hoti hai, threads ki sankhya ke saath nahi. Agar saare 32 ne branch A liya, toh cost $10$ hi rehti hai — bilkul bhi divergence penalty nahi.

Har block size ke liye, har limit se resident-block cap compute karo, minimum lo, phir warps mein convert karo.

Registers. Per thread cost $32$, toh per SM register file $65,536/32=2048$ threads support karta hai regardless of block size → yahan binding limit nahi (yeh exactly $2048$-thread warp cap se match karta hai).

Shared memory. Har block $4$ KB chahta hai aur SM ke paas $96$ KB hai, toh $\lfloor \frac{96\text{ KB}}{4\text{ KB}}\rfloor =24$ blocks max (KB units cancel ho jaate hain, pure count bachta hai).

Block/thread caps, per block size ke hisaab se:

• 128 threads (4 warps): thread cap $2048/128=16$ blocks; block cap $32$; SMEM $24$. Min $=16$ blocks $\Rightarrow 16\times 4=64$ warps → 100%.
• 256 threads (8 warps): thread cap $2048/256=8$ blocks; SMEM $24$; block cap $32$. Min $=8\Rightarrow 8\times 8=64$ warps → 100%.
• 512 threads (16 warps): thread cap $2048/512=4$ blocks; SMEM $24$. Min $=4\Rightarrow 4\times 16=64$ warps → 100%.

Answer: teeno $64$ warps = $100\%$ occupancy tak pahunchte hain, kyunki yeh kernel registers aur shared memory par light hai. Insight: jab koi resource scarce nahi hai, occupancy raw warp cap se decide hoti hai, aur block size ek secondary choice ban jaata hai (aisa size lo jo caps mein evenly fit ho aur tumhari shared-memory tiling ke liye accha ho — yahan 256 ek safe default hai).

Shared memory cap: $\lfloor \frac{96\text{ KB}}{32\text{ KB}}\rfloor =3$ blocks. Register cap: $2048/256=8$ blocks. Block cap: $32$. Min $=3$ blocks. Warps: $3\times 8=24$. Occupancy: $24/64=37.5\%$. Binding limit: shared memory. Kya karna chahiye: tile chhota karo (per block kam SMEM) ya buffer ko split karo, thoda data reuse trade karo zyada resident blocks ke liye. Yeh classic Occupancy-optimization tug-of-war hai reuse aur parallelism ke beech.

(a) Diagnosis. L2.1/L2.2 se: $128$ regs/thread → $\lfloor 65536/(256\times 128)\rfloor =2$ blocks → $16$ warps → $25\%$. Ek $400$-cycle stall ko poori tarah chhupane ke liye $\sim 400$ warps chahiye (L3.1 logic); $16$ warps per scheduler pass sirf $\sim 16$ cycles cover karte hain, toh SM har stall ka zyaatar hissa idle baitha rehta hai. Root cause: register pressure resident warps ko us level se kahin neeche cap kar deti hai jo latency-hiding ko chahiye.

(b) $64$ regs/thread par refactor ke baad. Registers per block: $256\times 64=16,384$. Blocks: $\lfloor 65,536/16,384\rfloor =4$. Warps: $4\times 8=32$. Occupancy: $32/64=50\%$.

(c) Latency hidden. Warps double hue $16\to 32$, toh per scheduler pass hideable slice roughly double ho jaata hai ($\sim 16\to \sim 32$ cycles). Phir bhi $400$ se kaafi kam hai, toh occupancy akele yeh poori tarah theek nahi kar sakti — tum saath mein memory-coalescing aur reuse (shared-memory tiling) bhi pursue karoge taaki stalls ki sankhya kam ho. Mastery point: occupancy us ceiling ko raise karti hai jitni latency tum chhupa sakte ho, lekin complementary — aur aksar bada — lever yeh hai ki latency khud ko reduce karo, accesses coalesce karke aur data shared memory mein reuse karke taaki chhupane ke liye pehle hi kam stalls hon.

$8$ blocks resident rakhne ke liye: Registers. Saare resident threads mein total $\le 65,536$. Threads resident $=8\times 256=2048$. Toh per-thread budget: $\frac{65,536}{2048}=32\text{ registers/thread (max).}$ Shared memory. $8$ blocks $96$ KB share karte hain: $\frac{96\text{ KB}}{8}=12\text{ KB/block (max).}$ Check: $8$ blocks $\le$ block cap ($32$) ✓; warps $=64=$ cap ✓. Answer: $100\%$ hit karne ke liye $32$ regs/thread aur $12$ KB SMEM/block par ya neeche raho. Kisi ek ko bhi exceed karo aur resident blocks $8$ se neeche gir jaate hain, aur occupancy khatam ho jaati hai. Yeh Occupancy-optimization ka poora budget-balancing act hai — aur isliye Tensor Cores aur thread block sizing ko haath mein haath milaake tune kiya jaata hai.

Ek warp hamesha 32 threads hota hai. Occupancy = active warps divided by ::: max warps per SM. Resident block count ::: minimum hota hai sab resource limits ka (registers, shared memory, warp cap, block cap). Extra warps ::: latency hide karte hain — yeh ALU throughput nahi badhate. Divergence cost ::: liye gaye distinct branch paths ki sankhya ke saath scale hoti hai, threads ki sankhya ke saath nahi.