Occupancy and latency hiding: Worked examples — Hinglish deep dive

6.2.10 · D3 · Hardware › GPU Architecture › Occupancy and latency hiding

Yeh page sirf ek kaam ke liye hai: practice. Hum parent note ke ideas lete hain aur unhe har tarah ki situation mein grind karte hain jo yeh topic de sakta hai — har resource limit, har degenerate input, har "trick" twist jo exam pasand karta hai. Shuru karne se pehle, ek promise: yahan koi bhi symbol use nahi hoga jo aapne pehle nahi dekha. Agar aapko warps ke baare mein refresher chahiye, woh 6.2.1-SM-Architecture aur 6.2.8-Warp-Scheduling mein milega.

Poore note mein, hamara reference chip A100 hai, jiske per-SM budgets hain:

Koi bhi example touch karne se pehle, yahan bataya gaya hai ki har ek resource-limited warp count kaise compute karte hain. Har example sirf inhi teen definitions mein numbers plug karta hai.

Scenario matrix

Occupancy problems sab reduce hote hain: "kaun sa resource pehle khatam hota hai?" Yahan har case-class hai jo answer decide kar sakti hai. Har row ek cell hai; aane wale examples tagged hain us cell ke saath jo woh cover karte hain.

Cell	Kya khatam hota hai (ya twist)	Degenerate/limit angle	Covered by
A	Registers bottleneck hain	bahut high regs/thread	Ex 1, Ex 6
B	Shared memory bottleneck hai	block granularity waste	Ex 2
C	Block-count cap bottleneck hai	tiny blocks, $B_{m a x}$ hit	Ex 3
D	Kuch bhi limit nahi → 100% possible	"free" case	Ex 4
E	Zero / degenerate input	0 shared mem, 1 thread/block	Ex 4, Ex 5
F	Do limits tie karte hain	min lo, dono equal	Ex 6
G	Real-world word problem	latency-hiding decision	Ex 7
H	Exam twist: high occupancy ≠ faster	spills vs. occupancy	Ex 8

Workflow hamesha same rehta hai, toh ise ek baar seekho (chart nodes P1–P5 label hain taaki woh upar ke Cell letters se kabhi clash na karein):

Example 1 — Cell A: registers pehle bite karte hain

Worked example Statement

Kernel 64 registers/thread use karta hai, koi shared memory nahi, 256 threads/block. A100 par occupancy kya hai?

Forecast: Abhi guess karo — kya yeh 25%, 50%, ya 100% hai? Aage padhne se pehle likh lo.

Step 1 — Registers per warp. $64 \frac{regs}{thread} \times 32 \frac{threads}{warp} = 2048 \frac{regs}{warp}$ Yeh step kyun? Registers per thread diye jaate hain, lekin occupancy warps count karta hai. Budget ko "warps' worth" mein convert karne ke liye 32 se multiply karo.

Step 2 — Register file mein kitne warps fit hote hain. $W_{regs} = ⌊ \frac{65536}{2048} ⌋ = 32 warps$ Floor kyun? Aap aadha warp host nahi kar sakte. Jo bhi bache hue registers ek pura warp complete nahi kar sakte, woh simply waste ho jaate hain.

Step 3 — Baaki limits. Koi shared memory nahi ⇒ $W_{smem} = \infty$ . Blocks: 256 threads/block $= 8$ warps/block, toh $W_{blocks} = B_{m a x} \times 8 = 32 \times 8 = 256$ warps — roster se kaafi upar, toh blocks bhi tight nahi hain.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 32 , \infty , 256 , 64 )}{64} = \frac{32}{64} = 50%$

Verify: $32 \times 2048 = 65536$ — register file exactly full hai, zero waste. Yeh ek clean register-bound kernel ki signature hai: budget evenly divide hoti hai. Agar forecast ≠ 50% tha, to dobara check karo ki aapne regs ko 32 se multiply kiya ya nahi.

Example 2 — Cell B: shared memory pehle bite karta hai

Worked example Statement

Kernel 32 registers/thread, 48 KB shared memory per block, 256 threads/block (= 8 warps/block) use karta hai. A100 par Occupancy?

Forecast: Registers yahan tame lagte hain. Kya iska matlab 100% hai? Guess karo.

Step 1 — Register limit (generous hona chahiye). $W_{regs} = ⌊ \frac{65536}{32 \times 32} ⌋ = ⌊ \frac{65536}{1024} ⌋ = 64 warps$ Kyun? Sirf 32 regs/thread par hum pura 64-warp roster fit kar sakte hain — registers yahan villain nahi hain.

Step 2 — Shared-memory limit, blocks mein. $⌊ \frac{164 KB}{48 KB} ⌋ = 3 blocks$ Blocks mein floor kyun? Shared memory per block allocate hoti hai, all-or-nothing. $3 \times 48 = 144$ KB use hua, 20 KB bacha — 4th block ke liye enough nahi, toh woh waste ho jaata hai.

Step 3 — Blocks ko warps mein convert karo. $W_{smem} = 3 blocks \times 8 \frac{warps}{block} = 24 warps$ Aur bhi $W_{blocks} = B_{m a x} \times 8 = 32 \times 8 = 256$ warps — binding nahi.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 64 , 24 , 256 , 64 )}{64} = \frac{24}{64} = 37.5%$

Verify: Shared mem sabse scarcest hai (24 < 64), toh wahi decide karta hai — $min$ se consistent. Leftover par sanity: $164 - 144 = 20 KB < 48$ , confirm karta hai ki koi 4th block fit nahi hoga. Answer Example 1 ke 50% se neeche hai kyunki shared memory ki block-granular waste registers ke exact fit se zyada harsh hai.

Example 3 — Cell C: block-count cap bite karta hai

Worked example Statement

Ek "wide grid" kernel tiny blocks of 32 threads (= 1 warp/block), 16 registers/thread, koi shared memory nahi launch karta hai. Occupancy ko kya limit karta hai?

Forecast: Sirf 16 regs/thread aur koi shared mem nahi — kya 100% pakka hai? Careful.

Step 1 — Register limit. $W_{regs} = ⌊ \frac{65536}{16 \times 32} ⌋ = ⌊ \frac{65536}{512} ⌋ = 128 warps$ Floor kyun? Aap ek warp ka fraction host nahi kar sakte — jo registers ek whole warp complete nahi karte woh waste hote hain. Yahan $128 > W_{m a x} = 64$ , toh sirf registers se pura roster allow hoga.

Step 2 — Block-count cap. Har block 1 warp hai. 64 warps reach karne ke liye hume SM par 64 blocks chahiye honge. Lekin hardware blocks ko $B_{m a x} = 32$ par cap karta hai. $W_{blocks} = B_{m a x} \times 1 \frac{warp}{block} = 32 \times 1 = 32 warps$ Yeh step kyun? Blocks hardware slots hain jinki hard ceiling hai. Tiny blocks block slots "spend" karte hain warp slots se faster, toh aap warps se pehle slots khatam kar lete ho.

Step 3 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 32 , 64 )}{64} = \frac{32}{64} = 50%$

Verify: Bottleneck $B_{m a x}$ hai, registers ya shared memory nahi — classic "blocks too small" trap. Fix: ≥64 threads/block (2 warps) use karo taaki 32 blocks pura 64 warps dein. Sanity: $32 \times 2 = 64 = W_{m a x}$ . ✓

Example 4 — Cells D & E: "free" case aur zero input

Worked example Statement

Kernel 16 registers/thread, 0 KB shared memory, 128 threads/block (= 4 warps) use karta hai. Occupancy?

Forecast: Teen limited cases ke baad, kya kuch rasta rok raha hai yahan?

Step 1 — Registers. $W_{regs} = ⌊ \frac{65536}{16 \times 32} ⌋ = ⌊ \frac{65536}{512} ⌋ = 128 warps$ Floor kyun? Hamesha same reason: ek partial warp host nahi ho sakta, toh round down karo. $128 > 64$ , toh registers koi limit nahi dete.

Step 2 — Shared memory (zero input, Cell E). Shared mem per block $= 0$ . Budget ko zero se divide karna "infinity blocks" hoga, matlab shared memory kabhi constrain nahi karti. $W_{smem} = \infty$ maano. Kyun? Ek resource jo aapne request nahi kiya woh khatam nahi ho sakta.

Step 3 — Blocks. 4 warps/block; $W_{blocks} = B_{m a x} \times 4 = 32 \times 4 = 128$ warps. 64 warps ke roster tak pahunchne ke liye sirf $64/4 = 16$ blocks chahiye $\leq B_{m a x} = 32$ , toh blocks theek hain.

Step 4 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 128 , 64 )}{64} = \frac{64}{64} = 100%$

Verify: Har candidate $\geq 64$ hai, toh $W_{m a x}$ cap khud binding "limit" hai — ek resource-light kernel ki healthy sign. Degenerate check: 0 shared mem correctly $min$ se drop out ho gayi rather than crash karke.

Example 5 — Cell E extreme: ek thread per block

Worked example Statement

Kisi ne (galti se!) A100 par 1 thread per block, 16 regs/thread, koi shared memory nahi launch kiya. Occupancy ki ceiling kya hai?

Forecast: 1 thread 32 se kaafi kam hai. Kya warp "round up" karta hai?

Step 1 — Ek block phir bhi pura warp cost karta hai. Chahe 1 thread ho, woh ek pura warp slot occupy karta hai (baaki 31 lanes masked off hain — idle lekin reserved). Toh 1 thread/block = 1 warp/block, 31 lanes waste hain. Kyun? Warps schedulable unit ka sabse chhota form hain; aapke paas warp ka fraction nahi ho sakta. Dekho 6.2.8-Warp-Scheduling.

Step 2 — Har limit compute karo. $r = 16$ , $n = 1$ ke saath:

Registers: $W_{regs} = ⌊ 65536/ (16 \times 32)⌋ = 128$ warps $> 64$ — binding nahi.
Shared mem: kuch request nahi $\Rightarrow W_{smem} = \infty$ — binding nahi.
Blocks: $W_{blocks} = B_{m a x} \times n = 32 \times 1 = 32$ warps — yeh binds karta hai. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 128 , \infty , 32 , 64 )}{64}$

Step 3 — Warps ki Occupancy. $\frac{32}{64} = 50%$

Step 4 — Effective useful work. Warp occupancy 50% hai, lekin har active warp sirf $1/32$ lanes use karta hai. Useful-thread fraction: $50% \times \frac{1}{32} \approx 1.56%$

Verify: $0.50/32 = 0.015625 = 1.5625%$ . Isliye akeli "occupancy" jhooth bol sakti hai — masked lanes ke saath high warp occupancy machine waste karti hai. Hamesha 32 ke multiples mein blocks launch karo.

Example 6 — Cell A + F: ek register limit jo tie karta hai

Worked example Statement

Kernel 128 registers/thread, koi shared memory nahi, 128 threads/block (4 warps) use karta hai. Register limit dikhao aur pata lagao ki kya block cap tie karta hai.

Forecast: High registers → low occupancy. Lekin kya koi aur cheez us number se match karti hai?

Step 1 — Register limit. $W_{regs} = ⌊ \frac{65536}{128 \times 32} ⌋ = ⌊ \frac{65536}{4096} ⌋ = 16 warps$ Kyun? 128 regs/thread par har warp 4096 registers gobble karta hai — sirf 16 fit hote hain (floor: ek partial 17th warp host nahi ho sakta).

Step 2 — Blocks mein. 16 warps at 4 warps/block $= 4$ blocks. Block cap deta hai $W_{blocks} = B_{m a x} \times 4 = 32 \times 4 = 128$ warps — plenty. Toh registers, blocks nahi, bind karte hain.

Step 3 — Occupancy. $\frac{m i n ( W _{regs} , W _{smem} , W _{blocks} , W _{m a x} )}{64} = \frac{m i n ( 16 , \infty , 128 , 64 )}{64} = \frac{16}{64} = 25%$

Step 4 — Tie kahan hoga (Cell F)? Agar uski jagah har block mein exactly 4 warps hote aur block cap 4 hota, toh $W_{regs}$ aur $W_{blocks}$ dono 16 ke barabar hote — ek saccha tie. $min$ use gracefully handle karta hai: equal values same answer deti hain, toh koi ambiguity nahi.

Verify: $16 \times 4096 = 65536$ — register file phir se exactly full. Registers ko 64/thread par halve karne se $W_{regs}$ double hokar 32 → 50% ho jaayega (Example 1 se match karta hai). ✓

Register–occupancy relationship ek picture ki deserve karti hai. Figure s01 A100 par registers/thread ke against occupancy plot karta hai, hamare worked example points marked hain:

Stepped pink line dekho: occupancy tabhi neeche jump karta hai jab ek naya warp fit nahi ho sakta — yeh staircase hai, smooth slope nahi, floor function ki wajah se. Yellow dots mark karte hain Examples 1 (64 regs → 50%), 4 (16 regs → 100%), aur 6 (128 regs → 25%).

Example 7 — Cell G: real-world latency-hiding decision

Worked example Statement

Ek memory-bound kernel: har warp loads issue karta hai jo $L = 400$ cycles lete hain. Aapki best tuning 50% occupancy (32 warps) deti hai. Ek colleague insist karta hai ki aapko 100% (64 warps) reach karna hi chahiye. Little's Law thinking use karke decide karo ki kya yeh worth it hai.

Forecast: Zyada warps → zyada latency hiding → hamesha faster? Yes/no guess karo.

Step 1 — "Throughput" precisely define karo. 5.3.4-Littles-Law kehta hai $outstanding work = L \times throughput,$ jahan yahan throughput = SM-wide, memory instructions issued per cycle. Ek A100 SM ke paas 4 warp schedulers hain, har ek 1 instruction/cycle issue karne mein capable hai, toh best case mein throughput $= 4$ memory-issues/cycle. Yeh step kyun? Little's Law ek time (latency) ko ek count (kitni cheezein in flight honi chahiyen) mein convert karta hai — woh count hi hai jo occupancy supply karni chahiye.

Step 2 — Kitne outstanding loads chahiye. $outstanding loads = L \times throughput = 400 cycles \times 4 \frac{issues}{cycle} = 1600 loads in flight.$ Lekin ek single warp sirf 1 memory instruction outstanding at a time rakhta hai (per dependent load). Toh warps needed $\approx 1600/ (loads per warp in flight)$ . Modest ILP ke saath har warp roughly 2–4 loads outstanding rakhta hai ek unrolled loop mein, deta hai $warps \approx \frac{1600}{4 schedulers \times a few loads/warp} \approx 16 - 32 warps.$ Kyun? Yahin se "16–32 warps" ki band aati hai — woh $1600$ outstanding loads hain 4 schedulers aur kuch loads jo har warp overlap karta hai mein divide karke.

Step 3 — Apne 32 warps se compare karo. 32 warps us band ke top par hai, toh 50% occupancy par pipeline essentially kabhi idle nahi hoti warps ki kami ki wajah se. 64 tak double karna sirf chhota residual hide karta hai — ≤5% speedup at best, aur yeh register spills (Example 8) cost kar sakta hai jo cheezein worse bana dete hain.

Step 4 — Decision. 50% par raho. Effort 6.2.9-Memory-Coalescing ya 7.1.3-Roofline-Model par lagao, jo true bottleneck move karte hain.

Verify: $400 \times 4 = 1600$ outstanding loads; 4 schedulers se divide karo $= 400$ per scheduler; ~dozen-plus loads overlapped per scheduler across warps ke saath, ~ $16$ – $32$ warps sufficient hain, aur $32 \geq$ woh. ✓ Answer: 100% mat chase karo.

Example 8 — Cell H: exam twist (higher occupancy, slower kernel)

Worked example Statement

Compute-bound kernel abhi 128 regs/thread → 25% occupancy use karta hai aur time $T$ par run hota hai. Occupancy "fix" karne ke liye aap compiler ko 64 regs/thread → 50% occupancy force karte ho, lekin yeh values ko local memory mein spill karta hai, har inner iteration mein ~300 cycles mein 2 spill loads add karta hai. Kya performance improve hoti hai?

Forecast: Occupancy 25% se 50% double ho gayi. Faster ya slower?

Step 1 — Occupancy before/after. $⌊ 65536/ (128 \cdot 32)⌋ = 16$ warps $= 25%$ ; $⌊ 65536/ (64 \cdot 32)⌋ = 32$ warps $= 50%$ . Number better lagta hai.

Step 2 — Lekin yeh kernel compute-bound hai. Iska bottleneck ALU throughput hai, jise high occupancy raise nahi karta jab scheduling bubbles already gone hain (woh hain, ILP/unrolling se). Toh 25→50 jump compute side par ~kuch nahi khareedta.

Step 3 — Spills ki cost. Har inner iteration ab $2 \times 300 = 600$ extra cycles pay karta hai spilled data move karne mein. Agar iteration ~256 FMA cycles ke useful work ka tha, toh naya per-iteration time hai $\frac{256 + 600}{256} = \frac{856}{256} \approx 3.34 \times the original.$

Step 4 — Verdict. "50% occupancy" version ~3.34× slower per iteration hai. Occupancy ek means hai (latency hide karna), end nahi. Registers rakho aur 25% pe raho. 6.2.11-Tensor-Cores kernels se compare karo, jo deliberately isi wajah se low occupancy run karte hain.

Verify: $2 \times 300 = 600$ ; $(256 + 600) /256 = 856/256 = 3.34375 > 1$ ⇒ change hurt karta hai. ✓ Yeh parent note ka canonical "100% occupancy ≠ best performance" trap hai.

Recall

Recall Occupancy kaun sa resource decide karta hai?

Sabse scarcest wala — occupancy $= min (W_{regs}, W_{smem}, W_{blocks}, W_{m a x}) / W_{m a x}$ . ::: Minimum, kyunki ek warp ko har resource simultaneously chahiye hoti hai.

Recall Block sizes 32 ke multiples kyun hone chahiye?

Ek block hamesha whole warps cost karta hai; ek partial warp mein extra lanes masked aur wasted hoti hain (Example 5). ::: 1 thread/block 50% warp occupancy deta hai lekin ~1.56% useful work.

Recall Kya 100% occupancy hamesha fastest hota hai?

Nahi — Examples 7 aur 8: "enough to hide latency" ke baad, extra warps diminishing returns dete hain aur register spills trigger kar sakte hain jo aapko slow karte hain. ::: Occupancy ek means hai latency hide karne ka, khud goal nahi.

Related: parent topic · 6.2.1-SM-Architecture · 6.2.8-Warp-Scheduling · 6.2.9-Memory-Coalescing · 6.2.11-Tensor-Cores · 7.1.3-Roofline-Model · 5.3.4-Littles-Law