Visual walkthrough — Occupancy and latency hiding

6.2.10 · D2 · Hardware › GPU Architecture › Occupancy and latency hiding

---

Step 1 — Warp kya hota hai, aur "issue" ka matlab kya hai?

KYA. Figure dekho. Har cyan row ek warp hai. Timeline par ek chhota box ek cycle hai — GPU ke clock ki sabse chhoti tick. Jab ek warp choose hota hai, SM uski agla instruction issue karta hai: woh instruction ko ek execution unit ko de deta hai. Iske issue hone mein exactly ek cycle lagta hai.

YE PEHLE KYUN. Baad ke saare counts "kitne warps" aur "kitne cycles" ke baare mein hain. Agar hum yeh nail down nahi karte ki ek warp ek instruction per cycle issue karta hai jab woh chosen warp hota hai, toh baad ka arithmetic meaningless ho jayega.

PICTURE. Amber marker woh single warp dikhata hai jise is cycle mein issue karne ka mauka milta hai. Baaki sab apni baari ka wait karte hain.

$\text{issue rate of one warp}=1\frac{\text{instruction}}{\text{cycle}}(\text{only when it is the chosen warp})$

• $\text{instruction}$ — ek operation, 32 threads ke liye ek saath
• $\text{cycle}$ — clock ki ek tick, woh unit jisme hum latency measure karte hain

---

Step 2 — Latency kya hai? Gap draw karo.

KYA. Ek warp ld.global issue karta hai (DRAM se read). Number sirf $L\approx 400$ cycles baad wapas aata hai. Un 400 cycles ke dauran woh warp loaded value ke saath kuch nahi kar sakta — value abhi wahan hai hi nahi.

YEH TOOL KYUN — ek "gap", ek "cost" nahi. Log latency ko throughput se confuse karte hain. Latency ek delay on one operation hai; throughput hai kitne per cycle complete hote hain. Hum latency ko ek khaali horizontal gap ki tarah picture karte hain taaki aap literally dekh sako ki fill karne ke liye idle room hai.

PICTURE. White bracket 400-cycle gap hai. Jis warp ne yeh start kiya woh grey ho gaya hai — so raha hai — poore bracket ke liye.

$L=400\text{cycles}\text{(the length of the empty bracket)}$

• $L$ — wait; picture mein gap ki horizontal length
• $400$ — typical DRAM read latency, parent note se

---

Step 3 — Akela ek warp gap waste kar deta hai

KYA. Ek single warp ke saath, load ke baad timeline mein ek instruction, phir 400 khaali cycles, phir agla instruction hota hai. SM sirf $401$ mein se $1$ cycle par useful kaam karta hai.

KYUN. Yeh problem statement hai. Hum chahte hain ki scheduler (dekho 6.2.8-Warp-Scheduling) har cycle kuch useful issue kare. Abhi woh nahi kar sakta, kyunki uske paas sirf ek warp hai aur woh so raha hai.

PICTURE. Khaale boxes gino — woh wasted cycles hain. Utilisation poore bar par woh chhoti amber sliver hai.

${\text{utilisation}}_{1\text{ warp}}=\frac{1\text{busy cycle}}{1+L\text{cycles}}=\frac{1}{401}\approx 0.25\%$

• numerator $1$ — woh single cycle jab warp busy tha issuing
• denominator $1+L$ — woh busy cycle plus poora khaali gap

---

Step 4 — Gap ko doosre warps se bharo

KYA. Aur warps leke aao. Jab Warp A apne 400-cycle gap mein so raha hota hai, scheduler Warp B se issue karta hai, phir Warp C se, phir Warp D se… ek per cycle. In mein se har ek apna load fire off kar ke so bhi sakta hai — lekin tab tak koi aur jaag chuka hota hai.

KYUN YEH POORI TRICK HAI. Gap mein $L$ khaale cycles hain. Agar hamare paas us gap ke dauran issue karne ke liye $L$ aur warp-instructions hain, toh har khaala box fill ho jayega. Stall gaayab ho jaata hai — is liye nahi ki load faster ho gaya (nahi hua), balki is liye ki SM ko hamesha doosra kaam mil gaya.

PICTURE. Warps ko rows ke roop mein stack karo. Diagonal amber staircase scheduler ko har cycle ek fresh warp ki taraf hop karte dikhata hai, gap ko solid paint karta hua.

$\underset{\text{kitni rows}}{\underbrace{N_{\text{warps}}}}\ge \underset{\text{fill karne wale empty boxes}}{\underbrace{L}}\times \underset{\text{instr issued per cycle}}{\underbrace{1}}$

• $N_{\text{warps}}$ — warp-rows ki ginti jo humne stack kiye
• $L$ — khaali cycles ki ginti jo cover karni hai
• $\times 1$ — ek scheduler ek cycle mein ek instruction issue karta hai

Yeh exactly Little's Law hai: items in flight = arrival rate × time in system. Yahan "items in flight" hain warps issuing, "time in system" hai latency $L$.

---

Step 5 — Generalise karo: throughput factor

KYA. Step 4 mein humne assume kiya tha ek scheduler 1 instruction/cycle issue karta hai, toh $\Theta =1$ aur $N=L$. Ek real SM mein 4 schedulers hote hain (dekho 6.2.1-SM-Architecture), har ek per cycle issue karta hai, toh $\Theta =4$.

EK PRODUCT KYUN. Har cycle aap $\Theta$ instructions chahte ho. Wait $L$ cycles tak chalti hai. Poori wait ke dauran aapko isliye $L\times \Theta$ instructions queued aur ready chahiye — yeh figure mein rectangle ka area hai.

PICTURE. Ek rectangle: width $L$ (waiting ke cycles), height $\Theta$ (instructions per cycle). Uska area in-flight operations ki woh ginti hai jo aapko supply karni hai.

$N_{\text{required}}=\underset{\text{width: wait in cycles}}{\underbrace{L}}\times \underset{\text{height: issue/cycle}}{\underbrace{\Theta }}=\text{area of the rectangle}$

---

Step 6 — "400 warps" ka number overshoot kyun karta hai (crucial edge case)

KYA. Naïve reading kehti hai "400 warps chahiye". Real kernels same latency 16–32 warps se hide karte hain. Koi cheating nahi hui — teen effects milke instructions per warp per cycle ko actually multiply karti hain jo aapko milti hai.

YEH KYUN MATTER KARTA HAI. Agar aap maan lete ki 400 warps chahiye toh aap panic kar jaate — hardware sirf 64 allow karta hai. Solution yeh hai ki ek warp us window mein ek se zyada outstanding operations supply karta hai.

PICTURE. Teen stacked bars teen multipliers dikhate hain jo required warp count ko 400 se ~16 tak shrink karte hain.

$N_{\text{warps}}=\frac{L\times \Theta }{\underset{\text{schedulers}}{\underbrace{S}}\times \underset{\text{ILP per warp}}{\underbrace{I}}}$

• $L\times \Theta$ — total in-flight ops needed (Step 5)
• $S=4$ — schedulers jo parallel mein issue karte hain, burden share karte hain
• $I$ — instruction-level parallelism: ek warp ke stall hone se pehle uske kaafi saare independent loads/FMAs in flight ho sakte hain (dekho 6.2.9-Memory-Coalescing ki kyun ek load kaafi saare transactions ho sakta hai lekin issue ek instruction ki tarah hota hai)

$L=400,\Theta =1$ effective per-scheduler, $S=4$, $I\approx 3$–$6$ ke saath: $N\approx \frac{400}{4\times 6}\approx 16$ warps. Isliye parent note kehta hai 16–32 active warps usually kaafi hote hain.

---

Step 7 — Degenerate cases: jab zyada warps KUCH NAHI karte

KYA. Do boundary situations jahan rectangle logic aapko batata hai ki warps add karna band karo.

1. Pehle se hide ho chuka hai (memory-bound, gap full). Jab ek baar $N\ge N_{\text{required}}$ ho jaaye toh har cycle busy hai. Is point ke baad warps add karne se zero throughput milta hai lekin registers chheen lete hain — aapko spills ki taraf dhakelte hain. Yeh parent ka "50% kaafi hai" observation hai.
2. Koi gap hi nahi (compute-bound). Agar ek warp 256 back-to-back FMAs karta hai, pipeline ILP se pehle se full hai — fill karne ke liye bahut kam gap hai, toh $N_{\text{required}}$ tiny hai (~4–8 warps).

KYUN. Requirement $L\times \Theta$ hai — ek fixed target, "zyada better hai" nahi. Jab aap ise meet kar lete ho, marginal warp idle capacity hai jiske liye aapne registers pay kiye.

PICTURE. Left: gap pehle se fully painted, extra warps unused hover kar rahe hain. Right: ek almost gapless compute bar — hide karne ko almost kuch nahi.

---

Ek-picture summary

KYA. Sab ek saath: ek warp load issue karta hai (Step 2), ek gap kholta hai (Step 3); doosre warps diagonally march karte hain ise fill karne ke liye (Step 4); aapko jitni zaroorat hai woh rectangle $L\times \Theta$ hai (Step 5); schedulers aur ILP warp count shrink karte hain (Step 6); aur "full" ke baad, extra warps waste hain (Step 7).

$\boxed{N_{\text{warps required}}=\frac{L\times \Theta }{S\times I}}\text{stop as soon as the gap is painted solid.}$

Recall Feynman retelling — simple words mein wapas bolo

Ek warp memory se ek number maangta hai aur phir kareeb 400 clock ticks tak so jaata hai jab tak number aata hai. Agar yeh tumhara akela warp hai, toh poora SM uske saath so jaata hai — ek bekar waste. Toh tum warps ka ek crowd paas rakhte ho: jab ek sota hai, scheduler doosre ko jagata hai, aur doosra, har idle tick ko real kaam se paint karta hua. Crowd mein kitne nappers chahiye? Exactly itne ki nap cover ho jaaye: nap ki lambai (400 ticks) times kitne instructions per tick tum issue karna chahte ho — yeh ek rectangle hai, aur uska area tumhara jawab hai. Lekin har warp ek instruction se zyada smart hai: 4 schedulers aur har warp mein pehle se kaafi saare independent operations in flight hone ke saath, sirf 16–32 warps ka crowd 400-tick nap cover kar leta hai. Aur sabse important baat — jab har tick paint ho jaaye, zyada warps add karna kuch nahi karta sirf registers hog karta hai. Full gap aim karo, full stadium nahi.

Recall

"Gap" jo hide hota hai woh physically kya hai? ::: Latency $L$ — woh ~400 idle cycles jo global load issue karne aur result ready hone ke beech mein hain. Required count ek product $L\times \Theta$ kyun hai? ::: Kyunki tumhe har cycle $\Theta$ instructions chahiye, $L$ cycles ke liye — yeh rectangle ka area hai. Tumhe literally 400 warps kyun nahi chahiye? ::: 4 schedulers parallel mein issue karte hain aur har warp ke kaafi saare independent ops in flight hote hain (ILP), requirement ko ~16 tak divide karte hue. Extra warps kab zero benefit dete hain? ::: Jab gap puri tarah cover ho jaaye (memory-bound) ya jab pipeline ILP se pehle se full ho (compute-bound).