Foundations — Occupancy and latency hiding

6.2.10 · D1 · Hardware › GPU Architecture › Occupancy and latency hiding

Pehle aapko parent topic padhne ke liye chip ke andar kuch physical objects ki ek choti vocabulary chahiye. Hum har ek ko bilkul zero se build karte hain — pehle picture, phir word, phir symbol.

1. Thread — ek worker jo ek recipe follow karta hai

Akela thread koi badi baat nahi — actually CPU thread se bhi slow hai. Iska power tab dikhta hai jab aapke paas bahut saare hon. Isliye neeche har symbol threads count karta hai, ya threads ke groups, kisi ek ki parwah kiye bina.

2. Warp — 32 threads lockstep mein march karte hue

Yahan pehla idea hai jo parent note assume karta hai ki aap already jaante ho. Threads ek-ek karke schedule nahi hote. Inhe 32 ke fixed groups mein bundle kiya jaata hai, aur saare 32 ek hi instruction execute karte hain ek hi moment mein, har ek apne data par.

Jab bhi parent likhta hai "active warps", matlab hai: 32 threads ke groups jo abhi hardware par live hain, kuch karne ko ready hain.

3. Streaming Multiprocessor (SM) — rasoi ghar

Is page par har quantity per SM measure ki jaati hai. "64 warps per SM" matlab hai ek rasoi ghar ek time mein apne counters par zyada se zyada 64 warps ke workers rakh sakta hai.

4. Warp scheduler — caller jo decide karta hai aage kaun kaam karega

Modern SMs mein 4 aise schedulers hote hain, toh ek hi cycle mein 4 tak warps issue ho sakte hain.

5. Cycles aur latency — time ki unit, aur wait ki length

Parent note ke key numbers, saare cycles mein:

6. Throughput — aap har cycle mein kitna finish karte ho

Latency aur throughput ek hi pipeline ke baare mein do alag sawaal hain, aur inhe confuse karna classic galti hai. Ek pipeline mein lamba latency phir bhi high throughput ho sakta hai — jaise ek assembly line jahan har car mein ghante lagte hain (latency) lekin har minute ek roll off hoti hai (throughput).

7. Little's Law — latency se "kitne warps" tak ka bridge

Ab hum parent note ka central formula build kar sakte hain. Sawaal hai: kisi cheez ko har cycle busy rakhne ke liye, given ki har request $L$ cycles mein return hoti hai, kitni requests ek saath in flight mein honi chahiye?

Picture (s03) padhna: har row ek warp ki timeline hai. Ek warp load fire karta hai (red) phir $L$ cycles wait karta hai. Agar aap enough warps stack karo, har column (cycle) mein kam se kam ek warp useful issuing kar raha hoga — koi empty column nahi, koi idle SM nahi.

Worked plug-in. Ek scheduler $\lambda =1$ instruction per cycle issue karta hai. Ek DRAM load $L=400$ cycles leta hai. Tab: $N=1\times 400=400\text{ instructions in flight}.$ Ye naïve single-scheduler number hai jo parent quote karta hai. Ye practice mein overshoot karta hai kyunki concurrency SM-wide supply hoti hai (4 schedulers, plus per warp kaafi instructions in flight), toh 16–32 active warps zyaatar kaafi hote hain.

8. Occupancy — headline ratio

Ab har symbol earn ho gaya hai, toh hum topic ki namesake quantity state kar sakte hain.

Occupancy matter karti hai sirf isliye Little's Law ki wajah se: zyada active warps matlab zyada requests in flight matlab fixed latency $L$ ka zyada hissa hide hota hai — us point tak jahan aapke paas already enough hain, uske baad extra warps kuch nahi karte.

9. Floor function $\lfloor \cdot \rfloor$ — poori units tak round down karo

Parent ke occupancy-limit formulas $\lfloor \rfloor$ se bhare hain. Ye scary notation nahi hai.

Prerequisite map

Equipment checklist

Khud test karo — sirf zor se jawab dene ke baad reveal karo.