Visual walkthrough — SIMT (single instruction multiple thread)

6.2.4 · D2 · Hardware › GPU Architecture › SIMT (single instruction multiple thread)

Parent note ne ek gehri result almost passing mein bata di thi:

$W\ge L\cdot T$

"Latency $L$ ko hide karne ke liye, tumhe kam se kam $L\cdot T$ warps chahiye." Yeh akela line hi poori wajah hai ki GPUs aaj jaise dikhte hain. Yeh page ise bilkul scratch se build karta hai — koi formula pehle se assume nahi, har symbol pehle earn kiya — aur har step par pictures use karta hai.

Hum discover karenge kyun ek GPU hazaron threads ko ek saath flight mein rakhta hai, aur kyun ise woh clever speculation tricks nahi chahiye jo ek CPU use karta hai. Contrast ke liye GPU vs CPU Architecture dekho, aur is derivation se jo tuning knob milta hai uske liye GPU Occupancy dekho.

---

Step 1 — Warp kya hai, ek moving object ke roop mein?

KYA. Kisi bhi formula se pehle, humein ek clean picture chahiye us cheez ki jo hum count kar rahe hain. Ek warp 32 threads ka ek bundle hai jo lockstep mein chalta hai: Streaming Multiprocessor ek instruction issue karta hai aur saare 32 lanes ise usi cycle mein obey karte hain.

KYUN. Poori derivation warps count karti hai, threads nahi. Agar "warp" fuzzy raha, toh baad ke har number bhi fuzzy honge. Isliye hum ise ek single object ke roop mein pin down karte hain jo time mein ek instruction at a time move karta hai.

PICTURE. Neeche, ek warp ek horizontal track hai. Ek filled block = "is cycle mein, is warp ne kaam kiya". Abhi sirf ek warp hai, aur woh exactly ek instruction per cycle karta hai.

Abhi kuch compute nahi karna — humne sirf time ($\text{cycle}$) aur goal ($T$) ka naam rakha.

---

Step 2 — Stall: "latency" actually kya cost karti hai

KYA. Kuch instructions ek cycle mein finish nahi hoti. Ek memory read ko off-chip aur wapas travel karna padta hai — is waiting time ko latency $L$ bolte hain, cycles mein measure kiya hua.

KYUN. Hum $L$ ko abhi introduce kar rahe hain kyunki yeh poori story ka villain hai. Iske baad jo bhi aata hai, woh sab $L$ ko hide karne ke liye exist karta hai.

PICTURE. Warp cycle $0$ par ek load issue karta hai, phir track $L$ cycles ke liye blank ho jaata hai — warp frozen hai, data ka wait kar raha hai. Sirf $L$ cycles ke baad hi woh phir move kar sakta hai.

$\underset{\text{cycles jisme warp frozen hai}}{\underbrace{L}}\approx 400\text{ (typical global-memory read)}$

---

Step 3 — Fix: wait mat karo, switch karo

KYA. Frozen Warp 0 ko ghoorte rehne ki jagah, warp scheduler simply ek alag warp pick karta hai jo ready hai, aur uski instruction issue karta hai. Warp 0 ka stall background mein chala jaata hai — free time jo hum kisi aur par spend karte hain.

KYUN. Yahi key move hai. Hum memory ko faster nahi banaate; hum wait ko invisible banate hain hamesha haath mein doosra kaam rakh ke. Isliye GPUs itne saare warps run karte hain: "parallel for its own sake" nahi, balki latency ko paper over karne ke liye.

PICTURE. Ab chaar tracks hain. Warp 0 frozen hai (blank), lekin har cycle par scheduler alag ready warp par jaata hai aur uski instruction fire karta hai (colored blocks). Core busy rehta hai chahe Warp 0 so raha ho.

---

Step 4 — Counting: kitne warps stall ko exactly hide karte hain?

KYA. Ab hum count karte hain. Un $L$ cycles ke dauran jab Warp 0 so raha hai, hum chahte hain ki doosre warps se har cycle $T$ instructions issue hon taaki core kabhi idle na ho.

KYUN. Isse picture arithmetic mein convert hoti hai. Hum precise question poochhte hain: itne warps exist karne chahiye ki gap hamesha covered rahe?

PICTURE. Tracks ko stack karo. Warp 0, width $L$ ki ek window ke liye sota hai. Us window mein humein har column busy rakhne ke liye kaafi doosre filled blocks chahiye. Unhe count karo.

Window ki har cycle mein $T$ instructions chahiye. Window $L$ cycles wide hai. Toh fill karne ke liye total instruction-slots hain:

$$\underset{\begin{array}{c}\text{gap ki}\\ \text{width}\end{array}}{\underbrace{L}}\times \underset{\begin{array}{c}\text{slots jo hum}\\ \text{per cycle fill karein}\end{array}}{\underbrace{T}}=\underset{\begin{array}{c}\text{stall cover karne ke liye}\\ \text{total slots}\end{array}}{\underbrace{L\cdot T}}$$

Ab, woh slots kitne warps supply karte hain? Ek warp, ek instruction fire karne ke baad, thoda busy/unavailable rehta hai dobara fire karne se pehle. Sabse simple case mein har warp ek cycle-slot par ek instruction contribute kar sakta hai, toh humein ek slot per ek warp chahiye:

$$\boxed{W\ge L\cdot T}$$

---

Step 5 — Refinement: warps jo har cycle fire nahi kar sakte

KYA. Real warps har cycle issue nahi kar sakte — issue karne ke baad, ek warp apni pipeline mein busy rehta hai aur average par sirf har $c$ cycles mein ek baar fire kar sakta hai. Toh ek warp ek slot per cycle cover nahi karta; yeh har $c$ cycles mein ek slot cover karta hai.

KYUN. Is correction ke bina count zyada optimistic hai. Humein divide karna hoga is baat se ki ek single warp kitni baar available hota hai.

PICTURE. Ek single warp ki track mein ab blocks $c$ cycles apart hain (issue, phir gap, phir issue). Har cycle ek column filled rakhne ke liye clearly zyada warps chahiye jo un gaps mein interleave hon.

$$W\ge \frac{L\cdot T}{c}\underset{\begin{array}{c}\text{ek warp ke issues ke beech}\\ \text{cycles}\end{array}}{\underbrace{c}}$$

---

Step 6 — Reality wall: SMs 32–64 warps par cap out ho jaate hain

KYA. Yahan twist hai. Ek real SM zyada se zyada 32–64 warps hold kar sakta hai — woh 100 ke kareeb bhi nahi jo humne abhi compute kiya. Toh ek single SM akela latency fully hide karne mein often nahi kar sakta.

KYUN. Yeh edge case ek footnote nahi hai — yeh wahi wajah hai ki GPUs jaise bane hain waise bane hain. Jab ek SM enough warps hold nahi kar sakta, tum deeper SM nahi banaate; tum bahut saare SMs banaate ho aur poore chip ki occupancy ko hiding karne do.

PICTURE. Left: humein kya chahiye (100 warps). Right: ek SM ke paas kya hai (maan lo 64). Shortfall bar woh gap hai jo architecture ko die par SMs replicate karne par majboor karti hai.

---

Step 7 — Degenerate cases (koi bhi scenario chhode bina)

KYA. Hum $L$, $T$, aur $W$ ki boundary values check karte hain taaki koi reader kisi aisi case par na aaye jise humne skip kiya.

KYUN. Woh formula jis par tum trust karte ho woh hai jiske extremes sense banate hain. Inhe test karte hain.

PICTURE. Teen mini-tracks: (a) $L=0$ koi latency nahi; (b) $W=1$ sirf ek warp; (c) $W$ bahut bada, kaafi se zyada.

Case	Matlab	Kya hota hai
$L=0$	instruction instantly finish ho jaati hai	$W\ge 0$ — ek warp bhi core ko busy rakhta hai; koi hiding zaruri nahi.
$W=1$, $L$ bada	single warp, lamba stall	core pure $L$ ke liye idle rehta hai — Step 2 ka $400\times$ waste wapas aata hai.
$W\ge L\cdot T/c$	enough (ya zyada) warps	stall fully hidden; extra warps ready baithte hain, koi nuksaan nahi (jab tak registers khatam na hon).
$T=0$	hum koi kaam nahi karna chahte	trivially satisfied — lekin pointless; hum hamesha $T\ge 1$ chahte hain.

---

Ek-picture summary

Upar ki saari baat ek image mein compress ho jaati hai: width $L$ ka ek stall, $L\cdot T/c$ interleaved warps se column-by-column fill kiya gaya, SM jo physically hold kar sakta hai usse cap kiya gaya.

Recall Feynman retelling — ise simple words mein wapas bolo

Sochho ek worker hai jo ek package order fire karti hai aur phir uske aane ke liye 400 seconds wait karti hai. Agar woh bas wait karti hai, toh 400 seconds waste ho jaate hain. Toh ek boss (scheduler) kehta hai: "Wait mat karo — jab tumhara package aa raha hai, kisi aur ka next step karo." Boss ko har ek second kaam dete rehne ke liye, humein itne workers chahiye ki har moment par kam se kam ek ke paas ek step ready ho. Kitne? Wait time (400) ko har second ke kaam ki matra (1) se multiply karo → 400 slots fill karne hain. Agar har worker sirf har 4 seconds mein act kar sake, divide karo → lagbhag 100 workers chahiye. Lekin ek room (ek SM) sirf 64 fit karta hai. Toh ek giant room banane ki jagah, chip bahut saare rooms (bahut saare SMs) banata hai. Lesson: GPU slowness ko clever hoke nahi, balki hamesha ready kaam ki ek badi crowd rakh ke hide karta hai — woh crowd occupancy kehlati hai, aur isliye hum ise high rakhte hain.

Recall

$L\cdot T$ kya quantity count karta hai? ::: Un instruction-slots ki total sankhya jo doosre warps ko fill karni hoti hain $L$ cycles ke stall ko throughput $T$ par cover karne ke liye. $c$ se divide kyun karte hain? ::: Kyunki ek single warp sirf har $c$ cycles mein ek baar issue kar sakta hai, toh har warp har $c$ cycles mein ek slot cover karta hai, na ki ek per cycle. $L=400$, $T=1$, $c=4$ ke saath kitne warps? ::: $400/4=100$ warps. Real GPUs deeper pipelines ki jagah bahut saare SMs kyun use karte hain? ::: Ek SM ~32–64 warps par cap karta hai, ~100 ki zarurat se kaafi kam, toh poore chip ka warp count (bahut saare SMs ke across) latency hiding supply karta hai. SIMT latency ko CPU se alag kaise hide karta hai? ::: Massive multithreading se (ready warp par switch karo), speculation ya out-of-order execution se nahi.

Yeh bhi dekho: SIMT (single instruction multiple thread) · Warp Divergence · GPU Memory Coalescing · CUDA Thread Hierarchy · Streaming Multiprocessor · Instruction-Level Parallelism · SIMD vs SIMT Comparison