Heterogeneous computing concepts: Foundations — Hinglish deep dive

6.1.12 · D1 · Hardware › Parallelism & Multicore › Heterogeneous computing concepts

Parent note padhne se pehle, tumhe har word aur symbol apna banana hoga jo woh throw karta hai. Yeh page unhe ek ek karke, zero se build karta hai. Upar se neeche padho — har block sirf wahi ideas use karta hai jo uske upar already define ho chuke hain.

1. Ek "processor" aur woh kya karta hai

Agar tum ek room mein bahut saari desks rakho, to tumhare paas bahut saare cores hain. Agar saari desks identical hain, to woh room homogeneous hai. Agar kuch desks badi-aur-smart hain aur kuch choti-aur-bahut-zyada hain, to room heterogeneous hai — yeh word literally Greek hai jiska matlab hai "different kinds".

Figure s01 neeche exactly yahi contrast draw karta hai: left mein, ek badi blue desk = CPU (ek powerful worker); right mein, choti green desks ka ek grid = GPU (bahut saare simple workers). Jaise jaise tum next callout padhte ho, do blocks dekho aur notice karo ki blue block ek box hai jabki green block chaubees hai — yeh size-vs-count trade-off hi poora picture hai.

Figure — Heterogeneous computing concepts

2. FLOP, FLOPS, aur kyun woh letter S matter karta hai

Parent note kaam ko "FLOPs" mein count karta hai. Yahan do almost-identical words chhupe hain — inhe mix karne se har formula kharab ho jaata hai.

Prefixes jo tumhe milenge (har ek pichle se ×1000 hai):

symbol	word	meaning
$G$	Giga	$1 0^{9}$ (ek billion)
$T$	Tera	$1 0^{12}$ (ek trillion)

To "10 TFLOPS" = har second dus trillion operations.

3. Time = Work ÷ Speed (master equation)

4. Bandwidth $B$ , data size $D$ , aur " $2 D / B$ " tax

GPU us data ko touch nahi kar sakta jo CPU ki memory mein hai. Data ko pehle ek wire ke upar copy karna padta hai. Woh wire slow hai, aur copy mein time lagta hai.

Same master equation, naye naam: box move karne ka time = box size ÷ pipe width = $D / B$ .

Figure s02 neeche round trip ko concrete banata hai. Blue box (CPU memory) aur green box (GPU memory) ek grey pipe ke dono siro par hain jiska width $B$ hai. Yellow arrow send leg hai (CPU→GPU) jisme $D / B$ lagta hai; neeche red arrow return leg hai (GPU→CPU) jisme ek aur $D / B$ lagta hai. Dono arrows follow karo aur tum literally un do trips ko count karte ho jo neeche likhe tax mein jod ke aate hain.

Intuition Tax sirf

pay nahi karna padta, hide bhi ho sakta hai $2 D / B$ (ya $D / B$ ) bytes move karne ki raw cost hai. Lekin agar pipe aur GPU ek saath run kar sakein, to tum agli chunk of data copy kar sakte ho jabki GPU current chunk crunch kar raha ho — transfer time compute time ke peeche chhup jaata hai instead of add hone ke. Yeh overlapping hai (a.k.a. communication ko computation ke saath pipelining karna). Jab overlap perfect ho, effective tax jo tumhe actually feel hota hai woh neeche ki taraf jaata hai: $T_{transfer, effective} = max (T_{transfer}, T_{compute}) - T_{compute},$ jo zero hota hai jab bhi compute, transfer se lamba ho. Isi wajah se real GPU code data ko chunks mein split karta hai aur stream karta hai — parent note ka "pipelined model" exactly yahi idea hai; Parallel Programming Models dikhata hai ki ise kaise express karein.

5. Arithmetic intensity — make-or-break ratio

Ab punchline. Do chips race karte hain, lekin sirf fast chip transfer tax pay karta hai. Kab yeh pay karna worth it hai?

Figure s03 neeche us ratio ko problem size $n$ ke against plot karta hai. Green curve (matrix × matrix) hamesha upar chadhti rehti hai kyunki uski intensity $n$ ki tarah badhti hai; red curve (vector add) flat aur low rehti hai. Yellow dashed line transfer tax ka break-even threshold hai — tum sirf shaded green region mein "jeette" ho jo line ke upar hai, jisme green curve enter karti hai aur red curve kabhi nahi pahunchti. Woh single picture explain karti hai ki kyun kuch workloads GPU ke liye hain aur kuch kabhi nahi hote.

6. Fractions $f_{s}$ aur $f_{p}$ — ek program ko do mein split karna

Amdahl's Law ko ek aur idea chahiye: ek program part "must-be-sequential" aur part "can-be-parallel" hota hai.

Ab hum woh last symbol define karte hain jis par ceiling tiki hai — normalized transfer overhead.

Figure s04 neeche payoff dikhata hai. Har curve overall speedup hai jaise GPU faster hota jaata hai (right ki taraf move karte hue). Notice karo ki har curve ek horizontal ceiling mein flatten ho jaata hai — same colour ki dotted line — chahe tum kitna bhi right jaao. Chota serial fraction (green, $f_{s} = 0.05$ ) ka ceiling high hai; bada wala (red, $f_{s} = 0.5$ ) low pe cap hai. Woh flattening hi Amdahl's Law hai.

Intuition

Bounded speedup kyun hota hai — ceiling step by step derive karna Heterogeneous run time se shuru karo, har term original run ke fraction ke roop mein: $T_{hetero} = serial, no speedup f_{s} + parallel, S_{GP U} \times faster \frac{f _{p}}{S _{GP U}} + normalized tax t_{xfer} .$ HAM AAGE KYA KARTE HAIN: pucho kya hota hai jab GPU infinitely fast ho, yani $S_{GP U} \to \infty$ let karo. KYUN: yeh best case hai — yeh hard floor expose karta hai jo koi hardware beat nahi kar sakta. Step: jaise $S_{GP U} \to \infty$ , beech wala term $\frac{f _{p}}{S _{GP U}} \to 0$ (kisi bahut badi cheez se divide karne par). Serial slice $f_{s}$ aur tax $t_{xfer}$ nahi shrink karte — faster GPU ke baare mein kuch bhi unhe touch nahi karta. To time yahan bottom out karta hai: $T_{m i n} = f_{s} + t_{xfer} .$ Kyunki $S = T_{old} / T_{new}$ aur $T_{old} = 1$ (normalized), best possible speedup hai: $S_{m a x} = \frac{1}{f _{s} + t _{xfer}} .$ Yeh kaisa dikhta hai: figure s04 mein horizontal dotted ceilings. $f_{s} = 0.05$ aur koi transfer tax nahi hone par, $S_{m a x} = 1/0.05 = 20 \times$ — tum chahe GPU kitna bhi monstrous ho, 20× se zyada nahi beat kar sakte. Amdahl's Law and Scalability mein deeper dive karo.

7. Performance-per-watt — accelerators aslaan kyun exist karte hain

Prerequisite map

Left par har foundation beech mein ek formula ko feed karta hai, aur sab milke right par parent topic ko feed karte hain: Heterogeneous Computing. Jab tum yeh splits code mein express bhi karna chahte ho, to Parallel Programming Models padho.

Equipment checklist

Right-hand side cover karo aur dekho ki kya tum har ek ka jawab reveal karne se pehle de sakte ho.

Homogeneous aur heterogeneous cores mein kya fark hai?

Homogeneous = sare identical cores/instruction sets; heterogeneous = do ya zyada alag tarah ke cores, har ek alag kaam ke liye tuned.

FLOP aur FLOPS mein kya fark hai?

FLOP ek floating-point operation hai (kaam ka count); FLOPS FLOPs per second hai (ek speed).

Work, speed aur time ko link karne wali master equation batao.

T = W / F

— time equals work divided by processing rate; iske units FLOP ÷ (FLOP/s) = seconds hain.

Round-trip transfer time mein 2 ka factor kyun hai, aur kab yeh sirf 1 hota hai?

Data CPU→GPU aur phir GPU→CPU travel karta hai (do trips →

2 D / B

); agar answer kabhi wapas nahi aata (one-way), to sirf

D / B

hota hai.

Overlapping kya hai aur yeh effective tax kaise reduce karta hai?

Copy aur GPU compute ek saath run karna taaki transfer compute ke peeche chhup jaaye; jab compute lamba ho, effective transfer tax zero ki taraf jaata hai.

Arithmetic intensity define karo aur batao high vs low ka kya matlab hai.

W / D

(FLOPs per byte); high = har byte move karne ke liye bahut compute (offload karna worth it), low = mostly data movement (worth it nahi).

f_{s} + f_{p} = 1

kyun hai?

Yeh ek poore program ke serial aur parallel slices hain, isliye unke fractions poore (1) mein add hote hain.

t_{xfer}

kya hai aur yeh

f_{s}

ke saath kaise reh sakta hai?

Transfer time divided by original run time — ek dimensionless fraction, isliye yeh

f_{s}

aur

f_{p}

ke units se match karta hai.

Infinitely fast GPU hone par bhi speedup ko kya cap karta hai?

Fixed costs jo shrink nahi karte: serial fraction plus normalized transfer overhead, jo

S_{m a x} = 1/ (f_{s} + t_{xfer})

deta hai.

Performance-per-watt kya measure karta hai aur yeh accelerators ko kyun justify karta hai?

Bijli ke har watt ke liye FLOPS; specialized hardware har unit energy ke liye bahut zyada compute karta hai, jo phones aur data centres mein asli budget hai.