GPU memory bandwidth optimization: Foundations — Hinglish deep dive

6.2.14 · D1 · Hardware › GPU Architecture › GPU memory bandwidth optimization

Is page par assume kiya gaya hai ki aapne kuch nahi dekha. Coalescing, banks, tiling, ya roofline ki baat karne se pehle, parent note mein use hone wala har letter earn karna padega. Upar se neeche padho — har symbol uske upar wale pe build hota hai.

0. Byte, address, aur "consecutive" access kya hota hai?

Parent note mein sab kuch data kahan rakha hai iske baare mein hai. To shuruat wahan se karte hain.

Figure — GPU memory bandwidth optimization

Figure s01 — address street. Memory ko ek lamba row of numbered byte-boxes ki tarah draw kiya gaya hai. Har box ek byte hai; uska number uska address hai. Blue block bytes A se A+3 tak cover karta hai — ye ek 4-byte float hai. Orange block uske bilkul baad A+4 se A+7 tak cover karta hai — agla float. Neeche green arrow mark karta hai ki ye do floats street par neighbours hain: dono ke beech koi gap nahi. Ye picture "consecutive" ki definition hai jis par poora page tika hai.

1. Threads, warps, aur wo lockstep mein kyun chalte hain

Warps kaise choose aur schedule hote hain, uske liye dekho GPU-Warp-Scheduling.

2. Memory transaction — truck ka fixed cargo size

Figure s02 — ek full truck vs. kai empty ones. Do warps dikhaye gaye hain, har ek ke 32 threads ek-ek float maang rahe hain. Top (green) row mein 32 floats side-by-side rakhe hain aur ek single 128-byte transaction mein fit ho jaate hain — ek full truck. Bottom (red) row mein wahi 32 floats bahut door-door bikhar gaye hain, isliye hardware ko kai alag-alag trucks bhejna padta hai, har ek mein sirf ek useful float aur bahut saari empty air. Chaahi gayi data ki amount same hai, trips ki sankhya bilkul alag.

Ab un pieces ko naam dete hain jo formula ko chahiye.

Best case, zero se worked out. 32 threads consecutive floats padhte hain, to $b = 4$ , lowest address $A$ , highest address $A + 31 \times 4$ : $span = (A + 124) - A + 4 = 128 bytes \Rightarrow ⌈ \frac{128}{128} ⌉ = 1 transaction.$

Worst case (stride 32). Thread $i$ reads $A + 32 i \times 4$ , to highest hai $A + 31 \times 128$ : $span = (A + 3968) - A + 4 = 3972 bytes \Rightarrow ⌈ \frac{3972}{128} ⌉ = 32 transactions.$

3. Efficiency $η$ — har truck kitna bhara hai

Consecutive case: $η = \frac{128}{1 \times 128} = 100%$ . Stride-32 case: $η = \frac{128}{32 \times 128} = 3.125%$ .

4. $B$ khud kahan se aata hai (raw bandwidth)

Aap usually $B$ calculate nahi karoge — datasheet deta hai (e.g. A100 ≈ 2 TB/s). Formula isliye matter karta hai taaki symbol $B$ mystery na rahe.

Parent note kai efficiencies ko saath multiply bhi karta hai: $η_{total} = η_{coalesce} \times η_{cache} \times η_{occupancy} .$ Har ek ek alag loss se ≤ 1 ka fraction hai. $η_{cache}$ (kam trips kyunki data already paas tha) Cache-Hierarchy ka kaam hai; $η_{occupancy}$ (road ko kabhi idle na rehne do) Memory-Latency-Hiding se jud jaata hai.

5. Shared memory aur banks — fast local shelf

Aap ise actually kaise program karte hain, uske liye dekho CUDA-Shared-Memory.

Bank formula se pehle, arithmetic notation ka ek chhota piece chahiye.

Definition Banks, bank-slot size

$s$ , aur Bank Conflicts Shared memory $N = 32$ banks mein split hoti hai — socho 32 parallel checkout counters. Har counter ek cycle mein ek thread serve kar sakta hai. $s$ ko bank-slot size mano — ek bank ke cell mein bytes ki sankhya (32-bit banks ke liye, $s = 4$ bytes). Ye wahi doosra "word" hai jiske baare mein hum §0 mein warn kar chuke hain — bank-slot, data-word nahi, halanki dono yahan 4 bytes hain.

Ek byte-address $A$ ko pehle ek slot-index $⌊ A / s ⌋$ mein convert kiya jaata hai (kaun sa cell number hai), aur phir us index ko ek counter par wrap kiya jaata hai: $Bank = ⌊ \frac{A}{s} ⌋ mod N$ Ek bank conflict tab hota hai jab warp ke kai threads alag-alag addresses chahte hain jo same counter par fall karte hain — unhe queue karna padta hai: ek bank par $k$ threads ko $k$ cycles lagte hain.

Figure s03 — banks as a clock face. 32 banks ek circle ke around draw kiye gaye hain, jaise clock par numbers hote hain. Slot-indices clockwise assign hote hain: 0, 1, 2, … 31, aur phir slot-index 32 seedha bank 0 par wrap back ho jaata hai. Orange dot byte-address 70 highlight karta hai: pehle slot size se floor-divide karo, $⌊ 70/4 ⌋ = 17$ , jisse slot-index 17 milta hai; phir $17 mod 32 = 17$ , to ye bank 17 par land karta hai. Clock ki shape hi modulo ko visible banati hai: 31 ke baad count karna loop kar jaata hai; yahan 17 abhi pehle lap mein hai, isliye bank = slot-index.

6. Work count karna: FLOPs aur arithmetic intensity

Figure s04 — the roofline. Horizontal axis arithmetic intensity $I$ hai (math per byte); vertical axis wo speed hai jo aap actually attain kar sakte hain. Black line ke do parts hain: left par ek slanted ramp (jahan memory road aapko limit karti hai) aur right par ek flat roof (jahan raw compute limit karta hai). Wo dotted vertical line par milte hain, machine balance $I^{⋆}$ . Red dot (naive, $I = 0.25$ ) bahut left aur neeche hai — starved. Orange dot (tiled, $I = 8$ ) ramp par chadh gaya hai lekin abhi bhi corner ke left mein hai. Green dot ( $I = 40$ ) flat roof par upar baitha hai — compute-bound. Poora message: $I$ badhana aapko ramp par rightward walk karta hai jab tak aap roof tak nahi pahunch jaate.

Parent ka tiling trick (parameter $T_{tile}$ = tile edge length) $I$ ko $0.25$ (no tiling) se utha kar $I^{⋆}$ tak aur usse age le jaata hai. Tiling data reuse kyun karta hai ye Parallel-Algorithm-Design ki kahani hai — aap ek value compute karte ho jab uske neighbours abhi shared memory mein hot hain.

7. Sab kuch kaise connect hota hai

Neeche ka diagram ek prerequisite map hai: ek arrow " $X \to Y$ " padho as " $Y$ samajhne se pehle $X$ samajhna zaroori hai." Har box ek concept hai jo humne abhi build kiya, same order mein jaise upar sections hain; arrows follow karo aur aap poora page re-walk kar lete ho. Sab kuch neeche parent topic mein funnel hota hai.

Equipment checklist

Right side cover karo aur zor se jawab do. Agar koi atkao, us section ko dobara padho.

Ek saadhi si sentence mein address kya hai?

Memory mein ek byte-box ki numbered position, jaise street par ek ghar ka number.

Is page par do alag "words" kaun se hain?

Machine data-word (DRAM mein 4-byte float) aur shared-memory bank-slot (4-byte bank cell) — dono yahan 4 bytes hain lekin alag ideas hain.

Memory ke liye warp size 32 kyun matter karta hai?

Saare 32 threads apne requests ek hi instant par issue karte hain, isliye hardware un simultaneous requests ko wide transactions mein merge kar sakta hai.

Symbol

b

kya represent karta hai?

Ek single thread jo bytes ek access mein request karta hai unki sankhya (e.g. ek float ke liye 4).

"Address span" ko formula ke roop mein define karo.

highest address touched − lowest address touched +

b

Transaction size

T_{tx}

kya hai, aur uski typical value?

Sabse chhota fixed chunk (≈128 bytes) jo controller kabhi bhi fetch karta hai; ek 4-byte request bhi poora

T_{tx}

-byte load kheench laati hai.

Transactions count karte waqt ceiling function

⌈ \cdot ⌉

ki zaroorat kyun hai?

Aap partial truck nahi bhej sakte — koi bhi bachha hua bytes ek aur full transaction force karta hai.

Ek perfectly consecutive 128-byte access phir bhi do transactions kyun cost kar sakta hai?

Agar wo

T_{tx}

-byte boundary se off shuru ho to wo do aligned segments mein straddle karta hai, do trucks mein split ho jaata hai.

Efficiency

η

ko words mein define karo.

Threads ne actually jo useful bytes use kiye, divided by transactions ne jo total bytes move kiye (0 = sab waste, 1 = perfectly full).

Delivered-throughput formula likho.

M = B \times η

Bandwidth formula mein 2 ka factor kyun hai?

DDR memory har clock tick ke rising aur falling edge dono par transfer karti hai, har dhadkan mein data double ho jaata hai.

Bank formula mein

s

kya hai aur address

A

se bank kaise milta hai?

s

bank-slot size hai (4 bytes);

⌊ A / s ⌋

compute karo slot-index ke liye, phir wo

mod 32

lo.

s = 4

ke saath byte-address 70 ka bank compute karo.

⌊ 70/4 ⌋ = 17

, phir

17 mod 32 = 17

→ bank 17.

Saare 32 threads active hone par bhi kab koi bank conflict nahi hota, aur catch kya hai?

Jab wo saare same address READ karein (hardware broadcast karta hai) ya saare distinct banks hit karein — lekin same address par writes merge nahi hote.

T_{tx}

aur

T_{tile}

mein fark karo.

T_{tx}

fixed 128-byte hardware DRAM transaction hai;

T_{tile}

software-chosen edge length hai ek matrix tile ka (e.g. 32).

B_{FLOP}

aur arithmetic intensity

I

define karo.

B_{FLOP}

peak compute FLOP/s mein hai;

I

hai FLOPs performed divided by bytes moved from DRAM.

Bandwidth-bound aur compute-bound mein kya fark hai?

I

ko machine balance

I^{⋆} = B_{FLOP} / B

se compare karo: neeche ho to bandwidth-bound, upar ho to compute-bound.