GPU memory bandwidth optimization: Question bank — Hinglish deep dive

6.2.14 · D5 · Hardware › GPU Architecture › GPU memory bandwidth optimization

Yeh page ek misconceptions ka question bank hai. Har item ek one-line reveal hai: prompt padho, apne dimag mein jawab socho, phir check karo. Goal arithmetic nahi hai (woh kahin aur hai) — yeh stress-test karna hai ki tum sochte kya ho GPU bytes kaise move karta hai ke baare mein.

Shuru karne se pehle, is page ko self-contained banate hain: har woh term jo traps mein use hoti hai, yahan define hai, picture ke saath.

Neeche ki picture mein ek warp ke 32 requests DRAM chunks ko hit kar rahe hain. Upar wali row (green) dekho: 32 saare addresses ek 128-byte chunk ke andar aate hain — ek trip. Neeche wali row (red): wahi 32 requests 32 chunks mein bikhre hue — 32 trips, har ek mostly wasted.

Figure — GPU memory bandwidth optimization

Do aur anchors jinpe traps rely karti hain

Jo efficiency tum actually paate ho woh teen alag-alag fractions ka product hai, har ek 0 aur 1 ke beech. Agar koi ek bhi chota ho, toh poora product chota ho jaata hai.

$η_{total} = η_{c} \times η_{h} \times η_{o}, M = B \times η_{total}$

Yahan $B$ raw bus bandwidth hai (bytes per second) aur $M$ woh throughput hai jo tum actually dekhte ho. Picture: series mein teen gates — har gate sirf ek fraction pass karta hai, toh end tak paani jo pohonchta hai woh product hai.

Arithmetic intensity aur machine balance — yahan derive kiya, kहीं se borrow nahi

Neeche ke traps mein do symbols aate hain. Tiled matrix multiply use karke inhe scratch se earn karte hain.

$I = T /4$ ki derivation (step by step):

Math count karo. Do $N \times N$ matrices multiply karne mein $N^{3}$ multiply-adds hote hain; har ek 2 FLOPs hai → total $2 N^{3}$ FLOPs.
Tiling ke saath DRAM bytes count karo. Tiling ke bina har output element DRAM se $2 N$ elements padhta hai. Tiling ek tile ko ek baar per block load karke reuse karta hai, global reads ko $T$ factor se cut karta hai, toh ab har output element $\frac{2 N}{T}$ global reads trigger karta hai. $N^{2}$ outputs aur 4 bytes per float ke saath: $Bytes = N^{2} \times \frac{2 N}{T} \times 4 = \frac{8 N ^{3}}{T}$
Divide karo. Intensity FLOPs over bytes hai: $I = \frac{2 N ^{3}}{8 N ^{3} / T} = \frac{T}{4} FLOPs/byte$ $N^{3}$ cancel ho jaata hai — intensity sirf tumhare chune hue tile par depend karta hai. $T = 32$ ke liye: $I = 8$ .

Machine balance example: ek GPU jiske paas 20 TFLOP/s peak compute aur 1 TB/s bandwidth hai: $I_{bal} = \frac{20 \times 1 0 ^{12}}{1 \times 1 0 ^{12}} = 20 FLOPs/byte$

Yeh rule set karta hai: agar $I < I_{bal}$ toh tum bandwidth-bound ho (data ki bhookh); agar $I > I_{bal}$ toh tum compute-bound ho (ALUs limit hain). Neeche ki picture yahi plot karti hai — roofline: ek sloped bandwidth ceiling jo ek flat compute ceiling se exactly $I_{bal}$ par milti hai.

Neeche jo bhi hai woh in ideas mein se kisi ek ke aas-paas, ya caches, shared memory banks, ya roofline ke aas-paas bana ek trap hai.

True or false — justify karo

Zyada threads hamesha zyada effective bandwidth dete hain

False. Bandwidth physical bus se cap hoti hai; extra threads sirf latency hide karke (Memory-Latency-Hiding) occupancy badhate hain — ek baar bus saturate ho jaye, toh zyada threads bas wait karte hain.

Ek fully coalesced kernel automatically fast hoti hai

False. Coalescing tumhe har transaction par high efficiency deta hai, lekin agar kernel compute-bound hai ya occupancy par bottleneck hai, toh memory kabhi limit thi hi nahi — coalescing ek aisi problem fix karta hai jo tumhare paas ho hi na.

Sabhi 32 threads se same address padhna 32-way bank conflict hai

False. Ek shared-memory broadcast identical address ko har lane ko ek cycle mein serve karta hai — conflict rule tabhi kaata hai jab threads same bank ko alag-alag addresses par hit karen.

Structure-of-Arrays hamesha Array-of-Structures se better hai

GPUs ke liye mostly true, lekin universal nahi. SoA field-wise access coalesce karta hai; lekin agar ek thread genuinely ek element ke saare fields ek saath chahta hai (use ki locality), toh AoS cache reuse par jeet sakta hai — "best" layout access pattern follow karta hai, koi slogan nahi.

Stride 32 ke saath strided access phir bhi "thoda sa" coalesce karta hai

Stride-32 floats ke liye False. 32 lanes mein se har ek ek alag 128-byte chunk mein land karta hai, toh tum 32 transactions pay karte ho aur har ek ka ek tees-wahaan fraction use karte ho — effectively fully un-coalesced (lagbhag teen percent efficiency).

Ek 32-by-32 shared array ko 32-by-33 pad karna memory bina kisi wajah ke waste karta hai

False. Extra column har row ki bank mapping ko ek se shift karta hai, transpose ke dauran column-access bank conflict tod deta hai — thodi si memory cost 32-times shared-throughput win khareedhti hai.

GPU par caches CPU ki tarah kaam karte hain

Emphasis mein False. GPU caches per-thread chote hain lekin aggregate demand bahut badi face karte hain; yeh mainly reuse pakadne aur transactions smooth karne ke liye exist karte hain, ek single thread ka working set CPU ke L1 ki tarah hold karne ke liye nahi.

Agar arithmetic intensity machine balance se neeche hai, toh tum bandwidth-bound ho

True. Roofline par, intensity

I_{bal}

se neeche matlab sloped bandwidth ceiling cap kar rahi hai — koi bhi faster ALUs tab tak help nahi karte jab tak tum

I

nahi badhate (jaise bade tiles se).

Coalescing aur bank conflicts ek hi phenomenon hain

False. Coalescing global DRAM transactions ke baare mein hai warp ke across; bank conflicts shared-memory parallel ports ke baare mein hain. Alag memories, alag fixes.

Error dhundho

"AoS with a 24-byte struct fully serializes into 32 separate reads."

Error: yeh partially coalesce karta hai. Ek warp

32 \times 24 = 768

bytes touch karta hai, jisme

⌈ 768/128 ⌉ = 6

transactions chahiye, 32 nahi. Yeh lagbhag paanch-chhahee bandwidth waste karta hai lekin worst case nahi hai.

"Tiling total arithmetic operations ki sankhya reduce karta hai."

Error: tiling memory traffic change karta hai, FLOP count nahi. Tum abhi bhi

2 N^{3}

FLOPs karte ho; bas operands bahut kam baar fetch karte ho, arithmetic intensity

I

badhate hue.

"Shared memory fast hai, toh poori matrix usme copy kar do."

Error: shared memory tiny hai (tens of KB per block). Tum tile exactly isliye karte ho kyunki woh poori matrix hold nahi kar sakti — tum

T \times T

tiles iske through stream karte ho.

"Bank = address mod 32, toh consecutive floats hamesha conflict karte hain."

Error: consecutive 4-byte words consecutive banks

0, 1, 2, \dots

se map hote hain, toh consecutive floats padhne wala ek warp 32 alag-alag banks hit karta hai — conflict-free ideal, koi conflict nahi.

"100 percent bandwidth paane ke liye bas har access coalesced karo."

Error: total efficiency

η_{c} \times η_{h} \times η_{o}

hai. Perfect coalescing (

η_{c} = 1

) low occupancy ke saath (

η_{o}

chota, controllers idle) phir bhi throughput peak se kaafi neeche chhod deta hai.

"Kyunki A100 ke paas 2 TB/s bandwidth hai, meri kernel 2 TB/s paayegi."

Error: 2 TB/s peak

B

hai. Real throughput

M = B \times η_{total}

hai; realistic

η_{total}

0.5 se 0.8 ke aas-paas hone par, tum peak ka ek fraction dekhte ho, well-optimized hone par bhi.

Why questions

Hardware 128-byte chunks kyun padhta hai exactly requested bytes ki jagah?

DRAM burst transfers ke liye optimize hai; fixed transaction addressing aur row-activation overhead amortize karta hai. Fine-grained single-byte reads overhead mein dub jaate, isliye bus wide design ki gayi hai.

Shared-memory banks full crossbar ki jagah kyun exist karte hain?

Ek sachi 32-by-32 crossbar (any thread to any address in one cycle) hardware-expensive hai. Banks memory ko partition karte hain taaki har address ek port se map ho, cheap 32-way parallelism deta hai — mapping collide karne par conflicts ki cost par. Dekho CUDA-Shared-Memory.

SoA 6 transactions per field ko 1 mein kyun badal deta hai?

SoA har field ko contiguously store karta hai, toh 32 lanes ek field padhte hue 32 consecutive floats maangti hain, exactly ek 128-byte chunk. AoS fields interleave karta hai, chahiye gaya field chunks mein scatter karta hai.

Tile size badhana ek kernel ko bandwidth-bound se compute-bound kyun flip kar sakta hai?

Arithmetic intensity tile size ke saath scale karta hai (

I = T /4

, upar derive kiya). Bade tiles har fetched byte ko zyada FLOPs ke liye reuse karte hain,

I

ko roofline par machine balance

I_{bal}

se aage push karte hue.

Broadcast bank conflict kyun avoid karta hai lekin scattered same-bank reads nahi karte?

Bank hardware ek fetched word ko sabhi lanes mein free mein fan out kar sakta hai (broadcast), lekin woh ek bank se alag-alag words simultaneously fetch nahi kar sakta — unhe serialize karna padta hai, ek per cycle.

Memory bandwidth, compute nahi, GPU bottleneck itni baar kyun hota hai?

Hazaaron cores data itni tezi se consume kar sakte hain jitna bus supply nahi kar sakti. Peak FLOP/s, peak bytes/s times typical intensity se bahut zyada hai, isliye zyaatar kernels data ke liye bhookhe rehte hain — roofline ka sloped region.

Edge cases

Coalescing efficiency kya hai jab ek warp global memory se single 4-byte float broadcast kare sabhi lanes ko?

Ek 128-byte transaction warp serve karta hai, aur haalaanki sirf 4 bytes unique hain, har lane genuinely woh value use karta hai, toh yeh fully coalesced hai (1 transaction) useful reuse ke saath, waste nahi.

Kya hota hai jab ek warp mein sirf lane 0 active ho (baaki 31 predicated off)?

Tum phir bhi lane 0 ke data ke liye ek full transaction (minimum chunk) pay karte ho akele — efficiency

η_{c}

4/128 \approx 0.031

tak gir jaati hai. Divergence aur masking bandwidth ko utna hi hurt karte hain jitna buri strides.

Ek pure memory copy (out[i]=in[i]) ki arithmetic intensity kya hai?

Zero FLOPs per byte moved, toh

I = 0

— roofline ki extreme left, purely bandwidth-bound. Iski ek hi optimization hai perfect coalescing; compute ke peechhe kuch bhi hide karne ke liye nahi hai.

Matmul derivation mein tile size

T = 1

ke liye, "tiled" formula kya collapse karta hai?

I = T /4 = 0.25

FLOPs/byte aur traffic reduction

= T = 1

times — yani exactly naive un-tiled case, formula ka sahi se degenerate hona confirm karta hai.

Agar ek warp ke 32 threads saare ek hi shared-memory bank ko 32 alag-alag addresses par access karen, toh kitna time lagta hai?

32 cycles — fully serialized, worst-case bank conflict, 1-cycle broadcast ka ulta.

Recall Quick self-check

Coalescing kis memory mein hota hai? ::: Global/DRAM transactions warp ke across. Bank conflicts kis memory mein hote hain? ::: Shared memory. Total efficiency teen kis factors ka product hai? ::: Coalescing $η_{c}$ , cache hit rate $η_{h}$ , occupancy $η_{o}$ . $I_{bal}$ kya hai? ::: Machine balance — peak FLOP/s divided by peak bytes/s; woh intensity jahan compute aur bandwidth match karte hain. $I_{bal}$ se neeche tum hote ho ::: bandwidth-bound.