Tensor cores and matrix operations: Question bank — Hinglish deep dive

6.2.12 · D5 · Hardware › GPU Architecture › Tensor cores and matrix operations

Shuru karne se pehle, hum kuch specific words aur symbols par rely karte hain. Hum har ek ko use karne se pehle define karte hain — including floating-point formats aur software libraries jinhe baad ke questions assume karte hain.

Definition Neeche use hone wale software aur scheduling terms

CUDA core: GPU ka general-purpose scalar unit — ek FMA per clock, saara non-matrix work handle karta hai (activations, control flow). Tensor Core se contrast karo, jo ek instruction mein poora tile karta hai (dekho 6.2.1-GPU-vs-CPU-architecture).
cuBLAS / cuDNN: NVIDIA ki pre-tuned libraries dense linear algebra aur neural-network layers ke liye. Ye automatically qualifying operations ko Tensor Cores par dispatch karte hain, toh tumhe kernels haath se likhne nahi padte (dekho 7.3.6-cuBLAS-and-cuDNN).
Roofline model: ek picture jo achievable performance ko arithmetic intensity (FLOPs done per byte moved) ke against plot karta hai. Ye dikhata hai ki tum compute se limited ho ya memory bandwidth se (dekho 9.1.5-Roofline-model aur neeche figure s04).
Warp→fragment mapping: kaise ek matrix tile ke cells warp ke 32 threads mein scattered hote hain. Har thread ek chhota, non-obvious fragment rakhta hai — koi thread ek clean row ya output cell nahi own karta (neeche figure s03).

Definition Mixed precision aur MMA formula

Poora Tensor Core operation hai MMA $D = A \times B + C$ : $A$ ko $B$ se multiply karo, phir ek alag matrix $C$ (accumulator/bias) add karo, $D$ milega. Yahan $C$ product nahi hai — ye woh cheez hai jo tum add karte ho.

Mixed precision ka matlab hai: inputs ko low-precision format mein multiply karo (kam bits, cheap aur fast) lekin running sum ko high-precision accumulator mein rakho (zyada bits, error buildup se bachata hai). Formally, $K$ pe inner sum ke liye: $D [i] [j] = products in low precision k = 0 \sum K - 1 (A [i] [k] \cdot B [k] [j]) + C [i] [j] (sum kept in high precision)$ Figure s02 dikhata hai ki wide accumulator kyun matter karta hai — chhote FP16 products ek coarse bucket mein kho jaate hain unless bucket khud fine-grained ho.

Figure s02 — coarse vs fine accumulator. Wahi chhote products ek coarse FP16 bucket ke gaps se gir jaate hain lekin ek fine FP32 mein survive karte hain.

Do aur visuals jinhe tumhe baad mein point kiya jaayega. Abhi ek nazar dalo:

Figure s03 — warp→fragment mapping. 32 warp threads mein se har ek output tile ke scattered cells rakhta hai (colours dikhate hain kaunsa thread kaunsa cell own karta hai), toh koi thread ek clean row nahi own karta — isliye mapping ko "opaque" kehte hain.

Figure — Tensor cores and matrix operations

Figure s04 — roofline. Performance arithmetic intensity ke saath badhti hai jab tak compute ceiling hit nahi hoti; ridge ke left mein tum memory-bound ho aur Tensor Cores idle baithte hain, right mein woh full speed par chalte hain.

True or false — justify karo

True or false: Ek Tensor Core single thread ko 4×4 matrix multiply karwata hai.

False. Instruction warp-level hai (WMMA operation) — warp ke saare 32 threads cooperate karte hain, har ek sirf A, B, aur accumulator ka ek tiny fragment rakhta hai (dekho figure s03). Koi akela thread kabhi poora tile nahi dekhta.

True or false: Tensor Cores CUDA cores ko replace karte hain.

False. Woh usi SM ke andar CUDA cores ke saath rehte hain. CUDA cores ab bhi saab kuch handle karte hain jo dense matrix multiply nahi hai — activations, reductions, control flow, older chips par FP64.

True or false: FP16 inputs use karne ka matlab hai ki poora matrix multiply FP16 mein hota hai.

False. Inputs FP16 hain lekin running sum FP32 mein rakha jaata hai (accumulator). Yahi mixed precision ka poora point hai — saste multiplies, safe additions.

True or false: TF32 mein 32 bits of precision hoti hai.

False. Naam ke bawajood, TF32 sirf 10-bit mantissa rakhta hai (FP16 jaisa) FP32 ke exponent range ke saath. "32" exponent range refer karta hai, total precision nahi. Mantissa error

\approx 2^{- 10}

hai,

2^{- 23}

nahi.

True or false: BF16 aur FP16 same 16-bit float ke alag naam hain.

False. Dono 16 bits hain, lekin BF16 zyada bits exponent par spend karta hai (wide range, coarse steps) aur FP16 zyada mantissa par (narrow range, fine steps). BF16 gradients par rarely overflow hota hai; FP16 hota hai — dekho 8.3.4-Mixed-precision-training.

True or false: Tensor Cores koi bhi matrix multiply jo tum unhe do usse speed up karte hain.

False. Agar operation memory-bound hai (dekho roofline model, figure s04, aur 9.1.5-Roofline-model), cores data ka wait karte hue idle baithte hain. Woh tabhi help karte hain jab tumhare paas per byte kaafi arithmetic ho unhe feed karne ke liye, aur dimensions jo cleanly tile karein.

True or false: Kyunki matrix multiply associative hai, tiling aur partial products accumulate karna exactly FP32 result deta hai.

Floating point mein False. Exact math mein associativity hold karta hai, lekin additions ko reorder karna rounding change karta hai. Results close hain, bit-identical nahi — runs compare karte waqt surprise ka ek real source.

True or false: Sparse ("structured sparsity") Tensor Cores automatically tumhare liye zeros skip kar dete hain.

False. Ampere/Hopper par 2:4 structured sparsity feature ko weights ko ek fixed pattern mein prune karna padta hai (har 4 ke group mein 2 nonzeros), phir offline compress karna padta hai. Sirf woh enforced pattern extra ~2× earn karta hai — arbitrary zeros kuch nahi dete.

True or false: Tensor Cores turn on karne ke liye apna model WMMA API mein rewrite karna padta hai.

False. Libraries jaise cuBLAS aur cuDNN (aur unke upar PyTorch) automatically Tensor Cores par dispatch karte hain jab dtype aur shapes qualify karte hain. WMMA (warp-level MMA) API sirf haath se likhe kernels ke liye hai.

Error dhundho

"Maine apni matrices float16 set ki lekin koi speedup nahi mili, toh Tensor Cores zaroor broken hain."

Likely error shape hai. Agar

M

N

K

8/16 ke multiples nahi hain, toh library CUDA cores par fall back kar deti hai. Hardware ko blame karne se pehle

M

N

K

alignment check karo.

"Main FP16 mein bhi accumulate karunga — ye faster hai aur memory bachata hai."

Error accuracy ki hai.

K

additions mein, chhote FP16 rounding errors compound hote hain aur result ko swamp kar sakte hain. FP32 accumulator bilkul isi drift ko rokne ke liye exist karta hai; use drop karna mixed precision ko defeat kar deta hai.

"Mera inference model INT8 weights use karta hai, toh accumulator bhi INT8 hai."

Galat. INT8 products ek INT32 accumulator mein sum hote hain. Sau INT8×INT8 products add karna 8 bits ko almost immediately overflow kar dega — dekho 8.4.2-Quantization-techniques.

"Volta Tensor Cores physically 16×16 hain, isliye WMMA tile 16×16 hai."

Error do levels ko conflate karta hai. Hardware sub-partition 4×4 hai; 16×16 woh WMMA API-exposed tile hai jo warp ke across kai 4×4 sub-multiplies se bana hai. Programmer 16×16 dekhta hai; silicon 4×4 mein kaam karta hai.

"Warp ka har thread output tile ka ek element compute karta hai."

False mapping. Warp→fragment mapping opaque aur non-obvious hai (dekho figure s03) — ek thread scattered pieces rakhta hai, ek clean output cell nahi. Kabhi assume mat karo ki threadIdx ek output element index karta hai.

"Maine training faster banane ke liye INT8 mein quantize kiya."

Error INT8 ko training ke liye use karna hai. INT8 ek inference technique hai training ke baad; gradients ko BF16 ya FP16 ka range/precision chahiye. INT8 mein backprop ke liye dynamic range nahi hoti.

"cuBLAS slow hai, toh main apna WMMA kernel likhuga — ye use beat karega."

Aam taur par galat. cuBLAS tiling, memory staging, aur memory hierarchy ke across pipelining ke liye haath se tune kiya gaya hai. Naive WMMA kernels generally isse badly haarte hain.

Why questions

Tensor Core tile ko 4×4 ki jagah 64×64 kyun nahi banate — bada toh faster hoga, na?

Ek tile warp ke registers mein fit hona chahiye. Bahut bada hone par woh slower memory mein spill karta hai (net loss), aur edge cases mein hardware waste karta hai jahan matrices ise fill nahi karti. 4×4 register fit aur instruction-decode cost amortize karne ke beech balance banata hai.

Accumulator inputs se wider kyun hota hai?

Kyunki

K

products ka sum kisi bhi single product se bahut bada magnitude range span karta hai (dekho figure s02). Ek wide FP32/INT32 accumulator woh range capture karta hai bina overflow ke ya chhote terms rounding se khoye bina.

Dimensions ke 8 ke multiples hona kyun matter karta hai, jab API tile 16×16 hai?

Do alag granularities collide karti hain. 16 WMMA API tile size se aata hai; 8 is baat se aata hai ki warp ke fragments register file mein kaise pack hote hain (aur, kuch precisions ke liye,

K

-step 16 nahi 8 hota hai). Ek dimension jo chhoti granularity, 8, ka multiple nahi hai, already clean fragment packing tod deta hai aur tumhe slow path par le jaata hai — toh 8, 16 nahi, "kya ye Tensor Cores hit karega" ke liye real floor hai.

TF32 "free" speedup kyun deta hai bina koi code changes ke?

Kyunki ye ordinary FP32 tensors accept karta hai aur Tensor Core ke andar silently mantissa truncate karta hai. Tumhara code FP32 declare karta rehta hai; hardware trimming karta hai, toh API surface unchanged rehta hai (Ampere+ default).

Tensor Cores exist kyun karte hain agar CUDA cores wahi math kar sakte?

Ek CUDA core ek FMA per clock karta hai; deep learning ko billions chahiye. Ek Tensor Core 64 FMAs ek instruction mein pack karta hai, instruction count 64× cut karta hai aur throughput 8–16× boost karta hai — dekho 6.2.1-GPU-vs-CPU-architecture.

Ek Tensor Core ek bade matrix multiply ke dauran idle kyun ho sakta hai?

Agar data kaafi fast nahi aa sakta (bandwidth-bound region of the roofline, figure s04), toh cores starve ho jaate hain. Peak FLOPS tabhi appear hoti hai jab arithmetic intensity itni high ho ki memory latency hide ho sake — dekho 9.1.5-Roofline-model.

Edge cases

16×16 tiles par 17×17 matrix multiply ka kya hoga?

Library ko har dimension ko 16 ke agle multiple tak round up karna padega: dono dimensions mein

17 \to 32

, toh ek

32 \times 32

padded region chaar 16×16 tiles cover karta hai. Useful work

17 \times 17 = 289

cells hai

32 \times 32 = 1024

mein se, toh

1 - 289/1024 \approx 71.8%

wasted padding hai — ek tiny overshoot utna hi costly hai jitna agle tile boundary tak.

Agar

K

(inner dimension) 1 ho — ek vector outer product?

Tile ke overhead ko amortize karne ke liye almost koi arithmetic nahi hai, aur loaded data reuse karne ke liye koi accumulation loop nahi hai. Ye strongly memory-bound hai; Tensor Cores yahan little se no benefit dete hain.

Agar tum ek Volta GPU par FP32 tensors feed karo (koi TF32 nahi)?

Volta Tensor Cores sirf FP16 inputs accept karte hain, toh pure FP32 matmul CUDA cores par chalta hai, Tensor Cores par nahi. Tum FP16 mein cast karne tak koi Tensor Core speedup nahi lete.

Agar INT8 quantization ke baad koi weight exactly zero ho?

Woh cleanly zero se multiply hota hai aur INT32 sum mein kuch contribute nahi karta — koi special handling nahi. Quantization ka real risk nonzero weights hai jo galat integer par snap hote hain, zeros nahi.

Early Tensor Core par FP64 scientific work ka kya hota hai?

Early (Volta/Turing) Tensor Cores ka koi FP64 mode nahi hai, toh kaam CUDA cores par chalta hai. Sirf Ampere/Hopper range-sensitive simulations ke liye FP64 Tensor Core paths add karte hain.

Agar ek matrix mein ek FP16 value ho jo multiply ke dauran overflow kare?

FP16 ka narrow range (max

\approx 65504

) accumulation hone se pehle hi

+ \infty

par overflow kar sakta hai. Isliye gradient-heavy kaam BF16 ya loss-scaling prefer karta hai — dekho 8.3.4-Mixed-precision-training.

Underflow ke baare mein kya — ek gradient FP16 ke liye bahut chhota?

\approx 6 \times 1 0^{- 8}

se neeche FP16 subnormals mein slide karta hai (reduced precision) aur phir zero par flush hota hai — gradient silently vanish ho jaata hai. Loss-scaling loss ko backprop se pehle upar multiply karta hai taaki ye values FP16 ki range mein waapis aa sakein.

Kya hoga agar ek input ek NaN ho (e.g. upstream

0 \times \infty

NaN propagate karta hai: koi bhi FMA jo isse touch kare NaN produce karta hai, jo phir us output cell ke poore accumulator ko poison kar deta hai. Ek kharab value poore output tile ko NaN mein badal sakti hai, isliye source par

\infty

aur

0 \times \infty

se guard karo.

Structured sparsity ko Ampere/Hopper par actually faster chalane ke liye kya chahiye?

Weights ko 2:4 pattern mein prune karna padta hai (har 4 ke group mein exactly 2 nonzeros) aur offline compress karna padta hai; hardware phir known zeros ko skip karta hai ~2× zyada throughput ke liye. Random sparsity se zero benefit milta hai — dekho 8.4.2-Quantization-techniques.

Recall Quick self-test

Upar ek trap jo tumse galat hua ::: Aage badhne se pehle uska justification dobara padho aur underlying rule ko apne words mein phrase karo. Yahan sabse common myth ::: "Tensor Cores sab kuch low precision mein karte hain" — nahi, inputs low precision hain, accumulation high precision mein hoti hai. M, N, K ka kya matlab hai? ::: $M$ = output rows, $N$ = output columns, $K$ = shared inner dimension jiske upar sum kiya jaata hai. MMA aur WMMA mein kya difference hai? ::: MMA matrix multiply-accumulate operation khud hai; WMMA wahi operation hai jo NVIDIA ke API ke through ek poori 32-thread warp dwara cooperatively issue hoti hai.