Worked examples — Tensor cores and matrix operations

6.2.12 · D3 · Hardware › GPU Architecture › Tensor cores and matrix operations

Yeh page Tensor Cores ka drill ground hai. Parent note ne tumhe bataya tha ki Tensor Core kya hota hai aur kyun exist karta hai. Yahan hum har tarah ki situation enumerate karte hain jo tumhe mil sakti hai — har awkward matrix shape, har precision choice, har "kya yeh fit bhi hoga?" edge case — aur har ek ko number tak kaam karte hain.

Agar tum tiled matrix multiply ki machinery se abhi nahi mile ho, pehle parent ko skim karo. Baaki sab kuch hum yahan zero se build karte hain.

---

Do notation pieces, pehle banaye gaye

Do symbols har line pe repeat hote hain. Hum inhe use karne se pehle define karte hain.

---

Scenario matrix

Problems work karne se pehle, aao cases enumerate karein. Ek Tensor Core problem kuch knobs se poori tarah describe hoti hai. Agar ek knob aise value le sakta hai jo naive approach ko tod de, us value ko apni row milti hai.

Cell	Case class	Kya test ho raha hai	Example
A	Clean fit	Dimensions tile ke exact multiples hain	Ex 1
B	Ragged fit (padding chahiye)	Dimension tile ka multiple nahi → padding / performance cliff	Ex 2
C	Degenerate: $K=0$, aur $M=0$ / $N=0$	Sum / output ka size zero hai — answer kya hai?	Ex 3
D	Skinny / vector-like ($N=1$ ya $M=1$)	Tile mostly wasted — utilisation ka sawaal	Ex 4
E	Instruction-count / speedup arithmetic	FMAs aur Tensor Core ops count karna	Ex 5
F	Precision choice (FP16 vs BF16 vs TF32 vs INT8)	Kaun sa mode, aur accuracy ki cost	Ex 6
G	Accumulator overflow / range	Jab low precision silently fail kare	Ex 7
H	Real-world word problem (roofline / timing)	Bytes vs FLOPs, kya yeh compute-bound hai?	Ex 8
I	Exam-style twist (transpose / layout)	row_major vs col_major, off-by-tile traps	Ex 9

Hum yeh symbols poore page mein use karenge — ek baar, yahan define kiye gaye:

---

Ex 1 — Cell A · Clean fit

Hum kya kar rahe hain: count karna ki tile-sized bites poore output ko kitna cover karte hain.

Niche figure (s01). Alt-text: ek blue $256\times 128$ output rectangle ko ek patli gray grid se $16$ tiles niche aur $16$ tiles across mein kaat diya gaya hai; ek tile orange se bhari hai aur labelled hai "ek $16\times 8$ output tile ($D$)"; ek green arrow labelled "$K/k=32$ accumulation steps" us orange tile mein andar point kar raha hai yeh dikhane ke liye ki yeh $32$ K-chunks sum karke bana hai. Teen grid counts picture se seedhe padho.

1. Output-row tiles count karo. $M/m=256/16=16$ (s01 mein grid ki $16$ rows). Yeh step kyun? Output $256\times 128$ hai; iske height ke along hum height $m=16$ ke tiles rakhte hain. Evenly divide hota hai → Cell A, clean.
2. Output-column tiles count karo. $N/n=128/8=16$ (grid ke $16$ columns). Kyun? Width ke across same logic, tile width $n=8$.
3. Inner (K) tiles count karo. $K/k=512/16=32$ (green arrow ki depth). Kyun? Har output tile ek shot mein complete nahi hota — shared dimension $K$ ko $k=16$ ke chunks mein kaata jaata hai, aur hum un chunks ko accumulate karte hain. Har chunk ka product running tile pe add hota hai $D=A\times B+C$ mein $+C$ ke zariye: pichle step ka $D$ is step ka $C$ ban jaata hai.
4. Teen counts ko multiply karo. $16\times 16\times 32=8192$ instructions. Multiply kyun? Har output tile (jitne $16\times 16$ orange squares hain) ko har K-chunk ke liye ek mma chahiye (jinka count $32$ hai).

Verify: total FMAs doosre tarike se $=M\cdot N\cdot K=256\cdot 128\cdot 512=16,777,216$. Har mma karta hai $m\cdot n\cdot k=16\cdot 8\cdot 16=2048$ FMAs. Toh $16,777,216/2048=8192$. ✓ Do tareekon se same number.

---

Ex 2 — Cell B · Ragged fit (padding & performance cliff)

Hum kya kar rahe hain: aisi dimension handle karna jise tile size evenly divide nahi karta.

Niche figure (s02). Alt-text: $17$ tiles ki horizontal strip; pehle $16$ tiles ($128$ columns) solid blue hain aur labelled hain "real, clean"; $17$waan tile split hai — ek orange slice ($2$ columns, "real: 129–130") ke bagal mein ek gray hatched slice ($6$ columns, "padding: 131–136, wasted"). $17$we tile ke upar ek red bracket likha hai "2 extra columns ke liye ek extra tile = the cliff." Dekho kaise tiny orange sliver ek poore gray tile ko saath le aata hai.

1. Divisibility check karo. $130/8=16.25$ — integer nahi. Yeh ragged case hai. Yeh step kyun? Tensor Cores sirf whole tiles mein bolte hain. Ek partial column strip "16.25 tiles" ke roop mein issue nahi ho sakti.
2. Ceiling se round up karo. Column-tiles ki count $=\lceil 130/8\rceil =17$ (yaad karo $\lceil 16.25\rceil =17$, upar define kiya) — s02 mein $17$ tiles. Round up kyun, down nahi? Extra $2$ columns drop karne se data loss hoga. Hume columns 129 aur 130 (orange sliver) cover karne hain, toh hum poora $17$waan tile allocate karte hain.
3. Padded width compute karo. $17\times 8=136$ tile space ke columns $130$ real columns ke liye. Kyun? Last tile columns 129–136 span karta hai; columns 131–136 padding hain ($B$ mein zeros, $D$ mein garbage-then-discarded) — gray hatched slice.
4. Wasted columns $=136-130=6$. Care kyun? Un $6$ columns ko phir bhi full Tensor Core cycles lagte hain. Yeh hai performance cliff: sirf $2$ real columns add karne se ek poora extra tile lag gaya.

Verify: last tile ka wasted fraction $=6/8=75\%$ idle us strip mein. Overall extra work vs. clean $128$: instructions $16\cdot 16\cdot 32$ se $16\cdot 17\cdot 32=8704$ ho jaate hain, yaani $8704/8192=1.0625$, 2 extra columns serve karne ke liye 6.25% overhead. ✓

---

Ex 3 — Cell C · Degenerate sizes ($K=0$, aur $M=0$ ya $N=0$)

Hum kya kar rahe hain: har boundary size pe tile counts aur defining sum evaluate karna.

Niche figure (s03). Alt-text: teen chhote panels side by side. Panel (a) "$K=0$": ek solid blue $16\times 16$ output square exist karta hai lekin label "$0$" se bhara hai — output tiles real hain, contraction empty hai. Panel (b) "$M=0$": zero height ki ek flat blue line labelled "$0\times 128$ — koi rows nahi, koi tiles nahi." Panel (c) "$N=0$": zero width ki ek patli blue vertical line labelled "$256\times 0$ — koi cols nahi, koi tiles nahi." Teeno ke niche ek gray caption likha hai "instructions run $=0$ har case mein."

1. Case (a), $K=0$ (empty contraction). Defining sum $C[i][j]={\sum }_{k=0}^{K-1}A[i][k]B[k][j]$ ka ek empty index range hai. Universal convention ke mutabik zero terms ka sum additive identity $0$ hai, toh $C$ hai $16\times 16$ zero matrix (panel (a): ek real square, $0$ se bhara). 0 kyun aur "undefined" kyun nahi? Ek empty sum accumulator ko unchanged rehne deta hai; identity jo "kuch nahi add karta" woh $0$ hai. Instruction count: $\lceil M/m\rceil \cdot \lceil N/n\rceil \cdot \lceil K/k\rceil =1\cdot 2\cdot 0=0$ mma calls. Output tiles exist karte hain ($16\times 16$ non-empty hai) lekin accumulate karne ke liye kuch nahi hai.
2. Case (b), $M=0$ (koi output rows nahi). $C$ hai $0\times 128$ — ek empty matrix (koi elements nahi; panel (b) ek zero-height line hai). Kyun? Agar $A$ mein zero rows hain, toh product mein zero rows hain; literally koi $C[i][j]$ nahi hai compute karne ke liye. Instruction count: $\lceil 0/16\rceil \cdot \lceil 128/8\rceil \cdot \lceil 512/16\rceil =0\cdot 16\cdot 32=0$.
3. Case (c), $N=0$ (koi output columns nahi). $C$ hai $256\times 0$ — phir ek empty matrix (panel (c) ek zero-width line hai). Kyun? (b) se symmetric: zero output columns → likhne ke liye kuch nahi. Instruction count: $\lceil 256/16\rceil \cdot 0\cdot \lceil 512/16\rceil =0$.
4. Unifying rule. Agar koi bhi teen tile counts mein se $0$ hai, product zero instructions run karta hai. Lekin wajah alag hai: (a) mein output tiles exist karte hain aur phir bhi zero se initialise hone chahiye; (b)/(c) mein initialise karne ke liye koi output tiles hi nahi hain. Initialisation mention kyun? Cell C ka classic bug: $K=0$ ke liye phir bhi fill_fragment(c_frag, 0.0f) karna padta hai warna garbage return hoga — chahe koi multiply kabhi na ho.

Verify: (a) $C$ hai $16\times 16$ zeros, $1\cdot 2\cdot 0=0$ ops; (b) $C$ hai $0\times 128$, $0$ ops; (c) $C$ hai $256\times 0$, $0$ ops. ✓

---

Ex 4 — Cell D · Skinny matrix (matrix–vector, $N=1$)

Hum kya kar rahe hain: waste measure karna jab ek dimension tile se bahut chhota ho.

Niche figure (s04). Alt-text: ek $16\times 8$ output tile ek $8$-wide, $16$-tall blue grid ke roop mein drawn. Sabse left column orange se bhari hai aur labelled hai "useful output ($y$ ka 1 column)"; baaki $7$ columns diagonal hatching ke saath gray hain, labelled "wasted tile width (8 mein se 7 columns padding)." Niche ek red caption likha hai "utilisation $=1/8=12.5\%$." Orange strip ko blue tile ka exactly ek-aathwaan hissa occupy karte hue dekho.

1. Output shape. $y$ hai $4096\times 1$. Column-tiles chahiye $=\lceil 1/8\rceil =1$. Kyun? Ek akela column bhi full width-$8$ tile force karta hai — ek orange column, lekin poora blue tile allocated (figure dekho).
2. Tile mein useful columns $=1$ out of $n=8$ (s04 mein orange strip vs. poora blue tile). Kyun? Us tile ke columns 2–8, $x$ ke padding (zero) columns ke against compute karte hain — real cycles, discarded results (gray hatched region).
3. Column utilisation $=1/8=12.5\%$. Yeh kyun matter karta hai? Tensor Cores fat GEMMs pe chamakte hain. Ek matrix–vector product width dimension ka $87.5\%$ waste karta hai — yeh memory-bound hai, compute-bound nahi (Ex 8 dekho).
4. Row dimension theek hai. $M=4096$, $16$ ka multiple hai, toh height utilisation $100\%$ hai. Overall tile utilisation $=100\%\times 12.5\%=12.5\%$. Dimensions alag kyun karein? Utilisation independent tile axes ke across multiply hoti hai.

Verify: column util $=1/8=0.125$; overall $=1\cdot 0.125=0.125=12.5\%$. s04 mein, orange strip (real output) blue tile ke aath columns mein se exactly ek hai. ✓

---

Ex 5 — Cell E · Speedup arithmetic

Hum kya kar rahe hain: total work ko work-per-instruction se divide karna.

Niche figure (s05). Alt-text: logarithmic x-axis (instruction count) wala ek horizontal bar chart. Ek lamba orange bar $\approx 1.07\times 10^9$ tak jaata hai labelled "scalar FMAs (ek per instruction)"; ek bahut chhota blue bar $\approx 5.24\times 10^5$ tak jaata hai labelled "Tensor Core mma ops (2048 FMAs each)." Unke beech ek red annotation likha hai "$\times 2048$ fewer instructions." Blue bar visibly orange ka ek tiny fraction hai.

1. Total FMAs. $M\cdot N\cdot K=1024^3=1,073,741,824\approx 1.07\times 10^9$ (orange bar). Cube kyun? $M\cdot N$ outputs mein se har ek ko $K$ FMAs chahiye.
2. FMAs per mma. $m\cdot n\cdot k=16\cdot 8\cdot 16=2048$. Kyun? Ek instruction ek $m\times n$ output tile fill karta hai length-$k$ contraction use karke $=mnk$ FMAs.
3. Reduction factor. $1024^3/2048=524,288$ instructions (blue bar), yaani instruction count $2048$ factor se gir jaata hai. Tile product kyun? Ek instruction ke andar har chhupa FMA ek kam instruction hai jo tum issue karte ho.

Verify: $1024^3=1073741824$; $/2048=524288$ Tensor Core ops; factor $=2048$. ✓

Recall 2048× kam instructions ka matlab 2048× faster kyun nahi hota?

Kyunki raw speedup clock, memory bandwidth, aur pipeline fill se bounded hota hai — instruction count sirf ceiling hai. Real speedup $8$–$16\times$ hota hai, hardware throughput se set, FMA-per-op ratio se nahi. ::: Instruction reduction ek upper bound hai; memory aur clock actual gain cap karte hain.

---

Ex 6 — Cell F · Precision choose karna (FP16 vs BF16 vs TF32 vs INT8)

Hum kya kar rahe hain: saare char precision modes ko concrete numbers pe exercise kar rahe hain.

Niche figure (s06). Alt-text: do stacked panels. Top panel — teen horizontal bit-layout bars (FP32, BF16, FP16) equal total length ke, har ek blue "sign" cell, orange "exponent" segment, aur green "mantissa" segment mein split; FP32 aur BF16 same wide orange exponent share karte hain (labelled "range: 8 exp bits"), jabki FP16 ka exponent visibly shorter hai ("range: 5 exp bits") aur iska mantissa BF16 se longer hai. Ek red arrow mark karta hai "$6\times 10^{-8}$ falls off FP16's range." Bottom panel — $-1$ se $1$ tak ek number line $9$ evenly spaced orange ticks ke saath INT8 ki coarse grid illustrate karta hai, annotated "step $=2/256=0.0078$."

Hum kya kar rahe hain: saare char precision modes ko concrete numbers pe exercise kar rahe hain.

1. (i) FP16 vs BF16 — range test. $6\times 10^{-8}$ ko har format ke smallest normal se compare karo. $6\times 10^{-8}<6.1\times 10^{-5}$ → FP16 ise zero ki taraf underflow kar deta hai. $6\times 10^{-8}\gg 1.2\times 10^{-38}$ → BF16 ise rakhta hai. Kyun? FP16 exponent pe sirf 5 bits spend karta hai (narrow range, s06 ka top panel); BF16 FP32 ka full 8-bit exponent rakhta hai, range ke liye mantissa bits trade karta hai. Training ke liye BF16 choose karo.
2. (ii) TF32 — free speed test. Relative rounding error lagbhag last mantissa bit mein ek unit hai: $2^{-10}\approx 9.77\times 10^{-4}$, FP32 ke $2^{-23}\approx 1.19\times 10^{-7}$ se compare mein. Care kyun? $\approx 10^{-3}$ relative error $0.1\%$ se kam hai — zyaadatar models ke liye invisible — aur TF32 ko koi code change nahi chahiye jabki FP32 se $\sim 8\times$ faster run karta hai. Default fast FP32 ke roop mein TF32 choose karo.
3. (iii) INT8 — quantisation step test. $256$ levels ($2^8$) ke saath range $1-(-1)=2$ width ke across, neighbouring representable weights ke beech step hai $2/256=0.0078125$ (tick spacing, s06 ka bottom panel). Yeh kyun matter karta hai? Koi bhi weight $\approx 0.0078$ ke nearest multiple pe round hota hai; INT8 yeh coarse grid accept karta hai exchange mein $16\times$ throughput aur FP32 vs $4\times$ smaller weights ke liye — training ke baad inference ke liye theek. Post-training inference ke liye INT8 choose karo.

Verify: (i) $6\times 10^{-8}<6.1\times 10^{-5}$ aur $>1.2\times 10^{-38}$; (ii) $2^{-10}=0.0009765625$, $2^{-23}\approx 1.19\times 10^{-7}$, aur $2^{-10}<10^{-3}$; (iii) $2/256=0.0078125$. ✓ Dekho 8.3.4-Mixed-precision-training aur 8.4.2-Quantization-techniques.

---

Ex 7 — Cell G · Accumulator overflow (kyun FP32 accumulate exist karta hai)

Hum kya kar rahe hain: accumulator ko bound karna aur iske capacity se compare karna.

1. Worst-case per-product magnitude. $127\times 127=16129$. Max kyun? $127$ sabse bada signed INT8 value hai; worst single term.
2. Worst-case sum. $K\times 16129=4096\times 16129=66,064,384$. $K$ se multiply kyun? Contraction $K$ aise terms add karta hai; worst case sabhi max sign pe hain.
3. INT16 ceiling se compare karo ($32767$). $66,064,384>32767$ → sirf kuch terms ke baad massive overflow. Yeh sab kuch kyun barbaad karta hai? Ek overflowed INT16 silently wrap around karta hai → garbage output, koi error nahi.
4. INT32 ceiling se compare karo ($2,147,483,647$). $66,064,384<2.1\times 10^9$ → comfortably fit karta hai. Hardware INT32 accumulate kyun use karta hai: yeh guarantee karta hai realistic $K$ ke liye koi overflow nahi, aur yahi reason hai ki Tensor Cores INT8 inputs ko ek INT32 accumulator ke saath pair karte hain (aur FP16 inputs ko FP32 ke saath) — accumulate result $D$ wide type mein rehta hai chahe $A,B$ narrow hon.

Verify: $127^2\cdot 4096=66064384$; $>32767$ (INT16 overflows); $<2147483647$ (INT32 safe). ✓ Yeh concrete reason hai ki "mixed precision" hamesha wide accumulate karta hai.

---

Ex 8 — Cell H · Real-world word problem (compute- vs memory-bound)

Hum kya kar rahe hain: roofline model apply karna — woh single test jo kehta hai "compute ya memory."

1. FLOPs. Matrix–vector $M\cdot K$ FMAs karta hai $=4096\cdot 4096=16,777,216$ FMAs $=2\times$ FLOPs mein $=33,554,432\approx 3.36\times 10^7$ FLOPs. $\times 2$ kyun? Har FMA ek multiply aur ek add hai.
2. Bytes moved. Hume $W$ read karna hai (dominant): $4096^2$ FP16 elements $\times 2$ bytes $=33,554,432$ bytes $\approx 3.36\times 10^7$ B. ($x$ aur $y$ negligible hain.) $W$ kyun dominate karta hai? $x$ sirf $4096$ elements hai; $W$ hai $4096^2$ — ek million times zyaada data.
3. Arithmetic intensity $=\text{FLOPs}/\text{bytes}=33,554,432/33,554,432=1.0$ FLOP/byte. Yeh ratio kyun? Yeh batata hai ki har fetched byte se hum kitna compute nikalte hain.
4. Machine balance $=312\times 10^{12}/2000\times 10^9=156$ FLOP/byte. Compare: hamari intensity $1.0\ll 156$ → bahut bade margin se memory-bound.
5. Time estimate (memory). $t=\text{bytes}/\text{BW}=33,554,432/(2000\times 10^9)\approx 1.68\times 10^{-5}$ s $=16.8\mu$s. Bandwidth kyun use karo, FLOP/s nahi? Jab memory-bound ho, memory pipe time set karta hai; Tensor Cores idle rehte hain.

Verify: intensity $=1.0$; balance $=156$; $1.0<156$ (memory-bound); $t=3.3554432\times 10^7/2\times 10^{12}=1.6777\times 10^{-5}$ s. ✓ (Compute-only time hoga $3.36\times 10^7/3.12\times 10^{14}\approx 0.1\mu$s — bahut chhota, confirm karta hai ki memory dominate karta hai.)

---

Ex 9 — Cell I · Exam twist (layout / transpose trap)

Hum kya kar rahe hain: layout flags ko transpose operators ke roop mein track karna, kyunki exactly yahi woh hain.

1. Layout ka matlab kya hai. Row-major buffer ko col-major ki tarah padhna mathematically transpose padhne ke barabar hai. Toh col_major load flag jo bhi bytes read karta hai us par ek transpose $(\cdot {)}^{\top }$ apply karta hai. Kyun? Row-major $B$ stored, column-major padha, mathematically $B^{\top }$ deta hai.
2. Student ne memory mein $B$ pre-transpose kiya → buffer ab $B^{\top }$ hold karta hai. Unhone kyun kiya: woh transpose chahte the jo unhe chahiye tha.
3. Phir col_major flag dobara transpose karta hai → fragment hold karta hai $(B^{\top }{)}^{\top }=B$. Yeh trap kyun hai: do transposes cancel ho jaate hain. $(X^{\top }{)}^{\top }=X$.
4. Result: mma_sync compute karta hai $A\times B$ — desired $A\times B^{\top }$ nahi. Do flips cancel ho gaye; student silently galat product leta hai (koi error nahi). Fix: exactly ek transpose karo — ya buffer flip karo ya layout flag set karo, dono kabhi nahi.

Verify: $A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]$, $B=\left[\begin{array}{cc}5& 6\\ 7& 8\end{array}\right]$ se: intended $A{B}^{\top }=\left[\begin{array}{cc}17& 23\\ 39& 53\end{array}\right]$ lekin double-flip deta hai $AB=\left[\begin{array}{cc}19& 22\\ 43& 50\end{array}\right]$. Dono matrices alag hain → bug confirm hota hai. ✓ (Dekho 7.3.6-cuBLAS-and-cuDNN: libraries ek explicit transA/transB flag expose karti hain taki tum ek baar, deliberately flip karo.)

---

Recap

Recall Kaun sa cell tumhe tile count

upar ceiling se round karne force karta hai? Cell B (ragged fit): tile size se divide na hone wali dimension $\lceil D/t\rceil$ tiles use karti hai, padding aur performance cliff create karti hai. ::: Ragged / padding case — ceiling use karo, wasted columns expect karo.

Recall Empty sizes:

$K=0$, $M=0$, aur $N=0$ mein kya fark hai? Teeno 0 mma instructions run karte hain. Lekin $K=0$ phir bhi ek real (zero-filled) output tile produce karta hai jo initialise hona chahiye, jabki $M=0$ ya $N=0$ ek empty matrix produce karte hain jisme koi output tiles nahi hote. ::: Zero instructions har case mein; sirf $K=0$ ko phir bhi accumulator init chahiye.

Recall INT8 Tensor Core hardware INT32 mein accumulate kyun karta hai?

Kyunki INT8 products ka length-$K$ contraction $\sim K\cdot 127^2$ tak pahunch sakta hai, jo modest $K$ par bhi INT16 overflow karta hai lekin INT32 ke $\pm 2.1\times 10^9$ range mein comfortably rehta hai. ::: Accumulated sum ka silent overflow avoid karne ke liye.

Recall Skinny matmul (

$N=1$): compute-bound ya memory-bound? Memory-bound — arithmetic intensity $\approx 1$ FLOP/byte machine balance ($\sim 156$) se bahut niche hai, aur tile utilisation sirf $12.5\%$ hai. ::: Memory-bound aur low utilisation; fix karne ke liye batch karo.

Related depth: 6.2.1-GPU-vs-CPU-architecture, 6.2.8-Memory-hierarchy-in-GPUs, 6.2.10-Warp-scheduling-and-execution.