Tensor cores and matrix operations: Visual walkthrough

6.2.12 · D2 · Hardware › GPU Architecture › Tensor cores and matrix operations

Yeh page Tensor cores and matrix operations ka central idea bilkul scratch se build karta hai. Hum shuru karte hain "do number-grids ko multiply karne ka matlab kya hai?" se, aur khatam karte hain "dedicated hardware 64× denser kyun hota hai?" pe. Har step mein ek picture hai. Coloured arrows follow karo.

Agar koi word yahan assumed lagta hai — woh hai nahin. Hum sab kuch build karte hain.

Step 1 — Matrix actually hota kya hai?

KYA HAI. Ek matrix bas numbers ki ek rectangular grid hoti hai. Hum isse ek naam dete hain (ek capital letter jaise $A$ ) aur andar ke kisi bhi ek number ko do labels se address karte hain: woh kis row mein hai aur kis column mein hai.

KYUN. Grids ko "multiply" karne se pehle, humein agree karna hoga ki ek grid ke andar ek number ko point kaise karein. Wahi pointing poora game hai.

PICTURE. Figure dekho. Row $i$ , column $j$ mein jo number hai woh $A [i] [j]$ likha jaata hai. Jo chhota square light up hota hai woh exactly wahi entry hai.

Figure — Tensor cores and matrix operations

Jo grid $M$ rows lambi aur $K$ columns chaudi hai use " $M \times K$ " grid kehte hain. " $\times$ " ko "by" padho: "M by K". Yeh size hai, abhi tak multiplication nahin.

Step 2 — "Do grids ko multiply karna" ka matlab kya hai? (ek output cell)

KYA HAI. Grid $A$ (size $M \times K$ ) ko grid $B$ (size $K \times N$ ) se multiply karne ke liye, hum ek nayi grid $C$ (size $M \times N$ ) produce karte hain. Hum ek baar mein $C$ ka ek cell compute karte hain.

KYUN. Ek single rule, har output cell pe apply hoti hai, poori operation define karta hai. Toh humein sirf ek cell samajhna hai. Woh master karo, sab kuch master ho jaata hai.

PICTURE. $C [i] [j]$ (output ka highlighted cell) paane ke liye, hum $A$ ki row $i$ (ek horizontal strip) aur $B$ ki column $j$ (ek vertical strip) lete hain. Hum dono ke saath saath slide karte hain, jis pair pe land karte hain unhe multiply karte hain, aur ek running total rakhte hain.

Step 3 — Yeh expensive part kyun hai? (kaam ginana)

KYA HAI. Pehle, basic operation ka naam. Har "ek pair multiply karo, running total mein add karo" ko fused multiply-add (FMA) kehte hain — ek multiply aur ek add ek saath glue kiya hua. Ab hum count karte hain ki poori grid kitne FMAs lagti hai.

KYUN. Deep learning mostly yahi operation hai. Agar humein FMA cost pata ho, toh hum jaante hain ki kisi ne special hardware banane ki taklif kyun uthaayi.

PICTURE. Har output cell ko $K$ FMAs chahiye (har slide position ke liye ek). $M \times N$ output cells hain. Toh kaam ka tower $M \times N \times K$ tall hai.

Recall

Matrix multiply ki cost $M \cdot N \cdot K$ kyun hoti hai? ::: $M \cdot N$ output cells, har ek ko fill karne ke liye $K$ multiply-adds chahiye.

Yahi wall hai. Ek CUDA core ek clock mein ek FMA karta hai. Ek billion FMAs, ek per clock, bahut saare clocks hain. Enter the tile.

Step 4 — Trick: grids ko 4×4 tiles mein kaato

KYA HAI. Ek giant multiply ki jagah, $A$ , $B$ , $C$ ko chhote $4 \times 4$ blocks mein kaat lo jinhein tiles kehte hain. Bada multiply bahut saare chhote tile-multiplies ban jaata hai jinhe hum add up karte hain.

KYUN. Kisi cell ke liye single dot-product sum (Step 2) ko 4 slide positions ke groups mein split kiya ja sakta hai aur group-totals wapas add kiye ja sakte hain — kyunki addition associative hoti hai (products ke sum ko regroup karna kabhi total nahin badalta). 4 ka har group exactly ek $4 \times 4$ tile-multiply hai, toh hum har tile ko dedicated hardware unit ko de sakte hain aur results add kar sakte hain.

PICTURE. $C$ ki ek output tile (top-left $4 \times 4$ block) $A$ -tiles ki ek strip ko across aur $B$ -tiles ki ek strip ko neeche chalate hue, tiles ke har opposite pair ko multiply karte hue, aur usi output tile mein accumulate karte hue build hoti hai.

Step 5 — Ek tile = ek instruction (Tensor Core hai kya)

KYA HAI. Ek Tensor Core woh hardware hai jo pura $4 \times 4$ tile multiply-accumulate, $D = A \times B + C$ , ek instruction mein compute karta hai — 64 alag steps mein nahin.

KYUN. Woh $4 \times 4$ tile $4 \times 4 \times 4 = 64$ FMAs chhupata hai. Saare 64 ek shot mein karna poora point hai: aap instruction-decode cost ek baar pay karte ho 64 arithmetic operations ke liye.

PICTURE. Left: ek CUDA core 64 tiny FMAs ek clock at a time grind karta hua. Right: ek Tensor Core saare 64 ek single instruction cycle mein swallow karta hua. Sama answer, 64× kam instructions.

Step 6 — Scoreboard: tiling ne kitna save kiya?

KYA HAI. $102 4^{3}$ job ke liye Tensor Core instructions count karo aur scalar FMA count se compare karo.

KYUN. Yeh payoff line hai — woh number jo poore chapter ko justify karta hai.

PICTURE. Do towers side by side: 1.07 billion scalar FMAs vs 16.8 million tile-instructions. Ratio ek clean $64$ hai.

Yeh 64× 64× wall-clock speedup mein badle ya nahin, yeh depend karta hai ki cores ko data kitna fast feed kiya jaata hai — woh memory hierarchy aur roofline ki kahaani hai. Compute density zaroori hai, sufficient nahin.

Step 7 — Degenerate cases (reader ko kabhi stranded mat chhodna)

KYA HAI. Awkward inputs handle karo: ek dimension jo 4 se divisible nahin, $K = 0$ , aur genuinely empty output ( $M = 0$ ya $N = 0$ ).

KYUN. Real matrices hamesha neat multiples nahin hoti. Reader ko pata hona chahiye ki har edge pe kya hota hai.

PICTURE. Ek grid jiska width 4 ka multiple nahin hai: last tile edge se hang off karti hai. Hum missing cells ko zeros se pad karte hain taaki tile abhi bhi $4 \times 4$ rahe.

Ek-picture summary

Upar sab kuch, ek canvas pe: ek cell index karo → ek row ko ek column se dot karo → FMAs ka mountain count karo → $4 \times 4$ tiles mein kaato → ek tile = ek Tensor Core instruction → 64× kam instructions.

Recall Feynman retelling — plain words mein wapas bolo

Ek matrix numbers ki ek grid hai. Do grids ko multiply karne ke liye, har output number pehli grid ki ek row aur doosri grid ke ek column ke saath slide karke banta hai, jis numbers pe land karo unhe multiply karo aur sab add karo. Har multiply-and-add ek FMA hai. Har output cell ke liye woh karo aur tumne $M \times N \times K$ FMAs pay kar liye — 1024 size ki grids ke liye woh ek billion se zyada hai, ek ek karke karne ke liye bahut zyada.

Clever move: grids ko chhote $4 \times 4$ blocks mein kaato. Kyunki products ki badi pile ko bundles mein add kar sakte ho aur same total milta hai jaisa ek line mein add karne pe (addition associative hai), tum block-by-block multiply kar sakte ho aur block-answers add kar sakte ho. Ek Tensor Core silicon ka ek tukda hai jo ek single instruction mein ek poora $4 \times 4$ block-multiply-and-add khaata hai — ek instruction ki price mein 64 multiply-adds. Poore $102 4^{3}$ job mein exactly 64× kam instructions hain.

Edge cases ise nahin todte: agar grid ka size 4 ka multiple nahin hai, toh gap ko zeros se pad karo (zeros kuch add nahin karte, toh answer unchanged rehta hai — thoda instruction waste hota hai). Agar shared dimension zero hai, toh sum empty hai aur accumulator $C$ bas pass through karta hai. Agar poora dimension zero hai, toh output mein koi cells hi nahin hain aur simply kuch karna hi nahin hai. Yahi poora idea hai: ordinary matrix multiply jaisi hi arithmetic, fixed-size tiles mein packed taaki dedicated hardware usse blast through kar sake.

Yeh bhi dekho: mixed-precision training aur quantization yeh tiles FP16/BF16/INT8 pe use karte hain; Hinglish note same ground cover karta hai.