Tensor cores and matrix operations: Foundations — Hinglish deep dive

6.2.12 · D1 · Hardware › GPU Architecture › Tensor cores and matrix operations

Parent note padhne se pehle, tumhe har woh symbol khud banana hoga jo woh tumhe deta hai. Hum unhe yahaan order mein banate hain, har ek pehle wale pe lean karta hua. Koi assumption nahi.

1. Ek number sirf ek value hai — lekin kitne bits?

Parent note baar baar FP16, FP32, INT8, TF32 kehta hai. Ye formats hain — ek number ko fixed on/off switches (bits) mein store karne ki recipes.

Figure — Tensor cores and matrix operations

Figure dekho. Har bar ek number ka bits ka "budget" hai, teen kamon mein baanta gaya:

Sign bit (slate, hamesha exactly 1 bit) decide karta hai positive ya negative: $0$ matlab $+$ , $1$ matlab $-$ . Yahi tarika hai jis se har floating-point format ek negative value store karta hai — bilkul shuru mein ek dedicated switch.
Exponent bits (lavender) decide karte hain ki number kitna bada ya chhota ho sakta hai — range.
Mantissa bits (mint) decide karte hain kitne decimal places ki detail — precision.

Table se pehle, iske do entries theek se samjho:

Format	Layout (sign · exp · mantissa)	Total bits	Kya milta hai
FP32	1 · 8 · 23	32	full range + full detail (safe default)
FP16	1 · 5 · 10	16	aadhi memory, chhoti range
BF16	1 · 8 · 7	16	FP32 ki range, coarse detail
TF32	1 · 8 · 10 (internal, 19)	19	FP32 range, FP16-jaisi detail
INT8	1 sign · 7 value (two's-complement)	8	whole numbers $- 128 \dots 127$ , FP32 se 4× chhota

Ise apni pocket mein rakho — yeh parent ke poore "Precision Modes" section ko power karta hai, aur 8.3.4-Mixed-precision-training aur 8.4.2-Quantization-techniques se link karta hai.

2. Matrix numbers ki ek grid hai

Parent A, B, C, D likhta hai aur unhe "4×4 tiles" ya "1024×1024 matrices" kehta hai. Ek matrix simply rows aur columns mein arranged numbers ka ek rectangle hai.

Figure mein, red pointer ko $A [1] [2]$ tak follow karo — row $1$ (doosri row neeche), column $2$ (teesra column across). Ek symbol, ek cell. Jab ye click kare, parent mein C[i][j] scary nahi lagega — yeh bas "output grid ka ek cell" hai.

3. Matrix multiplication: dot product disguise mein

Yahi sab kuch ka dil hai. Parent ka central formula hai

$C [i] [j] = \sum_{k = 0}^{K - 1} A [i] [k] \cdot B [k] [j]$

Hume isme har symbol earn karna hoga.

3a. Multiply-add (yahi ek core karta hai)

3b. Summation symbol $\sum$

Chhota worked example: $\sum_{k = 0}^{2} k^{2} = 0^{2} + 1^{2} + 2^{2} = 0 + 1 + 4 = 5$ . $\sum$ hamesha bas yahi karta hai.

3c. Ab poora formula, pictures mein

Ab formula zor se padho: " $C [i] [j]$ equals the sum, jaise $k$ $0$ se $K - 1$ tak jaata hai, $A$ -in-row- $i$ -column- $k$ times $B$ -in-row- $k$ -column- $j$ ka." Har symbol ab kuch aisa hai jise tum picture mein point kar sakte ho.

4. Ek cell se poore tile tak: `D = A×B + C`

Parent ka Tensor Core operation hai $D = A \times B + C$ jahan charon chhoti grids hain.

4a. Jab sizes evenly divide nahi hoti

Parent ke tiled loops $4$ (ya $16$ ) at a time aage badhte hain. Lekin agar $K = 1000$ ho, jo $4$ ka multiple nahi hai? Aakhri chunk matrix ke edge se baahar chala jayega.

5. Kaam kahan hota hai: SM, warp, thread

Parent kehta hai ki 32 threads ka ek warp cooperate karta hai, groups mein baanta gaya. Do hardware words define karne hain.

Numbers ko slow memory se in engines tak bhi travel karna hota hai — woh path 6.2.8-Memory-hierarchy-in-GPUs hai, aur isliye FP16 ka chhota size (section 1) speed ke liye matter karta hai.

6. Speed count karna: FLOP, FLOPS, throughput

Yeh parent ke performance math aur 9.1.5-Roofline-model ki vocabulary hai. Parent ke khud ke numbers pe quick self-check: $102 4^{3} \approx 1.07 \times 1 0^{9}$ FMAs, aur $(1024/4)^{3} = 25 6^{3} \approx 1.68 \times 1 0^{7}$ tile-ops, ratio $4^{3} = 64$ deta hai. Iska har piece ab ek aisa symbol hai jis par tumhara ownership hai.

Prerequisite map

Upar ki sab cheez parent topic 6.2.12 Tensor cores and matrix operations (Hinglish) aur uske companions 7.3.6-cuBLAS-and-cuDNN aur 8.3.4-Mixed-precision-training mein flow karti hai.

Equipment checklist

Dayi taraf cover karo aur khud test karo. Agar koi bhi jawab surprise kare, uska section dobara padho.

Bit kya hai, aur sign, exponent, aur mantissa bits mein se har ek kya control karta hai?

Bit ek on/off switch hai; sign bit positive/negative set karta hai, exponent bits range set karte hain, mantissa bits precision set karte hain.

Ek floating-point format negative number kaise store karta hai?

Ek dedicated sign bit ke saath shuru mein —

0

matlab

+

1

matlab

-

BF16 ka layout kya hai aur yeh training ke liye kyun accha hai?

1 sign + 8 exponent + 7 mantissa; yeh FP32 ka exponent copy karta hai isliye same wide range tak pahunchta hai, detail ki jagah range leta hai — gradients ke liye ideal.

INT8 kya range cover karta hai aur negatives kaise store hote hain?

- 128

+ 127

tak, 1 sign bit + 7 value bits two's-complement mein use karke.

Ek matrix ke liye shape

M \times N

ka matlab kya hai?

M

rows neeche jaate hue,

N

columns across jaate hue.

A [1] [2]

kis number ko refer karta hai (0-indexed)?

Doosri row, teesre column mein entry.

Row-major aur column-major storage mein kya fark hai?

Row-major row 0 phir row 1 lay out karta hai (entry at

i \cdot cols + j

); column-major column 0 phir column 1 lay out karta hai (entry at

j \cdot rows + i

Alfaz mein,

\sum_{k = 0}^{K - 1}

tumhe kya karne kehta hai?

k

0

K - 1

tak step karo, har ke liye term compute karo, aur unhe add karo.

FMA kya hai, aur yeh kitne FLOPs ka hai?

Ek fused multiply-add,

a \cdot b + c

; yeh 2 FLOPs count hota hai.

Ek output cell

C [i] [j]

ka formula likho.

C [i] [j] = \sum_{k = 0}^{K - 1} A [i] [k] \cdot B [k] [j]

A

aur

B

ka inner dimension

K

kyun match karna chahiye?

Yeh us row aur column ki length hai jis par tum saath saath slide karte ho; unmatched lengths ko pair nahi kiya ja sakta.

D = A \times B + C

mein,

C

kya kaam karta hai?

Yeh accumulator hai — yeh running total hold karta hai taaki partial products add hote rahen.

Tiling ko actually koun si properties correct banati hain?

Addition ki associativity aur scalar multiplication ki

k

-sum par distributivity — matrix-multiply associativity nahi, aur matrix multiply commutative nahi hai.

Partial tile kya hai aur ise kaise handle karte hain?

Edge tile jo sirf partly real data se bhara hai; missing cells ko

0

se pad karo (jo kuch contribute nahi karte) aur extra output rows/columns ignore karo.

8 ya 16 ke multiples wale dimensions fastest kyun run karte hain?

Woh partial tiles avoid karte hain, isliye koi hardware cycles padded arithmetic par waste nahi hoti.

FLOP aur FLOPS mein kya fark hai?

FLOP ek operation hai (ek kaam); FLOPS operations per second hai (ek rate).

Chhote formats jaise FP16 Tensor Cores ko faster kyun banate hain?

Fewer bits matlab zyada numbers chip par fit hote hain aur memory se faster move karte hain, jabki wide FP32 accumulator accuracy preserve karta hai.

Recall Quick fire — summation drill

$\sum_{k = 0}^{2} k^{2}$ compute karo. ::: $0 + 1 + 4 = 5$ . $2 \times 2$ matrices ke liye, $C = A \times B$ compute karne mein kitne FMAs lagte hain? ::: $2 \times 2 \times 2 = 8$ .