Neural processing units (NPUs): Foundations — Hinglish deep dive

6.5.8 · D1 · Hardware › Advanced & Emerging Architectures › Neural processing units (NPUs)

Parent note padhne se pehle, tumhe us har symbol ko apna banana hoga jo woh use karta hai. Yeh page har ek ko zero se build karta hai — pehle plain words mein, phir ek picture, phir why the topic needs it. Upar se neeche padho; har block sirf wahi cheezein use karta hai jo uske upar define ki gayi hain.

1. Ek number, aur unhe multiply karna

Bilkul flat se shuru karte hain. Ek number bas ek quantity hai — jaise $3$ ya $0.5$ . Jab hum do numbers ke beech $\times$ likhte hain, hum multiply karte hain: $3 \times 4 = 12$ .

Why the topic needs it: ek neural network multiplications se bana hai. Do neurons ke beech har connection ek signal ko ek strength se multiply karta hai. Isliye hardware ka sabse chhota useful piece sabse pehle multiply kar sakna chahiye.

2. Running total, aur "accumulate" ka idea

Ab maan lo tumhare paas numbers ki ek list hai aur tum unka total chahte ho. Tum ek running total rakhte ho — ek box jo $0$ se shuru hota hai aur jaise-jaise tum numbers daalo, badhta jaata hai.

$box \leftarrow box + new number$

Arrow $\leftarrow$ ka matlab hai "ban jaata hai": box ki nayi value uski purani value plus nayi number hai. Yeh accumulation hai — step by step sum build karna.

Why the topic needs it: poora NPU is ek operation ki hazaaron copies hai. Agar tum MAC samajh gaye, tum woh atom samajh gaye jisse machine bani hai.

3. Numbers ki lists: vector, aur dot product

Ek vector bas numbers ki ek ordered list hai, jaise $(2, 5, 1)$ . Ise labelled boxes ki ek row samjho.

Ab do vectors lo jo same length ke hon. Unhe position by position multiply karo, phir woh saare products add kar do. Yahi dot product hai:

$(a_{1}, a_{2}, a_{3}) \cdot (b_{1}, b_{2}, b_{3}) = a_{1} b_{1} + a_{2} b_{2} + a_{3} b_{3}$

Why the topic needs it: neural-network layer ka har ek output number ek dot product hai. Parent ki line $y_{ij} = \sum_{k} X_{ik} W_{k j}$ ek chhupa hua dot product hai — us $\sum$ ko hum aage unpack karte hain.

4. Summation symbol $\sum$

Bada Greek letter $\sum$ (sigma, "S" for "Sum") ka shorthand hai "ek series of terms add karo."

$\sum_{k = 1}^{K} t_{k} = t_{1} + t_{2} + t_{3} + \dots + t_{K}$

Ise zor se padho: "sum, jaise $k$ $1$ se $K$ tak jaata hai, $t_{k}$ ka."

$\sum$ ke neeche wala letter ( $k = 1$ ) counter hai aur uski starting value hai.
Upar wala letter ( $K$ ) woh hai jahan counter rukta hai.
$t_{k}$ har term ki recipe hai — ek template jisme tum current $k$ plug in karte ho.

Why the topic needs it: $\sum$ the compact way hai yeh kehne ka ki "har $k$ ke liye MAC karo aur total rakho." Parent ka dot product $y = \sum_{k = 1}^{K} X_{k} W_{k}$ precisely ek $K$ -step MAC chain hai. Counter $k$ hardware ko batata hai ki ek output ke liye kitne multiply-adds chahiye.

5. Numbers ki grids: matrix, aur subscripts

Ek matrix numbers ka ek rectangle hai — rows aur columns, jaise ek spreadsheet. Ise ek capital letter se name karte hain, jaise $W$ , aur ek single cell ko do subscripts se point karte hain:

$W_{k j} = row k, column j mein number$

Why the topic needs it: parent weights ko $W_{k j}$ aur inputs ko $X_{ik}$ likhta hai. Row/column convention ke bina, $y_{ij} = \sum_{k} X_{ik} W_{k j}$ gibberish hai. Iske saath, tum formula dekh sakte ho: yeh $X$ ki row $i$ ke column $k$ ko $W$ ke column $j$ ke row $k$ ke saath chalata hai.

6. Matrix multiply $Y = X W$

Matrix $X$ (size $M \times K$ ) ko matrix $W$ (size $K \times N$ ) se multiply karne ke liye, tum har output cell ko ek dot product ke roop mein compute karte ho: $X$ ki row $i$ aur $W$ ke column $j$ ka dot product.

$Y_{ij} = \sum_{k = 1}^{K} X_{ik} W_{k j}$

Shapes ko beech mein match karna chahiye: $X$ mein $K$ columns hain aur $W$ mein $K$ rows — yeh shared $K$ har dot product ki length hai. Result $Y$ $M \times N$ ka hai.

Why the topic needs it: matrix multiply neural inference ka ~90% hai. Poora NPU sirf yeh ek cheez fast karne ke liye exist karta hai. Iske teen dimensions $M, K, N$ decide karte hain ki PEs ki grid kitni bhari hui hai (utilisation, parent note mein).

7. Processing element (PE) aur grid

Ek PE (Processing Element) chip par ek physical MAC unit hai: hardware jo har clock cycle mein $box \leftarrow box + a \times b$ karta hai.

Kai PEs ko $R$ rows aur $C$ columns ki 2-D grid mein rakho aur numbers ko neighbours ke beech flow karne do — yahi systolic array hai. Saare $R \times C$ PEs har tick fire karte hain, toh grid har cycle mein $R \cdot C$ MACs karta hai.

8. Reuse, aur memory kyun dushman hai

Har woh number jo ek PE multiply karta hai memory se aana chahiye. Do types hain:

DRAM — bada, door, slow, energy-hungry (ek read ~ $640 pJ$ lagta hai).
On-chip SRAM / scratchpad — chhota, paas, sasta.

Ek MAC khud sirf ~ $0.2 pJ$ lagta hai. Toh number fetch karna ~1000× zyada mahenga ho sakta hai usse use karne se.

Why the topic needs it: parent kehta hai "data reuse hi poora game hai." Ab tum jaante ho kyun: arithmetic almost free hai; data move karna expensive hai.

9. Precision: ek number kitne bits use karta hai

Hardware mein ek number fixed number of bits (0/1 switches) mein store hota hai. Zyada bits = zyada fine detail lekin bade, zyada hungry circuits.

fp32 — 32-bit floating point, bahut precise.
fp16 / bfloat16 — 16-bit, half the bits.
int8 — 8-bit integer, coarse lekin tiny.

Why the topic needs it: "low precision wins" parent ka ek headline claim hai. $(bits)^{2}$ scaling hi iska reason hai.

Prerequisite map

Har woh box jo tum ab samajhte ho seedha parent topic mein feed hota hai, Neural processing units (NPUs). Grid idea Systolic arrays ki taraf jaata hai; reuse idea Dataflow and data reuse ki taraf; memory-vs-math cost Energy per operation aur Roofline model ki taraf; aur bit-width idea Quantization and int8 inference ki taraf. "Special chip kyun at all" thread Domain-specific architectures se connect hota hai, aur comparisons GPUs and SIMT aur Tensor cores mein milte hain.

Equipment checklist

Khud test karo — right side cover karo aur reveal karne se pehle answer do.

a \times b

ek picture ke roop mein kya matlab hai?

a

wide aur

b

tall rectangle ka area.

MAC kya hai?

Do numbers multiply karo aur result ek running total mein add karo:

box \leftarrow box + a \times b

box \leftarrow box + x

mein arrow

\leftarrow

ka kya matlab hai?

"Ban jaata hai" — box ki nayi value uski purani value plus

x

hai.

Vector kya hai?

Numbers ki ek ordered list.

Dot product sirf repeated MAC kyun hai?

Tum pairs ko position-by-position multiply karte ho aur unhe ek running total mein accumulate karte ho.

\sum_{k = 1}^{K} t_{k}

words mein kya kehta hai?

t_{k}

terms add karo jaise counter

k

1

K

tak run karta hai.

X_{ik}

mein kaun sa subscript row hai?

Pehla wala,

i

(row

i

, column

k

Y = X W

ke liye jahan

X

M \times K

ka hai aur

W

K \times N

ka hai,

Y

ka size kya hai?

M \times N

Ek output cell

Y_{ij}

mein kitne MACs lagte hain?

K

— ek har shared-dimension index ke liye.

PE kya hai?

Ek physical MAC unit jo har clock cycle mein ek multiply-accumulate karta hai.

Clock cycle kya hai?

Chip ki drumbeat ka ek tick; clock

f

baar per second tickti hai (Hz).

R \times C

grid har cycle mein kitne MACs karta hai?

R \cdot C

(saare PEs har tick fire karte hain).

Memory real cost kyun hai, arithmetic kyun nahin?

Ek DRAM read (~640 pJ) ek MAC (~0.2 pJ) se ~1000× zyada costly hai.

"Data reuse" ka matlab kya hai aur care kyun karein?

Har loaded number ko kai multiplications mein feed karna taaki mahenga fetch ek baar pay ho, kai baar nahin.

int8 fp32 se ~16× zyada MACs kyun deta hai?

Multiplier area/energy (bit-width)² ke saath scale karta hai, aur

(32/8)^{2} = 16