Neural processing units (NPUs): Visual walkthrough

6.5.8 · D2 · Hardware › Advanced & Emerging Architectures › Neural processing units (NPUs)

Step 1 — Do number lists, ek answer

KYA HAI. Matrices bhool jao. Sabse chhote possible neural-network idea se shuru karo: ek output neuron. Uske paas ek inputs ki list hai (unhe $x_{1}, x_{2}, x_{3}, \dots$ kaho) aur ek weights ki list hai (unhe $w_{1}, w_{2}, w_{3}, \dots$ kaho). Neuron ka kaam hai in do lists se ek number produce karna.

YEH OPERATION KYUN. Ek neuron poochta hai "har input kitna matter karta hai?" — woh har input ko uske weight se scale karta hai aur votes add karta hai. Woh "scale-and-add-up" hi neural computing mein sabse zyada repeat hone wala kaam hai, isliye yahi woh cheez hai jise hume sasta banana hai.

PICTURE. Rangeen boxes ki do rows lined up hain. Har purple input box seedha neeche wale coral weight box se pair karti hai. Hum har pair ko multiply karte hain, phir har product ek bucket mein daalnge aur sum karte hain.

$y = x_{1} w_{1} + x_{2} w_{2} + x_{3} w_{3} + \dots + x_{K} w_{K}$

$x_{k}$ ::: $k$ -vaan input number (purple). $k$ sirf ek counter hai, $1$ se $K$ tak.
$w_{k}$ ::: woh weight jo $x_{k}$ ke saath pair hai (coral) — uska "importance dial."
$K$ ::: kitne pairs hain (dono lists ki length).
$y$ ::: ek single output number, saare products ka sum.

Step 2 — Atomic move: multiply karo, phir running total mein add karo

KYA HAI. Dekho ki aap woh sum haath se actually kaise compute karte. Aap apne dimag mein ek running total rakhte ho. $0$ se shuru karo. Pehla pair multiply karo, usme add karo. Agla pair multiply karo, usme add karo. Repeat.

YEH ALAG KYUN KARO. Kyunki hardware sirf ek chhoti si cheez achhi bana sakta hai. Agar har dot product usi ek chhote kaam ki repetition hai — "do numbers multiply karo, result jo already hai usme add karo" — to hume sirf woh ek gadget banana hai, aur hajaaron copies banana hai.

PICTURE. Ek single box jis par MAC (Multiply-ACcumulate) likha hai. Do inputs enter karte hain ( $a$ aur $b$ ), ek running total $c$ pehle se aata hai, aur nikalta hai $c + a \times b$ . Purana total replace ho jaata hai naye se.

$acc_{new} = running total so far acc_{old} + one input a \times its weight b$

Step 3 — Ek saath bahut saare neurons = matrix multiply

KYA HAI. Ek neuron ek output deta hai. Ek layer mein bahut neurons hote hain, aur hum usually bahut inputs ek saath feed karte hain (ek batch). Input rows ko ek grid $X$ mein stack karo, weight columns ko ek grid $W$ mein stack karo. Har output $y_{ij}$ $X$ ki row $i$ aur $W$ ke column $j$ ka dot product hai.

KYUN. Real networks kabhi ek dot product nahin karte — woh poora rectangle karte hain. Rectangle ko matrix multiply naam dene se hum ek clean formula mein total kaam count kar sakte hain (agla step) har neuron ko track karne ki bajay.

PICTURE. Baayein ek grid $X$ , upar ek grid $W$ , aur bottom-right mein answer grid $Y$ . $Y$ ke cell $(i, j)$ ko bharne ke liye, $X$ ki row $i$ ke saath slide karo aur $W$ ke column $j$ ke saath neeche jao, pairs ko MAC-ing karte hue.

$y_{ij} = \sum_{k = 1}^{K} X_{ik} W_{k j}$

$X_{ik}$ ::: row $i$ , position $k$ mein input (hum kaun sa pair hain).
$W_{k j}$ ::: column $j$ , position $k$ mein weight.
$i$ ::: output row kaun si ( $1 \dots M$ ) — ek per batch item.
$j$ ::: output column kaun sa ( $1 \dots N$ ) — ek per neuron.
Andar wala $\sum_{k}$ exactly wahi dot product hai jo Step 1 mein tha.

Yahi woh $Y = X W$ hai jo parent note ne diya tha — ab har letter earned hai.

Step 4 — Kaam gino, aur asli dushman dhundho

KYA HAI. $M \times N$ output cells mein se har ek cell ko $K$ MACs lagte hain (ek per pair). To total ek simple product hai.

KYUN. Hume jaanna hai ki yeh kitna bura hota hai, aur dard kahan rehta hai — arithmetic mein, ya memory se numbers fetch karne mein?

$total MACs = rows M \cdot cols N \cdot pairs each K$

PICTURE. Do side-by-side bars. Lamba coral bar DRAM se ek number fetch karne ki energy hai ( ~~$640$ pJ). Chhota mint bar multiply ki khud ki energy hai (~~ $0.2$ pJ). Fetch, math se ~ $1000 \times$ zyada hai.

Step 5 — Reuse: har number ko bahut saare multiplies feed karne chahiye

KYA HAI. Step 3 se notice karo: ek akela weight $W_{k j}$ har output row $i$ mein appear karta hai — woh $M$ baar use hota hai. Ek akela input $X_{ik}$ har output column $j$ mein appear karta hai — $N$ baar use hota hai. Har number kai baar reusable hai.

KYUN. Agar ek number ek baar fetch kiya jaaye lekin $M$ baar multiply kiya jaaye, to crushing memory cost $M$ se divide ho jaati hai. Isi tarah hum Step 4 ki coral bar ko defeat karte hain — use cheaper banake nahin, balki use bahut kam baar pay karke.

PICTURE. Ek weight box still baith a hai. Ek arrow dikhata hai ki wahi same weight poori ek row of multiplies mein reuse ho raha hai, jabki fresh inputs uske paas se guzarte hain. "Load once, multiply many."

Step 6 — MACs ko grid mein wire karo: systolic array

KYA HAI. Step 2 ka MAC gadget lo aur plane mein tile karo: MACs ki rows aur columns, har ek sirf apne neighbours se wired (right aur down — koi long wires nahin, koi global memory nahin). Har weight apne cell mein load karo aur wahan hold karo (weight-stationary). Ab inputs ko baayi taraf se stream karo, ek column of data per clock tick.

GRID KYUN, SIRF NEIGHBOURS KYUN. Ek grid se $R \times C$ multiplies ek hi waqt hoti hain (parallelism, Step 4 ka ilaaj). Neighbour-only wiring ka matlab hai data door memory tak travel karne ki bajay ek chhota sa step hop karta hai per tick — sasta aur scalable. Yahi woh systolic array hai jise parent note ne naam diya tha.

PICTURE. Cells ki ek grid. Weights (coral) har cell ke andar baith e hain, fixed. Inputs (purple) left→right march karte hain. Partial sums (mint) top→bottom girte hain, har cell apna product girte total mein add karta hai. Dekho ek input diagonally poori grid sweep karta hai.

$psum_{out} = sum from cell above psum_{in} + input flowing in from left X_{in} \times weight held in this cell W_{stored}$

Kyunki ek input, ek baar inject hone ke baad, cells ki poori row mein march karta hai, woh single fetch poori row of multiplies ko feed karta hai — reuse (Step 5) physically bana diya. Related ideas: Tensor cores, GPUs and SIMT.

Step 7 — Fill, drain, aur steady-state ki jeet

KYA HAI. Grid instant nahin hai. Data staggered enter karta hai (ek diagonal wavefront), to early cells kaam karte hain jabki baad ke cells abhi bhi khali wait kar rahe hain — pipeline fill. Ek baar full hone ke baad, array steady state mein hota hai: har clock pe results ka ek poora column finish hota hai. Ant mein, aakhiri data khaalee hote cells ke through drain ho jaata hai.

YEH KYUN DIKHAO. Kyunki honest speed busy cycles aur filling/draining cycles ke ratio par depend karti hai. Badi matrix ke liye fill negligible hai; chhoti matrix ke liye fill zyaadatar run hai aur array mostly idle hai. Yahi fark hai marketing TOPS aur real speed mein.

PICTURE. Teen snapshots side by side: fill (live cells ka diagonal, corners idle), steady state (poori grid live), drain (live diagonal exit kar raha hai, entry corner phir se idle). Busy fraction shaded area hai.

$Peak throughput = MACs per tick R \cdot C \cdot ticks per second f_{clock} [MACs/s]$

$TOPS = \frac{2 R C f _{clock}}{1 0 ^{12}}, Utilisation = \frac{actual MACs}{R \cdot C \cdot ( cycles run )}$

$R, C$ ::: MAC cells ki rows aur columns.
$f_{clock}$ ::: cycles per second (Hz).
woh $2$ ::: har MAC = ek multiply plus ek add = do operations.
Utilisation ::: cells ka woh fraction jo actually useful kaam kar raha hai — kabhi $1$ assume mat karo.

Step 8 — Degenerate cases jo miss nahin karne chahiye

KYA HAI AUR KYUN. Ek formula tab hi trusted hota hai jab uske edges check kiye jaayein. Har corner chalo:

Case	Kya hota hai	Kyun
$K = 1$	Har output ke liye ek pair; accumulator ek single product add karta hai — koi chaining nahin.	Length one ka dot product sirf ek multiply hai.
Layer array se chhoti ( $N < C$ )	$C$ columns mein se sirf $N$ use hote hain → utilisation $\leq N / C$ .	Idle columns area burn karte hain lekin kaam nahin karte.
Chhota $M$ , badi grid	Fill/drain dominate karta hai; steady state barely reach hoti hai.	Rows itni nahin ki pipeline latency amortise ho sake.
Memory-bound layer	PEs data ka wait karte hue bhookhe rahte hain; effective TOPS $≪$ peak.	Bandwidth, grid nahin, limit hai (Roofline model).
Zero inputs / zero weights	MAC phir bhi run karta hai ( $0 \times b = 0$ ); result correct, energy waste.	Hardware zeros skip nahin karta jab tak specially build na ho.
int8 vs fp32 precision	int8 multiplier ≈ $(32/8)^{2} = 16 \times$ chhota.	Multiplier area $\propto$ (bit-width) $^{2}$ → 16× zyada MACs per area.

PICTURE. Ek chhota array jisme sirf left ke do columns lit hain (ek narrow layer). Right par dark idle columns dikhate hain wasted silicon — utilisation $= 2/ C$ .

Ek-picture summary

Ek canvas par poori derivation: do lists → ek dot product → ek MAC gadget → dot products ki ek matrix → memory tax → reuse → ek marching systolic grid → fill/steady/drain → peak aur utilisation.

Recall Feynman retelling — plain words mein wapas bolo

Ek neuron sirf har input ko ek dial se multiply karta hai aur votes add karta hai — yahi ek dot product hai. Aapke hardware ko ek hi trick chahiye: "do numbers multiply karo aur unhe ek running total mein add karo": ek chhota sa MAC cell. Poori layer in dot products ka ek rectangle hai — ek matrix multiply — aur woh billions of MACs hain. Lekin multiplying basically free hai; jo cheez maar daala hai woh hai numbers ko memory se fetch karna, jo ~1000× zyada cost karta hai. To winning idea hai reuse: har number ek baar fetch karo aur usse bahut saare multiplies se guzaaro. Isko physically karne ke liye hum ek plane ko MAC cells se tile karte hain jo sirf apne neighbours se wired hain, har cell mein ek weight freeze karte hain, aur inputs ko left-to-right march karne dete hain jabki partial sums top-to-bottom girte hain. Har input, ek baar inject hone ke baad, poori row sweep karta hai — ek fetch, bahut saare multiplies. Ek chhoti si fill ke baad grid har clock tick pe answers ka ek poora column finish karta hai. Uski top speed sirf (number of cells) × (clock) × 2 hai — lekin tabhi jab cells busy rahein, jo chhoti ya memory-starved layers kharaab kar deti hain. Aur kyunki chhote numbers ka multiplier kaafi chhota hota hai, int8 par jaane se ~16× zyada cells fit ho jaate hain — zyada math, kam energy, same silicon.