Neural processing units (NPUs): Question bank — Hinglish deep dive

6.5.8 · D5 · Hardware › Advanced & Emerging Architectures › Neural processing units (NPUs)

Shuru karne se pehle, har woh symbol aur unit jo in questions mein use hote hain, plain language mein defined hain.

Definition Woh vocabulary jo tumhe chahiye hogi

Ek PE (processing element) grid mein ek chhota sa compute box hota hai. Har PE ke andar exactly ek MAC (multiply-accumulate) hota hai — yeh operation hai "do numbers multiply karo, phir result ko ek running total mein add karo." Ek aisa box socho jisme do inputs hain aur ek number jisme woh baar baar add karta rehta hai.
Jab hum $(M \times K)$ aur $(K \times N)$ shape ki do matrices multiply karte hain: == $M$ pehli matrix ke rows ki sankhya hai (jaise ek batch mein kitne input samples hain), $N$ doosri matrix ke columns ki sankhya hai (jaise kitne output neurons hain), aur $K$ == woh shared inner dimension hai jis par sum kiya jaata hai (jaise har neuron ko kitne inputs feed hote hain). Ek output element, length $K$ ka dot product hota hai.
PEs ka array ek rectangle hota hai: == $R$ rows of PEs, $C$ columns of PEs, aur $f$ == clock frequency hai — chip ek second mein kitni baar tick karta hai (ek "cycle" ek heartbeat hai jisme har PE ek MAC karta hai). $f = 1 GHz$ matlab ek billion ticks per second.
TOPS = tera-operations per second = har second mein trillions ( $1 0^{12}$ ) operations. Ek MAC do operations count hota hai (ek multiply + ek add), isliye ek array jo frequency $f$ par $R \cdot C$ MACs per cycle karta hai woh $\frac{2 \cdot R \cdot C \cdot f}{1 0 ^{12}}$ TOPS tak pohonchta hai.

Neeche ke do diagrams mushkil questions ko anchor karte hain: kaise ek systolic array fill aur drain hota hai, aur kaise Roofline model compute-bound aur memory-bound mein split karta hai.

True or false — justify karo

CPU simply ek slow NPU hai.

False — CPU ek general machine hai jo apni zyaatar energy control par spend karta hai (fetch, decode, branch). NPU ek dataflow hard-wire karta hai, isliye woh branchy code mein bura hai lekin matrix-multiply par per joule 10–100× better hai. Dekho Domain-specific architectures.

Do chips jinke equal peak TOPS hain woh same network ko same speed par chalayenge.

False — peak TOPS 100% utilisation aur unlimited memory bandwidth assume karta hai. Jis chip ki memory actually apne array ko feed kar sakti hai woh jeetta hai; doosra stall karta hai. Roofline model se judge karo, spec sheet se nahi.

Ek systolic array isliye kaam karta hai kyunki yeh normal ALU se zyada clock rate par run karta hai.

False — "systolic" data kahan jaata hai ke baare mein hai, clock rate ke baare mein nahi. Numbers sirf neighbouring PEs ke beech hop karte hain, isliye wires chhoti rehti hain aur har step mein koi global memory access nahi hota; yahi efficiency deta hai, raw speed nahi. Dekho Systolic arrays.

fp32 se int8 mein switch karne se har network unusable ho jaata hai.

False — proper calibration ke saath inference ke liye, int8 typically 1% se kam accuracy khota hai kyunki nets training ke saath noise ke saath train hue the aur unki nonlinearities rounding ko absorb karti hain. Dekho Quantization and int8 inference.

Har weight ko memory se ek baar load karna, phir use reuse karna, matmul ke multiplications ki sankhya change karta hai.

False — multiply count

M \cdot N \cdot K

(rows × output-columns × shared inner dimension) par fixed hai. Reuse badalta hai ki tum kitne expensive memory reads pay karte ho, arithmetic nahi. Dekho Dataflow and data reuse.

Ek int8 multiplier ek fp32 se approximately 4× chhota hota hai.

False — multiplier area bit-width

b

ke square ke saath scale karta hai (woh

b \times b

partial-product grid), isliye yeh approximately

(32/8)^{2} = 16 \times

chhota hai, 4× nahi.

Clock frequency

f

double karna aur array width double karna dono same throughput gain dete hain.

Raw throughput number ke liye True, lekin array-width path usually smarter hai — yeh similar voltage par 2× MACs/cycle deliver karta hai, jabki

f

double karne ke liye aksar higher voltage

V

chahiye aur, kyunki power

\propto V^{2} f

hai, disproportionately zyada energy burn hoti hai (dekho Energy per operation).

Agar ek layer memory-bandwidth bound hai, toh zyada PEs add karne se woh speed up hoti hai.

False — extra compute useless hai jab data itni tez aa hi nahi sakta; array sirf zyada idle baith jaata hai. Tumhe zyada bandwidth ya better reuse chahiye, jise roofline visible karta hai.

Error dhundho

"Peak TOPS

= R \cdot C \cdot f

2× ka error hai — har MAC ek multiply aur ek add hai, isliye operations

= 2 \cdot R \cdot C \cdot f

hain (phir "tera" ke liye

1 0^{12}

se divide karo).

R \cdot C \cdot f

MACs/second count karta hai, ops/second nahi.

"Ek weight-stationary array har clock mein har PE mein weight reload karta hai."

Contradiction — stationary ka matlab hai weight ek baar load hota hai aur held rehta hai jabki bahut saare activations uske paas se stream hote hain. Har clock mein reload karna us reuse ko destroy kar dega jo is design ko justify karta hai.

"Utilisation = actual MACs ÷ peak TOPS."

Units mismatch — tumhe

R \cdot C \cdot (cycles run)

se divide karna chahiye, jo ek MAC count hai, na ki ek rate. Ek count ko ek per-second rate (TOPS) se divide karna seconds deta hai, fraction nahi.

"Systolic array cycle 1 par apna pehla result produce karta hai."

Pipeline fill ko ignore karta hai — data ko grid ke across march karna hoga pehle full column nikle; woh initial fill cycles (aur end mein drain cycles) utilisation ko girate hain, especially chhoti matrices ke liye.

"Kyunki ek MAC ~0.2 pJ cost karta hai aur ek DRAM read ~640 pJ, arithmetic expensive part hai."

Ulta hai — DRAM read picojoules mein MAC se ~1000× hai, isliye memory movement expensive part hai; poora NPU design un reads ko on-chip reuse ke zariye avoid karne ke liye exist karta hai.

"Bada array hamesha better hota hai, isliye use

4096 \times 4096

banao."

Utilisation ko ignore karta hai — ek bada array typical layer shapes aur fill/drain ke dauran mostly idle rehta hai, silicon waste karta hai. Array size ko workload ki matrix dimensions

M, N, K

se match karna chahiye.

Why questions

Matrix multiply kyun hai, activation functions kyun nahi, jis cheez ke around NPUs shaped hain?

Kyunki inference runtime se ~90% matmul hai — woh 80/20 rule. Dominant operation ko optimize karo; cheap nonlinearities chhote dedicated units par ride kar sakti hain.

Data reuse, multipliers add karne se zyada kyun matter karta hai?

Kyunki bottleneck woh energy hai jo data move karne mein kharch hoti hai, arithmetic karne mein nahi. Ek number ek baar load karna (640 pJ) aur use bahut saare multiplies (0.2 pJ each) ko feed karna ~1000× fetch cost ko un saare MACs mein amortise karta hai.

Neural networks int8 tolerate kar sakti hain jab scientific computing nahi kar sakta?

Nets noise ke saath train hote hain aur values saturating nonlinearities se pass karti hain, isliye chhoti rounding errors amplify hone ki jagah absorb ho jaati hain; scientific codes aisi operations chain karte hain jo tiny errors ko bade ones mein compound karte hain.

"Column of results per cycle" sirf array fill hone ke baad kyun appear hoti hai?

Activations aur partial sums ko physically far edge tak propagate karna hota hai pehle. Jab tak pipeline full nahi hoti, kuch PEs mein koi valid inputs nahi hain, isliye koi complete output nahi ja sakta.

GPUs aur NPUs coexist kyun karte hain rather than ek doosre ko replace karne ke?

Ek GPU (GPUs and SIMT) bahut saare kernels par flexible rehta hai; ek NPU flexibility ko ek dataflow par peak efficiency ke liye trade karta hai. Training aur irregular kaam GPU ko favour karta hai; fixed high-volume inference NPU ko favour karta hai. Tensor cores GPUs ke andar middle ground hain.

Peak TOPS ki jagah effective throughput kyun quote karte hain?

Peak sirf perfectly shaped, perfectly fed workload se achievable hai. Effective throughput real utilisation reflect karta hai — chhote layers, fill/drain, aur memory limits routinely ise peak ke 20–60% tak cut kar dete hain.

Edge cases

Ek layer jisme

N = 1

hai (ek single output column, jaise batch-size-1 vector) ek

256 \times 256

array par run karta hai — utilisation ka kya hota hai?

Sirf ek column of PEs useful kaam karta hai; baaki 255 idle baithe rehte hain, isliye utilisation roughly

1/256

tak collapse ho jaati hai. Aisi matrix-vector shapes memory-bound hain aur wide arrays ke liye poor fit hain.

Agar ek matrix dimension array se chhota hai (jaise

K = 64

256

-tall array par)?

Unused PE rows har cycle waste hote hain, aur array under-fills karta hai; pipeline fill consider karne se pehle bhi utilisation girta hai. Tiling ya ek chhota/mapped array efficiency recover karta hai.

Ek systolic array mein last cycles mein throughput kya hai jab last activations exit karte hain (drain)?

Yeh taper off hoti hai — pipeline empty hone ke saath kam PEs valid data hold karte hain, fill phase ko mirror karta hai. Drain cycles partial output produce karte hain aur overall utilisation ke against count hote hain.

Bit-width ko extreme tak le jao: kya 1-bit ("binary") precision, square law ke predict kiye

(32/1)^{2} = 1024 \times

multiplier savings deta hai?

Area law roughly hold karta hai, lekin accuracy usually break down hoti hai — binary nets ko special training chahiye aur phir bhi hard tasks par accuracy lose hoti hai, isliye theoretical silicon win us limit par cap hoti hai jo model tolerate kar sakta hai.

Agar do operands dono zero hain, toh kya PE phir bhi multiply par energy spend karta hai?

Haan, jab tak hardware mein explicit zero-skipping (sparsity) logic na ho — ek naive dense PE multiply clock karta hai regardless. Zeros exploit karna exactly wahi hai jo sparse-accelerator designs add karte hain.

1 \times 1

array (ek PE) ki degenerate limit par, NPU kya ban jaata hai?

Bas ek single MAC unit — ek sequential multiplier-adder bina kisi parallelism ya spatial reuse ke, essentially woh CPU-style ek-ek karke karne wala path jise NPU escape karne ke liye banaya gaya tha.

Recall Self-check: woh chaar trap families name karo jo is page par attack hoti hain

Memory-vs-math cost ::: ~1000× DRAM tax (640 pJ vs 0.2 pJ) jo reuse ko real design goal banata hai. Peak-vs-effective ::: peak TOPS ek marketing ceiling hai; utilisation aur roofline sach bataate hain. Precision fear ::: int8 inference theek hai; $b^{2}$ area law silicon win drive karta hai. Shape/degenerate cases ::: chhota $N$ , chhota $K$ , fill/drain, aur $1 \times 1$ arrays utilisation ko khatam kar dete hain.