Neural processing units (NPUs): Exercises — Hinglish deep dive

6.5.8 · D4 · Hardware › Advanced & Emerging Architectures › Neural processing units (NPUs)

Shuru karne se pehle, ek shared reference card taaki koi bhi symbol unexplained na rahe:

Level 1 — Recognition

L1.1

Kaunsi single operation neural-network inference runtime ka roughly 90% banati hai, aur wo kis atomic hardware operation se bani hoti hai?

Recall Solution

Kya: Neural-net inference mein matrix multiplication dominant hai. Kyun: fully-connected aur convolution layers dono $Y = X W$ par reduce ho jaate hain, aur har output element ek dot product $y_{ij} = \sum_{k} X_{ik} W_{k j}$ hai. The atom: har dot product multiply-accumulate (MAC) operations ki ek chain hai — ek pair multiply karo, running sum mein add karo. Toh poora workload "billions of MACs" hai.

L1.2

Har term ko uske ek-line meaning se match karo: (a) PE, (b) systolic array, (c) weight-stationary, (d) on-chip SRAM.

Recall Solution

(a) PE — Processing Element: ek MAC unit jo $a \times b + c$ compute karta hai.
(b) Systolic array — PEs ki ek 2-D grid jahan data rhythmically neighbouring PEs ke beech flow karta hai (har step par koi global memory read nahi hoti).
(c) Weight-stationary — weights ek baar load hote hain aur har PE ke andar hold rehte hain jabki activations stream karte hain.
(d) On-chip SRAM — compute ke paas fast scratchpad memory jo weights/activations hold karti hai taaki DRAM ka expensive trip avoid ho.

Level 2 — Application

L2.1

Ek matmul $(256 \times 512)$ matrix ko $(512 \times 128)$ matrix se multiply karta hai. Kitne MACs lagte hain?

Recall Solution

Formula: $MACs = M \cdot N \cdot K$ jahan yahan $M = 256$ , $K = 512$ , $N = 128$ . Yeh formula kyun: har $M \cdot N$ outputs mein se har ek length $K$ ka dot product hai, yaani $K$ multiplies each. $MACs = 256 \times 128 \times 512 = 16, 777, 216 \approx 1.68 \times 1 0^{7} .$

L2.2

Ek array $R = C = 256$ PEs ka hai jo $f = 1$ GHz par run karta hai. Iska peak TOPS do.

Recall Solution

Formula: $TOPS = \frac{2 R C f}{1 0 ^{12}}$ . Yeh $2$ har MAC mein multiply aur add dono count karta hai. $TOPS = \frac{2 \times 256 \times 256 \times 1 0 ^{9}}{1 0 ^{12}} = \frac{2 \times 65536 \times 1 0 ^{9}}{1 0 ^{12}} = 131.07 TOPS .$ Yeh kaisa dikhta hai: neeche figure dekho — jeet grid ka area hai ( $25 6^{2} = 65536$ MACs har single tick), clock speed nahi.

L2.3

Wohi $256 \times 256$ array ek matmul run karta hai $M = N = K = 256$ ke saath perfectly (fill/drain ignore karo). Kitne clock cycles, aur $1$ GHz par microseconds mein kitna time?

Recall Solution

Total MACs: $25 6^{3} = 16, 777, 216$ . MACs per cycle: poora grid $R \cdot C = 65536$ per cycle karta hai. $cycles = \frac{25 6 ^{3}}{65536} = \frac{16 , 777 , 216}{65536} = 256 cycles .$ Time: $256$ cycles $\times 1 ns/cycle = 256 ns = 0.256 μ s$ . Exactly $K$ kyun: weight-stationary $K \times N = 256 \times 256$ grid ke saath jo poori weight matrix hold karta hai, tum saari $M = 256$ activation rows stream karte ho — fill ke baad har cycle mein ek row — toh $M = 256$ result rows $256$ cycles mein.

Level 3 — Analysis

L3.1

Ek layer $128 \times 128$ array par map hoti hai lekin sirf $96$ of the $128$ columns aur $128$ of the $128$ rows use karti hai. Fill/drain ignore karte hue, utilisation aur effective TOPS kya hogi agar peak ( $2$ GHz par) $65.5$ TOPS hai?

Recall Solution

Peak check: $TOPS = \frac{2 \times 12 8 ^{2} \times 2 \times 1 0 ^{9}}{1 0 ^{12}} = \frac{2 \times 16384 \times 2 \times 1 0 ^{9}}{1 0 ^{12}} = 65.54$ TOPS. ✓ Busy PEs: $128 \times 96 = 12288$ out of $128 \times 128 = 16384$ . $Utilisation = \frac{12288}{16384} = 0.75 = 75%.$ Effective TOPS: $0.75 \times 65.54 = 49.15$ TOPS. Kyun: idle $32$ columns silicon hain jo kuch nahi kar rahe — peak number assume karta hai ki har PE har cycle fire karta hai, jo ek mismatched layer shape tod deta hai.

L3.2

Naïve dataflow har MAC ke liye dono operands DRAM se load karta hai. Ek DRAM read $\approx 640$ pJ aur ek MAC $\approx 0.2$ pJ. $16.78 \times 1 0^{6}$ MACs ke liye (L2.1 se), total energy compute karo (a) naïve — 2 reads per MAC — vs (b) ek ideal reuse case jahan dono operands mein se har ek sirf ek baar read hota hai aur poore traversal ke liye reuse hota hai. Maan lo reuse case mein $2$ DRAM reads per PE-load hain lekin hum sirf arithmetic count karte hain jo MACs par dominate hai. Ratio do (naïve DRAM energy) : (MAC arithmetic energy).

Recall Solution

Naïve memory energy: $2$ reads/MAC $\times 640$ pJ $\times 16.78 \times 1 0^{6} = 2 \times 640 \times 16.78 \times 1 0^{6}$ pJ. $= 2 \times 640 \times 16.777216 \times 1 0^{6} = 2.147 \times 1 0^{10} pJ = 21.47 mJ (memory only) .$ Arithmetic energy: $0.2 pJ \times 16.78 \times 1 0^{6} = 3.36 \times 1 0^{6}$ pJ $= 3.36 μ J$ . Ratio: $\frac{2.147 \times 1 0 ^{10}}{3.36 \times 1 0 ^{6}} = 6400$ . Iska matlab: naïvely, memory movement 6400× arithmetic energy jalata hai — yahi woh number hai jo poori systolic reuse machine ko justify karta hai. $2 \times 640/0.2 = 6400$ factor bas "do DRAM reads har ek $640$ pJ ka vs ek $0.2$ pJ MAC" hai.

Level 4 — Synthesis

L4.1

Tumhe ek silicon budget diya gaya hai aur bataya gaya hai: ek fp32 multiplier area $A$ occupy karta hai. Multiplier area roughly (bit-width)² scale karta hai. (a) Ek int8 multiplier kitna chota hai? (b) Agar tum saare fp32 MACs ko int8 se swap karo, toh same area mein kitne zyada MACs fit honge? (c) Peak TOPS gain kya hai (same clock par)?

Recall Solution

(a) Area ratio: $Energy/Area_{mult} \propto (bits)^{2}$ , toh $\frac{A _{fp32}}{A _{int8}} = (\frac{32}{8})^{2} = 4^{2} = 16.$ Ek int8 multiplier 16× chota hai. (b) Same area ⇒ 16× zyada MAC units. (c) Peak TOPS $= 2 R C f$ ; $16 \times$ zyada PEs ke saath (bada effective $R \cdot C$ ) aur unchanged $f$ , peak bhi 16× scale karta hai. Kyun NN ise accept karte hain: networks quantisation noise ke liye robust hoti hain (Quantization and int8 inference) — noise ke saath trained aur tolerant nonlinearities use karte hue — toh int8 par inference typically <1% accuracy khoata hai jabki yeh 16× density milti hai.

L4.2

Design choice: tumhe back-to-back chhote $32 \times 32 \times 32$ matmuls ki ek stream run karni hai. Utilisation ke liye tum $256 \times 256$ array ya $32 \times 32$ array choose karoge, aur kyun? (Estimate ke liye fill/drain differences ignore karo; ek $K \times N = 32 \times 32$ layer ek $32 \times 32$ region par map hoti hai.)

Recall Solution

Mapping: ek $K \times N = 32 \times 32$ weight matrix sirf $32 \times 32$ block of PEs occupy karta hai. $256 \times 256$ array par: busy PEs $= 32 \times 32 = 1024$ out of $256 \times 256 = 65536$ ⇒ $Utilisation = \frac{1024}{65536} = 0.0156 = 1.56%.$ $32 \times 32$ array par: busy PEs $= 1024$ out of $1024$ ⇒ 100% utilisation. Conclusion: chhotey array ki small layers ke liye utilisation kaafi better hai. Yahan giant array ka peak TOPS waste marketing hai — zyaattar PEs idle baithe hain. Yahi reason hai ki real designs chhote layers tile/batch karte hain ya array size ko typical layer shapes ke matmul shapes se match karne ke liye choose karte hain (Domain-specific architectures, Dataflow and data reuse).

Level 5 — Mastery

L5.1

Ek $256 \times 256$ weight-stationary array $f = 1$ GHz par ek matmul run karta hai $M = 1024$ , $K = 256$ , $N = 256$ ke saath. Pipeline fill latency $R + C - 1$ cycles hai pehla full column aane se pehle, phir $M$ columns ke liye ek result column per cycle. (a) Fill samet total cycles. (b) Fill ignore karte hue ideal cycles. (c) Utilisation as (ideal / actual). (d) Effective TOPS.

Recall Solution

Weight matrix exactly $K \times N = 256 \times 256$ hai, toh yeh poore $256 \times 256$ grid ko tile karta hai — har PE ek weight hold karta hai, steady state mein sab busy. (a) Fill: $R + C - 1 = 256 + 256 - 1 = 511$ cycles. Steady state: $M = 1024$ activation rows stream karo ⇒ $1024$ cycles. Total: $cycles_{actual} = 511 + 1024 = 1535.$ (b) Ideal (fill nahi): useful work $M = 1024$ result columns hai ek per cycle $= 1024$ cycles. (Equivalently $MACs / (R C) = 1024 \cdot 256 \cdot 256/65536 = 1024$ .) ✓ (c) Utilisation: $Util = \frac{1024}{1535} = 0.6671 = 66.7%.$ (d) Effective TOPS: peak $= 2 \times 25 6^{2} \times 1 0^{9} /1 0^{12} = 131.07$ TOPS. $Effective = 0.6671 \times 131.07 = 87.4 TOPS .$ Yeh kaisa dikhta hai: fill/drain neeche figure mein ramp hai — array $511$ cycles tak "warm up" karta hai partial work karte hue phir full rhythm pe aata hai. Bade $M$ ke saath yeh overhead shrink hoti hai; tiny $M$ ke saath yeh dominate kar leti hai.

L5.2

Same array aur clock. Ab tumhare paas choice hai: upar wali layer run karo (bada $M = 1024$ ) ya ek layer $M = 16$ ke saath (same $K = N = 256$ ). $M = 16$ case ke liye utilisation compute karo ( $511$ -cycle fill ke saath) aur ek sentence mein explain karo Roofline model se tie karte hue ki batching kyun help karta hai.

Recall Solution

Actual cycles: $511 + 16 = 527$ . Ideal useful cycles: $16$ . $Util = \frac{16}{527} = 0.03036 = 3.04%.$ Batching kyun help karta hai: tiny $M$ ke saath fixed $511$ -cycle fill $16$ cycles ke real work ko overwhelm kar deta hai, toh array 97% idle rahta hai — $M$ badhana (batching) fill ko amortise karta hai aur tumhe roofline ke compute-bound ridge ki taraf push karta hai, jahan tum actually $131$ TOPS exploit karte ho.

Recall Har numeric answer ki ek-nazar summary

L2.1 $= 1.68 \times 1 0^{7}$ MACs · L2.2 $= 131.07$ TOPS · L2.3 $= 256$ cycles $= 0.256 μ s$ · L3.1 $= 75%$ , $49.15$ TOPS · L3.2 ratio $= 6400 : 1$ · L4.1 $= 16 \times /16 \times /16 \times$ · L4.2 $= 1.56%$ vs $100%$ · L5.1 $= 1535$ cycles, $66.7%$ , $87.4$ TOPS · L5.2 $= 3.04%$ .