Worked examples — Neural processing units (NPUs)

6.5.8 · D3 · Hardware › Advanced & Emerging Architectures › Neural processing units (NPUs)

Yeh page Neural processing units (NPUs) ko har possible case class ke through drill karti hai jo is topic mein aa sakti hai: full arrays, half-filled arrays, chote degenerate layers, pipeline fill/drain edge, memory-bound limit, aur precision trade-offs. Hum har number scratch se banate hain — koi bhi formula ya jargon tab tak use nahi hota jab tak hum uska matlab yahan, isi page par, explain na kar den.

---

The scenario matrix

Kuch bhi work karne se pehle, yeh map hai har case ka jo ek NPU throughput/efficiency problem mein aa sakti hai. Neeche har worked example ko us cell ke saath tag kiya gaya hai jo woh cover karta hai.

Cell	Case class	Kya tricky banata hai	Example
A	Full array, perfect fit	"Textbook" peak — baseline	Ex 1
B	Partial columns/rows used	Utilisation 100% se neeche girti hai	Ex 2
C	Layer bigger than array (tiling)	Ek pass kaafi nahi; array ko loop karo	Ex 3
D	Pipeline fill/drain overhead	Small layer → latency throughput pe dominate karti hai	Ex 4
E	Degenerate input ($K=1$, ya $1\times 1$)	Length 1 ka dot product; array mostly idle	Ex 5
F	Memory-bound limit (roofline)	Compute fast hai par data aa nahi sakta	Ex 6
G	Precision trade (int8 vs fp32)	Same silicon, alag MAC count	Ex 7
H	Real-world word problem	Phone-chip spec ko numbers mein translate karo	Ex 8
I	Exam twist (solve for unknown)	TOPS diya, missing clock/size nikalo	Ex 9

---

Teen quantities jo hum har jagah reuse karte hain

Har example inhi teen ideas mein plug hota hai. Hum har ek ko plain words mein state karte hain pehle, symbols use karne se pehle.

Ek dataflow jo hum Example 4 se pehle define karna chahte hain: weight-stationary

Ab scenarios.

---

Example 1 — Cell A: full array, perfect fit

Recall Forecast first

Padhne se pehle guess karo: kya yeh 1 TOPS, 100 TOPS, ya 10000 TOPS ke kareeb hai? Tumhara guess ::: Lagbhag 131 TOPS — width kaam karta hai, clock nahi.

Step 1 — PEs count karo. $R\cdot C=256\times 256=65536$ multiply-accumulators. Yeh step kyun? Peak throughput matlab "sab ek saath kaam karte hain," toh pehle hum count karte hain ki kitne calculators exist karte hain.

Step 2 — Ek MAC = do ops. Har PE 1 MAC/cycle = 2 ops/cycle karta hai. Yeh step kyun? TOPS operations count karta hai, aur $2$ ka factor mult + add se aata hai (upar define kiya).

Step 3 — Clock se multiply karo. $\text{ops/s}=2\cdot 65536\cdot 10^9=1.31\times 10^{14}$ Yeh step kyun? Clock $f$ matlab "cycles per second," toh ops/s = ops-per-cycle × cycles-per-second.

Step 4 — TOPS mein convert karo ($1\text{ TOPS}=10^{12}$ ops/s). $\text{TOPS}=\frac{1.31\times 10^{14}}{10^{12}}=131\text{ TOPS}$

Recall Verify karo

Ek single-MAC CPU at 3 GHz $2\cdot 3\times 10^9=6\times 10^9$ ops/s karta hai. Ratio $\frac{1.31\times 10^{14}}{6\times 10^9}\approx 2.2\times 10^4$ — array ~22000× wider-per-second hai. Units check: (PEs)(ops/PE/cycle)(cycles/s) = ops/s. ✓

---

Example 2 — Cell B: partial array (utilisation < 100%)

Recall Forecast

Agar aadhe PEs dead weight hain, effective throughput hona chahiye… ::: peak ka aadha.

Step 1 — Peak (sab PEs). $\text{Peak}=\frac{2\cdot 128\cdot 128\cdot 2\times 10^9}{10^{12}}=65.5\text{ TOPS}$ Yeh step kyun? Peak hardware se fixed hai, layer se nahi — yeh ceiling hai.

Step 2 — Kitne PEs busy hain? Sirf 64 columns loaded hain → $128\times 64=8192$ of $16384$ PEs active. Yeh step kyun? Utilisation = busy ÷ total; ek unloaded column clock chahe kuch bhi ho zero MACs contribute karta hai.

Step 3 — Utilisation. $U=\frac{8192}{16384}=0.5=50\%$

Step 4 — Effective TOPS. $\text{Effective}=U\times \text{Peak}=0.5\times 65.5=32.8\text{ TOPS}$ Yeh step kyun? Real speed = ceiling × ceiling ka woh fraction jo actually reach hoti hai.

Figure s01 — half-loaded array. $128\times 128$ grid ki jagah ek $16\times 16$ miniature use ki gayi hai. Orange PEs baayi taraf weights hold karti hain aur real MACs karti hain; dark daahini taraf idle hai. Picture ko ek fraction ki tarah padho: lit area ÷ total area = 16 mein se 8 columns = $0.5$, jo exactly woh utilisation $U$ hai jo hum Step 3 mein compute karte hain. Figure utilisation formula ka drawn version hai.

Recall Verify karo

Figure mein 64 lit columns, 64 dark dikhte hain → lit fraction $=0.5$. Effective = 32.8 TOPS, exactly 65.5 ka aadha. ✓

---

Example 3 — Cell C: layer bigger than the array (tiling)

Recall Forecast

Output $Y$ $512\times 512$ hai. Isme kitne $256\times 256$ output blocks fit hote hain? ::: $2\times 2=4$ tiles.

Step 1 — Formula se total MACs. Yahan $M=512,K=256,N=512$: $\text{MACs}=512\cdot 512\cdot 256=6.71\times 10^7$ Yeh step kyun? Total arithmetic hardware se independent hai — yeh math ki property hai.

Step 2 — Output tiles. Output $M\times N=512\times 512$ hai. Array ek residency mein $256\times 256$ output block produce karta hai. Tiles $=\frac{512}{256}\cdot \frac{512}{256}=4$. Yeh step kyun? Jab matrix grid se bada hota hai, hum grid ko sub-blocks par loop karte hain (tiling). Har tile ek full array residency hai.

Step 3 — MACs per tile. Har output block ($256\times 256$) $K=256$ par sum karta hai: $256\cdot 256\cdot 256=1.68\times 10^7$. Times 4 tiles $=6.71\times 10^7$. ✓ Step 1 se match karta hai. Yeh step kyun? Cross-check karo ki tiling total work nahi badlata — yeh sirf time mein split karta hai.

Recall Verify karo

$4\times 1.6777\times 10^7=6.71\times 10^7$, $512\cdot 512\cdot 256$ ke barabar. Tiling work conserve karta hai. ✓

---

Example 4 — Cell D: pipeline fill and drain on a small layer

Recall Forecast

Agar fill mein ~256 cycles lagte hain aur drain mein ~256 aur, lekin sirf 16 rows real data flow karta hai, toh kya array mostly kaam kar raha hai ya mostly start/stop ho raha hai? ::: Zyaadatar overhead — fill + drain 16 ko dwarf karte hain.

Step 1 — Fill latency. Data ko array width traverse karna hai: 256 cycles pehle full result aane se pehle. Yeh step kyun? Fill definition se: wavefront ko (array-width) hops chahiye far edge tak pahunchne ke liye.

Step 2 — Real data stream karna. Ek baar fill hone par, $M=16$ rows mein se har ek ka 1 aur cycle cost hoti hai: 16 cycles useful streaming ke. Yeh step kyun? Pipe full hone ke baad, throughput 1 row/cycle hai — yeh sirf "full speed" phase hai.

Step 3 — Drain latency. Last row inject hone ke baad, uske partial sums ko ab bhi far edge se bahar walk karna hai: 256 cycles drain. Yeh step kyun? Drain definition: wave ki tail ko exit karna hota hai pehle ki last answer complete ho. Isse ignore karna true latency ko undercount karta hai.

Step 4 — Total cycles aur overhead fraction. $\text{total}=\underset{\text{fill}}{\underbrace{256}}+\underset{\text{stream}}{\underbrace{16}}+\underset{\text{drain}}{\underbrace{256}}=528\text{cycles}$ $\text{overhead fraction}=\frac{256+256}{528}=\frac{512}{528}\approx 0.970=97.0\%$ Yeh step kyun? Yeh honest "small layer" penalty hai: 97% cycles fill+drain hain, full-throughput work nahi.

Recall Verify karo

$512/528=0.9697$. Sanity: ek bada layer ($M=10000$) $\frac{512}{512+10000}\approx 4.9\%$ overhead deta hai — negligible. Fill+drain sirf small layers ko hurt karta hai. ✓

---

Example 5 — Cell E: degenerate input ($K=1$, aur $1\times 1$)

Recall Forecast

$K=1$ ke saath, har output sirf ek multiply hai (koi accumulation nahi). Kya array ki depth (accumulation direction) madad karti hai? ::: Nahi — depth collapse ho ke 1 ho jaati hai, toh woh poora dimension waste hota hai.

Step 1 — Total MACs. $M=256,N=256,K=1$: $\text{MACs}=256\cdot 256\cdot 1=65536$ Yeh step kyun? Ek degenerate layer bhi har output ko ek baar touch karti hai.

Step 2 — Total cycles = fill + compute + drain. Inject karne ke liye sirf $K=1$ row of activation hai, toh useful streaming 1 cycle hai. Uske around 256-wide grid ka fill aur drain baitha hai: $\text{cycles}=\underset{\text{fill}}{\underbrace{256}}+\underset{\text{compute}}{\underbrace{1}}+\underset{\text{drain}}{\underbrace{256}}=513$ Yeh step kyun? Cycle count explicitly likhe bina neeche utilisation number kahin se aata nahi dikhega. Yahan "1 compute cycle" sirf productive tick hai; baaki sab pipe ka startup/shutdown hai.

Step 3 — Run ke dauran Utilisation. Grid could do $65536$ MACs har cycle mein; 513 cycles mein uski capacity $65536\times 513$ hai. Actual useful MACs $=65536$: $U=\frac{65536}{65536\cdot 513}=\frac{1}{513}\approx 0.195\%$ Yeh step kyun? Degenerate case worst utilisation expose karta hai — fill+drain single compute cycle ko swamp kar dete hain.

Recall Verify karo

$1/513=0.001949=0.195\%$. Truly degenerate $1\times 1$ ($M=N=K=1$) ko 65536 PEs par 1 MAC chahiye → apne 513-cycle run mein, $U=\frac{1}{65536\cdot 513}\approx 3.0\times 10^{-8}$. Tiny inputs array waste karte hain — exactly yahi wajah hai ki NPUs bade matmuls ke liye suit karte hain. ✓

---

Example 6 — Cell F: memory-bound limit (roofline)

Recall Forecast

Agar har byte par sirf 4 ops hote hain aur 100 billion bytes per second aate hain, toh bandwidth se math ceiling $4\times 100\times 10^9=?$ hai ::: $4\times 10^{11}$ ops/s = 0.4 TOPS — 131 TOPS se bahut neeche, toh memory-bound.

Step 1 — Bandwidth-limited compute. ${\text{ops/s}}_{\text{mem}}$ woh maximum operations per second hai jo bandwidth sustain kar sakti hai — woh throughput ceiling jo sirf is baat se set hoti hai ki bytes kitni fast arrive karti hain: ${\text{ops/s}}_{\text{mem}}=I\times B=4\times 100\times 10^9\frac{\text{bytes}}{\text{s}}=4\times 10^{11}\text{ops/s}$ Yeh step kyun? Throughput (bytes/s jo aate hain) × (ops done per byte) se zyaada nahi ho sakta. Yeh Roofline model ki slanted memory roof hai. Units: $\frac{\text{ops}}{\text{byte}}\cdot \frac{\text{bytes}}{\text{s}}=\frac{\text{ops}}{\text{s}}$. ✓

Step 2 — Woh ceiling topline metric, TOPS, mein convert karo. ${\text{TOPS}}_{\text{mem}}=\frac{{\text{ops/s}}_{\text{mem}}}{10^{12}}=\frac{4\times 10^{11}}{10^{12}}=0.4\text{TOPS}$ Yeh step kyun? Har doosra example speed TOPS mein report karta tha; $131$ TOPS compute roof se compare karne ke liye hum memory ceiling ko same unit mein daalna chahte hain.

Step 3 — Classify: compute-bound ya memory-bound? Compute ceiling = $131$ TOPS; memory ceiling = $0.4$ TOPS. Kyunki $0.4\ll 131$, achievable throughput kam wala hai → memory-bound, $0.4$ TOPS par capped. Yeh step kyun? Roofline hamesha lower roof pick karta hai; yahan PEs nahi, bandwidth bottleneck hai.

Step 4 — Array ka effective utilisation. $U=\frac{0.4}{131}\approx 0.31\%$ Yeh step kyun? Yeh dikhata hai ki peak TOPS ek fiction hai jab data starved ho — "peak TOPS speed batata hai" ka trap concretely demonstrate hota hai.

Figure s02 — the roofline. Horizontal orange line compute ceiling hai (131 TOPS): chahe kitna bhi data feed karo, PEs isse exceed nahi kar sakte. Rising plum dashed line memory roof $I\times B$ hai: thodi reuse ke saath (small $I$, left side) bandwidth tumhe PEs ki ability se bahut neeche cap karti hai. Teal curve woh hai jo actually milta hai — dono mein se lower. Hamare layer ka orange dot $I=4$ par baitha hai, slope par 0.4 TOPS par pinned, ridge ke left mein memory-bound region mein $I=1310$ par. Flat top ki taraf chadhne ke liye $I$ badhana hoga (data Dataflow and data reuse ke through reuse karo), zyaada PEs nahi.

Recall Verify karo

$4\times 100\times 10^9=4\times 10^{11}$ ops/s $=0.4$ TOPS; $0.4/131=0.00305$. Ridge: compute-bound hone ke liye $I\cdot B\ge 131\text{ TOPS}$ chahiye, matlab $I\ge 1310$ ops/byte. Usse neeche, bandwidth ya reuse badhao, PEs nahi. ✓

---

Example 7 — Cell G: precision trade (int8 vs fp32)

Step 1 — Area ratio. $\frac{{\text{area}}_{\text{fp32}}}{{\text{area}}_{\text{int8}}}={\left(\frac{32}{8}\right)}^2=16$ Yeh step kyun? fp32 32 bits use karta hai, int8 8; abhi define kiya gaya square rule apply karo.

Step 2 — Naya PE count. fp32 grid: $64\times 64=4096$ PEs. int8 $16\times$ fit karta hai: $4096\times 16=65536$ PEs → ek $256\times 256$ grid. Yeh step kyun? Same area ÷ 16× smaller multiplier = 16× count; $\sqrt{16}=4$ toh har side quadruple hoti hai ($64\to 256$).

Step 3 — Peak TOPS at 1 GHz. $\text{int8 peak}=\frac{2\cdot 65536\cdot 10^9}{10^{12}}=131\text{TOPS},\text{fp32 peak}=\frac{2\cdot 4096\cdot 10^9}{10^{12}}=8.19\text{TOPS}$ Yeh step kyun? Same silicon aur clock ke liye $16\times$ throughput multiplier confirm karta hai.

Recall Verify karo

$131.07/8.19=16.0$. Accuracy caveat: int8 inference mein typically calibration ke saath <1% accuracy loss hoti hai — trade iske laayak hai (dekho Quantization and int8 inference aur Tensor cores, jo precisions mix karte hain). ✓

---

Example 8 — Cell H: real-world word problem

Recall Forecast

Chip ka MAC budget per second vs model ki demand — kaunsa bada hai? ::: Chip mein huge headroom hai; demand sirf ek slice chahti hai.

Step 1 — Chip MAC budget per second. $\text{MAC/s}=R\cdot C\cdot f=128\cdot 128\cdot 0.8\times 10^9=1.31\times 10^{13}\text{MAC/s}$ Yeh step kyun? Hardware ko "MACs per second" ceiling mein convert karo (note: MACs, ops nahi — yahan ×2 nahi).

Step 2 — Model demand per second. $3\times 10^9\frac{\text{MAC}}{\text{frame}}\times 30\frac{\text{frame}}{\text{s}}=9\times 10^{10}\text{MAC/s}$ Yeh step kyun? Frame rate per-frame work ko ek sustained rate mein badal deta hai jo hum compare kar sakte hain.

Step 3 — Fit hoga? $9\times 10^{10}\le 1.31\times 10^{13}$ → haan, bahut bade margin ke saath.

Step 4 — Required utilisation. $U=\frac{9\times 10^{10}}{1.31\times 10^{13}}=0.687\%\approx 0.69\%$ Yeh step kyun? Batata hai ki chip 99%+ time idle hai → hum clock/voltage gira sakte hain battery bachane ke liye (Energy per operation).

Recall Verify karo

$128^2\cdot 0.8\times 10^9=1.31072\times 10^{13}$; $9\times 10^{10}/1.31072\times 10^{13}=0.00687$. Battery insight: kyunki sirf 0.69% chahiye, $f$ ~150× giraa do → same 30 fps bahut kam power par. ✓

---

Example 9 — Cell I: exam twist (solve for the unknown)

Recall Forecast

$\text{TOPS}=2R^{2}f/10^{12}$ ko $R$ ke liye rearrange karne mein square root chahiye. Roughly, kya $R$ 100, 200, ya 500 ke kareeb hai? ::: Lagbhag 200.

Step 1 — $R=C$ ke saath TOPS formula likho. $\text{TOPS}=\frac{2R^{2}f}{10^{12}}$ Yeh step kyun? "Square array" matlab $R=C$, toh PE count $R^2$ hai; ab ek unknown hai.

Step 2 — $R^2$ solve karo. $R^2=\frac{\text{TOPS}\times 10^{12}}{2f}=\frac{100\times 10^{12}}{2\times 1.25\times 10^9}=\frac{10^{14}}{2.5\times 10^9}=4\times 10^4$ Yeh step kyun? Pehle count isolate karo — mid-expression square root lene se zyaada clean hai.

Step 3 — Root lo. $R=\sqrt{4\times 10^4}=200\Rightarrow \text{200}\times \text{200}\text{array}$ Yeh step kyun? $R$ ek side length hona chahiye, toh PE count ka square root lete hain.

Recall Verify karo

Forward-check: $\frac{2\cdot 200^2\cdot 1.25\times 10^9}{10^{12}}=\frac{2\cdot 40000\cdot 1.25\times 10^9}{10^{12}}=\frac{10^{14}}{10^{12}}=100$ TOPS. ✓

---

Matrix, filled

Recall Kya har cell cover hua?

A→Ex1, B→Ex2, C→Ex3, D→Ex4, E→Ex5, F→Ex6, G→Ex7, H→Ex8, I→Ex9 ::: sab nau cells cover hue, full array se degenerate $1\times 1$, memory-bound, precision, real-world, aur reverse-solve tak.

---

Quick self-test