Worked examples — Neural processing units (NPUs)

This page drills Neural processing units (NPUs) through every case class the topic can throw at you: full arrays, half-filled arrays, tiny degenerate layers, the pipeline fill/drain edge, the memory-bound limit, and precision trade-offs. We build each number from scratch — no formula and no jargon is used before we say what it means, right here on this page.

---

The scenario matrix

Before working anything, here is the map of every case an NPU throughput/efficiency problem can land in. Each worked example below is tagged with the cell(s) it covers.

Cell	Case class	What makes it tricky	Example
A	Full array, perfect fit	The "textbook" peak — baseline	Ex 1
B	Partial columns/rows used	Utilisation drops below 100%	Ex 2
C	Layer bigger than array (tiling)	One pass isn't enough; loop the array	Ex 3
D	Pipeline fill/drain overhead	Small layer → latency dominates throughput	Ex 4
E	Degenerate input ($K=1$, or $1\times 1$)	Dot product of length 1; array mostly idle	Ex 5
F	Memory-bound limit (roofline)	Compute is fast but data can't arrive	Ex 6
G	Precision trade (int8 vs fp32)	Same silicon, different MAC count	Ex 7
H	Real-world word problem	Translate a phone-chip spec into numbers	Ex 8
I	Exam twist (solve for unknown)	Given TOPS, find the missing clock/size	Ex 9

---

The three quantities we reuse everywhere

Every example plugs into the same three ideas. We state each in plain words before using its symbols.

One dataflow we must define before Example 4: weight-stationary

Now the scenarios.

---

Example 1 — Cell A: full array, perfect fit

Recall Forecast first

Guess before reading: is it closer to 1 TOPS, 100 TOPS, or 10000 TOPS? Your guess ::: About 131 TOPS — width, not clock, does the work.

Step 1 — Count the PEs. $R\cdot C=256\times 256=65536$ multiply-accumulators. Why this step? Peak throughput is "everyone works at once," so we first count how many calculators exist.

Step 2 — One MAC = two ops. Each PE does 1 MAC/cycle = 2 ops/cycle. Why this step? TOPS counts operations, and the factor of $2$ comes from mult + add (defined above).

Step 3 — Multiply by clock. $\text{ops/s}=2\cdot 65536\cdot 10^9=1.31\times 10^{14}$ Why this step? The clock $f$ is "how many cycles per second," so ops/s = ops-per-cycle × cycles-per-second.

Step 4 — Convert to TOPS ($1\text{ TOPS}=10^{12}$ ops/s). $\text{TOPS}=\frac{1.31\times 10^{14}}{10^{12}}=131\text{ TOPS}$

Recall Verify

A single-MAC CPU at 3 GHz does $2\cdot 3\times 10^9=6\times 10^9$ ops/s. Ratio $\frac{1.31\times 10^{14}}{6\times 10^9}\approx 2.2\times 10^4$ — the array is ~22000× wider-per-second. Units check: (PEs)(ops/PE/cycle)(cycles/s) = ops/s. ✓

---

Example 2 — Cell B: partial array (utilisation < 100%)

Recall Forecast

If half the PEs are dead weight, effective throughput should be… ::: half of peak.

Step 1 — Peak (all PEs). $\text{Peak}=\frac{2\cdot 128\cdot 128\cdot 2\times 10^9}{10^{12}}=65.5\text{ TOPS}$ Why this step? Peak is fixed by the hardware, not the layer — it's the ceiling.

Step 2 — How many PEs are busy? Only 64 columns are loaded → $128\times 64=8192$ of $16384$ PEs active. Why this step? Utilisation = busy ÷ total; an unloaded column contributes zero MACs no matter the clock.

Step 3 — Utilisation. $U=\frac{8192}{16384}=0.5=50\%$

Step 4 — Effective TOPS. $\text{Effective}=U\times \text{Peak}=0.5\times 65.5=32.8\text{ TOPS}$ Why this step? Real speed = ceiling × fraction of ceiling actually reached.

Figure s01 — half-loaded array. A $16\times 16$ miniature stands in for the $128\times 128$ grid. The orange PEs on the left half hold weights and do real MACs; the dark right half is idle. Read the picture as a fraction: lit area ÷ total area = 8 of 16 columns = $0.5$, which is exactly the utilisation $U$ we compute in Step 3. The figure is the utilisation formula, drawn.

Recall Verify

The figure shows 64 lit columns, 64 dark → lit fraction $=0.5$. Effective = 32.8 TOPS, exactly half of 65.5. ✓

---

Example 3 — Cell C: layer bigger than the array (tiling)

Recall Forecast

The output $Y$ is $512\times 512$. How many $256\times 256$ output blocks fit in it? ::: $2\times 2=4$ tiles.

Step 1 — Total MACs from the formula. Here $M=512,K=256,N=512$: $\text{MACs}=512\cdot 512\cdot 256=6.71\times 10^7$ Why this step? Total arithmetic is independent of hardware — it's a property of the math.

Step 2 — Output tiles. Output is $M\times N=512\times 512$. The array produces a $256\times 256$ output block per residency. Tiles $=\frac{512}{256}\cdot \frac{512}{256}=4$. Why this step? When the matrix exceeds the grid, we loop the grid over sub-blocks (tiling). Each tile is one full array residency.

Step 3 — MACs per tile. Each output block ($256\times 256$) sums over $K=256$: $256\cdot 256\cdot 256=1.68\times 10^7$. Times 4 tiles $=6.71\times 10^7$. ✓ matches Step 1. Why this step? Cross-check that tiling doesn't change the total work — it only splits it in time.

Recall Verify

$4\times 1.6777\times 10^7=6.71\times 10^7$, equal to $512\cdot 512\cdot 256$. Tiling conserves work. ✓

---

Example 4 — Cell D: pipeline fill and drain on a small layer

Recall Forecast

If it takes ~256 cycles to fill and another ~256 to drain, but only 16 rows of real data flow, is the array mostly working or mostly starting/stopping? ::: Mostly overhead — fill + drain dwarf the 16.

Step 1 — Fill latency. Data must traverse the array width: 256 cycles before the first full result emerges. Why this step? From the fill definition: the wavefront needs (array-width) hops to reach the far edge.

Step 2 — Streaming the real data. Once filled, each of the $M=16$ rows costs 1 more cycle: 16 cycles of useful streaming. Why this step? After the pipe is full, throughput is 1 row/cycle — this is the only "at full speed" phase.

Step 3 — Drain latency. After the last row is injected, its partial sums must still walk out the far edge: 256 cycles of drain. Why this step? The drain definition: the tail of the wave must exit before the last answer is complete. Ignoring it undercounts the true latency.

Step 4 — Total cycles and overhead fraction. $\text{total}=\underset{\text{fill}}{\underbrace{256}}+\underset{\text{stream}}{\underbrace{16}}+\underset{\text{drain}}{\underbrace{256}}=528\text{cycles}$ $\text{overhead fraction}=\frac{256+256}{528}=\frac{512}{528}\approx 0.970=97.0\%$ Why this step? This is the honest "small layer" penalty: 97% of cycles are fill+drain, not full-throughput work.

Recall Verify

$512/528=0.9697$. Sanity: a big layer ($M=10000$) gives $\frac{512}{512+10000}\approx 4.9\%$ overhead — negligible. Fill+drain hurt only small layers. ✓

---

Example 5 — Cell E: degenerate input ($K=1$, and $1\times 1$)

Recall Forecast

With $K=1$, each output is just one multiply (no accumulation). Does the array's depth (the accumulation direction) help? ::: No — depth collapses to 1, so that whole dimension is wasted.

Step 1 — Total MACs. $M=256,N=256,K=1$: $\text{MACs}=256\cdot 256\cdot 1=65536$ Why this step? Even a degenerate layer still touches every output once.

Step 2 — Total cycles = fill + compute + drain. There is only $K=1$ row of activation to inject, so useful streaming is 1 cycle. Around it sit the fill and drain of a 256-wide grid: $\text{cycles}=\underset{\text{fill}}{\underbrace{256}}+\underset{\text{compute}}{\underbrace{1}}+\underset{\text{drain}}{\underbrace{256}}=513$ Why this step? Without writing the cycle count explicitly the utilisation number below would appear from nowhere. Here the "1 compute cycle" is the only productive tick; everything else is startup/shutdown of the pipe.

Step 3 — Utilisation over the run. The grid could do $65536$ MACs each cycle; over 513 cycles its capacity is $65536\times 513$. Actual useful MACs $=65536$: $U=\frac{65536}{65536\cdot 513}=\frac{1}{513}\approx 0.195\%$ Why this step? The degenerate case exposes the worst utilisation — fill+drain swamp the single compute cycle.

Recall Verify

$1/513=0.001949=0.195\%$. The truly degenerate $1\times 1$ ($M=N=K=1$) needs 1 MAC on 65536 PEs → over its 513-cycle run, $U=\frac{1}{65536\cdot 513}\approx 3.0\times 10^{-8}$. Tiny inputs waste the array — exactly why NPUs suit large matmuls. ✓

---

Example 6 — Cell F: memory-bound limit (roofline)

Recall Forecast

If only 4 ops happen per byte and 100 billion bytes arrive per second, the math ceiling from bandwidth is $4\times 100\times 10^9=?$ ::: $4\times 10^{11}$ ops/s = 0.4 TOPS — far below 131 TOPS, so memory-bound.

Step 1 — Bandwidth-limited compute. Let ${\text{ops/s}}_{\text{mem}}$ be the most operations per second the bandwidth can sustain — the throughput ceiling set purely by how fast bytes arrive: ${\text{ops/s}}_{\text{mem}}=I\times B=4\times 100\times 10^9\frac{\text{bytes}}{\text{s}}=4\times 10^{11}\text{ops/s}$ Why this step? Throughput can't exceed (bytes/s that arrive) × (ops done per byte). This is the slanted memory roof of the Roofline model. Units: $\frac{\text{ops}}{\text{byte}}\cdot \frac{\text{bytes}}{\text{s}}=\frac{\text{ops}}{\text{s}}$. ✓

Step 2 — Convert that ceiling to the topline metric, TOPS. ${\text{TOPS}}_{\text{mem}}=\frac{{\text{ops/s}}_{\text{mem}}}{10^{12}}=\frac{4\times 10^{11}}{10^{12}}=0.4\text{TOPS}$ Why this step? Every other example reported speed in TOPS; to compare against the $131$ TOPS compute roof we must put the memory ceiling in the same unit.

Step 3 — Classify: compute-bound or memory-bound? Compute ceiling = $131$ TOPS; memory ceiling = $0.4$ TOPS. Since $0.4\ll 131$, the achievable throughput is the lower of the two → memory-bound, capped at $0.4$ TOPS. Why this step? The roofline always picks the lower roof; here bandwidth, not the PEs, is the bottleneck.

Step 4 — Effective utilisation of the array. $U=\frac{0.4}{131}\approx 0.31\%$ Why this step? Shows peak TOPS is a fiction when data is starved — the "peak TOPS tells you the speed" trap made concrete.

Figure s02 — the roofline. The horizontal orange line is the compute ceiling (131 TOPS): no matter how much data you feed, the PEs cannot exceed it. The rising plum dashed line is the memory roof $I\times B$: with little reuse (small $I$, left side) bandwidth caps you far below the PEs' ability. The teal curve is what you actually attain — the lower of the two. Our layer's orange dot sits at $I=4$, pinned on the slope at 0.4 TOPS, deep in the memory-bound region left of the ridge at $I=1310$. To climb toward the flat top you must raise $I$ (reuse data via Dataflow and data reuse), not add more PEs.

Recall Verify

$4\times 100\times 10^9=4\times 10^{11}$ ops/s $=0.4$ TOPS; $0.4/131=0.00305$. Ridge: to become compute-bound you need $I\cdot B\ge 131\text{ TOPS}$, i.e. $I\ge 1310$ ops/byte. Below that, add bandwidth or reuse, not PEs. ✓

---

Example 7 — Cell G: precision trade (int8 vs fp32)

Step 1 — Area ratio. $\frac{{\text{area}}_{\text{fp32}}}{{\text{area}}_{\text{int8}}}={\left(\frac{32}{8}\right)}^2=16$ Why this step? fp32 uses 32 bits, int8 uses 8; apply the square rule just defined.

Step 2 — New PE count. fp32 grid: $64\times 64=4096$ PEs. int8 fits $16\times$: $4096\times 16=65536$ PEs → a $256\times 256$ grid. Why this step? Same area ÷ 16× smaller multiplier = 16× the count; $\sqrt{16}=4$ so each side quadruples ($64\to 256$).

Step 3 — Peak TOPS at 1 GHz. $\text{int8 peak}=\frac{2\cdot 65536\cdot 10^9}{10^{12}}=131\text{TOPS},\text{fp32 peak}=\frac{2\cdot 4096\cdot 10^9}{10^{12}}=8.19\text{TOPS}$ Why this step? Confirms the $16\times$ throughput multiplier for the same silicon and clock.

Recall Verify

$131.07/8.19=16.0$. Accuracy caveat: int8 inference typically loses <1% accuracy with calibration — the trade is worth it (see Quantization and int8 inference and Tensor cores, which mix precisions). ✓

---

Example 8 — Cell H: real-world word problem

Recall Forecast

The chip's MAC budget per second vs the model's demand — which is bigger? ::: The chip has huge headroom; demand needs only a slice.

Step 1 — Chip MAC budget per second. $\text{MAC/s}=R\cdot C\cdot f=128\cdot 128\cdot 0.8\times 10^9=1.31\times 10^{13}\text{MAC/s}$ Why this step? Convert hardware to a "MACs I can do per second" ceiling (note: MACs, not ops — no ×2 here).

Step 2 — Model demand per second. $3\times 10^9\frac{\text{MAC}}{\text{frame}}\times 30\frac{\text{frame}}{\text{s}}=9\times 10^{10}\text{MAC/s}$ Why this step? Frame rate turns per-frame work into a sustained rate we can compare.

Step 3 — Fits? $9\times 10^{10}\le 1.31\times 10^{13}$ → yes, with large margin.

Step 4 — Required utilisation. $U=\frac{9\times 10^{10}}{1.31\times 10^{13}}=0.687\%\approx 0.69\%$ Why this step? Tells us the chip is idle 99%+ of the time → we can lower clock/voltage to save battery (Energy per operation).

Recall Verify

$128^2\cdot 0.8\times 10^9=1.31072\times 10^{13}$; $9\times 10^{10}/1.31072\times 10^{13}=0.00687$. Battery insight: since only 0.69% is needed, drop $f$ ~150× → same 30 fps at far lower power. ✓

---

Example 9 — Cell I: exam twist (solve for the unknown)

Recall Forecast

Rearranging $\text{TOPS}=2R^{2}f/10^{12}$ for $R$ needs a square root. Roughly, is $R$ near 100, 200, or 500? ::: Around 200.

Step 1 — Write the TOPS formula with $R=C$. $\text{TOPS}=\frac{2R^{2}f}{10^{12}}$ Why this step? "Square array" means $R=C$, so PE count is $R^2$; we now have one unknown.

Step 2 — Solve for $R^2$. $R^2=\frac{\text{TOPS}\times 10^{12}}{2f}=\frac{100\times 10^{12}}{2\times 1.25\times 10^9}=\frac{10^{14}}{2.5\times 10^9}=4\times 10^4$ Why this step? Isolate the count first — cleaner than square-rooting mid-expression.

Step 3 — Take the root. $R=\sqrt{4\times 10^4}=200\Rightarrow \text{200}\times \text{200}\text{array}$ Why this step? $R$ must be a side length, so we square-root the PE count.

Recall Verify

Forward-check: $\frac{2\cdot 200^2\cdot 1.25\times 10^9}{10^{12}}=\frac{2\cdot 40000\cdot 1.25\times 10^9}{10^{12}}=\frac{10^{14}}{10^{12}}=100$ TOPS. ✓

---

The matrix, filled

Recall Did every cell get covered?

A→Ex1, B→Ex2, C→Ex3, D→Ex4, E→Ex5, F→Ex6, G→Ex7, H→Ex8, I→Ex9 ::: all nine cells covered, from full array to degenerate $1\times 1$, memory-bound, precision, real-world, and reverse-solve.

---

Quick self-test