Neural processing units (NPUs): Exercises — Hardware deep dive

Before we start, one shared reference card so no symbol appears unexplained:

Level 1 — Recognition

L1.1

Which single operation makes up roughly 90% of neural-network inference runtime, and what atomic hardware operation is it built from?

Recall Solution

What: Neural-net inference is dominated by matrix multiplication. Why: fully-connected and convolution layers both reduce to $Y = X W$ , and each output element is a dot product $y_{ij} = \sum_{k} X_{ik} W_{k j}$ . The atom: each dot product is a chain of multiply-accumulate (MAC) operations — multiply a pair, add to the running sum. So the whole workload is "billions of MACs."

L1.2

Match each term to its one-line meaning: (a) PE, (b) systolic array, (c) weight-stationary, (d) on-chip SRAM.

Recall Solution

(a) PE — Processing Element: one MAC unit computing $a \times b + c$ .
(b) Systolic array — a 2-D grid of PEs where data flows rhythmically between neighbouring PEs (no global memory read per step).
(c) Weight-stationary — weights are loaded once and held inside each PE while activations stream past.
(d) On-chip SRAM — fast scratchpad memory near the compute holding weights/activations so we avoid the expensive trip to DRAM.

Level 2 — Application

L2.1

A matmul multiplies a $(256 \times 512)$ matrix by a $(512 \times 128)$ matrix. How many MACs does it take?

Recall Solution

Formula: $MACs = M \cdot N \cdot K$ where here $M = 256$ , $K = 512$ , $N = 128$ . Why this formula: each of the $M \cdot N$ outputs is a dot product of length $K$ , i.e. $K$ multiplies each. $MACs = 256 \times 128 \times 512 = 16, 777, 216 \approx 1.68 \times 1 0^{7} .$

L2.2

An array is $R = C = 256$ PEs running at $f = 1$ GHz. Give its peak TOPS.

Recall Solution

Formula: $TOPS = \frac{2 R C f}{1 0 ^{12}}$ . The $2$ counts the multiply and the add in each MAC. $TOPS = \frac{2 \times 256 \times 256 \times 1 0 ^{9}}{1 0 ^{12}} = \frac{2 \times 65536 \times 1 0 ^{9}}{1 0 ^{12}} = 131.07 TOPS .$ What it looks like: see the figure below — the win is the area of the grid ( $25 6^{2} = 65536$ MACs every single tick), not the clock speed.

L2.3

The same $256 \times 256$ array runs a matmul with $M = N = K = 256$ perfectly (ignore fill/drain). How many clock cycles, and how long in microseconds at $1$ GHz?

Recall Solution

Total MACs: $25 6^{3} = 16, 777, 216$ . MACs per cycle: the full grid does $R \cdot C = 65536$ per cycle. $cycles = \frac{25 6 ^{3}}{65536} = \frac{16 , 777 , 216}{65536} = 256 cycles .$ Time: $256$ cycles $\times 1 ns/cycle = 256 ns = 0.256 μ s$ . Why it's exactly $K$ : with a weight-stationary $K \times N = 256 \times 256$ grid holding the whole weight matrix, you stream all $M = 256$ activation rows through — one row per cycle after fill — so $M = 256$ result rows in $256$ cycles.

Level 3 — Analysis

L3.1

A layer maps onto a $128 \times 128$ array but only uses $96$ of the $128$ columns and $128$ of the $128$ rows. Ignoring fill/drain, what is the utilisation and the effective TOPS if peak (at $2$ GHz) is $65.5$ TOPS?

Recall Solution

Peak check: $TOPS = \frac{2 \times 12 8 ^{2} \times 2 \times 1 0 ^{9}}{1 0 ^{12}} = \frac{2 \times 16384 \times 2 \times 1 0 ^{9}}{1 0 ^{12}} = 65.54$ TOPS. ✓ Busy PEs: $128 \times 96 = 12288$ out of $128 \times 128 = 16384$ . $Utilisation = \frac{12288}{16384} = 0.75 = 75%.$ Effective TOPS: $0.75 \times 65.54 = 49.15$ TOPS. Why: the idle $32$ columns are silicon doing nothing — the peak number assumes every PE fires every cycle, which a mismatched layer shape breaks.

L3.2

The naïve dataflow loads both operands from DRAM for every MAC. A DRAM read $\approx 640$ pJ and a MAC $\approx 0.2$ pJ. For $16.78 \times 1 0^{6}$ MACs (from L2.1), compute total energy (a) naïve — 2 reads per MAC — vs (b) an ideal reuse case where each of the two operands is read once and reused for its whole traversal. Assume the reuse case does $2$ DRAM reads per PE-load but we count only the arithmetic dominated by MACs. Give the ratio (naïve DRAM energy) : (MAC arithmetic energy).

Recall Solution

Naïve memory energy: $2$ reads/MAC $\times 640$ pJ $\times 16.78 \times 1 0^{6} = 2 \times 640 \times 16.78 \times 1 0^{6}$ pJ. $= 2 \times 640 \times 16.777216 \times 1 0^{6} = 2.147 \times 1 0^{10} pJ = 21.47 mJ (memory only) .$ Arithmetic energy: $0.2 pJ \times 16.78 \times 1 0^{6} = 3.36 \times 1 0^{6}$ pJ $= 3.36 μ J$ . Ratio: $\frac{2.147 \times 1 0 ^{10}}{3.36 \times 1 0 ^{6}} = 6400$ . What it means: naïvely, memory movement burns 6400× the arithmetic energy — this is the number that justifies the whole systolic reuse machine. The $2 \times 640/0.2 = 6400$ factor is just "two DRAM reads costing $640$ pJ each vs one $0.2$ pJ MAC."

Level 4 — Synthesis

L4.1

You are given a silicon budget and told: an fp32 multiplier occupies area $A$ . Multiplier area scales roughly as (bit-width)². (a) How much smaller is an int8 multiplier? (b) If you swap all fp32 MACs for int8, how many times more MACs fit in the same area? (c) What is the peak TOPS gain (same clock)?

Recall Solution

(a) Area ratio: $Energy/Area_{mult} \propto (bits)^{2}$ , so $\frac{A _{fp32}}{A _{int8}} = (\frac{32}{8})^{2} = 4^{2} = 16.$ An int8 multiplier is 16× smaller. (b) Same area ⇒ 16× more MAC units. (c) Peak TOPS $= 2 R C f$ ; with $16 \times$ more PEs (larger effective $R \cdot C$ ) and unchanged $f$ , peak scales 16× too. Why NN accept it: networks are robust to quantisation noise (Quantization and int8 inference) — trained with noise and using tolerant nonlinearities — so inference at int8 typically loses <1% accuracy while giving this 16× density.

L4.2

Design choice: you must run a stream of small $32 \times 32 \times 32$ matmuls back-to-back. Would you choose a $256 \times 256$ array or a $32 \times 32$ array for utilisation, and why? (Ignore fill/drain differences for the estimate; a $K \times N = 32 \times 32$ layer maps onto a $32 \times 32$ region.)

Recall Solution

The mapping: a $K \times N = 32 \times 32$ weight matrix occupies only a $32 \times 32$ block of PEs. On the $256 \times 256$ array: busy PEs $= 32 \times 32 = 1024$ out of $256 \times 256 = 65536$ ⇒ $Utilisation = \frac{1024}{65536} = 0.0156 = 1.56%.$ On the $32 \times 32$ array: busy PEs $= 1024$ out of $1024$ ⇒ 100% utilisation. Conclusion: the small array is far better utilised for small layers. The giant array's peak TOPS is wasted marketing here — most PEs sit idle. This is exactly why real designs tile/batch small layers or pick array size to match typical layer shapes (Domain-specific architectures, Dataflow and data reuse).

Level 5 — Mastery

L5.1

A $256 \times 256$ weight-stationary array at $f = 1$ GHz runs a matmul with $M = 1024$ , $K = 256$ , $N = 256$ . The pipeline fill latency is $R + C - 1$ cycles before the first full column emerges, then one result column per cycle for $M$ columns. (a) Total cycles including fill. (b) Ideal cycles ignoring fill. (c) Utilisation as (ideal / actual). (d) Effective TOPS.

Recall Solution

The weight matrix is exactly $K \times N = 256 \times 256$ , so it tiles the whole $256 \times 256$ grid — every PE holds one weight, all busy during steady state. (a) Fill: $R + C - 1 = 256 + 256 - 1 = 511$ cycles. Steady state: stream $M = 1024$ activation rows ⇒ $1024$ cycles. Total: $cycles_{actual} = 511 + 1024 = 1535.$ (b) Ideal (no fill): the useful work is $M = 1024$ result columns at one per cycle $= 1024$ cycles. (Equivalently $MACs / (R C) = 1024 \cdot 256 \cdot 256/65536 = 1024$ .) ✓ (c) Utilisation: $Util = \frac{1024}{1535} = 0.6671 = 66.7%.$ (d) Effective TOPS: peak $= 2 \times 25 6^{2} \times 1 0^{9} /1 0^{12} = 131.07$ TOPS. $Effective = 0.6671 \times 131.07 = 87.4 TOPS .$ What it looks like: the fill/drain is the ramp on the figure below — the array "warms up" for 511 cycles doing partial work before hitting full rhythm. With a large $M$ this overhead shrinks; with tiny $M$ it dominates.

L5.2

Same array and clock. You now have a choice: run the layer above (large $M = 1024$ ) or a layer with $M = 16$ (same $K = N = 256$ ). Compute utilisation for the $M = 16$ case (with the $511$ -cycle fill) and explain, in one sentence tying to the Roofline model, why batching helps.

Recall Solution

Actual cycles: $511 + 16 = 527$ . Ideal useful cycles: $16$ . $Util = \frac{16}{527} = 0.03036 = 3.04%.$ Why batching helps: with tiny $M$ the fixed $511$ -cycle fill overwhelms the $16$ cycles of real work, so the array is 97% idle — increasing $M$ (batching) amortises the fill and pushes you toward the compute-bound ridge of the roofline, where you actually exploit the $131$ TOPS.

Recall One-look summary of every numeric answer

L2.1 $= 1.68 \times 1 0^{7}$ MACs · L2.2 $= 131.07$ TOPS · L2.3 $= 256$ cycles $= 0.256 μ s$ · L3.1 $= 75%$ , $49.15$ TOPS · L3.2 ratio $= 6400 : 1$ · L4.1 $= 16 \times /16 \times /16 \times$ · L4.2 $= 1.56%$ vs $100%$ · L5.1 $= 1535$ cycles, $66.7%$ , $87.4$ TOPS · L5.2 $= 3.04%$ .