Neural processing units (NPUs): Question bank — Hardware deep dive

Before we start, every symbol and unit these questions lean on, defined in plain speech.

Definition The vocabulary you'll need

A PE (processing element) is one tiny compute box in the grid. Inside each PE lives exactly one MAC (multiply-accumulate) — the operation "multiply two numbers, then add the result onto a running total." Picture a box with two inputs and one number it keeps adding onto.
When we multiply two matrices of shape $(M \times K)$ and $(K \times N)$ : == $M$ is the number of rows of the first matrix (e.g. how many input samples in a batch), $N$ is the number of columns of the second matrix (e.g. how many output neurons), and $K$ == is the shared inner dimension being summed over (e.g. how many inputs feed each neuron). One output element is a dot product of length $K$ .
The array of PEs is a rectangle: == $R$ rows of PEs, $C$ columns of PEs, and $f$ == is the clock frequency — how many cycles the chip ticks per second (a "cycle" is one heartbeat where every PE does one MAC). $f = 1 GHz$ means a billion ticks per second.
TOPS = tera-operations per second = trillions ( $1 0^{12}$ ) of operations each second. One MAC counts as two operations (one multiply + one add), so an array doing $R \cdot C$ MACs per cycle at frequency $f$ hits $\frac{2 \cdot R \cdot C \cdot f}{1 0 ^{12}}$ TOPS.

The two diagrams below anchor the trickier questions: how a systolic array fills and drains, and how the Roofline model splits compute-bound from memory-bound.

True or false — justify

A CPU is simply a slow NPU.

False — a CPU is a general machine that spends most of its energy on control (fetch, decode, branch). An NPU hard-wires one dataflow, so it is worse at branchy code but 10–100× better per joule on matrix-multiply. See Domain-specific architectures.

Two chips with equal peak TOPS will run the same network at the same speed.

False — peak TOPS assumes 100% utilisation and unlimited memory bandwidth. The chip whose memory can actually feed its array wins; the other stalls. Judge with the Roofline model, not the spec sheet.

A systolic array works because it runs at a higher clock than a normal ALU.

False — "systolic" is about where data goes, not clock rate. Numbers hop only between neighbouring PEs, so wires stay short and no global memory is touched per step; that gives efficiency, not raw speed. See Systolic arrays.

Switching from fp32 to int8 makes every network unusable.

False — for inference with proper calibration, int8 typically loses under 1% accuracy because nets were trained with noise and their nonlinearities absorb rounding. See Quantization and int8 inference.

Loading each weight from memory once, then reusing it, changes the number of multiplies the matmul needs.

False — the multiply count is fixed at

M \cdot N \cdot K

(rows × output-columns × shared inner dimension). Reuse changes how many expensive memory reads you pay for, not the arithmetic. See Dataflow and data reuse.

An int8 multiplier is about 4× smaller than an fp32 one.

False — multiplier area scales with the square of bit-width

b

(the

b \times b

partial-product grid), so it is about

(32/8)^{2} = 16 \times

smaller, not 4×.

Doubling the clock frequency

f

and doubling the array width give the same throughput gain.

True for the raw throughput number, but the array-width path is usually smarter — it delivers 2× MACs/cycle at similar voltage, while doubling

f

often needs higher voltage

V

and, since power

\propto V^{2} f

, burns disproportionately more energy (see Energy per operation).

If a layer is memory-bandwidth bound, adding more PEs speeds it up.

False — extra compute is useless when data can't arrive fast enough; the array just idles harder. You need more bandwidth or better reuse, which the roofline makes visible.

Spot the error

"Peak TOPS

= R \cdot C \cdot f

Off by 2× — each MAC is one multiply and one add, so operations

= 2 \cdot R \cdot C \cdot f

(then divided by

1 0^{12}

for the "tera").

R \cdot C \cdot f

counts MACs/second, not ops/second.

"A weight-stationary array reloads the weight into each PE every clock."

Contradiction — stationary means the weight is loaded once and held while many activations stream past it. Reloading every clock would destroy the reuse that justifies the design.

"Utilisation = actual MACs ÷ peak TOPS."

Units mismatch — you must divide by

R \cdot C \cdot (cycles run)

, a MAC count, not a rate. Dividing a count by a per-second rate (TOPS) gives seconds, not a fraction.

"The systolic array produces its first result on cycle 1."

Ignores pipeline fill — data must march across the grid before the first full column emerges; those initial fill cycles (and drain cycles at the end) lower utilisation, especially for small matrices.

"Because a MAC costs ~0.2 pJ and a DRAM read ~640 pJ, arithmetic is the expensive part."

Backwards — the DRAM read is ~1000× the MAC in picojoules, so memory movement is the expensive part; the whole NPU design exists to avoid those reads via on-chip reuse.

"Bigger array is always better, so make it

4096 \times 4096

Ignores utilisation — a huge array is mostly idle on typical layer shapes and during fill/drain, wasting silicon. Array size should match the workload's matrix dimensions

M, N, K

Why questions

Why is matrix multiply, not activation functions, the thing NPUs are shaped around?

Because inference is ~90% matmul by runtime — the 80/20 rule. Optimise the dominant operation; the cheap nonlinearities can ride along on small dedicated units.

Why does data reuse matter more than adding multipliers?

Because the bottleneck is energy spent moving data, not doing arithmetic. Loading a number once (640 pJ) and feeding it to many multiplies (0.2 pJ each) amortises the ~1000× fetch cost across all those MACs.

Why can neural networks tolerate int8 when scientific computing cannot?

Nets are trained with noise and pass values through saturating nonlinearities, so small rounding errors get absorbed rather than amplified; scientific codes chain operations that compound tiny errors into large ones.

Why does one "column of results per cycle" only appear after the array fills?

The activations and partial sums must physically propagate to the far edge first. Until the pipeline is full, some PEs have no valid inputs yet, so no complete output can leave.

Why do GPUs and NPUs coexist rather than one replacing the other?

A GPU (GPUs and SIMT) stays flexible across many kernels; an NPU trades flexibility for peak efficiency on one dataflow. Training and irregular work favour the GPU; fixed high-volume inference favours the NPU. Tensor cores are the middle ground inside GPUs.

Why quote effective throughput instead of peak TOPS?

Peak is achievable only with a perfectly shaped, perfectly fed workload. Effective throughput reflects real utilisation — small layers, fill/drain, and memory limits routinely cut it to 20–60% of peak.

Edge cases

A layer with

N = 1

(a single output column, e.g. a batch-size-1 vector) runs on a

256 \times 256

array — what happens to utilisation?

Only one column of PEs does useful work; the other 255 sit idle, so utilisation collapses to roughly

1/256

. Such matrix-vector shapes are memory-bound and a poor fit for wide arrays.

What if a matrix dimension is smaller than the array (say

K = 64

on a

256

-tall array)?

The unused PE rows are wasted every cycle, and the array under-fills; utilisation drops even before considering pipeline fill. Tiling or a smaller/mapped array recovers efficiency.

What is the throughput of a systolic array in the very last cycles as the last activations exit (drain)?

It tapers off — fewer PEs hold valid data as the pipeline empties, mirroring the fill phase. Drain cycles produce partial output and count against overall utilisation.

Take bit-width to the extreme: does 1-bit ("binary") precision give the

(32/1)^{2} = 1024 \times

multiplier savings the square law predicts?

The area law roughly holds, but accuracy usually breaks down — binary nets need special training and still lose accuracy on hard tasks, so the theoretical silicon win is capped by what the model can tolerate.

If two operands are both zero, does the PE still spend energy on the multiply?

Yes, unless the hardware has explicit zero-skipping (sparsity) logic — a naive dense PE clocks the multiply regardless. Exploiting zeros is exactly what sparse-accelerator designs add.

At the degenerate limit of a

1 \times 1

array (one PE), what has the NPU become?

Just a single MAC unit — a sequential multiplier-adder with no parallelism or spatial reuse, essentially the CPU-style one-at-a-time path the NPU was built to escape.

Recall Self-check: name the four trap families this page attacks

Memory-vs-math cost ::: the ~1000× DRAM tax (640 pJ vs 0.2 pJ) that makes reuse the real design goal. Peak-vs-effective ::: peak TOPS is a marketing ceiling; utilisation and roofline give the truth. Precision fear ::: int8 inference is fine; the $b^{2}$ area law drives the silicon win. Shape/degenerate cases ::: tiny $N$ , small $K$ , fill/drain, and $1 \times 1$ arrays kill utilisation.