Neural processing units (NPUs): Visual walkthrough

Step 1 — Two lists of numbers, one answer

WHAT. Forget matrices. Start with the smallest possible neural-network idea: one output neuron. It has a list of inputs (call them $x_{1}, x_{2}, x_{3}, \dots$ ) and a list of weights (call them $w_{1}, w_{2}, w_{3}, \dots$ ). The neuron's job is to produce one number from these two lists.

WHY this operation. A neuron asks "how much does each input matter?" — it scales each input by its weight and adds up the votes. That "scale-and-add-up" is the single most repeated act in all of neural computing, so it is the thing we must make cheap.

PICTURE. Two rows of coloured boxes are lined up. Each purple input box pairs with the coral weight box directly below it. We multiply each pair, then drop every product into one bucket and sum.

$y = x_{1} w_{1} + x_{2} w_{2} + x_{3} w_{3} + \dots + x_{K} w_{K}$

$x_{k}$ ::: the $k$ -th input number (purple). $k$ is just a counter, $1$ to $K$ .
$w_{k}$ ::: the weight paired with $x_{k}$ (coral) — its "importance dial."
$K$ ::: how many pairs there are (the length of both lists).
$y$ ::: the single output number, the sum of all the products.

Step 2 — The atomic move: multiply, then add to a running total

WHAT. Look at how you would actually compute that sum by hand. You keep a running total in your head. Start at $0$ . Multiply the first pair, add it in. Multiply the next pair, add it in. Repeat.

WHY isolate this. Because the hardware can only build one small thing well. If every dot product is nothing but the same tiny act repeated — "multiply two numbers, add the result to what I already have" — then we only need to build that one gadget, and build thousands of copies.

PICTURE. A single box labelled MAC (Multiply-ACcumulate). Two inputs enter ( $a$ and $b$ ), a running total $c$ arrives from before, and out comes $c + a \times b$ . The old total is replaced by the new one.

$acc_{new} = running total so far acc_{old} + one input a \times its weight b$

Step 3 — Many neurons at once = a matrix multiply

WHAT. One neuron gives one output. A layer has many neurons, and we usually feed many inputs at once (a batch). Stack the input rows into a grid $X$ , stack the weight columns into a grid $W$ . Each output $y_{ij}$ is the dot product of row $i$ of $X$ with column $j$ of $W$ .

WHY. Real networks never do one dot product — they do a whole rectangle of them. Naming the rectangle a matrix multiply lets us count the total work in one clean formula (next step) instead of tracking each neuron.

PICTURE. A grid $X$ on the left, a grid $W$ on top, and the answer grid $Y$ at the bottom-right. To fill cell $(i, j)$ of $Y$ , slide across row $i$ of $X$ and down column $j$ of $W$ , MAC-ing pairs.

$y_{ij} = \sum_{k = 1}^{K} X_{ik} W_{k j}$

$X_{ik}$ ::: input in row $i$ , position $k$ (which pair we're on).
$W_{k j}$ ::: weight in column $j$ , position $k$ .
$i$ ::: which output row ( $1 \dots M$ ) — one per batch item.
$j$ ::: which output column ( $1 \dots N$ ) — one per neuron.
The inner $\sum_{k}$ is exactly the dot product from Step 1.

This is the same $Y = X W$ the parent note gave — now every letter is earned.

Step 4 — Count the work, and discover the real enemy

WHAT. Each of the $M \times N$ output cells costs $K$ MACs (one per pair). So the total is a simple product.

WHY. We need to know how bad this gets, and where the pain lives — in the arithmetic, or in fetching numbers from memory?

$total MACs = rows M \cdot cols N \cdot pairs each K$

PICTURE. Two side-by-side bars. The tall coral bar is the energy to fetch a number from DRAM ( ~~$640$ pJ). The tiny mint bar is the energy of the multiply itself (~~ $0.2$ pJ). The fetch dwarfs the math by ~ $1000 \times$ .

Step 5 — Reuse: each number should feed many multiplies

WHAT. Notice from Step 3: a single weight $W_{k j}$ appears in every output row $i$ — it gets used $M$ times. A single input $X_{ik}$ appears in every output column $j$ — used $N$ times. Each number is reusable many times.

WHY. If a number is fetched once but multiplied $M$ times, the crushing memory cost is divided by $M$ . That is how we defeat the coral bar of Step 4 — not by making it cheaper, but by paying it far less often.

PICTURE. One weight box sits still. An arrow shows the same weight being reused across a whole row of multiplies, while fresh inputs stream past it. "Load once, multiply many."

Step 6 — Wire the MACs into a grid: the systolic array

WHAT. Take the MAC gadget from Step 2 and tile the plane with it: rows and columns of MACs, each wired only to its neighbours (right and down — no long wires, no global memory). Load each weight into its own cell and hold it there (weight-stationary). Now stream inputs in from the left, one column of data per clock tick.

WHY a grid, why neighbours only. A grid gives us $R \times C$ multiplies happening at the same instant (parallelism, Step 4's cure). Neighbour-only wiring means data hops one short step per tick instead of travelling to far memory — cheap and scalable. This is the systolic array the parent note named.

PICTURE. A grid of cells. Weights (coral) sit inside each cell, fixed. Inputs (purple) march left→right. Partial sums (mint) fall top→bottom, each cell adding its own product to the falling total. Watch one input diagonally sweep the whole grid.

$psum_{out} = sum from cell above psum_{in} + input flowing in from left X_{in} \times weight held in this cell W_{stored}$

Because an input, once injected, marches across an entire row of cells, that single fetch feeds a whole row of multiplies — reuse (Step 5) made physical. Related ideas: Tensor cores, GPUs and SIMT.

Step 7 — Fill, drain, and the steady-state win

WHAT. The grid isn't instant. Data enters staggered (a diagonal wavefront), so early cells work while later cells still wait empty — the pipeline fill. Once full, the array is in steady state: every clock it finishes a full column of results. At the end, the last data drains out through empty-ing cells.

WHY show this. Because the honest speed depends on the ratio of busy cycles to filling/draining cycles. For a big matrix the fill is negligible; for a tiny matrix the fill is most of the run and the array is mostly idle. This is the difference between marketing TOPS and real speed.

PICTURE. Three snapshots side by side: fill (a diagonal of live cells, corners idle), steady state (whole grid live), drain (live diagonal exiting, entry corner idle again). The busy fraction is the shaded area.

$Peak throughput = MACs per tick R \cdot C \cdot ticks per second f_{clock} [MACs/s]$

$TOPS = \frac{2 R C f _{clock}}{1 0 ^{12}}, Utilisation = \frac{actual MACs}{R \cdot C \cdot ( cycles run )}$

$R, C$ ::: rows and columns of MAC cells.
$f_{clock}$ ::: cycles per second (Hz).
the $2$ ::: each MAC = one multiply plus one add = two operations.
Utilisation ::: the fraction of cells actually doing useful work — never assume $1$ .

Step 8 — The degenerate cases you must not miss

WHAT & WHY. A formula is only trusted once its edges are checked. Walk every corner:

Case	What happens	Why
$K = 1$	One pair per output; the accumulator adds a single product — no chaining.	Dot product of length one is just a multiply.
Layer smaller than array ( $N < C$ )	Only $N$ of $C$ columns used → utilisation $\leq N / C$ .	Idle columns still burn area but do no work.
Tiny $M$ , huge grid	Fill/drain dominates; steady state barely reached.	Not enough rows to amortise pipeline latency.
Memory-bound layer	PEs starve waiting for data; effective TOPS $≪$ peak.	Bandwidth, not the grid, is the limit (Roofline model).
Zero inputs / zero weights	MAC still runs ( $0 \times b = 0$ ); result correct, energy wasted.	Hardware doesn't skip zeros unless specially built.
int8 vs fp32 precision	int8 multiplier ≈ $(32/8)^{2} = 16 \times$ smaller.	Multiplier area $\propto$ (bit-width) $^{2}$ → 16× more MACs per area.

PICTURE. A small array with only its left two columns lit (a narrow layer). Dark idle columns on the right show wasted silicon — utilisation $= 2/ C$ .

The one-picture summary

The whole derivation on one canvas: two lists → a dot product → one MAC gadget → a matrix of dot products → the memory tax → reuse → a marching systolic grid → fill/steady/drain → peak & utilisation.

Recall Feynman retelling — say it back in plain words

A neuron just multiplies each input by a dial and adds up the votes — that's a dot product. The only trick your hardware ever needs is "multiply two numbers and add them to a running total": one little MAC cell. A whole layer is a rectangle of these dot products — a matrix multiply — and it's billions of MACs. But the multiplying is basically free; what kills you is fetching numbers from memory, which costs ~1000× more. So the winning idea is reuse: fetch each number once and shove it through many multiplies. To do that physically, we tile a plane with MAC cells wired only to their neighbours, freeze a weight inside each cell, and let inputs march left-to-right while partial sums fall top-to-bottom. Each input, once injected, sweeps a whole row — one fetch, many multiplies. After a short fill the grid finishes a full column of answers every clock tick. Its top speed is just (number of cells) × (clock) × 2 — but only if the cells stay busy, which small or memory-starved layers ruin. And because a small-number multiplier is far tinier, dropping to int8 lets ~16× more cells fit — more math, less energy, same silicon.