Foundations — Neural processing units (NPUs)

Before you can read the parent note, you need to own every symbol it throws at you. This page builds each one from nothing — plain words, then a picture, then why the topic needs it. Read top to bottom; each block uses only things defined above it.

1. A number, and multiplying two of them

Start absolutely flat. A number is just a quantity — like $3$ or $0.5$. When we write two numbers next to each other with a $\times$, we multiply: $3\times 4=12$.

Why the topic needs it: a neural network is made of multiplications. Every connection between two neurons multiplies a signal by a strength. So the smallest useful piece of hardware must, above all, multiply.

2. A running total, and the "accumulate" idea

Now suppose you have a list of numbers and you want their total. You keep a running total — a box that starts at $0$ and grows as you toss numbers into it.

$\text{box}\leftarrow \text{box}+\text{new number}$

The arrow $\leftarrow$ means "becomes": the box's new value is its old value plus the new number. This is accumulation — building up a sum step by step.

Why the topic needs it: the entire NPU is thousands of copies of this one operation. If you understand MAC, you understand the atom the machine is built from.

3. Lists of numbers: the vector, and the dot product

A vector is just an ordered list of numbers, like $(2,5,1)$. Think of it as a row of labelled boxes.

Now take two vectors of the same length. Multiply them position by position, then add up all those products. That is the dot product:

$(a_1,a_2,a_3)\cdot (b_1,b_2,b_3)=a_1{b}_1+a_2{b}_2+a_3{b}_3$

Why the topic needs it: every single output number of a neural-network layer is one dot product. The parent's line $y_{ij}={\sum }_k{X}_{ik}{W}_{kj}$ is a dot product in disguise — we unpack that $\sum$ next.

4. The summation symbol $\sum$

The big Greek letter $\sum$ (sigma, an "S" for "Sum") is shorthand for "add up a series of terms."

${\sum }_{k=1}^K{t}_k=t_1+t_2+t_3+\cdots +t_K$

Read it aloud: "sum, as $k$ goes from $1$ up to $K$, of $t_k$."

• The letter under the $\sum$ ($k=1$) is the counter and its starting value.
• The letter on top ($K$) is where the counter stops.
• $t_k$ is the recipe for each term — a template where you plug in the current $k$.

Why the topic needs it: $\sum$ is the compact way to say "do a MAC for every $k$ and keep the total." The parent's dot product $y={\sum }_{k=1}^K{X}_k{W}_k$ is precisely a $K$-step MAC chain. The counter $k$ tells the hardware how many multiply-adds one output takes.

5. Grids of numbers: the matrix, and subscripts

A matrix is a rectangle of numbers — rows and columns, like a spreadsheet. We name one with a capital letter, e.g. $W$, and we point at a single cell with two subscripts:

$W_{kj}=\text{the number in row }k,\text{column }j$

Why the topic needs it: the parent writes weights as $W_{kj}$ and inputs as $X_{ik}$. Without the row/column convention, $y_{ij}={\sum }_k{X}_{ik}{W}_{kj}$ is gibberish. With it, you can see the formula: it walks column $k$ of row $i$ of $X$ against row $k$ of column $j$ of $W$.

6. Matrix multiply $Y=XW$

To multiply matrix $X$ (size $M\times K$) by matrix $W$ (size $K\times N$), you compute each output cell as a dot product: row $i$ of $X$ dotted with column $j$ of $W$.

$Y_{ij}={\sum }_{k=1}^K{X}_{ik}{W}_{kj}$

The shapes must match at the middle: $X$ has $K$ columns and $W$ has $K$ rows — that shared $K$ is the length of every dot product. The result $Y$ is $M\times N$.

Why the topic needs it: matrix multiply is ~90% of neural inference. The whole NPU exists to do this one thing fast. Its three dimensions $M,K,N$ decide how well the grid of PEs is filled (utilisation, in the parent note).

7. The processing element (PE) and the grid

A PE (Processing Element) is one physical MAC unit on the chip: hardware that does $\text{box}\leftarrow \text{box}+a\times b$ once per clock cycle.

Put many PEs in a 2-D grid of $R$ rows and $C$ columns and let numbers flow between neighbours — that is the systolic array. All $R\times C$ PEs fire every tick, so the grid does $R\cdot C$ MACs per cycle.

8. Reuse, and why memory is the enemy

Every number a PE multiplies has to come from memory. There are two kinds:

• DRAM — big, far away, slow, energy-hungry (a read costs ~$640\text{ pJ}$).
• On-chip SRAM / scratchpad — small, close, cheap.

A MAC itself costs only ~$0.2\text{ pJ}$. So fetching the number can cost ~1000× more than using it.

Why the topic needs it: the parent says "data reuse is the whole game." Now you know why: arithmetic is nearly free; moving data is expensive.

9. Precision: how many bits a number uses

A number in hardware is stored in a fixed number of bits (0/1 switches). More bits = finer detail but bigger, hungrier circuits.

• fp32 — 32-bit floating point, very precise.
• fp16 / bfloat16 — 16-bit, half the bits.
• int8 — 8-bit integer, coarse but tiny.

Why the topic needs it: "low precision wins" is a headline claim of the parent. The $(\text{bits}{)}^2$ scaling is the reason.

Prerequisite map

Every box you now understand feeds directly into the parent topic, Neural processing units (NPUs). The grid idea leads to Systolic arrays; the reuse idea to Dataflow and data reuse; the memory-vs-math cost to Energy per operation and the Roofline model; the bit-width idea to Quantization and int8 inference. The "why a special chip at all" thread connects to Domain-specific architectures, and comparisons live in GPUs and SIMT and Tensor cores.

Equipment checklist

Test yourself — cover the right side and answer before revealing.