Neural processing units (NPUs)

WHY do NPUs exist?

This is the 80/20: matrix-multiply (the 20% of operation types) is 80%+ of the runtime. Optimise that, ignore the rest.

WHAT is an NPU?

Key building blocks:

• Processing Element (PE): one MAC unit — computes $a \times b + c$.
• Systolic array: a 2-D grid of PEs where data flows rhythmically (like a heartbeat — Greek systole) between neighbours, no global memory access per step.
• On-chip SRAM / scratchpad: holds weights & activations near the compute.
• Low precision datapaths: int8, fp16, bfloat16.

HOW does the core math map to hardware?

Step 1 — The operation we must accelerate

A fully-connected layer computes $\mathbf{y} = W\mathbf{x}$, and a batch is a matrix multiply $Y = XW$. One output element is:

$y_{ij} = \sum_{k=1}^{K} X_{ik}\, W_{kj}$

Step 2 — Counting the work (derive the cost)

A matrix multiply of $(M\times K)\cdot(K\times N)$ needs, per output element, $K$ multiplies and $K-1$ adds. There are $MN$ outputs:

$\text{MACs} = M \cdot N \cdot K$

Step 3 — The naïve cost is memory, not math

Do it the dumb way: for each MAC, load $X_{ik}$ and $W_{kj}$ from memory. That's $2 \times MNK$ memory reads. WHY bad? A DRAM read ≈ 640 pJ; a MAC ≈ 0.2 pJ. The memory tax dominates by 1000×.

Step 4 — The systolic array (derivation of the dataflow)

Arrange PEs in a $K\times N$ grid. Weights $W_{kj}$ are loaded and held stationary inside PEs (this is called weight-stationary). Activations $X_{ik}$ flow left→right, one column of data entering per clock. Partial sums flow top→bottom, each PE doing:

$\text{psum}_{\text{out}} = \text{psum}_{\text{in}} + X_{\text{in}} \times W_{\text{stored}}$

Because each activation, once injected, marches across a whole row of PEs, one memory load feeds many multiplies. After the array fills (the "pipeline fill" latency), it produces a full result column every cycle.

Step 5 — Throughput and efficiency (derive the win)

An array of $R\times C$ PEs performs up to $R\cdot C$ MACs/cycle:

$\text{Peak throughput} = R \cdot C \cdot f_{\text{clock}} \quad [\text{MACs/s}]$

$\text{TOPS} = \frac{2 \cdot R \cdot C \cdot f_{\text{clock}}}{10^{12}} \quad (\text{the } 2 = \text{one mult} + \text{one add})$

Precision: WHY low bits win

$\text{Energy}_{\text{mult}} \propto (\text{bit-width})^2$

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Connections

• Systolic arrays — the dataflow engine inside most NPUs.
• GPUs and SIMT — general parallel compute vs fixed-function NPUs.
• Tensor cores — matmul units bolted onto GPUs (a hybrid step toward NPUs).
• Quantization and int8 inference — why low precision works.
• Roofline model — deciding compute-bound vs memory-bound.
• Dataflow and data reuse — weight/output/row-stationary strategies.
• Energy per operation — the physics motivating specialisation.
• Domain-specific architectures — the broader design philosophy.

Concept Map

Hinglish (regional understanding)