Google TPU architecture and systolic arrays: Question bank

Every word here is built from zero: a MAC is one multiply-accumulate — take two numbers, multiply them, add the result to a running total called an accumulator. A PE (processing element) is a tiny circuit that does exactly one MAC per clock tick. A systolic array is a grid of these PEs wired only to their immediate neighbours, through which data flows step-by-step like a bucket brigade. Keep those three pictures in mind.

True or false — justify

True or false: In a systolic array each input operand is fetched from off-chip memory once per MAC that uses it.

False. Each operand is read from memory once, then marched cell-to-cell (a_out = a_in); it is reused by many PEs without any further memory access. That single-read-many-uses trick is the whole point of "systolic".

True or false: A systolic array needs a global bus connecting every PE to shared memory.

False. PEs talk only to nearest neighbours — short local wires. Avoiding a global bus is exactly what lets the clock run fast and the energy stay low.

True or false: TPUv1 could train neural networks.

False. TPUv1 was inference-only with 8-bit integer MACs. Training needs floating-point range for gradients, which arrived with TPU v2+ using bfloat16.

True or false: Peak throughput of an

N \times N

MAC array grows linearly with

N

False. It grows as

N^{2}

: there are

N^{2}

MAC cells, all busy each clock, so

Peak = 2 N^{2} f

. Parallelism here is spatial, so doubling the side quadruples the compute.

True or false: The "systole" in systolic refers to a memory hierarchy.

False. It comes from systole, the heart's contraction — data is rhythmically pumped through the array each clock, like blood through a heart. It's a dataflow metaphor, not a memory one.

True or false: Higher arithmetic intensity means fewer memory accesses per unit of compute.

True. Arithmetic intensity is

\frac{MACs}{bytes read}

; raising it means each byte does more work, which is how you climb past the memory wall.

True or false: A single small matrix multiply runs at peak efficiency on a systolic array.

False. Every op pays a fixed pipeline fill/drain cost of

\approx 2 N - 1

cycles. One small op wastes those cycles; the array only reaches high efficiency on large batched work where fill is amortized.

True or false: In weight-stationary dataflow the weights move through the array while activations stay put.

False. It's the opposite — weights stay put (stationary) in each cell, and activations stream across while partial sums accumulate. That is why it is called weight-stationary.

Spot the error

Error hunt: "TPU throughput = clock × number of cores, just like a CPU."

The TPU has

N^{2}

MAC cells, not a handful of cores, so throughput

= 2 N^{2} f

— it scales with the square of the array side. CPU-style linear-in-cores reasoning badly underestimates it.

Error hunt: "We stagger (skew) the inputs to make the array look nicer on a diagram."

Skewing is functional, not cosmetic: data takes different numbers of cycles to reach different cells, so each row is delayed one clock so that the right operands meet at the right PE at the right time.

Error hunt: "The TPU has caches and branch prediction to speed up matmuls."

TPUv1 deliberately has no cache, no branch predictor, no out-of-order logic. All that die area goes to arithmetic instead — that is the domain-specific ASIC bet.

Error hunt: "For an

N \times N

multiply, latency is just

M

cycles for

M

rows."

You forgot the fill/drain: total

\approx (2 N - 1) + M

cycles. The first result only appears after data has propagated diagonally across the whole array.

Error hunt: "Reuse factor of a systolic array is fixed at 2 regardless of size."

Reuse factor is

\approx N

, the array dimension — a deeper array reuses each operand more. That's why a

256 \times 256

array cuts memory traffic by roughly

256 \times

versus a naïve read.

Error hunt: "The Unified Buffer is a small register file inside each PE."

The Unified Buffer is a large (24 MB) on-chip SRAM that holds activations and results for the whole array, feeding it and catching outputs to avoid DRAM round-trips. Per-PE storage is just the tiny accumulator plus stored weight.

Error hunt: "TPU is basically a fast general-purpose CPU."

It is a domain-specific ASIC that only does matrix/vector ops well. Its speed comes precisely from not being general — no OOO, no branch prediction, no caches, everything spent on MACs.

Why questions

Why does the systolic array break the memory wall while a naïve CPU loop does not?

The CPU re-fetches operands from memory for every MAC (data movement dominates); the array reads each operand once and reuses it across a whole row/column, so it does far more compute per byte fetched.

Why do systolic arrays "love" large batches?

The fixed fill/drain cost

\approx 2 N - 1

is paid once regardless of batch size, so a large batch

M ≫ N

spreads that overhead thin — efficiency

\frac{M}{M + 2 N}

approaches 1.

Why is the arithmetic intensity of an

n \times n

systolic multiply about

n /2

It performs

n^{3}

MACs but reads only the

2 n^{2}

input values once each, so intensity

= \frac{n ^{3}}{2 n ^{2}} = \frac{n}{2}

— reuse rises directly with array depth.

Why does each cell copy its activation forward with a_out = a_in?

That copy is the reuse mechanism — the same activation, read from memory once, is handed to the next cell so it can be used again there, achieving one-read-many-uses across the row.

Why did TPUv1 use 8-bit integers instead of floating point?

Inference tolerates low precision via quantization, and 8-bit MACs are far cheaper in area and energy — letting you pack many more MAC cells per chip. Training's gradients need more range, hence later floats.

Why is a systolic array wired only to nearest neighbours instead of a shared bus?

Local wires are short, so they carry signals fast at low energy and let the clock run high; a global bus would be long, slow, power-hungry, and a bandwidth bottleneck.

Edge cases

Edge case: What is the throughput of the array during the first

2 N - 1

cycles?

Below peak — the pipeline is filling, so not all

N^{2}

cells hold valid operands yet. Full

2 N^{2} f

throughput is only reached in steady state after fill.

Edge case: What happens if the matrix is smaller than the array (say

8 \times 8

fed to a

256 \times 256

array)?

Most PEs sit idle, so efficiency is terrible; you still pay the large fill cost with almost no useful work. Tiny ops are exactly where systolic arrays are least efficient.

Edge case: If you feed a batch of exactly

M = 1

row, what dominates the cycle count?

The fill/drain term

2 N - 1

completely dominates the single streamed row, so nearly all cycles are overhead — the classic latency-bound worst case.

Edge case: What limits performance when the layer is memory-bandwidth-bound rather than compute-bound?

You are on the sloped part of the roofline: arithmetic intensity is too low to keep the MACs fed, so DRAM bandwidth, not the

2 N^{2} f

peak, sets the achievable rate.

Edge case: Does doubling the clock

f

double throughput without limit?

Only up to power/thermal and wire-delay limits; the formula

2 N^{2} f

is linear in

f

, but physically the local-wire timing and heat cap how high

f

can go, which is why designers also grow

N

Recall Rebuild the two master formulas from memory

Peak throughput of $N \times N$ array ::: $2 N^{2} f$ FLOP/s ( $N^{2}$ cells $\times$ 2 FLOP/MAC $\times f$ ). Cycles to push $M$ rows through ::: $\approx (2 N - 1) + M$ (fill/drain plus stream).