Google TPU architecture and systolic arrays: Worked examples

This is the worked-examples companion to the parent TPU note. There we built the why and how of a dataflow systolic array. Here we drill every case class you can meet — tiny multiplies, degenerate shapes, batch limits, quantization edge cases, and an exam twist — each worked from the ground up.

Before we start, one promise from the contract: no symbol appears before it is defined. If the parent used it, we re-earn it here.

Definition The matrix-multiply symbols (define these once, use them everywhere below)

We are multiplying two number-grids to get a third: $C = A B$ .

$A$ has some rows and columns; call the row count $m$ and the column count $k$ . So $A$ is an " $m \times k$ " grid (read " $m$ by $k$ ": $m$ rows, $k$ columns).
$B$ must have $k$ rows (to match $A$ 's columns) and some number of columns $n$ ; so $B$ is " $k \times n$ ".
The result $C$ then has $m$ rows and $n$ columns, i.e. " $m \times n$ ".
$k$ is special: it is the shared / contraction dimension — the length of each dot product, the thing we sum over.

To point at one specific entry of a grid we use two little index letters written as subscripts:

$i$ = which row (runs $1, 2, \dots, m$ ).
$j$ = which column (runs $1, 2, \dots, n$ ).
$p$ = which term inside the dot product (runs $1, 2, \dots, k$ ).

So $A_{i p}$ means "the number in row $i$ , column $p$ of $A$ ", and the entry of $C$ in row $i$ , column $j$ is the sum of $k$ products: $C_{ij} = \sum_{p = 1}^{k} A_{i p} B_{p j}$ On the physical array we borrow the same letters: cell $(p, j)$ is the hardware cell in row $p$ , column $j$ that stores weight $B_{p j}$ .

The scenario matrix

Every question this topic throws at you falls into one of these cells. The examples below are labelled with the cell they cover.

#	Case class	What makes it tricky	Example
A	Plain square multiply	the "does the wiring really compute the dot product?" sanity check	Ex 1
B	Non-square / rectangular ( $m \neq = k \neq = n$ )	the array isn't square-shaped for your data	Ex 2
C	Degenerate: array bigger than data ( $N >$ matrix side)	wasted cells, low utilization	Ex 3
D	Limiting: tiny batch $M \to 1$ vs huge batch $M \to \infty$	pipeline fill dominates vs vanishes	Ex 4
E	Throughput / roofline (peak FLOP/s)	scales as $N^{2}$ , not linearly	Ex 5
F	Memory-traffic saving (arithmetic intensity)	reuse factor = $N$	Ex 6
G	Zero / sign / zero-point in quantization	8-bit overflow, negatives, dequant offset	Ex 7
H	Real-world word problem	translate a layer into cycles & time	Ex 8
I	Exam twist: matrix bigger than array (tiling)	you must split the multiply	Ex 9

Let's hit every cell.

Example 1 — Case A: plain $2 \times 2$ multiply (does the wiring work?)

Forecast: guess $C_{11}$ before reading on — is it $1 \cdot 5 + 2 \cdot 7$ , or $1 \cdot 5 + 3 \cdot 6$ ?

The figure below shows the whole picture at once: read it before the steps.

Figure — Google TPU architecture and systolic arrays

How to read the figure (annotation walk-through):

The four lavender rounded boxes are the hardware cells. Each is labelled with the weight $B_{p j}$ it stores: the box's vertical position is the row $p$ (top = $p = 1$ , bottom = $p = 2$ ), its horizontal position is the column $j$ (left = $j = 1$ , right = $j = 2$ ).
The two coral arrows on the left are the rows of $A$ marching rightward into the cells — an activation is read once and passed cell-to-cell (that is the one-read-many-uses trick).
The two mint arrows at the bottom are the partial sums dropping downward out of each column; the number that falls out of column $j$ is exactly $C_{ij}$ . Look at those two mint labels — they are the answer you are about to derive.

Steps.

Load the weights. Cell $(p, j)$ stores $B_{p j}$ . So the left column of cells (column $j = 1$ ) holds $B_{11} = 5$ (top, row $p = 1$ ) and $B_{21} = 7$ (bottom, row $p = 2$ ). The right column ( $j = 2$ ) holds $B_{12} = 6$ and $B_{22} = 8$ . (These are the four lavender boxes in the figure.) Why this step? The output $C_{ij} = \sum_{p} A_{i p} B_{p j}$ needs $B_{p j}$ sitting where the matching $A_{i p}$ will pass. Column $j$ owns output column $j$ .
Stream row $i = 1$ of $A$ , namely $(A_{11}, A_{12}) = (1, 2)$ , into the top (the top coral arrow) and it walks across. Column 1 accumulates: $acc = A_{11} B_{11} + A_{12} B_{21} = 1 \cdot 5 + 2 \cdot 7 = 5 + 14 = 19 = C_{11}$ Why this step? Each cell does $partial_{out} = partial_{in} + a_{in} \cdot w_{stored}$ . The partial sum travels down (mint arrow), picking up one product per cell — that is the dot product $\sum_{p} A_{1 p} B_{p 1}$ .
Same row into column $j = 2$ : $C_{12} = 1 \cdot 6 + 2 \cdot 8 = 6 + 16 = 22$ Why this step? The same activation row $(1, 2)$ is reused by a second column of cells with no new memory read — this is the reuse that column-parallelism buys. It computes the second dot product $\sum_{p} A_{1 p} B_{p 2}$ .
Stream row $i = 2$ of $A$ , $(3, 4)$ : $C_{21} = 3 \cdot 5 + 4 \cdot 7 = 15 + 28 = 43, C_{22} = 3 \cdot 6 + 4 \cdot 8 = 18 + 32 = 50$ Why this step? A new row of $A$ produces a new row of $C$ : the weights $B_{p j}$ stay stationary in the cells while successive activation rows stream through, so one weight-load serves the entire batch.

Forecast check: the correct $C_{11} = 1 \cdot 5 + 2 \cdot 7 = 19$ — the first option. If you guessed $1 \cdot 5 + 3 \cdot 6$ you paired a row of $A$ with a row (not column) of $B$ ; the dot product runs down $B$ 's column $j$ , so the second factor is $B_{p j}$ with $p$ varying, not $B_{i p}$ .

Verify: by hand, $A B = (19432250)$ . The array wired the sum in space instead of a loop in time — same answer. ✓

Example 2 — Case B: rectangular multiply ( $3 \times 2$ by $2 \times 4$ )

Forecast: guess the shape of $C$ and the total MAC count first.

Steps.

Shape rule. $A$ is $m \times k = 3 \times 2$ , $B$ is $k \times n = 2 \times 4$ . The shared dimension $k = 2$ (the length we sum over) must match — it does. So $C$ is $m \times n = 3 \times 4$ . Why this step? $C_{ij} = \sum_{p = 1}^{k} A_{i p} B_{p j}$ only makes sense if $A$ 's column count equals $B$ 's row count ( $= k$ ).
Count outputs. $C$ has $m \cdot n = 3 \cdot 4 = 12$ entries. Why this step? Each entry $C_{ij}$ needs its own accumulator cell in the array, so counting outputs tells you how many result cells (and columns of streamed activations) the computation occupies.
Count MACs. Each entry is a sum of $k = 2$ products, so total MACs $= m \cdot n \cdot k = 3 \cdot 4 \cdot 2 = 24$ . Why this step? Total work in a matmul is always $m n k$ MACs — this is the number that hits the array.
Array depth needed. The partial-sum direction must be at least $k = 2$ cells deep so a full dot product accumulates as it flows down. A $256 \times 256$ array handles $k \leq 256$ in one pass. Here $k = 2 ≪ 256$ , so most of the array is idle — foreshadowing Case C. Why this step? The contraction length $k$ sets how far a partial sum must travel to become complete; if $k$ exceeded the array depth you would have to tile (Example 9), so checking $k$ against $N$ decides whether one pass suffices.

Forecast check: $C$ is $3 \times 4$ (not square — the output inherits $m$ from $A$ and $n$ from $B$ ), and the MAC count is $24$ . If you guessed a $2 \times 2$ or $4 \times 4$ shape, remember the contraction dimension $k$ disappears from the result; only $m$ and $n$ survive.

Verify: $m n k = 3 \cdot 4 \cdot 2 = 24$ MACs; $C$ shape $3 \times 4$ has $12$ entries; $24/12 = 2 = k$ . Consistent. ✓

Example 3 — Case C: array bigger than the data (utilization)

Forecast: guess — is it near $100%$ , near $50%$ , or tiny?

Steps.

Useful cells. A $2 \times 2 \times 2$ multiply keeps at most $2 \times 2 = 4$ output cells busy (and depth $2$ ). Why this step? Only cells that own a real $C_{ij}$ and see real operands compute anything.
Total cells. $256 \times 256 = 65536$ . Why this step? The array is a fixed physical grid; its cell count is a constant of the hardware, so it is the denominator against which any problem's useful work is measured.
Utilization: $U = \frac{4}{65536} = 6.1 \times 1 0^{- 5} \approx 0.0061%$ Why this step? This is the degenerate case — the array is a fixed-size grid; small problems drown in unused silicon.

Forecast check: the answer is tiny ( $0.0061%$ ), not near $50%$ or $100%$ . The intuition trap is thinking " $2 \times 2$ uses a fair chunk"; in fact utilization is $(data side / N)^{2}$ , which collapses fast when the data is much smaller than $N$ .

Verify: $4/65536 = 0.00006103 \dots$ , i.e. $0.0061%$ . This is exactly why the parent note says TPUs are for big matrices — small ops are catastrophically wasteful. ✓

Example 4 — Case D: the two limits of batch size $M$

Forecast: guess $η (1)$ to the nearest percent, and what $η$ approaches for huge $M$ .

Steps.

Where the formula comes from. Total cycles $= fill+drain (2 N - 1) + stream M$ . Only $M$ of those cycles emit results; the fill is overhead. So $η = M / (M + 2 N - 1)$ . Why this step? The first result appears only after data has crawled diagonally across all $N$ rows and $N$ columns — that diagonal delay is $2 N - 1$ clocks.
Limit $M = 1$ with $2 N - 1 = 511$ : $η (1) = \frac{1}{1 + 511} = \frac{1}{512} \approx 0.00195 = 0.195%$ Why this step? One tiny op pays the full fill cost for a single result — worst case. (This is the "latency of one small matmul" trap from the parent.)
Limit $M \to \infty$ : $lim_{M \to \infty} \frac{M}{M + 511} = 1 = 100%$ Why this step? As the batch grows, the fixed 511-cycle fill becomes negligible — the array runs at full efficiency. TPUs are throughput machines.
Solve $η \geq 0.9$ , respecting that $M$ is a whole number. Setting the equality first: $\frac{M}{M + 511} = 0.9 \Rightarrow M = 0.9 M + 0.9 \cdot 511 \Rightarrow 0.1 M = 459.9 \Rightarrow M = 4599$ Why this step? $M$ counts activation rows, so it must be a non-negative integer — a fractional batch is meaningless. $η (M)$ is increasing in $M$ , so we need the smallest integer $M$ with $η (M) \geq 0.9$ . The equality boundary lands exactly on the integer $4599$ : check $η (4599) = 4599/5110 = 0.9000$ (exactly $\geq 0.9$ ), while $η (4598) = 4598/5109 = 0.89998 < 0.9$ . Had the boundary been fractional we would take the ceiling $⌈ M ⌉$ ; here it is already integral, so $M = 4599$ is the answer.

Forecast check: $η (1) \approx 0.2%$ (dismally small — one op wastes the whole pipeline fill), and the huge-batch limit is exactly $100%$ . If you guessed $η (1)$ near $50%$ , you underestimated how large the fixed $511$ -cycle fill is relative to a single streamed row.

Verify: $1/512 = 0.001953$ ; $lim = 1$ ; $4599/ (4599 + 511) = 4599/5110 = 0.9000$ ; and $4598/5109 = 0.89998 < 0.9$ confirms $4599$ is the smallest integer that qualifies. ✓

Example 5 — Case E: peak throughput scales as $N^{2}$

Forecast: guess the TOPS number, then guess how much doubling $N$ multiplies it (×2 or ×4?).

Steps.

Formula. $N^{2}$ cells, each 1 MAC/clock in steady state, 2 ops per MAC: $Peak = 2 N^{2} f ops/s$ Why this step? Every one of the $N^{2}$ cells fires each clock — parallelism is spatial, so throughput $\propto$ area $\propto N^{2}$ .
Plug in $N = 256$ : $2 \cdot 25 6^{2} \cdot 0.7 \times 1 0^{9} = 2 \cdot 65536 \cdot 0.7 \times 1 0^{9} = 9.175 \times 1 0^{13} ops/s \approx 92 TOPS$
Double the side, $N = 512$ : $2 \cdot 51 2^{2} \cdot 0.7 \times 1 0^{9} = 3.67 \times 1 0^{14} = 4 \times (9.175 \times 1 0^{13})$ Why this step? Doubling $N$ quadruples throughput — this is the "not clock×cores like a CPU" lesson.

Forecast check: the answer is $\approx 92$ TOPS, and doubling $N$ multiplies throughput by ×4, not ×2. If you guessed ×2 you were thinking linearly (like adding CPU cores); the array grows in area, so the factor is squared.

Verify: $2 \cdot 25 6^{2} \cdot 0.7 \times 1 0^{9} = 9.1750 \times 1 0^{13}$ ops/s; $(512/256)^{2} = 4$ ; ratio confirms the $N^{2}$ scaling. See Roofline model for how this peak becomes the flat "compute-bound" ceiling. ✓

Example 6 — Case F: memory-traffic saving (arithmetic intensity)

Forecast: guess the ratio — is it about ×16, ×256, or ×65536?

Steps.

Naïve reads. Every MAC re-fetches its two operands. Number of MACs $= m n k = 25 6^{3}$ , each needing 2 reads: $2 \cdot 25 6^{3} = 2 \cdot 16, 777, 216 = 3.355 \times 1 0^{7} reads$ Why this step? This is the memory-wall worst case — compute is cheap, moving data is not.
Systolic reads. Each of $A$ and $B$ ( $25 6^{2}$ entries each) is read once: $2 \cdot 25 6^{2} = 2 \cdot 65536 = 1.31 \times 1 0^{5} reads$
Ratio: $\frac{2 N ^{3}}{2 N ^{2}} = N = 256$ Why this step? The reuse factor equals the array side $N$ : each byte fetched feeds a whole row/column of cells.
Arithmetic intensity (MACs per input read): $AI = \frac{N ^{3}}{2 N ^{2}} = \frac{N}{2} = 128 MACs/read$

Forecast check: the ratio is ×256 (equals $N$ ), not ×16 or ×65536. If you guessed ×65536 you confused it with the cell count $N^{2}$ ; the reuse factor is the array side $N$ , because each input feeds one row/column of cells, not the whole grid.

Verify: $25 6^{3} = 16777216$ ; ratio $2 N^{3} /2 N^{2} = 256$ ; AI $= 256/2 = 128$ . ✓

Example 7 — Case G: zero, sign, zero-point, and overflow in 8-bit MACs

Forecast: before reading — guess the largest magnitude a single 8-bit×8-bit product can reach, and guess why we need an offset (zero-point) instead of just a scale.

Steps.

(a) Zero weight. If $w = 0$ then $a \cdot w = 0$ for any activation $a$ . The cell still passes $a_{out} = a_{in}$ along. Why this step? Zero is not special in hardware — it flows through like any number; this is why zeros don't automatically save time on a dense array.
(b) Most-negative and most-positive product (sign corners). The extreme positive signed product is two negatives: $(- 128) \cdot (- 128) = 16384$ and the most-negative single product is a negative times the largest positive: $(- 128) \cdot (127) = - 16256$ Both fit easily in 16 bits. Why this step? Covers the sign corner — two negatives give the largest positive, mixed signs give the most-negative, so we have shown both sign quadrants are handled without overflow at the multiplier.
(c) No accumulator overflow. Summing $k = 256$ worst-case products of magnitude $16384$ : $256 \cdot 16384 = 4, 194, 304$ A signed 32-bit register holds up to $2^{31} - 1 = 2, 147, 483, 647$ . Since $4.19 \times 1 0^{6} ≪ 2.15 \times 1 0^{9}$ , no overflow. Why this step? Explains why the accumulator (32-bit) is deliberately wider than the operands (8-bit): summing many products grows the running total, so it needs headroom the 8-bit inputs don't.
(d) Quantize then dequantize a concrete value. Take a ReLU activation range $[0, 6]$ mapped onto unsigned codes $[0, 255]$ . Then the scale is $s = 6/255 = 0.023529 \dots$ and the zero-point is $z = 0$ (real $0$ maps to code $0$ ). Quantize $r = 4.5$ : $q = round (\frac{4.5}{0.023529}) + 0 = round (191.25) = 191$ Dequantize back: $\overset{r}{^} = s (q - z) = 0.023529 \cdot (191 - 0) = 4.4941 \dots$ The round-trip error is $∣4.5 - 4.4941∣ = 0.0059$ , about half a scale step $s /2$ — the price of 8 bits. Why this step? Shows the full quantize → compute → dequantize loop the TPU actually runs: integers flow through the MAC array (cheap, dense), and results are rescaled by $s$ afterward. The zero-point $z$ is what makes a lopsided (all-positive) range fit.

Forecast check: the largest-magnitude single product is $16384$ (from $(- 128)^{2}$ ), reached by two negatives, not by two large positives (whose max is $12 7^{2} = 16129$ ). And the zero-point is needed because a scale alone can only stretch a range symmetric about $0$ ; the offset $z$ slides the range so an all-positive interval like $[0, 6]$ uses all 256 codes.

Verify: $(- 128) (- 128) = 16384$ ; $(- 128) (127) = - 16256$ ; $256 \cdot 16384 = 4194304 < 2147483647$ ; $round (4.5 \cdot 255/6) = 191$ ; and $6/255 \cdot 191 = 4.4941$ , error $\approx 0.006$ . See Quantization and 8-bit inference. ✓

Example 8 — Case H: real-world word problem (a dense layer's runtime)

Forecast: guess whether the fill cost ( $\sim 511$ cycles) noticeably changes the answer.

Steps.

Cycles. $cycles = (2 N - 1) + M = 511 + 8192 = 8703$ . Why this step? Fill/drain 511 cycles once, then one result-row per clock for the $M = 8192$ rows.
Efficiency check. $η = 8192/8703 = 0.941 = 94.1%$ — a fat batch amortizes the fill, exactly as Case D predicted.
Time. One cycle $= 1/ (0.7 \times 1 0^{9}) s = 1.4286 \times 1 0^{- 9} s$ . So $8703 \times 1.4286 \times 1 0^{- 9} s = 1.243 \times 1 0^{- 5} s = 12.43 μ s$ Why this step? Converts abstract cycles into a wall-clock number — the unit ( $μ$ s) is what an engineer actually reports.

Forecast check: the fill barely matters here — $511$ cycles on top of $8192$ is only a $\sim 6%$ overhead, so $η \approx 94%$ . With a big batch the fixed fill is nearly invisible, which is exactly why TPUs batch aggressively.

Verify: cycles $= 511 + 8192 = 8703$ ; $η = 8192/8703 = 0.9413$ ; time $= 8703/ (0.7 \times 1 0^{9}) = 1.2433 \times 1 0^{- 5}$ s $= 12.43 μ$ s. ✓

Example 9 — Case I: exam twist — matrix bigger than the array (tiling)

Forecast: guess the number of $256 \times 256$ tiles first, and whether the total MAC count differs from an untiled $51 2^{3}$ .

Steps.

Split each dimension. $512 = 2 \times 256$ , so break $A$ , $B$ , $C$ into $256 \times 256$ blocks. Because each of the three matrix dimensions ( $m, n, k$ ) splits into $512/256 = 2$ chunks, $C$ becomes a $2 \times 2$ grid of output tiles $= 4$ output tiles. Why this step? The array is a fixed $256 \times 256$ window; you slide it over the big problem — this is tiling, the standard fix when the matrix exceeds the hardware.
Passes per output tile. Each output block $C_{I J} = \sum_{P} A_{I P} B_{P J}$ sums over the contraction tile index $P$ , and $k = 512$ also splits into $512/256 = 2$ pieces. So each of the 4 output tiles needs $2$ accumulate-passes. $tile-passes = (output tiles) \times (contraction tiles) = 4 \times 2 = 8$ Why this step? One pass loads a $256 \times 256$ weight block and streams one $256$ -column slab; you need one pass per (output-tile, contraction-tile) pair, adding their partial sums.
Total MACs (must equal the untiled count). Each pass drives a full $256 \times 256 \times 256$ multiply, i.e. $25 6^{3} = 16, 777, 216$ MACs. Over all 8 passes: $8 \times 25 6^{3} = 8 \times 16, 777, 216 = 134, 217, 728$ Compare with an untiled $512 \times 512$ multiply, which does $m n k = 51 2^{3} = 134, 217, 728$ MACs — identical. Why this step? Tiling rearranges work in space/time but never changes the arithmetic total — that equality ( $8 \cdot 25 6^{3} = 51 2^{3}$ ) is the correctness invariant you check in an exam.

Forecast check: there are 4 output tiles and 8 total passes, and the MAC count is identical to $51 2^{3}$ . If you guessed the total MACs would shrink, remember tiling only reorders work — the arithmetic $m n k$ is conserved.

Verify: output tiles $= (512/256)^{2} = 4$ ; contraction tiles $= 512/256 = 2$ ; passes $= 8$ ; $8 \cdot 25 6^{3} = 134217728 = 51 2^{3}$ . ✓

Recap

Recall Which batch size gives 90% efficiency on a 256×256 array?

$M \geq (2 N - 1) \cdot \frac{0.9}{0.1} = 511 \cdot 9 = 4599$ rows. ::: ~4600 rows — TPUs need big batches.

Recall Why does doubling the array side quadruple throughput?

Throughput $= 2 N^{2} f$ ; it scales with area $N^{2}$ because all $N^{2}$ cells fire every clock. ::: It's spatial (area) parallelism, not linear.

Recall Does tiling change the total MAC count?

No — $8 \times 25 6^{3} = 51 2^{3}$ . Tiling only reorders the same arithmetic. ::: Same total work, split into passes.

Recall In 8-bit quantization, what does the zero-point do?

It is the integer code that stands for real value $0$ , letting a lopsided (all-positive, post-ReLU) range fit the 8-bit codes without waste. ::: The offset that anchors real 0 in integer space.

Related depth: GPU architecture and SIMT execution · ASIC vs FPGA vs general-purpose processors · Roofline model.