Intuition The one idea behind this whole topicA neural network is mostly one operation done billions of times: multiply two numbers and add the result to a running total . The TPU wins by arranging thousands of tiny multiply-add circuits in a grid so that each number, once fetched from memory, flows through the whole grid being reused — instead of being fetched again for every single multiply.
This page assumes you know nothing . Every letter, arrow, and word the parent topic leans on is built here, in order, each one earning its place before the next uses it.
Before any formula, fix the three shapes of data.
Definition Scalar, vector, matrix
A scalar is a single number, like 7
A vector is an ordered list of numbers, like ( 1 , 2 , 3 )
A matrix is a rectangle of numbers arranged in rows and columns. Picture: a grid of dots.
Why the topic needs this: a neural network layer stores its learned knowledge as a matrix of weights , and the data flowing through it as vectors of activations . Everything the TPU does is pushing vectors through matrices.
Definition The multiply symbol× a × b plain multiplication — the same × 3 × 4 = 12 A × B matrix multiply defined in §3 — a whole grid of these little multiplications wired together. Same symbol, but read it as "matrix multiply" the moment both sides are matrices. When we multiply two numbers we may also just write them side by side, like a p w p ×
Definition The subscript notationA ij A ij i j A always the row , the second is always the column . So A 23
Read Question ::: Answer:
What does A 31 The number in row 3, column 1 of matrix A
Why the topic needs it: the parent writes the core formula as C ij = ∑ p A i p B p j i j p
Definition Multiply–accumulate (MAC)One MAC is the two-step move: multiply two numbers, then add the product into a running total (an "accumulator"). Start the accumulator at zero , then repeat:
acc ← acc + a × w ( acc starts at 0 ) ← acc acc 0 k k
Picture: an empty jar labelled acc a × w
Intuition Why "multiply then add" and not something fancier?Because that is literally what a dot product is (next section), and a dot product is what every entry of a matrix multiply is. Neural networks are built almost entirely from matrix multiplies, so this one tiny operation is 95%+ of the work. Optimise the atom, win the war.
Definition FLOP, TOPS, and "peak"A FLOP is one fl oating-point op eration — one multiply, or one add, counts as one FLOP. So a single MAC = 1 multiply + 1 add = 2 FLOPs . "FLOP/s" means FLOPs per second — a raw speed.
TOPS means T era-OP erations per S econd (1 0 12 Peak throughput ("peak" for short) is the best-case speed: how many FLOPs/s the chip does if every arithmetic unit is busy every clock tick . It is a ceiling, not a promise.
Why the topic needs it: the parent's throughput formula Peak = 2 N 2 f N f 2 only because each MAC is 2 FLOPs. Miss this and the formula looks like magic.
Take two equal-length lists a = ( a 1 , a 2 , … , a k ) w = ( w 1 , … , w k ) dot product is: multiply matching entries, add them all up.
a ⋅ w = a 1 w 1 + a 2 w 2 + ⋯ + a k w k = ∑ p = 1 k a p w p
Now the strange symbols in that formula:
Definition The summation sign∑ p = 1 ∑ k p 1 k loop written as one symbol . p
Picture: a row of k a p w p ∑
∑ p = 1 k Why it feels right: the big Greek symbol looks intimidating, like it must be doing something exotic.
The fix: ∑ S um = add. Its cousin ∏
Why the topic needs it: the whole matrix-multiply formula is one dot product per output cell . The systolic array is nothing but a machine for computing many dot products in parallel.
Now we can read the parent's central equation.
Trace the indices with a picture:
i one row of A (the red row).j one column of B (the blue column).p together , multiplying the pair it lands on and adding to the running sum.
Definition The shape rulem × k k × n inner dimensions match : A k B k k outer dimensions m × n
Read:
Why must A B Each output is a dot product of a row of A B
Worked example Follow the formula on a2 × 2 A = ( 1 3 2 4 ) B = ( 5 7 6 8 ) C 11 = A 11 B 11 + A 12 B 21 = 1 ⋅ 5 + 2 ⋅ 7 = 19 C 12 = 1 ⋅ 6 + 2 ⋅ 8 = 22 C 21 = 3 ⋅ 5 + 4 ⋅ 7 = 43 C 22 = 3 ⋅ 6 + 4 ⋅ 8 = 50 in space, at once, instead of one after another.
Why the topic needs it: this equation is the workload. Everything else — the array, the streaming, the fill cost — exists to run this one formula cheaply.
The multiplies are cheap. The trouble is fetching the numbers .
Definition Byte, and what one memory read movesA byte is the standard small unit of computer data — think of it as one "box" that holds one number (an 8-bit integer number, the kind TPUv1 uses, fits in exactly one byte). When we say a value is "fetched from memory," one fetch moves the byte(s) of that one value into the compute unit. To keep counting simple on this page, treat one number = one read = a fixed small number of bytes , so "reads" and "bytes" grow in lockstep.
Definition Memory read, and the memory wallA memory read is the act of fetching one value out of memory (DRAM/SRAM) into the compute unit. It costs far more time and energy than the multiply itself. When a chip spends most of its time waiting for data instead of computing, it has hit the memory wall — see Memory wall and arithmetic intensity .
Definition Arithmetic intensityArithmetic intensity is the ratio
intensity = number of values fetched from memory number of arithmetic operations done . both the top and bottom in the same unit — here, count of values (equivalently bytes, since we fixed one value = a fixed number of bytes). High intensity = you do lots of math per fetched value = the memory wall stops hurting. Low intensity = you starve, waiting on memory.
Picture: a factory (compute) fed by a single narrow conveyor belt (memory). If each delivered part is used for many operations, the factory stays busy. If each part is used once and thrown away, the factory idles waiting for the belt.
Why the topic needs it: the entire justification for a systolic array is "read each number once , reuse it N Roofline model plots on its x-axis. The parent's claim "arithmetic intensity = n /2
Definition Big-O notationO ( ⋅ ) O ( n 3 ) n 3 n coarse growth label , not an exact count.
Intuition Where the countsn 3 2 n 2 n × n n 2 n n n 2 × n = n 3 O ( n 3 ) naïve machine, doing one MAC at a time, re-reads its two operands from memory for every MAC, so it makes about n 3 systolic array instead loads each input value once : there are n 2 A n 2 B 2 n 2 n 3 2 n 2
Definition Clock and cycleA chip's clock is a metronome ticking billions of times a second. Each tick is a cycle (or "clock"). Hardware does a fixed small amount of work per cycle. The clock frequency f 700 MHz = 700 × 1 0 6 side length N N × N
With N f N × N N 2 f = 2 N 2 f
Definition Pipeline fill / drainWhen data must travel across a grid before the first answer pops out, the early cycles produce no output — the pipe is still filling. Likewise the last data takes cycles to drain out. This fixed startup cost is why the parent says small jobs are wasteful and big batches are king.
Picture: a garden hose. Turn the tap on and water does not exit instantly — it must first travel the length of the hose (fill). That travel time is wasted if you only wanted one cup.
A batch is a group of input examples processed together in one go — say 64 images pushed through the network at once instead of one at a time. In our grid, a batch is many rows of activations streamed back-to-back through the array.
Read:
Why do systolic arrays prefer large batches? The fixed fill/drain cost (≈ 2 N − 1
Definition Inference vs. trainingTraining is the phase where a network learns its weights from example data, adjusting them repeatedly — it needs higher-precision math for the tiny corrections (gradients). Inference is the phase where an already-trained network is used to make predictions on new inputs — just the forward matrix multiplies, tolerant of low precision. TPUv1 did inference only; training came with TPU v2+.
Definition Cache and branch prediction (what the TPU throws away)A cache is a small fast memory a normal CPU keeps near its compute, holding recently-used values so it need not go to slow DRAM every time. Branch prediction is a CPU trick that guesses which way an "if" will go so it can keep working without waiting to find out. Both make general code faster but cost chip area and complexity. The TPU omits both — its dataflow is fixed and predictable, so it spends that saved area on more MAC units instead.
Definition ASIC and domain-specific acceleratorAn ASIC (Application-Specific Integrated Circuit) is a chip hard-wired for one job , with no general-purpose flexibility. A domain-specific accelerator is an ASIC aimed at one kind of workload (here: neural-net matrix multiplies). Contrast this with a CPU/GPU in ASIC vs FPGA vs general-purpose processors . The TPU trades flexibility for efficiency — no caches, no branch prediction, just arithmetic.
Definition Processing element (PE)A PE is one tiny circuit in the array that holds a weight and does a single MAC per cycle, then passes data to its neighbour. The array is an N × N N = 256 256 × 256 = 65 , 536
Definition Quantization / 8-bit8-bit integer math uses whole numbers in a tiny range instead of full-precision decimals, which makes each MAC circuit smaller and faster. This is quantization ; see Quantization and 8-bit inference . TPUv1 used it because inference tolerates low precision — this is also why TPUv1 could not train (training's gradients need more precision).
The diagram below is a dependency map : read each arrow →
Each foundation feeds the next; together they arrive at the systolic array and the TPU.
Test yourself — reveal only after you have answered aloud.
Can I read A ij Yes — i j
Do I know the two meanings of the symbol × Between numbers it is plain multiplication; between matrices it is the matrix multiply of §3.
Can I state a dot product as a sum of products? a ⋅ w = ∑ p = 1 k a p w p
Do I know why ∑ Sigma = S um; the multiply symbol would be ∏
Can I compute one entry C ij It is the dot product of row i A j B
Do I know why matrix shapes must match (k = k The shared dimension is the length of each dot product; both lists must be equally long.
Can I say what one MAC is, its starting value, and how many FLOPs it is? Accumulator starts at 0
Do I know what FLOP/s, TOPS, and "peak" mean? FLOP/s = float ops per second; TOPS = trillion integer ops per second; peak = best-case speed with every unit busy.
Can I define arithmetic intensity in one consistent unit? Operations done per value fetched from memory (top and bottom both counted as values/bytes).
Do I know where n 3 2 n 2 n 2 n n 3 2 n 2 A B
Can I say what a byte is? The standard small data unit; one 8-bit number fits in one byte.
Do I know what a clock cycle, fill/drain, and a batch are? A cycle is one tick; fill/drain is startup cost while data crosses the array; a batch is many inputs processed together.
Can I explain why the TPU drops caches and branch prediction? Its dataflow is fixed and predictable, so it spends that saved chip area on more MAC units instead.
Can I explain the difference between inference and training? Training learns the weights (needs precision); inference uses a trained network (tolerates low precision).
