Google TPU architecture and systolic arrays: Exercises

6.5.7 · D4 · Hardware › Advanced & Emerging Architectures › Google TPU architecture and systolic arrays

Formula Teen tools jo tum baar baar use karoge

Maano $N$ = square array ki side length hai (toh usme $N \times N$ cells hain), $f$ = clock frequency hertz mein (cycles per second), $M$ = batch mein rows ki sankhya jo hum stream karते hain.

1. Peak throughput — agar har cell busy ho toh kitni arithmetic operations per second: $Peak = 2 N^{2} f FLOP/s$ $N^{2}$ cells ka count hai; $2$ isliye hai kyunki ek multiply-accumulate (MAC) do operations hai — ek multiply aur ek add.

2. Latency (cycle count) — $M$ rows ko push karne mein kitne clock ticks lagte hain: $cycles \approx (2 N - 1) + M$ $(2 N - 1)$ fill/drain cost hai: data ko pehla jawab neeche se nikalne se pehle puri array mein diagonally ripple karna padta hai.

3. Arithmetic intensity — memory se fetch kiye gaye bytes (ya operands) per kitne MACs hue: $AI = \frac{MACs performed}{operands read}$ Bada number matlab tum memory par kam aur apne silicon par zyada depend karte ho — tum memory wall ke upar chad jaate ho.

Level 1 — Recognition

Recall Solution

KYA: "Systolic" systole se aaya hai, yani heartbeat — woh contraction jo blood pump karti hai. Metaphor kyun: jaise heartbeat blood ko rhythmically chamber se chamber mein push karti hai, array data ko rhythmically, ek hop per clock, ek cell se neighbouring cell mein push karta hai. Koi bhi cell computation ke beech global memory tak nahi pahunchti; sab kuch hand-to-hand pass hota hai. Captured key property: local, rhythmic dataflow with maximal reuse.

Recall Solution

Weights ruk jaate hain (yahi "weight-stationary" ka matlab hai) — har cell poori computation ke liye ek weight $B_{p j}$ hold karta hai.
Activations horizontally move karti hain (left → right), $a_{out} = a_{in}$ ke zariye pass hoti hain.
Partial sums vertically move karte hain (top → bottom), har cell apna product add karta hai: $partial_{out} = partial_{in} + a_{in} \cdot w_{stored}$ .

Level 2 — Application

Recall Solution

KYA: $Peak = 2 N^{2} f$ mein plug karo. $Peak = 2 \cdot 12 8^{2} \cdot (1 \times 1 0^{9}) = 2 \cdot 16384 \cdot 1 0^{9} = 3.2768 \times 1 0^{13} ops/s .$ Factor 2 kyun: har cell ek clock mein ek MAC complete karta hai = 1 multiply + 1 add = 2 ops. TOPS mein convert karo ( $1 TOPS = 1 0^{12}$ ops/s): $\approx 32.8 TOPS$ .

Recall Solution

KYA: $cycles \approx (2 N - 1) + M = (2 \cdot 128 - 1) + 1000 = 255 + 1000 = 1255$ cycles. Wasted fraction: fill/drain part $255$ cycles ka hai, toh $\frac{255}{1255} \approx 0.203 = 20.3% .$ Kyun matter karta hai: ek paanchwa hissa sirf pipeline fill karne mein gaya — yeh ek warning hai ki is array par chhote batches inefficient hain (ise hum L3 mein directly tackle karte hain).

Recall Solution

KYA — MACs count karo: ek $N \times N \times N \times N$ product $N^{3}$ MACs karta hai, yahan $51 2^{3}$ . Naïve reads: har MAC $A$ ka 1 element aur $B$ ka 1 element padhta hai → $2 \cdot 51 2^{3}$ reads. Systolic reads: $2 N^{2} = 2 \cdot 51 2^{2}$ inputs mein se har ek exactly ek baar padha jaata hai. $reuse = \frac{2 \cdot 51 2 ^{3}}{2 \cdot 51 2 ^{2}} = 512 = N .$ KYun: reuse factor array dimension $N$ ke barabar hai — har operand $N$ cells mein se guzra jaane se pehle.

Level 3 — Analysis

Recall Solution

KYA: $\frac{M}{M + 511} \geq 0.95$ solve karo. Multiply out karo: $M \geq 0.95 M + 0.95 \cdot 511$ , toh $0.05 M \geq 485.45$ , milta hai $M \geq \frac{485.45}{0.05} = 9709.$ Round up karo: $M \geq 9709$ rows. KYun: fixed 511-cycle fill tabhi "dilute" hoti hai jab useful stream $M$ use vastly outnumber kare. Zyada utilisation targets exponentially bade batches demand karte hain — isliye TPUs throughput machines hain, latency machines nahi.

Recall Solution

KYA — cycles, phir seconds (time = cycles ÷ $f$ ):

A: cycles $= (2 \cdot 128 - 1) + 2000 = 255 + 2000 = 2255$ . Time $= 2255/ (1 \times 1 0^{9}) = 2.255 μ s$ .
B: cycles $= (2 \cdot 256 - 1) + 2000 = 511 + 2000 = 2511$ . Time $= 2511/ (0.25 \times 1 0^{9}) = 10.044 μ s$ . Winner: A, aur yeh wall-clock time mein ~4.45× faster hai. KYun: equal peak throughput clock ko hide karta hai. B apna peak sirf perfect steady state mein reach karta hai; uska slow clock har cycle ko stretch karta hai, aur uska bada array zyada fill pay karta hai. Peak FLOP/s ek ceiling hai, delivered time nahi.

Figure — Google TPU architecture and systolic arrays

Level 4 — Synthesis

Recall Solution

KYA — N pick karo: $N = 65536 = 256$ . (Square yahan forced hai kyunki poora budget ek square array hai.) Peak: $2 \cdot 25 6^{2} \cdot (0.7 \times 1 0^{9}) = 2 \cdot 65536 \cdot 0.7 \times 1 0^{9} = 9.175 \times 1 0^{13} \approx 91.75 TOPS$ — bilkul TPUv1 ballpark. Utilisation for $M = 1024$ : fill $= 2 N - 1 = 511$ ; $\frac{1024}{1024 + 511} = \frac{1024}{1535} \approx 0.667 = 66.7% .$ Tension: ek fixed cell budget ke saath, bada $N$ peak throughput badhata hai (∝ $N^{2}$ ) lekin fill cost ( $2 N - 1$ ) bhi badhata hai, jo modest batches par utilisation hurt karta hai. $M = 1024$ par yeh 256-array sirf do-tihai efficient chalti hai — tum ya toh batch bada karte ya, agar latency-bound ho, chhoti array prefer karte. Koi free lunch nahi; tum peak ceiling aur fill overhead ke beech trade kar rahe ho. Iske general form ke liye Dataflow architectures dekho.

Recall Solution

KYA: array AI $= N /2 = 256/2 = 128$ MACs/byte. Balance point $50$ se compare karo: kyunki $128 > 50$ , tum compute-bound ho (ek achhi jagah — tum DRAM se nahi, silicon se limited ho). Margin: $128/50 = 2.56 \times$ balance point se aage. KYun: systolic layout ka poora reason AI ko itna high push karna hai ki memory wall clear ho sake; yahan tum use 2.56× se clear karte ho, toh zyada memory bandwidth add karna tumhe speed up nahi karega — sirf zyada/faster cells karengi.

Level 5 — Mastery

Recall Solution

KYA — build karo: ek batch mein useful MACs $= M \cdot N^{2}$ (har $M$ rows mein se ek sab $N^{2}$ cells ko ek baar drive karta hai). Total cycles $= (2 N - 1) + M$ , aur time $= cycles / f$ . Toh $delivered = \frac{2 \cdot M N ^{2}}{( 2 N - 1 + M ) / f} = 2 N^{2} f \cdot utilisation η \frac{M}{M + 2 N - 1} .$ KYun yeh shape hai: yeh exactly peak × utilisation hai. Jab $M \to \infty$ , $η \to 1$ aur delivered $\to 2 N^{2} f$ (ceiling). $M = 1$ par, $η = \frac{1}{2 N} \to$ tiny — single row fill amortise karne mein barely kamyab hota hai. Numbers: peak $= 91.75$ TOPS (4.1 se); $η = \frac{1024}{1024 + 511} = 0.6671$ ; $delivered = 91.75 \times 0.6671 \approx 61.2 TOPS .$ Padho ise: tumne 91.75 TOPS ka silicon kharida lekin 61.2 deliver kiya — missing third pipeline fill mein leak ho gaya. Woh gap hi L3/L4 lesson hai quantitatively.

Recall Solution

(a) Inference ke liye 8-bit kyun kaafi hai: ek 8-bit multiplier 32-bit float multiplier se kaafi chhota aur kam-energy wala hai, toh tum usi silicon mein zyada MAC cells fit karte ho → zyada $N^{2}$ → zyada peak. Inference sirf ek forward pass chahta hai, aur ek achhi tarah se quantised network 8-bit weights/activations ko negligible accuracy loss ke saath tolerate karta hai (dekho Quantization and 8-bit inference). Zyada cells + adequate precision = TPUv1 ka bet. (b) Training ke liye 8-bit kyun fail hota hai: training mein gradient descent use hota hai; gradients tiny ho sakte hain aur kai steps par accumulate hote hain. Unhe 8-bit integers mein round karna un chhoti values ko throw away karta hai jo learning signal carry karti hain — model converge karna band kar deta hai. Training ko floating point chahiye wide dynamic range ke saath (bfloat16 float32 ke 8 exponent bits rakhta hai), isliye TPU v2+ switch kiya. Yahi precision-vs-range tension hai jo quantization note develop karta hai. Synthesis: precision ek dial hai, constant nahi — tum minimum precision spend karo jo task tolerate kare aur savings se throughput kharido. Inference kam tolerate karta hai; training gradients ki rounding kaafi kam tolerate karta hai.