Google TPU architecture and systolic arrays: Question bank

6.5.7 · D5 · Hardware › Advanced & Emerging Architectures › Google TPU architecture and systolic arrays

Yahan har word zero se build hua hai: ek MAC ek multiply-accumulate hota hai — do numbers lo, multiply karo, aur result ko ek running total mein add karo jise accumulator kehte hain. Ek PE (processing element) ek chhoti si circuit hai jo exactly ek MAC per clock tick karti hai. Ek systolic array inhi PEs ki ek grid hoti hai jo sirf apne immediate neighbours se wired hoti hai, jiske through data step-by-step flow karta hai jaise ek bucket brigade. Ye teen pictures dimag mein rakho.

True or false — justify

True or false: Ek systolic array mein har input operand ko har us MAC ke liye off-chip memory se fetch kiya jaata hai jo use karta hai.

False. Har operand memory se sirf ek baar read hota hai, phir cell-to-cell march karta hai (a_out = a_in); wo bina kisi aur memory access ke kaafi saare PEs ke liye reuse hota hai. Ye single-read-many-uses trick hi "systolic" ka poora point hai.

True or false: Ek systolic array ko ek global bus ki zaroorat hoti hai jo har PE ko shared memory se connect kare.

False. PEs sirf nearest neighbours se baat karte hain — chhote local wires. Global bus avoid karna hi woh cheez hai jo clock fast chalane aur energy low rakhne deti hai.

True or false: TPUv1 neural networks train kar sakta tha.

False. TPUv1 sirf inference-only tha 8-bit integer MACs ke saath. Training ke liye gradients ke liye floating-point range chahiye, jo TPU v2+ ke saath bfloat16 use karke aaya.

True or false: Ek

N \times N

MAC array ki peak throughput

N

ke saath linearly badhti hai.

False. Ye

N^{2}

ke saath badhti hai:

N^{2}

MAC cells hain, sab har clock busy hain, isliye

Peak = 2 N^{2} f

. Parallelism yahan spatial hai, isliye side double karne se compute chaar guna ho jaata hai.

True or false: "Systole" in systolic ek memory hierarchy ko refer karta hai.

False. Ye systole se aaya hai, dil ki contraction — data har clock rhythmically array ke through pump hota hai, jaise dil mein khoon. Ye ek dataflow metaphor hai, memory wala nahi.

True or false: Zyada arithmetic intensity ka matlab hai compute ki har unit ke liye kam memory accesses.

True. Arithmetic intensity

\frac{MACs}{bytes read}

hai; ise badhane ka matlab hai har byte zyada kaam karta hai, jisse tum memory wall ke paar chadh jaate ho.

True or false: Ek single chhota matrix multiply ek systolic array pe peak efficiency pe run karta hai.

False. Har op ek fixed pipeline fill/drain cost

\approx 2 N - 1

cycles ka deta hai. Ek chhota op un cycles ko waste karta hai; array sirf large batched kaam pe high efficiency tak pahunchta hai jahan fill amortize hota hai.

True or false: Weight-stationary dataflow mein weights array ke through move karte hain jabki activations jagah pe rehte hain.

False. Ulta hota hai — weights jagah pe rehte hain (stationary) har cell mein, aur activations stream karte hain across jabki partial sums accumulate hote hain. Isliye ise weight-stationary kehte hain.

Spot the error

Error hunt: "TPU throughput = clock × number of cores, bilkul CPU ki tarah."

TPU mein

N^{2}

MAC cells hain, na ki muthi bhar cores, isliye throughput

= 2 N^{2} f

— ye array side ke square ke saath scale karta hai. CPU-style linear-in-cores reasoning isse bahut underestimate karti hai.

Error hunt: "Hum inputs ko stagger (skew) karte hain taaki array diagram pe accha dikhe."

Skewing functional hai, cosmetic nahi: data ko alag-alag cells tak pahunchne mein alag cycles lagte hain, isliye har row ko ek clock delay kiya jaata hai taaki sahi operands sahi time pe sahi PE pe milein.

Error hunt: "TPU mein matmuls speed up karne ke liye caches aur branch prediction hain."

TPUv1 mein jaanbujhkar no cache, no branch predictor, no out-of-order logic hai. Woh saari die area arithmetic ko jaati hai — yahi domain-specific ASIC ka bet hai.

Error hunt: "Ek

N \times N

multiply ke liye, latency sirf

M

rows ke liye

M

cycles hai."

Tum fill/drain bhool gaye: total

\approx (2 N - 1) + M

cycles. Pehla result tab tak nahi aata jab tak data poore array mein diagonally propagate na ho jaaye.

Error hunt: "Systolic array ka reuse factor size se independent

2

pe fixed hai."

Reuse factor

\approx N

hota hai, array dimension — ek deeper array har operand ko zyada reuse karta hai. Isliye ek

256 \times 256

array roughly

256 \times

memory traffic reduce karta hai ek naive read ke comparison mein.

Error hunt: "Unified Buffer har PE ke andar ek chhota register file hai."

Unified Buffer ek bada (24 MB) on-chip SRAM hai jo poore array ke liye activations aur results hold karta hai, use feed karta hai aur outputs pakadta hai taaki DRAM round-trips na hon. Per-PE storage sirf woh tiny accumulator hai plus stored weight.

Error hunt: "TPU basically ek fast general-purpose CPU hai."

Ye ek domain-specific ASIC hai jo sirf matrix/vector ops acche se karta hai. Iski speed precisely not being general se aati hai — no OOO, no branch prediction, no caches, sab kuch MACs pe spend.

Why questions

Why does the systolic array memory wall break karta hai jabki ek naive CPU loop nahi?

CPU har MAC ke liye operands memory se re-fetch karta hai (data movement dominate karta hai); array har operand ko ek baar read karta hai aur poori row/column mein reuse karta hai, isliye wo fetch kiye gaye har byte ke liye kaafi zyada compute karta hai.

Systolic arrays "large batches se pyaar" kyun karte hain?

Fixed fill/drain cost

\approx 2 N - 1

ek baar pay hoti hai batch size se independent, isliye ek bada batch

M ≫ N

us overhead ko phaila deta hai — efficiency

\frac{M}{M + 2 N}

1 ke paas jaati hai.

n \times n

systolic multiply ki arithmetic intensity

n /2

ke aas-paas kyun hoti hai?

n^{3}

MACs perform karta hai lekin sirf

2 n^{2}

input values ek baar each read karta hai, isliye intensity

= \frac{n ^{3}}{2 n ^{2}} = \frac{n}{2}

— reuse directly array depth ke saath badhta hai.

Har cell apna activation a_out = a_in ke saath forward kyun copy karta hai?

Woh copy hi reuse mechanism hai — wahi activation, memory se ek baar read ki gayi, agले cell ko hand off ki jaati hai taaki wahan phir use ho sake, poori row mein one-read-many-uses achieve karte hue.

TPUv1 ne floating point ki jagah 8-bit integers kyun use kiye?

Inference low precision tolerate karta hai quantization ke zariye, aur 8-bit MACs area aur energy mein kaafi saste hain — jo tumhe per chip kaafi zyada MAC cells pack karne dete hain. Training ke gradients ko zyada range chahiye, isliye baad mein floats aaye.

Systolic array sirf nearest neighbours se kyun wired hota hai instead of shared bus ke?

Local wires chhote hote hain, isliye signals fast aur low energy pe carry karte hain aur clock high run karne dete hain; ek global bus lamba, slow, power-hungry, aur ek bandwidth bottleneck hota.

Edge cases

Edge case: Array ki throughput pehle

2 N - 1

cycles mein kya hoti hai?

Peak se neeche — pipeline fill ho rahi hai, isliye abhi saare

N^{2}

cells mein valid operands nahi hain. Full

2 N^{2} f

throughput sirf steady state mein fill ke baad milti hai.

Edge case: Kya hota hai agar matrix array se chhoti ho (maano

8 \times 8

256 \times 256

array ko feed ki gayi)?

Zyattar PEs idle baithte hain, isliye efficiency bahut kharab hoti hai; tum phir bhi bada fill cost pay karte ho almost koi useful kaam kiye bina. Chhote ops exactly wahan hain jahan systolic arrays sabse kam efficient hote hain.

Edge case: Agar tum exactly

M = 1

row ka batch feed karo, toh cycle count mein kya dominate karta hai?

Fill/drain term

2 N - 1

us single streamed row ko completely dominate karta hai, isliye almost saare cycles overhead hain — classic latency-bound worst case.

Edge case: Kya performance limit karta hai jab layer memory-bandwidth-bound ho rather than compute-bound?

Tum roofline ke sloped part pe ho: arithmetic intensity itni kam hai ki MACs fed nahi reh sakte, isliye DRAM bandwidth, na ki

2 N^{2} f

peak, achievable rate set karta hai.

Edge case: Kya clock

f

double karna bina limit ke throughput double karta hai?

Sirf power/thermal aur wire-delay limits tak; formula

2 N^{2} f

f

mein linear hai, lekin physically local-wire timing aur heat cap karte hain ki

f

kitna high ja sakta hai, isliye designers

N

bhi badhate hain.

Recall Memory se do master formulas rebuild karo

$N \times N$ array ki peak throughput ::: $2 N^{2} f$ FLOP/s ( $N^{2}$ cells $\times$ 2 FLOP/MAC $\times f$ ). $M$ rows push karne ke cycles ::: $\approx (2 N - 1) + M$ (fill/drain plus stream).