6.5.7 · D1Google TPU architecture and systolic arrays
Intuition Is poore topic ke peeche ek hi idea haiEk neural network mostly ek hi operation karta hai, billions of times: do numbers ko multiply karo aur result ko ek running total mein add karo . TPU isliye jeetta hai kyunki wo hazaaron tiny multiply-add circuits ko ek grid mein arrange karta hai, taaki har number, ek baar memory se fetch hone ke baad, poore grid mein flow karta rahe aur reuse hota rahe — har single multiply ke liye dobara fetch karne ki zaroorat na pade.
Is page par assume kiya gaya hai ki aap kuch nahi jaante. Har letter, arrow, aur word jis par parent topic depend karta hai, yahan build kiya gaya hai — ek order mein, har ek apni jagah earn karta hai pehle, phir agla usse use karta hai.
Kisi bhi formula se pehle, data ki teen shapes fix karo.
Definition Scalar, vector, matrix
Ek scalar ek single number hota hai, jaise 7
Ek vector numbers ki ek ordered list hoti hai, jaise ( 1 , 2 , 3 )
Ek matrix numbers ka ek rectangle hota hai, rows aur columns mein arrange kiya hua. Picture: dots ka ek grid.
Topic ko ye kyun chahiye: ek neural network layer apni seekhi hui knowledge ko weights ki ek matrix mein store karti hai, aur us se flow hone wala data activations ke vectors ke roop mein hota hai. TPU jo bhi karta hai wo vectors ko matrices ke through push karna hai.
Definition Multiply ka symbol× a × b seedha multiplication — wahi × 3 × 4 = 12 A × B matrix multiply jo §3 mein define kiya gaya hai — in choti multiplications ka ek poora grid ek saath wired. Ek hi symbol, lekin jab dono sides matrices hon tab use "matrix multiply" padho. Jab hum do numbers multiply karte hain, hum unhe side by side bhi likh sakte hain, jaise a p w p ×
Definition Subscript notationA ij A ij A i j hamesha row hota hai, doosra hamesha column . Toh A 23
Padho Question ::: Answer:
A 31 Matrix A
Topic ko ye kyun chahiye: parent core formula ko C ij = ∑ p A i p B p j i j p
Definition Multiply–accumulate (MAC)Ek MAC do-step ka move hai: multiply karo do numbers ko, phir product ko ek running total ("accumulator") mein add karo. Accumulator ko zero se shuru karo, phir repeat karo:
acc ← acc + a × w ( acc starts at 0 ) ← acc acc 0 k k
Picture: ek khali jar jis par acc a × w
Intuition "Multiply then add" kyun, kuch fancier kyun nahi?Kyunki yahi literally dot product hai (agla section), aur dot product wahi hai jo matrix multiply ki har entry hai. Neural networks almost entirely matrix multiplies se bani hoti hain, toh ye ek tiny operation 95%+ kaam hai. Atom ko optimize karo, war jeeto.
Definition FLOP, TOPS, aur "peak"Ek FLOP ek fl oating-point op eration hai — ek multiply, ya ek add, ek FLOP count hota hai. Toh ek single MAC = 1 multiply + 1 add = 2 FLOPs . "FLOP/s" ka matlab hai FLOPs per second — ek raw speed.
TOPS ka matlab hai T era-OP erations per S econd (1 0 12 Peak throughput (short mein "peak") best-case speed hai: chip kitne FLOPs/s karta hai agar har ek arithmetic unit har clock tick par busy ho . Ye ek ceiling hai, promise nahi.
Topic ko ye kyun chahiye: parent ke throughput formula Peak = 2 N 2 f N f 2
Do equal-length lists lo a = ( a 1 , a 2 , … , a k ) w = ( w 1 , … , w k ) dot product hai: matching entries multiply karo, sab add kar lo.
a ⋅ w = a 1 w 1 + a 2 w 2 + ⋯ + a k w k = ∑ p = 1 k a p w p
Ab us formula mein strange symbols:
Definition Summation sign∑ p = 1 ∑ k p 1 k loop hai jo ek symbol mein likha gaya hai . p
Picture: k a p w p ∑
∑ p = 1 k Ye sahi kyun lagta hai: bada Greek symbol intimidating lagta hai, jaise zaroor kuch exotic kar raha hoga.
Fix: ∑ S um = add hota hai. Iska cousin ∏
Topic ko ye kyun chahiye: poora matrix-multiply formula har output cell ke liye ek dot product hai. Systolic array kuch nahi balki ek machine hai jo bahut saare dot products parallel mein compute kare.
Ab hum parent ki central equation padh sakte hain.
Ek picture ke saath indices trace karo:
i A j B p saath saath chalta hai, har baar jo pair milta hai use multiply karta hai aur running sum mein add karta hai.
m × k k × n inner dimensions match karein : A k B k k outer dimensions m × n
Padho:
A B Har output ek row of A B
Worked example Formula ko ek2 × 2 A = ( 1 3 2 4 ) B = ( 5 7 6 8 ) C 11 = A 11 B 11 + A 12 B 21 = 1 ⋅ 5 + 2 ⋅ 7 = 19 C 12 = 1 ⋅ 6 + 2 ⋅ 8 = 22 C 21 = 3 ⋅ 5 + 4 ⋅ 7 = 43 C 22 = 3 ⋅ 6 + 4 ⋅ 8 = 50 space mein, ek saath karta hai, ek ke baad ek ki jagah.
Topic ko ye kyun chahiye: ye equation hi workload hai. Baaki sab kuch — array, streaming, fill cost — is ek formula ko saste mein run karne ke liye exist karta hai.
Multiplications sasti hain. Mushkil numbers fetch karne mein hai.
Definition Byte, aur ek memory read kya move karta haiEk byte computer data ki standard small unit hai — ise ek "box" samjho jo ek number rakhta hai (ek 8-bit integer number, wali type jo TPUv1 use karta hai, exactly ek byte mein fit hoti hai). Jab hum kahte hain ek value "memory se fetch" ki gayi, ek fetch us ek value ke byte(s) ko compute unit mein le jaata hai. Is page par counting simple rakhne ke liye, maano ek number = ek read = bytes ki ek fixed chhoti sankhya , toh "reads" aur "bytes" lockstep mein badhte hain.
Definition Memory read, aur memory wallEk memory read ek value ko memory (DRAM/SRAM) se compute unit mein fetch karne ka act hai. Iska cost multiply se kahin zyada time aur energy lagti hai. Jab ek chip apna zyaadatar time compute karne ki jagah data ka intezaar karne mein lagaaye, toh usne memory wall hit kar liya — dekhein Memory wall and arithmetic intensity .
Definition Arithmetic intensityArithmetic intensity ye ratio hai:
intensity = number of values fetched from memory number of arithmetic operations done . Dono top aur bottom ko ek hi unit mein measure karo — yahan, values ka count (equivalently bytes, kyunki humne fix kiya ek value = bytes ki ek fixed sankhya). High intensity = har fetched value par bahut saara math = memory wall kam hurts karta hai. Low intensity = aap starve karte ho, memory ka intezaar karte hue.
Picture: ek factory (compute) jo ek single narrow conveyor belt (memory) se feed hoti hai. Agar har delivered part kai operations mein use hoti hai, toh factory busy rehti hai. Agar har part ek baar use hoke throw away ho jaaye, toh factory belt ka intezaar karte hue idle rahti hai.
Topic ko ye kyun chahiye: systolic array ka poora justification hai "har number ko ek baar read karo, use N Roofline model apne x-axis par plot karta hai. Parent ka claim "arithmetic intensity = n /2
Definition Big-O notationO ( ⋅ ) O ( n 3 ) n 3 n coarse growth label hai, exact count nahi.
n 3 2 n 2 n × n n 2 n n n 2 × n = n 3 O ( n 3 ) naïve machine, ek baar mein ek MAC karte hue, har MAC ke liye memory se apne do operands re-read karti hai, toh ye roughly n 3 Systolic array iske bajaye har input value ko ek baar load karta hai: A n 2 B n 2 2 n 2 n 3 2 n 2
Definition Clock aur cycleEk chip ka clock ek metronome hai jo billions of times per second tick karta hai. Har tick ek cycle hai (ya "clock"). Hardware har cycle mein ek fixed chhoti amount kaam karta hai. Clock frequency f 700 MHz = 700 × 1 0 6 side length N N × N
N f N × N N 2 f = 2 N 2 f
Definition Pipeline fill / drainJab data ko pehla answer bahar aane se pehle ek grid mein travel karna padta hai, early cycles koi output nahi produce karte — pipe abhi bhar raha hai. Isi tarah last data ko bahar drain hone mein cycles lagte hain. Ye fixed startup cost ki wajah se parent kehta hai ki chhote jobs wasteful hain aur bade batches king hain.
Picture: ek garden hose. Tap on karo aur paani instantly nahi nikalta — pehle use hose ki poori length travel karni padti hai (fill). Agar aap sirf ek cup chahte the toh wo travel time waste hai.
Ek batch ek saath process kiye jaane wale input examples ka group hai — jaise ek baar mein ek ki jagah 64 images network mein push ki jaayein. Hamare grid mein, ek batch kaafi saari rows of activations hain jo back-to-back array mein stream hoti hain.
Padho:
Systolic arrays bade batches kyun prefer karte hain? Fixed fill/drain cost (≈ 2 N − 1
Definition Inference vs. trainingTraining wo phase hai jahan ek network example data se apne weights seekhta hai, unhe baar baar adjust karta hai — tiny corrections (gradients) ke liye higher-precision math ki zaroorat hoti hai. Inference wo phase hai jahan ek already-trained network ko naye inputs par predictions karne ke liye use kiya jaata hai — bas forward matrix multiplies, low precision tolerant. TPUv1 sirf inference karta tha; training TPU v2+ ke saath aayi.
Definition Cache aur branch prediction (jo TPU throw away karta hai)Ek cache ek small fast memory hoti hai jo ek normal CPU apne compute ke paas rakhta hai, recently-used values hold karta hai taaki har baar slow DRAM mein na jaana pade. Branch prediction ek CPU trick hai jo guess karta hai ki ek "if" kis taraf jaayega taaki wo pata lagane ka intezaar kiye bina kaam karta rahe. Dono general code ko faster banate hain lekin chip area aur complexity cost karte hain. TPU dono omit karta hai — uska dataflow fixed aur predictable hai, toh wo bachaya hua area zyaada MAC units mein kharach karta hai.
Definition ASIC aur domain-specific acceleratorEk ASIC (Application-Specific Integrated Circuit) ek aisa chip hai jo ek kaam ke liye hard-wired hota hai, bina kisi general-purpose flexibility ke. Ek domain-specific accelerator ek aisa ASIC hai jo ek tarah ke workload ke liye aimed hai (yahan: neural-net matrix multiplies). Isse CPU/GPU se compare karo ASIC vs FPGA vs general-purpose processors mein. TPU flexibility trade karta hai efficiency ke liye — no caches, no branch prediction, sirf arithmetic.
Definition Processing element (PE)Ek PE array mein ek tiny circuit hai jo ek weight hold karta hai aur har cycle mein ek single MAC karta hai, phir data apne neighbour ko pass karta hai. Array identical PEs ka ek N × N N = 256 256 × 256 = 65 , 536
Definition Quantization / 8-bit8-bit integer math full-precision decimals ki jagah tiny range mein whole numbers use karta hai, jo har MAC circuit ko chhota aur faster banata hai. Ye quantization hai; dekhein Quantization and 8-bit inference . TPUv1 ne ise isliye use kiya kyunki inference low precision tolerate karta hai — isliye bhi TPUv1 train nahi kar sakta tha (training ke gradients ko zyaada precision chahiye).
Neeche wala diagram ek dependency map hai: har arrow →
Har foundation agla feed karta hai; saath mein ye systolic array aur TPU par pahunchte hain.
Khud ko test karo — sirf apne answer ke baad reveal karo.
Kya main A ij Haan — i j
Kya main symbol × Numbers ke beech ye plain multiplication hai; matrices ke beech ye §3 ka matrix multiply hai.
Kya main ek dot product ko products ke sum ke roop mein state kar sakta hoon? a ⋅ w = ∑ p = 1 k a p w p
Kya main jaanta hoon ∑ Sigma = S um; multiply symbol ∏
Kya main matrix product ki ek entry C ij Ye A i B j
Kya main jaanta hoon matrix shapes kyun match karni chahiye (k = k Shared dimension har dot product ki length hai; dono lists equally long honi chahiye.
Kya main bol sakta hoon ek MAC kya hai, uska starting value kya hai, aur ye kitne FLOPs hai? Accumulator 0
Kya main jaanta hoon FLOP/s, TOPS, aur "peak" ka matlab? FLOP/s = float ops per second; TOPS = trillion integer ops per second; peak = best-case speed jab har unit busy ho.
Kya main arithmetic intensity ek consistent unit mein define kar sakta hoon? Memory se fetch ki gayi har value par kitne operations hue (top aur bottom dono values/bytes mein count kiye).
Kya main jaanta hoon n 3 2 n 2 n 2 n n 3 2 n 2 A B
Kya main bol sakta hoon ek byte kya hai? Standard small data unit; ek 8-bit number ek byte mein fit hota hai.
Kya main jaanta hoon clock cycle, fill/drain, aur batch kya hain? Ek cycle ek tick hai; fill/drain startup cost hai jab data array cross karta hai; ek batch bahut saare inputs ek saath process hote hain.
Kya main explain kar sakta hoon TPU caches aur branch prediction kyun drop karta hai? Uska dataflow fixed aur predictable hai, toh wo bachaya hua chip area zyaada MAC units mein lagata hai.
Kya main inference aur training ke beech difference explain kar sakta hoon? Training weights seekhti hai (precision chahiye); inference ek trained network use karta hai (low precision tolerate hoti hai).
