Wafer-scale engines (Cerebras-style) — Hinglish notes

6.5.17 · Hardware › Advanced & Emerging Architectures

YEH EXIST KYUN KARTA HAI? (Problem kya hai)

Intuition anchor karne ke liye key numbers (Cerebras WSE-2/3 class, order-of-magnitude):

Ek wafer ≈ 215 mm × 215 mm active silicon (~46,000 mm²).
~850,000 cores, ~40 GB on-chip SRAM (sab fast, distributed).
On-chip fabric bandwidth petabytes/second range mein — off-chip se orders of magnitude zyada.

WAFER-SCALE ENGINE KYA HOTA HAI?

Teen defining design choices:

Dataflow / weight-stationary compute — cores sparse tensor operations process karte hain; zeros ko skip kiya jaata hai hardware mein (sparsity harvesting).
Distributed SRAM, no DRAM — memory cores mein spread hoti hai taaki har core apni memory tak ~1 cycle mein pahunch sake. Bandwidth = (cores) × (per-core BW), jo astronomically high hoti hai.
2-D mesh NoC (network-on-chip) — nearest-neighbor links; messages core-to-core hop karte hain.

EK DINNER PLATE JITNI BADI CHIP KAISE BANATE HAIN? (First-principles obstacles)

Normal wafers ko chips mein dice kiya jaata hai kyunki teen cheezein buri tarah scale hoti hain:

1. Yield killer derive karna

Ek wafer ko defect density $D$ (defects per mm²) aur die area $A$ ke saath model karo. Classic Poisson yield model ke under, ek die tabhi achha hota hai jab usmein zero killing defects hon.

Ek die mein expected defects $= D \cdot A$ . Zero ki probability (Poisson):

$Y = e^{- D A}$

Fix (steel-manned): WSEs "zero defects" ki guarantee chodh dete hain aur iske bajaye redundancy build karte hain. Extra spare cores aur reroutable mesh links fabric ko dead cores ke aas-paas route karne dete hain. Effective yield ban jaata hai "enough good cores," naki "ek perfect wafer."

Agar $f$ fraction cores fail ho sakti hain aur humein $M$ banaye gaye cores mein se $N$ working cores chahiye, toh hum simply chahte hain:

$M (1 - f) \geq N \Rightarrow M \geq \frac{N}{1 - f}$

Toh tum cores ko $\frac{1}{1 - f}$ factor se over-provision karte ho.

2. Scribe lines cross karna

Cerebras fabrication ke dauran dicing streets par extra metal wiring add karta hai taaki normally-alag reticle fields ek connected mesh ban jayein. Yahi crucial trick hai jo "wafer par kai chips" ko "ek chip" mein badalta hai.

3. Power delivery & cooling

Power back se vertically deliver ki jaati hai (current wafer ke perpendicular) aur ek cold plate se direct water/liquid ke zariye cool ki jaati hai, jo lambe in-plane resistive drops se bachata hai aur thermal expansion ko special connector material se handle karta hai.

Figure — Wafer-scale engines (Cerebras-style)

YEH KYUN JEET TA HAI: ek bandwidth back-of-envelope

Weight-stationary sparse dataflow (COMPUTE KAISE HOTA HAI)

Common mistakes

Steel-man: wafer-scale galat choice kab hai?

Agar tumhara workload poor locality wala hai (all-to-all traffic, tiny models, ya aise bade datasets jo on-wafer SRAM mein fit nahi hote), toh mesh ka nearest-neighbor advantage gaayab ho jaata hai aur off-chip DRAM help karta. WSEs tab shine karte hain jab model + working set on-wafer fit ho jaate hain aur traffic mostly local/streaming hoti hai.

Recall Feynman: ek 12-saal ke bacche ko samjhao

Ek sheher imagine karo jahan har ghar ek tiny worker hai jiske paas apni notebook (memory) hai. Normally hum sheher ko alag-alag kasbon ke roop mein banate hain aur unke beech slow post se chithiyaan bhejte hain — bahut intezaar hota hai. Ek wafer-scale engine poore sheher ko ek hi block mein banata hai jahan pehle se har ghar ke beech raaste bane hote hain, taaki workers apne neighbors ko notes turant de sakein. Kuch ghar toote hue honge (wafer mein hamesha flaws hoti hain), isliye hum extra ghar banate hain aur roads ko toote hue gharon ke aas-paas draw karte hain. Kyunki kisi ko zyada door chithiyaan mail nahi karni, poora sheher giant homework problems (AI training) super fast solve karta hai.

Connections

Dennard Scaling & the Memory Wall — kyun off-chip bandwidth woh bottleneck ban gayi jis par WSEs attack karte hain.
Network-on-Chip (NoC) & Mesh Topologies — woh fabric jo WSE cores ko link karti hai.
Systolic Arrays & TPUs — ek alternative dataflow accelerator; dense vs sparse contrast karo.
Chiplets & 2.5D/3D Integration — opposite philosophy: kai chote dies vs ek huge die.
Yield & Defect Density Models — $Y = e^{- D A}$ theory jo yahan use ki gayi.
Sparsity in Neural Networks — kyun zero-skipping WSE speedups ke liye matter karta hai.
Liquid Cooling & Power Delivery Networks — ek wafer par ~15 kW ke liye enabling technology.

Flashcards

Wafer-scale engine kya hota hai?

Ek processor jo poore silicon wafer se ek die ke roop mein banaya jaata hai, kai chote cores ka 2-D mesh jisme har core ke paas local SRAM aur ek router hota hai, koi external DRAM nahi.

Tum simply ek defect-free full-wafer die fabricate kyun nahi kar sakte?

Yield

Y = e^{- D A}

~0 ho jaati hai jab area

A

poore wafer ke paas pahunchta hai; random defects ek perfect large die essentially impossible bana dete hain.

Zero-defect yield formula derive karo.

Defects Poisson hain mean

λ = D A

ke saath; 0 defects ki probability hai

P (0) = e^{- D A}

WSEs near-zero monolithic yield se kaise deal karte hain?

Redundant/spare cores plus reroutable mesh dead cores ke aas-paas route karta hai; cores ko

1/ (1 - f)

factor se over-provision karo.

N

cores chahiye aur failure fraction

f

hai, toh kitne banane honge?

M \geq N / (1 - f)

Reticle fields ko connect karne wali special fabrication trick kya hai?

Scribe (dicing) lines par extra metal wiring daali jaati hai taaki alag fields ek mesh ban jayein.

On-wafer aggregate bandwidth itni huge kyun hoti hai?

Saare mesh links parallel mein chalte hain, isliye

B_{t o t a l} = M \cdot B_{cor e}

— hundreds of TB/s to PB/s.

WSE mein model state kahan rehta hai (no DRAM)?

Saare cores ke distributed on-chip SRAM mein sharded; total capacity cores ka sum hoti hai.

WSE vs GPU mein memory model kya hai?

WSE = distributed-memory mesh (sirf SRAM); GPU = shared-memory SIMT with external HBM DRAM.

WSE mein power kaise deliver aur heat kaise remove hoti hai?

Power wafer ke back face se vertically feed hoti hai; cooling direct liquid/water cold plate se.

Sparsity WSE ki GPU se zyada help kyun karta hai?

WSE hardware multiply-by-zero skip karta hai (unstructured sparsity), jabki GPUs ko dense/structured math chahiye.

Wafer-scale galat choice kab hai?

Poor locality/all-to-all traffic wale workloads ya working sets jo on-wafer SRAM mein fit nahi hote.

"Weight-stationary" dataflow kya hota hai?

Har core apna weights ka slice locally rakhta hai jabki activations stream hoti hain, baar baar weight movement avoid hota hai.