Wafer-scale engines (Cerebras-style): Exercises — Hinglish deep dive

6.5.17 · D4 · Hardware › Advanced & Emerging Architectures › Wafer-scale engines (Cerebras-style)

Shuru karne se pehle, ek symbol contract taaki kuch bhi unexplained na lage:

Silicon wafer ko normally hundreds of chhote chips mein kata jaata hai. Ek sentence mein batao ki wafer-scale engine alag kya karta hai, aur woh ek trick ka naam lo jo ise physically possible banati hai.

Recall Solution 1.1

Kya: ek WSE poore wafer ko ek single chip ke roop mein rakhta hai use dice karne ki jagah, toh saare cores fast on-silicon wires se connected rehte hain. Enabling trick: scribe lines ke across extra metal wiring add karna (dicing streets), taki normally-alag reticle fields ek connected mesh ban jaayein. Kyun important hai: scribe lines ke across wiring ke bina tumhare paas sirf ek uncut wafer par bohot saari chips hoti — ek single chip nahi. Woh ek fabrication step hi "many chips" ko "one chip" mein turn karta hai.

Problem 1.2 (L1)

Har item ko woh boundary match karo jise ek data byte ko WSE ke andar cross nahi karna padta, jabki GPU cluster mein karna padta: (a) package pins, (b) PCB traces, (c) inter-server network link, (d) ek on-core register.

Recall Solution 1.2

Ek WSE traffic ko on silicon rakhta hai, toh yeh (a) package pins, (b) PCB traces, aur (c) inter-server network link se bachta hai. Yeh (d) se nahi bachta — ek on-core register core ke andar hota hai; har architecture woh use karta hai. Kyun: parent ka poora argument yahi hai ki har boundary (core→pins→PCB→network) on-chip wire se "slower, hotter, hungrier" hoti hai. WSE outer boundaries delete karta hai, sabse inner wali nahi.

L2 — Application

Problem 2.1 (L2)

Ek process mein defect density $D = 0.01$ defects/mm² hai. Poisson yield model $Y = e^{- D A}$ use karke, yield compute karo (a) ek chhote die $A = 100$ mm² ke liye aur (b) ek full wafer die $A = 46, 000$ mm² ke liye. Dono ko interpret karo.

Recall Solution 2.1

Yeh formula kyun: defects randomly aur independently land karte hain, isliye ek fixed area mein count Poisson hoti hai mean $λ = D A$ ke saath. Ek die sirf zero killing defects hone par kaam karti hai, aur $P (0) = e^{- λ} = e^{- D A}$ .

(a) Small die: $D A = 0.01 \times 100 = 1$ . $Y = e^{- 1} \approx 0.368$ Lagbhag $37%$ dies kaam karti hain — ek chhoti chip ke liye routine; baaki throw away kar dete hain.

(b) Full wafer die: $D A = 0.01 \times 46, 000 = 460$ . $Y = e^{- 460} \approx 1 0^{- 200}$ Yeh effectively zero hai. Ek monolithic defect-free wafer kabhi nahi hota — isliye exactly WSEs "zero defects" abandon karke redundancy use karte hain. Yeh kaisa dikhta hai: neeche yield-vs-area curve dekho — yeh wafer scale se bahut pehle collapse ho jaata hai.

Figure — Wafer-scale engines (Cerebras-style)

Problem 2.2 (L2)

Ek job ko $N = 800, 000$ working cores chahiye. Process mein expect hai ki built cores ka fraction $f = 0.05$ dead rahega. Kitne cores $M$ build karne padenge?

Recall Solution 2.2

Over-provision kyun: hum perfect wafer ki demand nahi kar sakte, isliye hum spares build karte hain aur dead cores ke around reroute karte hain. Hume $M$ built mein se kam se kam $N$ survivors chahiye: $M (1 - f) \geq N \Rightarrow M \geq \frac{N}{1 - f}$ Plug in: $M \geq \frac{800 , 000}{1 - 0.05} = \frac{800 , 000}{0.95} \approx 842, 105$ Toh ≈ 842,106 cores build karo (round up — core ka fraction build nahi ho sakta). Matlab: over-provision factor hai $\frac{1}{1 - f} = \frac{1}{0.95} \approx 1.0526$ — lagbhag $5.3%$ extra hardware tumhe $5%$ death rate tolerate karne ki capability deta hai.

L3 — Analysis

Problem 3.1 (L3)

$N = 800, 000$ working cores deliver karne ke do options hain. Option A: over-provision ke saath ek wafer (2.2 se, $M_{A} \approx 842, 106$ build karne padte hain). Option B: wafer ko $A = 100$ mm² ke chhote dies mein dice karo yield $Y = 0.368$ par (2.1a se), har die mein $c = 1000$ cores hain. Option B mein $800, 000$ working cores worth yield karne ke liye kitne chhote dies build karne padenge, aur dono options mein built silicon ka kitna fraction waste hoga?

Recall Solution 3.1

Option A (wafer): built $M_{A} \approx 842, 106$ , needed $800, 000$ . Wasted-core fraction: $\frac{842 , 106 - 800 , 000}{842 , 106} \approx 0.05 = 5%.$

Option B (diced): built die ke per working cores $= c \cdot Y$ ? Nahi — dhyan se. Ek die ya toh poori good hoti hai ya poori throw away hoti hai (ek killing defect poori die ko kill karta hai). Toh ek good die $1000$ cores deti hai; ek bad die $0$ deti hai. Dies ka good fraction $= Y = 0.368$ . Dies jo build karni padein: $good dies needed = \frac{800 , 000}{1000} = 800 dies,$ $built dies = \frac{800}{Y} = \frac{800}{0.368} \approx 2174 dies .$ Wasted-die fraction $= 1 - Y = 0.632 = 63.2%$ .

Analysis: Option A ~ $5%$ silicon waste karta hai; Option B ~ $63%$ . Gap kyun: ek diced flow mein ek single defect ek saath $1000$ cores discard kar deta hai. Wafer flow mein ek defect sirf ek core ko maarta hai jo uspar land karta hai, kyunki mesh uske around reroute karta hai. Fine-grained redundancy bahut zyaada silicon-efficient hai — yahi deep reason hai ki wafer-scale wasteful lagte hue bhi pay off kar sakta hai.

Problem 3.2 (L3)

$1$ GHz par ek $1000 \times 1000$ mesh par ( $1$ hop = $1$ ns), do cores ke beech ek message ke liye average-case aur worst-case latency compare karo. Maano traffic average par nearest-neighbour hai.

Recall Solution 3.2

Hop-count = latency kyun: har core-to-core link ek clock hai, toh latency ≈ (number of hops) × (cycle time). $1$ GHz par, cycle $= 1$ ns.

Neighbour (average local traffic): $1$ hop $= 1$ ns.
Worst case, corner to corner: ek mesh mein tum rows phir columns ke along travel karte ho, toh hops $= (1000 - 1) + (1000 - 1) = 1998 \approx 2000$ hops $= 2000$ ns $= 2 μ s$ .

Yeh kaisa dikhta hai: neeche red path corner-to-corner route hai; almost sab real traffic chhoti local hops use karta hai. Interpretation: even the worst on-wafer case (~ $2 μ$ s) best inter-server network case se match karta hai, aur typical case ( $1$ ns) usse ~1000× beat karta hai. Locality sab kuch hai.

L4 — Synthesis

Problem 4.1 (L4)

Ek WSE mein $M = 850, 000$ cores hain, har ek $B_{cor e} = 1$ GB/s neighbours ko push karta hai. (a) Total on-wafer bandwidth compute karo. (b) Ek GPU cluster link $B_{g p u} = 900$ GB/s per link tak jaata hai. WSE ke aggregate ke barabar kitne GPU links lagenge? (c) Ek sentence mein batao ki mesh par aggregate bandwidth additive kyun hoti hai?

Recall Solution 4.1

(a) Sum kyun: saare mesh links parallel mein chalte hain, isliye aggregate bandwidth add hoti hai: $B_{t o t a l} = M \cdot B_{cor e} = 850, 000 \times 1 GB/s = 8.5 \times 1 0^{5} GB/s = 850 TB/s .$

(b) Equivalent GPU links: $\frac{B _{t o t a l}}{B _{g p u}} = \frac{8.5 \times 1 0 ^{5} GB/s}{900 GB/s} \approx 944 links .$ Toh ek WSE ka internal fabric ≈ ~944 top-end GPU links ki bandwidth ke barabar hai, sab ek piece of silicon par.

(c) Additive kyun: kyunki links physically alag wires hain jo simultaneously operate kar rahe hain — koi shared bottleneck channel nahi — isliye unki capacities stack hoti hain compete karne ki jagah.

Problem 4.2 (L4)

Ek model mein $40$ GB parameters hain. Har core $S = 48$ KB SRAM hold karta hai. (a) Kya yeh fit hoga? (b) Sirf ise store karne ke liye minimum kitne cores chahiye, compute ignore karke? (c) "Model ek core mein fit hona chahiye" intuition galat kyun hai?

Recall Solution 4.2

(a)/(b) Aggregate kyun: capacity saare cores ka sum hai, per core nahi. $40$ GB store karne ke liye minimum cores: $cores = \frac{40 GB}{48 KB} = \frac{40 \times 1 0 ^{9} bytes}{48 \times 1 0 ^{3} bytes} \approx 833, 333 cores .$ $\sim 850, 000$ cores available hone ke saath, yeh thodi jagah bachi ke saath fit ho jaata hai — exactly isliye parent ke " $40$ GB on-chip SRAM" aur " $850, 000$ cores" numbers design se match hote hain.

(c) Intuition galat kyun hai: model saare cores mein sharded hota hai; koi single core kabhi use hold nahi karta. Mesh partial results ko together stitch karta hai, isliye relevant capacity distributed total hai, na ki chhota per-core SRAM.

L5 — Mastery

Problem 5.1 (L5)

Design decision: tumhara model ReLU ke baad $60%$ zero activations rakhta hai (sparsity $s = 0.6$ ). Dense hardware par har multiply chalta hai; ek WSE par, multiply-by-zero skip hota hai. (a) Sparsity se ideal speedup kya hai? (b) Agar skipping remaining work par $20%$ overhead leta hai (nonzeros indexing), toh effective speedup kya hai? (c) Batao ki yeh hardware advantage kab disappear hota hai.

Recall Solution 5.1

(a) Zeros skip kyun: $0$ se multiply karna $0$ produce karta hai aur time aur energy waste karta hai. Agar activations ka fraction $s$ zero hai, toh sirf $(1 - s)$ kaam bachta hai. Dense ke against ideal speedup: $speedup_{i d e a l} = \frac{1}{1 - s} = \frac{1}{1 - 0.6} = \frac{1}{0.4} = 2.5 \times .$

(b) Overhead ke saath: nonzero work hai $(1 - s)$ , $20%$ indexing cost se inflate hoke, yaani $\times 1.2$ . Effective work fraction $= (1 - s) \times 1.2 = 0.4 \times 1.2 = 0.48$ . $speedup_{e f f} = \frac{1}{0.48} \approx 2.083 \times .$ Toh overhead $2.5 \times$ ko $2.08 \times$ tak trim karta hai — phir bhi ek real win.

(c) Kab disappear hota hai: agar sparsity low hai (kam zeros), $\frac{1}{1 - s} \to 1$ aur overhead ise even slower bana sakta hai. Agar sparsity unstructured hai lekin hardware ko structured patterns chahiye the, toh skipping machinery stall ho jaati hai. WSEs unstructured sparsity par shine karte hain, jise dense GPUs exploit karna mushkil paate hain — dekho Sparsity in Neural Networks.

Problem 5.2 (L5)

Apne khilaf steel-man: ek aisa workload propose karo jahan WSE galat choice hai, aur mesh se tied bandwidth/locality argument ke saath justify karo.

Recall Solution 5.2

Ek bad-fit workload: ek tiny model with ek huge streaming dataset jo all-to-all communication karta hai (jaise kuch graph/embedding-lookup jobs jahan har core ko har step arbitrary door ka data reach karna padta hai). Mesh yahan kyun haarta hai:

Mesh ka advantage nearest-neighbour locality hai — local hops ~1 ns hote hain. All-to-all average paths ko corner-to-corner ~ $2 μ$ s ke paas force karta hai, locality win erase karta hai (Problem 3.2).
"No external DRAM" fatal hai agar dataset ~ $40$ GB on-wafer SRAM se zyaada ho: ab tum off-wafer stream karte ho, wahi interconnect wall hit karte hue jo WSE avoid karne ke liye bana tha. Conclusion: WSEs jab jeette hain jab model + working set on-wafer fit ho aur traffic mostly local/streaming ho. Koi bhi ek assumption todho aur off-chip DRAM plus conventional cluster use use beat kar sakta hai.

Recall Ek-line self-test

Wafer-scale over-provision factor $\frac{1}{1 - f}$ tiny (~ $5%$ ) kyun hai jabki same defect density par ek diced flow ~ $63%$ waste karta hai? ::: Kyunki wafer flow ki redundancy ki unit ek single core hai (ek defect ek core ko maarta hai, mesh reroute karta hai), jabki diced flow ki unit ek whole die hai (ek defect ~1000 cores scrap karta hai).

Exercises — Wafer-scale engines (Cerebras-style)

L1 — Recognition

Problem 1.1 (L1)