Wafer-scale engines (Cerebras-style): Exercises — Hardware deep dive

Before we start, one symbol contract so nothing appears un-earned:

L1 — Recognition

Problem 1.1 (L1)

A silicon wafer is normally cut into hundreds of small chips. State, in one sentence, what a wafer-scale engine does differently, and name the single trick that makes it physically possible.

Recall Solution 1.1

What: A WSE keeps the entire wafer as one single chip instead of dicing it, so all cores stay connected by fast on-silicon wires. The enabling trick: adding extra metal wiring across the scribe lines (the dicing streets), so the normally-separate reticle fields become one connected mesh. Why it matters: without wiring across scribe lines you would just have many chips sitting on one uncut wafer — not a single chip. That one fabrication step is what turns "many chips" into "one chip."

Problem 1.2 (L1)

Match each item to the boundary a data byte does not have to cross inside a WSE that it would cross in a GPU cluster: (a) package pins, (b) PCB traces, (c) inter-server network link, (d) an on-core register.

Recall Solution 1.2

A WSE keeps traffic on silicon, so it avoids (a) package pins, (b) PCB traces, and (c) the inter-server network link. It does not avoid (d) — an on-core register is inside the core; every architecture uses those. Why: the parent's whole argument is that each boundary (core→pins→PCB→network) is "slower, hotter, hungrier" than on-chip wire. The WSE deletes the outer boundaries, not the innermost one.

L2 — Application

Problem 2.1 (L2)

A process has defect density $D = 0.01$ defects/mm². Using the Poisson yield model $Y = e^{- D A}$ , compute the yield for (a) a small die $A = 100$ mm² and (b) a full wafer die $A = 46, 000$ mm². Interpret each.

Recall Solution 2.1

Why this formula: defects land randomly and independently, so the count in a fixed area is Poisson with mean $λ = D A$ . A die works only with zero killing defects, and $P (0) = e^{- λ} = e^{- D A}$ .

(a) Small die: $D A = 0.01 \times 100 = 1$ . $Y = e^{- 1} \approx 0.368$ About $37%$ of these dies work — routine for a small chip; you throw away the rest.

(b) Full wafer die: $D A = 0.01 \times 46, 000 = 460$ . $Y = e^{- 460} \approx 1 0^{- 200}$ This is effectively zero. A monolithic defect-free wafer never happens — which is exactly why WSEs abandon "zero defects" and use redundancy instead. What it looks like: see the yield-vs-area curve below — it collapses long before wafer scale.

Figure — Wafer-scale engines (Cerebras-style)

Problem 2.2 (L2)

A job needs $N = 800, 000$ working cores. The process is expected to leave a fraction $f = 0.05$ of built cores dead. How many cores $M$ must be built?

Recall Solution 2.2

Why over-provision: we cannot demand a perfect wafer, so we build spares and reroute around dead cores. We need at least $N$ survivors out of $M$ built: $M (1 - f) \geq N \Rightarrow M \geq \frac{N}{1 - f}$ Plug in: $M \geq \frac{800 , 000}{1 - 0.05} = \frac{800 , 000}{0.95} \approx 842, 105$ So build ≈ 842,106 cores (round up — you can't build a fraction of a core). What it means: the over-provision factor is $\frac{1}{1 - f} = \frac{1}{0.95} \approx 1.0526$ — about $5.3%$ extra hardware buys you tolerance to a $5%$ death rate.

L3 — Analysis

Problem 3.1 (L3)

Two options deliver $N = 800, 000$ working cores. Option A: one wafer with over-provision (from 2.2, needs $M_{A} \approx 842, 106$ built). Option B: dice the wafer into small dies of $A = 100$ mm² at yield $Y = 0.368$ (from 2.1a), each containing $c = 1000$ cores. How many small dies must be built in Option B to yield $800, 000$ working cores' worth, and what fraction of built silicon is wasted in each option?

Recall Solution 3.1

Option A (wafer): built $M_{A} \approx 842, 106$ , needed $800, 000$ . Wasted-core fraction: $\frac{842 , 106 - 800 , 000}{842 , 106} \approx 0.05 = 5%.$

Option B (diced): working cores per built die $= c \cdot Y$ ? No — careful. A die is either all good or all thrown away (a killing defect kills the whole die). So a good die gives $1000$ cores; a bad die gives $0$ . Fraction of dies good $= Y = 0.368$ . Dies needed to be built: $good dies needed = \frac{800 , 000}{1000} = 800 dies,$ $built dies = \frac{800}{Y} = \frac{800}{0.368} \approx 2174 dies .$ Wasted-die fraction $= 1 - Y = 0.632 = 63.2%$ .

Analysis: Option A wastes ~ $5%$ of silicon; Option B wastes ~ $63%$ . Why the gap: in a diced flow a single defect discards $1000$ cores at once. In the wafer flow a defect kills only the one core it lands in, because the mesh reroutes around it. Fine-grained redundancy is far more silicon-efficient — this is the deep reason wafer-scale can pay off despite sounding wasteful.

Problem 3.2 (L3)

On a $1000 \times 1000$ mesh at $1$ GHz (1 hop = 1 ns), compare average-case and worst-case latency for a message between two cores. Assume traffic is nearest-neighbour on average.

Recall Solution 3.2

Why hop-count = latency: each core-to-core link is one clock, so latency ≈ (number of hops) × (cycle time). At $1$ GHz, cycle $= 1$ ns.

Neighbour (average local traffic): $1$ hop $= 1$ ns.
Worst case, corner to corner: in a mesh you travel along rows then columns, so hops $= (1000 - 1) + (1000 - 1) = 1998 \approx 2000$ hops $= 2000$ ns $= 2 μ s$ .

What it looks like: the red path below is the corner-to-corner route; almost all real traffic uses the tiny local hops instead. Interpretation: even the worst on-wafer case (~ $2 μ$ s) matches the best inter-server network case, and the typical case (1 ns) beats it by ~1000×. Locality is everything.

L4 — Synthesis

Problem 4.1 (L4)

A WSE has $M = 850, 000$ cores, each pushing $B_{cor e} = 1$ GB/s to neighbours. (a) Compute total on-wafer bandwidth. (b) A GPU cluster link tops out at $B_{g p u} = 900$ GB/s per link. How many such GPU links equal the WSE's aggregate? (c) In one sentence, why is aggregate bandwidth additive on a mesh?

Recall Solution 4.1

(a) Why sum: all mesh links run in parallel, so aggregate bandwidth adds: $B_{t o t a l} = M \cdot B_{cor e} = 850, 000 \times 1 GB/s = 8.5 \times 1 0^{5} GB/s = 850 TB/s .$

(b) Equivalent GPU links: $\frac{B _{t o t a l}}{B _{g p u}} = \frac{8.5 \times 1 0 ^{5} GB/s}{900 GB/s} \approx 944 links .$ So one WSE's internal fabric ≈ ~944 top-end GPU links' worth of bandwidth, all on one piece of silicon.

(c) Why additive: because the links are physically separate wires operating simultaneously — no shared bottleneck channel — so their capacities stack instead of competing.

Problem 4.2 (L4)

A model has $40$ GB of parameters. Each core holds $S = 48$ KB of SRAM. (a) Can it fit? (b) What is the minimum number of cores just to store it, ignoring compute? (c) Why is the "model must fit in one core" intuition wrong?

Recall Solution 4.2

(a)/(b) Why aggregate: capacity is the sum over all cores, not per core. Minimum cores to store $40$ GB: $cores = \frac{40 GB}{48 KB} = \frac{40 \times 1 0 ^{9} bytes}{48 \times 1 0 ^{3} bytes} \approx 833, 333 cores .$ With $\sim 850, 000$ cores available, it fits with a little room to spare — which is exactly why the " $40$ GB on-chip SRAM" and " $850, 000$ cores" numbers from the parent are matched by design.

(c) Why the intuition is wrong: the model is sharded across all cores; no single core ever holds it. The mesh stitches partial results together, so the relevant capacity is the distributed total, not the tiny per-core SRAM.

L5 — Mastery

Problem 5.1 (L5)

Design decision: your model has $60%$ zero activations after ReLU (sparsity $s = 0.6$ ). On dense hardware every multiply runs; on a WSE, multiply-by-zero is skipped. (a) What is the ideal speedup from sparsity? (b) If skipping incurs a $20%$ overhead on the remaining work (indexing nonzeros), what is the effective speedup? (c) State when this hardware advantage disappears.

Recall Solution 5.1

(a) Why skip zeros: a multiply by $0$ produces $0$ and wastes time and energy. If a fraction $s$ of activations are zero, only $(1 - s)$ of the work remains. Ideal speedup vs dense: $speedup_{i d e a l} = \frac{1}{1 - s} = \frac{1}{1 - 0.6} = \frac{1}{0.4} = 2.5 \times .$

(b) With overhead: the nonzero work is $(1 - s)$ , inflated by the $20%$ indexing cost, i.e. $\times 1.2$ . Effective work fraction $= (1 - s) \times 1.2 = 0.4 \times 1.2 = 0.48$ . $speedup_{e f f} = \frac{1}{0.48} \approx 2.083 \times .$ So overhead trims $2.5 \times$ down to about $2.08 \times$ — still a real win.

(c) When it disappears: if sparsity is low (few zeros), $\frac{1}{1 - s} \to 1$ and overhead can even make it slower. Also, if the sparsity is unstructured but the hardware needed structured patterns, the skipping machinery stalls. WSEs shine on unstructured sparsity, which dense GPUs struggle to exploit — see Sparsity in Neural Networks.

Problem 5.2 (L5)

Steel-man against yourself: propose one workload where a WSE is the wrong choice, and justify with a bandwidth/locality argument tied to the mesh.

Recall Solution 5.2

A bad-fit workload: a tiny model with a huge streaming dataset that does all-to-all communication (e.g. certain graph/embedding-lookup jobs where every core must reach arbitrary far-away data every step). Why the mesh loses here:

The mesh's advantage is nearest-neighbour locality — local hops are ~1 ns. All-to-all forces average paths near the corner-to-corner ~ $2 μ$ s, erasing the locality win (Problem 3.2).
"No external DRAM" is fatal if the dataset exceeds the ~ $40$ GB on-wafer SRAM: you now stream from off-wafer anyway, hitting the very interconnect wall the WSE was built to avoid. Conclusion: WSEs win when model + working set fit on-wafer and traffic is mostly local/streaming. Break either assumption and off-chip DRAM plus a conventional cluster can beat it.

Recall One-line self-test

Why is the wafer-scale over-provision factor $\frac{1}{1 - f}$ tiny (~ $5%$ ) while a diced flow wastes ~ $63%$ at the same defect density? ::: Because the wafer flow's unit of redundancy is a single core (a defect kills one core, the mesh reroutes), whereas the diced flow's unit is a whole die (one defect scraps ~1000 cores).