Domain-specific accelerators: Exercises — Hardware deep dive

Definition The Roofline model and its assumptions (read before the L2–L5 problems)

Roofline says attainable throughput is $P_{attain} = min (P_{peak}, B \cdot I)$ : you are limited by either the compute ceiling ( $P_{peak}$ , a flat line) or the memory ceiling ( $B \cdot I$ , a slanted line), whichever is lower. Its simplifying assumptions (know when it lies):

Compute and memory transfers fully overlap — while MACs run, data streams in. (If they don't overlap, real performance is worse than the roofline.)
Perfect streaming, no stalls; the array is always fed.
Cache/scratchpad effects are folded into $I$ : a bigger cache raises effective reuse (raises $I$ ), a cold cache lowers effective $B$ (shifts you left/down). We treat $B$ and $I$ as the effective values. These edge cases (partial overlap, cold caches) only make the real point sharper: the roofline is an upper bound.

Before we start, one figure fixes the two pictures every problem leans on. We reuse it directly: the pie is the picture behind Exercise 1.1 and 2.1 (energy per op), and the roofline is the picture behind Exercises 2.2, 2.3, 3.3 and 5.1 (which wall).

Read the left picture as: the actual add (teal sliver) is tiny; the fetch / decode / control (orange) is the bulk — this is exactly the waste Exercise 2.1 quantifies. Read the right picture as: your kernel is a dot placed by its intensity $I$ ; if it sits under the slanted line it is memory-bound (the $I = 25$ orange dot of Ex 2.2), and the vertical line at $I_{ridge} = 300$ is the ridge point of Ex 2.3 where the two ceilings cross.

Level 1 — Recognition

Recall Solution 1.1

Answer: (c) performance-per-watt. Once Dennard scaling died (~2006), voltage could no longer drop with each shrink, so power density rose. We can fit transistors but not power them all (Dark silicon). Therefore the binding constraint is energy per useful operation, not raw clock — look again at the energy pie in the figure: the teal ALU sliver is the only useful part. GHz (a) is limited by heat; transistor count (b) is limited by power; cache (d) is a means, not the goal.

Recall Solution 1.2

False. A systolic array typically runs at a lower clock than a CPU. Its win is massive parallelism ( $N \times N$ MAC cells firing every cycle), deleted control overhead, lower precision, and data reuse (each loaded value is used many times — quantified in Ex 3.2). Clock speed is not the lever.

Level 2 — Application

Recall Solution 2.1

Efficiency ratio $= \frac{84 pJ/op}{0.3 pJ/op} = 280 \times$ .

What we did: divided energy-per-op of the CPU by energy-per-op of the accelerator.
Why: ops/joule $= 1/ (joules per op)$ , so the ratio of energies is the ratio of efficiencies (units of pJ cancel). This is the correct metric because energy/op is the binding constraint (Exercise 1.1), and the figure's pie shows why the CPU number is so large — most of its joules go to the orange overhead a DSA deletes.

Recall Solution 2.2

First convert units so both ceilings are in ops/second: $B \cdot I = 400 GB/s 400 \times 1 0^{9} \frac{bytes}{s} \times I 25 \frac{ops}{byte} = 10 \times 1 0^{12} \frac{ops}{s} = 10 TOPS .$ The bytes cancel, leaving ops/s = TOPS. Then $P_{attain} = min (P_{peak}, B \cdot I) = min (120, 10) = 10 TOPS .$ Since $B \cdot I = 10 < 120 = P_{peak}$ , you are memory-bound — the orange dot in the figure sits under the slanted line. Only $10/120 \approx 8.3%$ of peak is usable.

Why this step: Roofline tells you the bottleneck before you optimize — here more MACs would sit idle (assumption: compute and memory overlap perfectly; the real machine can only be worse); the fix is to raise $I$ (fuse layers, larger tiles).

Recall Solution 2.3

Set the two ceilings equal: $B \cdot I_{ridge} = P_{peak}$ , so $I_{ridge} = \frac{P _{peak}}{B} = \frac{120 \times 1 0 ^{12} ops/s}{400 \times 1 0 ^{9} bytes/s} = 300 ops/byte .$ (The "per second" cancels top and bottom, leaving ops/byte.) Below $300$ ops/byte you are memory-bound; at or above it you are compute-bound. Our kernel's $I = 25$ is far below $300$ — deeply memory-bound. This is the vertical line in the figure.

Level 3 — Analysis

Recall Solution 3.1

(a) MACs/cycle $= 256 \times 256 = 65536$ (one multiply-add per cell per cycle). (b) Each loaded weight streams past all activations along one axis, so reuse factor $\approx$ the array dimension $= 256$ . (c) The "why" of $N^{3}$ : the output $C$ is an $N \times N$ matrix, so it has $N \times N = N^{2}$ output entries. Each output entry $C_{ij} = \sum_{k = 1}^{N} A_{ik} B_{k j}$ is a dot product of length $N$ , which costs $N$ multiply-adds. Total $= outputs N^{2} \times MACs each N = N^{3}$ . Here $N = 512$ : total MACs $= 51 2^{3} = 134, 217, 728 \approx 1.34 \times 1 0^{8}$ .

Recall Solution 3.2

Naive: total ops $= N^{3}$ (from Ex 3.1c). If every operand is refetched from memory, bytes moved $\approx 3 N^{3}$ (two reads + one write per MAC). $I_{naive} = \frac{N ^{3}}{3 N ^{3}} = \frac{1}{3} ops/byte — a *flat* line, order 1 (unchanged as N grows).$ Systolic: you load $\approx N^{2}$ data once into the grid and reuse each value $\approx N$ times, so bytes moved $\approx N^{2}$ while ops stay $N^{3}$ : $I_{systolic} = \frac{N ^{3}}{N ^{2}} = N ops/byte — a *rising* line, order N .$ Concretely: $N = 128 \Rightarrow I \approx 128$ ; $N = 256 \Rightarrow I \approx 256$ . Twice the width, twice the intensity.

Why it matters: intensity now grows with array size, so on the Roofline you slide right past the ridge point and become compute-bound — the whole point of the design.

Recall Solution 3.3

New memory ceiling $= B \cdot I = 400 \times 1 0^{9} \times 340 = 136 \times 1 0^{12} = 136$ TOPS. Since $340 > I_{ridge} = 300$ (Ex 2.3), you have crossed the ridge: $P_{attain} = min (120, 136) = 120 TOPS (now compute-bound).$ Speedup over the un-fused $10$ TOPS: $120/10 = 12 \times$ . Note the win came entirely from data reuse, not from buying hardware. Edge case $I ≫ I_{ridge}$ : raising $I$ still higher (say $500$ ) buys nothing more — you are pinned at the flat $120$ TOPS compute ceiling; only more MACs help now.

Level 4 — Synthesis

Recall Solution 4.1

With parallel fraction $p = 0.95$ and speedup $s = 50$ on that fraction: $Speedup = \frac{1}{( 1 - 0.95 ) + \frac{0.95}{50}} = \frac{1}{0.05 + 0.019} = \frac{1}{0.069} \approx 14.5 \times .$ Even a $50 \times$ accelerator yields only $\approx 14.5 \times$ overall because the untouched $5%$ now dominates (the immovable stub). Lesson: DSAs are narrow; you need a CPU to orchestrate the serial/irregular parts. Real systems are heterogeneous — CPU + DSA — not DSA-only.

Recall Solution 4.2

Starting attainable $= B \cdot I = 500 \times 1 0^{9} \times 8 = 4$ TOPS. Option A (double MACs): raises $P_{peak}$ but the kernel is memory-bound, so $P_{attain} = min (2 P_{peak}, B \cdot I) = 4$ TOPS — no change. Wasted money. Option B (INT8): operands are $4 \times$ smaller, so effective $I$ rises from $8$ to $32$ ops/byte. New ceiling $= 500 \times 1 0^{9} \times 32 = 16$ TOPS = $4 \times$ improvement. Choose B. Quantization attacks the actual bottleneck (bytes moved); adding MACs does not.

Level 5 — Mastery

Recall Solution 5.1

(a) $B \cdot I = 250 \times 1 0^{9} \times 12 = 3$ TOPS $< 60$ TOPS ⇒ memory-bound, using $3/60 = 5%$ of peak. (b) Ridge point $I_{ridge} = P_{peak} / B = 60 \times 1 0^{12} / (250 \times 1 0^{9}) = 240$ ops/byte. You need to raise $I$ from $12$ to $240$ — a $20 \times$ increase in reuse. (c) Decision: spend on scratchpad (option ii). The array is already $95%$ starved; a bigger array raises $P_{peak}$ (the flat ceiling) while the kernel is pinned to the slanted ceiling — the extra MACs sit idle. More scratchpad enables bigger tiles / operand reuse, pushing $I$ rightward toward the ridge, which is the only way to convert peak into attainable throughput. Edge case to respect: if a cold/too-small cache lowers effective $B$ , you slide down the slanted ceiling and lose even the $3$ TOPS — another reason the reuse (scratchpad) fix is the right lever.

Recall Solution 5.2

Scheme: Mixed precision training. Keep the sensitive parts — master weights and gradient accumulation — in higher precision (FP32), while doing the bulk matmuls (forward/backward) in low precision (bfloat16/FP16). Loss scaling prevents small gradients from underflowing. Why it works (domain tolerance): neural nets are statistical; the millions of forward-pass MACs average away rounding noise, so low precision there is nearly free (as in inference). But accumulation of tiny gradients is a sum where rounding errors compound — that step keeps high precision. You spend precision exactly where the domain is intolerant, and save it everywhere the domain is tolerant. That selective placement is the essence of domain-specific design.

Recall Solution 5.3

Pollack's Rule: single-core performance grows as $area$ . Doubling area gives $2 \approx 1.41 \times$ , not $2 \times$ — diminishing returns from a monolithic core. Contrast: the DSA instead spends that area on many small, identical MAC cells whose aggregate throughput scales roughly linearly with area (data-level parallelism), and deletes per-op control overhead. So the same transistors buy far more useful ops/mm² — provided arithmetic intensity keeps them fed (the Roofline caveat from Exercise 5.1).

Recall Self-check: which lever fixes which symptom?

Memory-bound kernel, MACs idle ::: Raise arithmetic intensity — bigger tiles, fused ops, more scratchpad, lower-precision operands. Serial fraction dominates total time ::: Amdahl limit — keep a CPU to orchestrate; heterogeneous system, not DSA-only. Gradient diverges in INT8 ::: Mixed precision — high precision only for sensitive accumulation. "Doubling core area = 2x speed" ::: False by Pollack's Rule; $2 \approx 1.41 \times$ for a monolithic core. Kernel already right of the ridge ( $I ≫ I_{ridge}$ ) ::: Compute-bound — more reuse is wasted; only more MACs raise throughput now.