Domain-specific accelerators: Worked examples — Hardware deep dive

This page is the drill ground. The parent note built the ideas (the Roofline model, Systolic arrays, energy-per-op). Here we run those ideas through every kind of situation an exam or a real chip can throw at you — every sign of the inequality, the zero/degenerate cases, the limits, a word problem, and a nasty twist.

Before any example, we agree on the tools and their symbols so nothing sneaks in undefined.

The figure below draws exactly this picture: the blue line is $P_{attain}$ , the yellow dashed line is the rising memory ceiling $B \times I$ , the red dashed line is the flat compute ceiling $P_{peak}$ , and the green dot marks the ridge $I^{⋆}$ where they cross. Keep it in mind — every roofline example below is a point placed on this curve.

The scenario matrix

Every DSA numeric question is one of these cells. The last column names the example that nails it.

#	Case class	The twist it tests	Example
A	Memory-bound ( $I < I^{⋆}$ )	extra MACs are wasted	Ex 1
B	Compute-bound ( $I > I^{⋆}$ )	more bandwidth is wasted	Ex 2
C	Balanced ( $I = I^{⋆}$ , boundary)	the exact ridge point	Ex 3
D	Degenerate: $I \to 0$ (no reuse)	streaming, zero-reuse limit	Ex 4
E	Degenerate: $I \to \infty$ (infinite reuse)	you saturate $P_{peak}$	Ex 4
F	Energy-per-op comparison	ops/joule, not GHz	Ex 5
G	Systolic reuse & utilisation	non-square tiles, edge fill/drain waste	Ex 6
H	Word problem (real deploy)	pick the bottleneck, then act	Ex 7
I	Exam twist: precision change moves $I$	INT8 halves bytes → doubles $I$	Ex 8
J	Heterogeneous / Amdahl limit	serial part caps total speedup	Ex 9

Worked examples

Worked example Example 1 — Cell A: memory-bound

A DSA has $P_{peak} = 90$ TOPS and $B = 300$ GB/s. Kernel intensity $I = 50$ ops/byte.

Forecast: Guess the ridge point $I^{⋆}$ first. Is $50$ left or right of it?

Ridge point. $I^{⋆} = P_{peak} / B = \frac{90 \times 1 0 ^{12}}{300 \times 1 0 ^{9}} = 300$ ops/byte. Why this step? The ridge tells us the boundary before we compute anything — it converts the problem into a simple left/right comparison.
Compare. $I = 50 < 300 = I^{⋆}$ ⇒ we are left of the ridge ⇒ memory-bound. Why this step? Side of the ridge = the bottleneck. No arithmetic on the ceilings needed to diagnose.
Attainable. Memory ceiling $= B \times I = 300 \times 1 0^{9} \times 50 = 15 \times 1 0^{12} = 15$ TOPS. So $P_{attain} = min (90, 15) = 15$ TOPS. Why this step? We finish the $min$ to report the actual usable rate.

Verify: $15 < 90$ , consistent with memory-bound. Utilisation $= 15/90 = 16.7%$ — most MACs idle, exactly the symptom of being left of the ridge. Units: (bytes/s)(ops/byte) = ops/s ✓.

Worked example Example 2 — Cell B: compute-bound

Same chip: $P_{peak} = 90$ TOPS, $B = 300$ GB/s. Now a fused kernel achieves $I = 500$ ops/byte.

Forecast: Ridge is still $300$ . Which side now?

Compare to ridge. $I = 500 > 300 = I^{⋆}$ ⇒ right of ridge ⇒ compute-bound. Why this step? We reuse the ridge from Ex 1 — the chip didn't change, only the kernel.
Memory ceiling. $B \times I = 300 \times 1 0^{9} \times 500 = 150 \times 1 0^{12} = 150$ TOPS. Why this step? We still need the memory ceiling's actual value to feed the $min$ — the diagnosis tells us which ceiling wins, but only the number tells us by how much and confirms the ordering.
Attainable. $P_{attain} = min (90, 150) = 90$ TOPS. Why this step? The compute ceiling now wins; we are limited by silicon, not memory.

Verify: $90 < 150$ ✓ compute-bound. Utilisation $= 90/90 = 100%$ — MACs fully fed. Buying more bandwidth here does nothing; buying more MACs would help. Opposite conclusion to Ex 1, as expected on the other side of the ridge.

Worked example Example 3 — Cell C: exactly balanced (boundary)

Same chip. What intensity makes it perfectly balanced, and what is attainable there?

Forecast: Guess: do the two ceilings agree at the ridge?

Solve for balance. By definition $I = I^{⋆} = P_{peak} / B = 300$ ops/byte. Why this step? "Balanced" means neither ceiling dominates, i.e. $B \times I = P_{peak}$ — solving that equation for $I$ is literally the definition of the ridge point, so the balance intensity is $I^{⋆}$ .
Both ceilings. Compute $= 90$ TOPS; memory $= 300 \times 1 0^{9} \times 300 = 90 \times 1 0^{12} = 90$ TOPS. Why this step? At the ridge the two ceilings must be numerically equal — that's the whole meaning of "balanced."
Attainable. $min (90, 90) = 90$ TOPS.

Verify: Both ceilings equal $90$ TOPS ✓. This is the cheapest intensity that still saturates the chip — any less wastes MACs, any more wastes bandwidth. It is the design target.

Worked example Example 4 — Cells D & E: the two limits (

$I \to 0$ and $I \to \infty$ ) Same chip ( $P_{peak} = 90$ TOPS, $B = 300$ GB/s). Explore the extremes.

Forecast: As reuse vanishes, does performance go to $0$ or to $B$ ? As reuse explodes, does it grow without bound?

$I \to 0$ (no reuse, pure streaming). $B \times I \to 300 \times 1 0^{9} \times 0 = 0$ , so $P_{attain} = min (90, 0) = 0$ TOPS. Why this step? Zero reuse means every operation waits on a fresh fetch; in the limit the compute engine starves completely.
$I \to \infty$ (infinite reuse). $B \times I \to \infty$ , so $P_{attain} = min (90, \infty) = 90$ TOPS — it clamps at $P_{peak}$ . Why this step? The $min$ caps performance; you can never exceed the silicon's peak no matter how much you reuse.

Verify: The roofline literally is a rising line ( $B \cdot I$ ) that flattens at $90$ — Ex 4's two limits are its two ends: $0$ at the origin, $90$ at the plateau. See the figure below.

Worked example Example 6 — Cell G: systolic reuse & real utilisation

A $128 \times 128$ MAC grid processes a $256 \times 256$ matmul.

Forecast: Peak MACs/cycle? And is utilisation really $100%$ , or does fill/drain steal some?

Peak MACs/cycle. $128 \times 128 = 16384$ . Why this step? Each cell does one multiply-add per cycle; the count is just the grid area.
Reuse factor. Each loaded weight streams past all $128$ activations in its row/column ⇒ reuse $\approx 128$ = the array dimension. Why this step? Reuse = array dimension is what lifts arithmetic intensity from $O (1)$ to $O (N)$ , pushing the design toward compute-bound.
Fill/drain overhead (the twist). Data takes $\sim 2 N - 1 = 255$ cycles to fill and drain the pipeline before/after steady state. Over a run that streams $256$ activation columns, useful cycles $\approx 256$ , wasted $\approx 255$ , so steady-state utilisation is roughly $\frac{256}{256 + 255} \approx 50%$ for one small tile. Why this step? Small matrices relative to the array pay a large edge tax — this is why real chips like big tiles.

Verify: $16384 = 12 8^{2}$ ✓; reuse $128 = N$ ✓; $256/ (256 + 255) = 0.5009 \approx 50%$ ✓. Lesson: peak $\neq =$ sustained; the parent's "reuse = array dimension" holds only once the pipe is full.

Worked example Example 7 — Cell H: real-world word problem

You deploy a recommender on a DSA: $P_{peak} = 100$ TOPS, $B = 1$ TB/s. The model's embedding-lookup stage has $I = 2$ ops/byte; its dense stage has $I = 400$ ops/byte. Which stage is the bottleneck, and what do you optimise?

Forecast: Ridge is $I^{⋆} = 100/1 = 100$ . Guess which stage sits on which side.

Embedding stage. $I = 2 < 100$ ⇒ memory-bound. Attainable $= B \times I = 1 0^{12} \times 2 = 2 \times 1 0^{12} = 2$ TOPS (only $2%$ of peak). Why this step? Diagnose each stage independently against the same ridge.
Dense stage. $I = 400 > 100$ ⇒ compute-bound. Attainable $= min (100, 400) = 100$ TOPS (full peak). Why this step? The dense stage sits right of the ridge, so its ceiling is $P_{peak}$ not memory — re-diagnosing it separately confirms the two stages live on opposite sides and must be treated differently.
Act. The embeddings dominate lost performance. Fix = raise $I$ there: fuse ops, cache hot embeddings, quantise the table (Quantization) to move fewer bytes. Adding MACs would be useless for that stage. Why this step? Optimise the binding constraint; the Roofline model told us which one it is.

Verify: Embedding $2$ TOPS $= 2%$ of $100$ ✓; dense $100$ TOPS $= 100%$ ✓; ridge $100/1 = 100$ ✓. The two stages land on opposite sides of the ridge — the classic heterogeneous-workload picture (Heterogeneous computing).

Worked example Example 8 — Cell I: exam twist — precision changes

$I$ A kernel does $W$ ops and, in FP32, moves each operand as $4$ bytes, giving $I_{32} = 25$ ops/byte on a chip with $B = 300$ GB/s, $P_{peak} = 90$ TOPS. Here $W$ is the total number of operations and $Q$ is the total number of bytes moved, so intensity is $I = W / Q$ . You switch the operands to INT8 ( $1$ byte). What is the new $I$ , and does the bottleneck flip?

Forecast: INT8 is $4 \times$ fewer bytes. Guess the new intensity and whether it crosses the ridge ( $I^{⋆} = 300$ ).

New intensity. Ops $W$ unchanged; bytes moved $Q$ drop by $4 \times$ . Since $I = W / Q$ , quartering $Q$ quadruples $I$ : $I_{8} = 4 \times 25 = 100$ ops/byte. Why this step? Intensity is ops per byte; fewer bytes per operand raises reuse-per-byte mechanically. This is why low precision helps even when op count is identical.
Re-diagnose. $I_{8} = 100$ , ridge $I^{⋆} = 300$ . Still $100 < 300$ ⇒ still memory-bound, but closer. Why this step? Changing precision moved $I$ , so the earlier diagnosis may no longer hold — we must re-check which side of the ridge we land on before trusting any ceiling.
Attainable before vs after. Before: $300 \times 1 0^{9} \times 25 = 7.5$ TOPS. After: $300 \times 1 0^{9} \times 100 = 30$ TOPS. A $4 \times$ jump purely from precision — no new silicon. Why this step? Shows precision is a bandwidth lever, not just a compute lever — the point of Mixed precision training and INT8 inference.

Verify: $4 \times 25 = 100$ ✓; $7.5 \times 4 = 30$ ✓; both attainable values below $90$ , so bottleneck stayed memory-side ✓. If we'd started at $I_{32} = 80$ , INT8 would give $320 > 300$ and flip to compute-bound — worth stating as the boundary case.

Worked example Example 9 — Cell J: heterogeneous / Amdahl ceiling

A pipeline spends fraction $f = 0.9$ of its time in a matmul that a DSA speeds up by $s = 50 \times$ . The remaining $0.1$ (control, I/O) stays on the CPU, unchanged. What is the whole-program speedup?

Forecast: With a $50 \times$ accelerator, guess the overall speedup — near $50 \times$ , or much less?

Amdahl's Law. Overall speedup $= \frac{1}{( 1 - f ) + f / s}$ . Why this step? Only part of the program is accelerated; the serial remainder sets a hard ceiling (Amdahl's Law).
Plug in. $= \frac{1}{0.1 + 0.9/50} = \frac{1}{0.1 + 0.018} = \frac{1}{0.118} \approx 8.47 \times$ . Why this step? The un-accelerated $10%$ now dominates the runtime.
Limit ( $s \to \infty$ ). $\frac{1}{0.1 + 0} = 10 \times$ — the absolute cap no accelerator can beat. Why this step? Shows why real systems are heterogeneous and why shrinking the serial part matters more than a faster DSA once $s$ is large.

Verify: $0.9/50 = 0.018$ ; $0.1 + 0.018 = 0.118$ ; $1/0.118 = 8.474 \dots$ ✓; limit $1/0.1 = 10$ ✓. Even a "perfect" DSA yields only $10 \times$ here — the Cell J lesson.

Recall Which cell is each example?

Ex1 memory-bound (A) ::: left of ridge, $I < I^{⋆}$ Ex2 compute-bound (B) ::: right of ridge, $I > I^{⋆}$ Ex3 balanced (C) ::: on the ridge, both ceilings equal Ex4 limits (D,E) ::: $I \to 0$ gives $0$ ; $I \to \infty$ clamps to $P_{peak}$ Ex5 energy (F) ::: divide energies, not clocks → $350 \times$ Ex6 systolic (G) ::: peak $\neq =$ sustained; fill/drain tax Ex7 word problem (H) ::: diagnose each stage, fix the memory-bound one Ex8 precision twist (I) ::: fewer bytes → higher $I$ → $4 \times$ free Ex9 Amdahl (J) ::: serial part caps speedup at $1/ (1 - f)$