Domain-specific accelerators

WHY do DSAs exist? (First principles)

Result: we have transistors (Moore's Law limped on) but can't power them all (the Dark Silicon problem). If you can't turn everything on, the smart move is to build many specialized blocks and light up only the one you need. That is the economic engine of DSAs.

HOW do you get the speedup? (The 5 levers — the 80/20)

The canonical example: the systolic array (TPU-style matmul)

Why it wins — the arithmetic intensity argument (derived):

For a naive matrix multiply $C = A\cdot B$ with $N\times N$ matrices:

Total MACs $= N^3$ .
If every operand came from memory each time, memory ops $= 3N^3$ → arithmetic intensity $\approx \tfrac{1}{3}$ ops/byte. Memory-bound, terrible.

In an $N\times N$ systolic array, you load $O(N^2)$ data and reuse each value $O(N)$ times inside the array:

\text{Arithmetic intensity} \;=\; \frac{\text{operations}}{\text{bytes moved}} \;\approx\; \frac{N^3}{N^2}\;=\;O(N).

So intensity grows with array size → you become compute-bound and can keep the MACs fed. That's the whole trick.

Formula Roofline model — deriving when a DSA helps

Attainable performance (FLOP/s) is capped by either compute or memory bandwidth: $P_{\text{attain}} = \min\big(P_{\text{peak}},\; B \times I\big)$ where $B$ = memory bandwidth (bytes/s) and $I$ = arithmetic intensity (FLOP/byte). Derivation: if a kernel does $W$ FLOPs and moves $Q$ bytes, its intensity is $I=W/Q$ . Time is at least $\max(W/P_{\text{peak}},\, Q/B)$ . Dividing $W$ by that time gives $\min(P_{\text{peak}}, B\cdot W/Q)=\min(P_{\text{peak}}, B\cdot I)$ . Insight: raising $P_{\text{peak}}$ (more MACs) is useless unless you also raise $I$ (data reuse) — which is exactly what a systolic array/scratchpad does.

Worked examples

Common mistakes (Steel-manned)

Forecast-then-Verify

Recall Predict before revealing

Q: A kernel has intensity $I=2$ ops/byte, on a chip with $B=1$ TB/s and $P_{\text{peak}}=100$ TOPS. Compute-bound or memory-bound? Attainable? Predict first... $B\cdot I = 10^{12}\times 2 = 2\times10^{12}=2$ TOPS $< 100$ TOPS ⇒ memory-bound, attainable $=2$ TOPS. Only 2% of peak used — classic reason to fuse ops and raise reuse.

Flashcards

What ended the "free lunch" of automatic per-generation CPU speedups?

The end of Dennard scaling (~2006): power density stopped staying constant as transistors shrank.

What is the dark-silicon problem?

We can fit more transistors than we can power/cool simultaneously, so we must specialize and light up only the needed blocks.

Define a domain-specific accelerator.

A chip specialized for one class of problems, trading generality for large gains in performance-per-watt and per-area.

Name the 5 sources of DSA advantage.

Data-level parallelism, lower/cheaper arithmetic precision, domain-specific memory (scratchpads), reduced control overhead, domain-specific language/mapping.

What does a systolic array do?

Streams data rhythmically through a grid of MAC cells, reusing each loaded value many times to minimize memory traffic.

State the Roofline model.

P_{\text{attain}}=\min(P_{\text{peak}}, B\times I)

where

B

=bandwidth,

I

=arithmetic intensity.

What is arithmetic intensity?

Operations performed per byte moved from memory (FLOP/byte); higher intensity means more compute-bound.

Why does an N×N systolic array raise arithmetic intensity to O(N)?

Each loaded operand is reused ~N times across the array, so ops/bytes ≈ N³/N² = N.

Why can DSAs use lower precision safely for ML inference?

Statistical averaging over many weights hides rounding noise, so INT8/bfloat16 costs ~4× less energy with negligible accuracy loss.

Why doesn't adding more MACs always help?

If the kernel is memory-bound (

B\cdot I<P_{\text{peak}}

) the extra MACs starve; you must raise arithmetic intensity instead.

Why is energy-per-op, not clock speed, the key DSA metric?

In the dark-silicon era the binding constraint is power; specialization deletes fetch/decode/control energy overhead.

Recall Feynman: explain to a 12-year-old

Imagine a super-smart student who can do any homework — math, art, history. But before every single question they waste time reading the question, deciding what to do, double-checking, and flipping pages. That's a CPU. Now imagine a factory machine that does only one thing — say, stamping cookies — but does it thousands at a time, super fast, using almost no energy, because it never has to "think" about what to do next. That's a domain-specific accelerator. It's dumb outside its one job, but for that job it's unbeatable. Since we can't power all our chip's transistors at once anymore, it's smarter to build lots of these specialized cookie-stampers and only switch on the one we currently need.

Connections

Dennard scaling & Dark silicon — the physical reasons DSAs exist.
Roofline model — the analysis tool for compute vs memory bound.
Systolic arrays — the canonical matmul accelerator structure.
TPU (Tensor Processing Unit) and GPU vs TPU — real-world DSAs.
Heterogeneous computing — CPUs orchestrating specialized accelerators.
Amdahl's Law & Pollack's Rule — limits of general-purpose scaling.
Quantization & Mixed precision training — enabling lower-precision arithmetic.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek normal CPU "sab kuch kar leta hai" wala student hai — par har chhoti operation se pehle instruction fetch karta hai, decode karta hai, branch predict karta hai, reorder karta hai. Yeh saara "sochne" ka kaam actual math se 10-100x zyada energy kha jaata hai. Domain-specific accelerator (DSA) is drama ko delete kar deta hai: woh sirf ek kaam ke liye banaya jaata hai (jaise matrix multiply), aur usi mein 10x se 1000x better performance-per-watt de deta hai.

Yeh zaroori kyun ho gaya? Kyunki Dennard scaling khatam ho gaya — ab transistor chhota karne se power density constant nahi rehti, matlab heat badh jaati hai. Isliye hum saare transistors ek saath on nahi kar sakte — isko dark silicon kehte hain. Toh smart move yeh hai ki bahut saare specialized blocks banao aur sirf jo abhi chahiye usi ko on karo.

Speed kahan se aati hai? Yaad rakho PALMS: Parallelism (ek saath hazaaron MAC operations), lower precision Arithmetic (INT8/bfloat16 — ML mein accuracy pe koi khaas farak nahi padta), Local scratchpad memory (cache ka overhead nahi), Minimal control overhead, aur Specialized compiler. Systolic array iska classic example hai — data ek grid mein "behta" hai aur har value baar baar reuse hoti hai, jisse arithmetic intensity O(N) ho jaati hai.

Ek cheez pakki yaad rakho: sirf zyada MACs (zyada TOPS) daalne se kaam nahi banta. Roofline model — $P=\min(P_{peak}, B\times I)$ — batata hai ki agar kernel memory-bound hai toh extra MACs bekaar baithe rahenge. Pehle intensity (data reuse) badhao, tab peak performance milegi. Isliye asli systems heterogeneous hote hain: CPU boss banke orchestrate karta hai, DSA hot kernel ka kaam nipta deta hai.

WHY do DSAs exist? (First principles)

HOW do you get the speedup? (The 5 levers — the 80/20)

The canonical example: the systolic array (TPU-style matmul)

Worked examples

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Advanced & Emerging Architectures

Connections