Domain-specific accelerators: Exercises — Hinglish deep dive

6.5.6 · D4 · Hardware › Advanced & Emerging Architectures › Domain-specific accelerators

Definition Roofline model aur uske assumptions (L2–L5 problems se pehle padho)

Roofline kehta hai attainable throughput $P_{attain} = min (P_{peak}, B \cdot I)$ hai: aap ya toh compute ceiling ( $P_{peak}$ , ek flat line) se limited ho ya memory ceiling ( $B \cdot I$ , ek slanted line) se — jo bhi lower ho. Iske simplifying assumptions (jaano kab ye jhooth bolta hai):

Compute aur memory transfers fully overlap karte hain — jab MACs chal rahe hote hain, data stream hota rehta hai. (Agar overlap nahi hota, to real performance roofline se buri hogi.)
Perfect streaming, koi stalls nahi; array ko hamesha data milta rehta hai.
Cache/scratchpad effects $I$ mein fold hain: bada cache effective reuse badhata hai (raises $I$ ), cold cache effective $B$ ghataata hai (aapko left/down shift karta hai). Hum $B$ aur $I$ ko effective values maante hain. Ye edge cases (partial overlap, cold caches) real point ko aur sharp banate hain: roofline ek upper bound hai.

Shuru karne se pehle, ek figure do woh pictures fix kar deta hai jis par har problem tikti hai. Hum ise directly reuse karte hain: pie Exercise 1.1 aur 2.1 (energy per op) ke peeche ki picture hai, aur roofline Exercises 2.2, 2.3, 3.3 aur 5.1 (which wall) ke peeche ki picture hai.

Left picture ko aise padho: actual add (teal sliver) tiny hai; fetch / decode / control (orange) bulk hai — yahi wo waste hai jo Exercise 2.1 quantify karta hai. Right picture ko aise padho: aapka kernel ek dot hai jo apni intensity $I$ ke hisaab se placed hai; agar wo slanted line ke neeche baitha hai to wo memory-bound hai ( $I = 25$ orange dot of Ex 2.2), aur $I_{ridge} = 300$ par vertical line Ex 2.3 ka ridge point hai jahan dono ceilings cross karti hain.

Level 1 — Recognition

Recall Solution 1.1

Answer: (c) performance-per-watt. Jab Dennard scaling khatam hua (~2006), voltage har shrink ke saath aur nahi gir sakta tha, toh power density badhne lagi. Hum transistors fit kar sakte hain par unhe power nahi de sakte (Dark silicon). Isliye binding constraint energy per useful operation hai, raw clock nahi — figure ki energy pie ko dobara dekho: teal ALU sliver hi ek aisa hissa hai jo useful hai. GHz (a) heat se limited hai; transistor count (b) power se limited hai; cache (d) ek means hai, goal nahi.

Recall Solution 1.2

False. Ek systolic array typically CPU se lower clock par chalta hai. Iska win hai massive parallelism ( $N \times N$ MAC cells har cycle mein fire karte hain), deleted control overhead, lower precision, aur data reuse (har loaded value kaafi baar use hoti hai — Ex 3.2 mein quantify kiya gaya hai). Clock speed lever nahi hai.

Level 2 — Application

Recall Solution 2.1

Efficiency ratio $= \frac{84 pJ/op}{0.3 pJ/op} = 280 \times$ .

Humne kya kiya: CPU ka energy-per-op accelerator ke energy-per-op se divide kiya.
Kyun: ops/joule $= 1/ (joules per op)$ , toh energies ka ratio hi efficiencies ka ratio hai (pJ ki units cancel ho jaati hain). Ye sahi metric hai kyunki energy/op binding constraint hai (Exercise 1.1), aur figure ki pie dikhati hai kyun CPU number itna bada hai — uske zyaadatar joules orange overhead pe jaate hain jo ek DSA delete kar deta hai.

Recall Solution 2.2

Pehle units convert karo taaki dono ceilings ops/second mein hon: $B \cdot I = 400 GB/s 400 \times 1 0^{9} \frac{bytes}{s} \times I 25 \frac{ops}{byte} = 10 \times 1 0^{12} \frac{ops}{s} = 10 TOPS .$ Bytes cancel ho jaate hain, ops/s = TOPS bachta hai. Phir $P_{attain} = min (P_{peak}, B \cdot I) = min (120, 10) = 10 TOPS .$ Kyunki $B \cdot I = 10 < 120 = P_{peak}$ , aap memory-bound ho — figure mein orange dot slanted line ke neeche baitha hai. Peak ka sirf $10/120 \approx 8.3%$ hi usable hai.

Ye step kyun: Roofline aapko bottleneck optimize karne se pehle bata deta hai — yahan zyaada MACs idle rahenge (assumption: compute aur memory perfectly overlap karte hain; real machine sirf worse ho sakti hai); fix hai $I$ badhana (fuse layers, larger tiles).

Recall Solution 2.3

Dono ceilings equal set karo: $B \cdot I_{ridge} = P_{peak}$ , toh $I_{ridge} = \frac{P _{peak}}{B} = \frac{120 \times 1 0 ^{12} ops/s}{400 \times 1 0 ^{9} bytes/s} = 300 ops/byte .$ ("Per second" upar aur neeche cancel ho jaata hai, ops/byte bachta hai.) $300$ ops/byte se neeche aap memory-bound ho; uske barabar ya upar compute-bound. Hamare kernel ka $I = 25$ bahut $300$ se neeche hai — deeply memory-bound. Ye figure mein vertical line hai.

Level 3 — Analysis

Recall Solution 3.1

(a) MACs/cycle $= 256 \times 256 = 65536$ (har cell mein har cycle mein ek multiply-add). (b) Har loaded weight ek axis ke saath saari activations ke past stream karta hai, toh reuse factor $\approx$ array dimension $= 256$ . (c) $N^{3}$ ka "why": output $C$ ek $N \times N$ matrix hai, toh uske $N \times N = N^{2}$ output entries hain. Har output entry $C_{ij} = \sum_{k = 1}^{N} A_{ik} B_{k j}$ length $N$ ka dot product hai, jisme $N$ multiply-adds lagte hain. Total $= outputs N^{2} \times MACs each N = N^{3}$ . Yahan $N = 512$ : total MACs $= 51 2^{3} = 134, 217, 728 \approx 1.34 \times 1 0^{8}$ .

Recall Solution 3.2

Naive: total ops $= N^{3}$ (Ex 3.1c se). Agar har operand memory se refetch hota hai, bytes moved $\approx 3 N^{3}$ (har MAC ke liye do reads + ek write). $I_{naive} = \frac{N ^{3}}{3 N ^{3}} = \frac{1}{3} ops/byte — ek *flat* line, order 1 (N badhne par unchanged).$ Systolic: aap $\approx N^{2}$ data grid mein ek baar load karte ho aur har value $\approx N$ baar reuse karte ho, toh bytes moved $\approx N^{2}$ jabki ops $N^{3}$ hi rehte hain: $I_{systolic} = \frac{N ^{3}}{N ^{2}} = N ops/byte — ek *rising* line, order N .$ Concretely: $N = 128 \Rightarrow I \approx 128$ ; $N = 256 \Rightarrow I \approx 256$ . Double width, double intensity.

Kyun matter karta hai: intensity ab array size ke saath badhti hai, toh Roofline par aap ridge point ke past right slide karte ho aur compute-bound ban jaate ho — yahi design ka poora point hai.

Recall Solution 3.3

New memory ceiling $= B \cdot I = 400 \times 1 0^{9} \times 340 = 136 \times 1 0^{12} = 136$ TOPS. Kyunki $340 > I_{ridge} = 300$ (Ex 2.3), aapne ridge cross kar liya: $P_{attain} = min (120, 136) = 120 TOPS (ab compute-bound).$ Un-fused $10$ TOPS ke upar speedup: $120/10 = 12 \times$ . Note karo ki ye win sirf data reuse se aaya, hardware khareedne se nahi. Edge case $I ≫ I_{ridge}$ : $I$ aur zyaada badhana (maan lo $500$ ) kuch nahi deta — aap flat $120$ TOPS compute ceiling par pin ho; ab sirf zyaada MACs help kar sakte hain.

Level 4 — Synthesis

Recall Solution 4.1

Parallel fraction $p = 0.95$ aur us fraction par speedup $s = 50$ ke saath: $Speedup = \frac{1}{( 1 - 0.95 ) + \frac{0.95}{50}} = \frac{1}{0.05 + 0.019} = \frac{1}{0.069} \approx 14.5 \times .$ $50 \times$ accelerator ke bawajood overall sirf $\approx 14.5 \times$ milta hai kyunki untouched $5%$ ab dominate karta hai (immovable stub). Lesson: DSAs narrow hote hain; aapko serial/irregular parts orchestrate karne ke liye CPU chahiye. Real systems heterogeneous hote hain — CPU + DSA — sirf DSA nahi.

Recall Solution 4.2

Starting attainable $= B \cdot I = 500 \times 1 0^{9} \times 8 = 4$ TOPS. Option A (double MACs): $P_{peak}$ badhata hai lekin kernel memory-bound hai, toh $P_{attain} = min (2 P_{peak}, B \cdot I) = 4$ TOPS — koi change nahi. Paisa waste. Option B (INT8): operands $4 \times$ chhote hain, toh effective $I$ $8$ se $32$ ops/byte ho jaata hai. New ceiling $= 500 \times 1 0^{9} \times 32 = 16$ TOPS = $4 \times$ improvement. B chunno. Quantization actual bottleneck (bytes moved) par attack karta hai; MACs add karna nahi karta.

Level 5 — Mastery

Recall Solution 5.1

(a) $B \cdot I = 250 \times 1 0^{9} \times 12 = 3$ TOPS $< 60$ TOPS ⇒ memory-bound, peak ka $3/60 = 5%$ use ho raha hai. (b) Ridge point $I_{ridge} = P_{peak} / B = 60 \times 1 0^{12} / (250 \times 1 0^{9}) = 240$ ops/byte. Aapko $I$ ko $12$ se $240$ tak badhana hoga — reuse mein $20 \times$ increase. (c) Decision: scratchpad par spend karo (option ii). Array already $95%$ starved hai; bada array $P_{peak}$ (flat ceiling) badhata hai jabki kernel slanted ceiling par pinned hai — extra MACs idle rahenge. Zyaada scratchpad bade tiles / operand reuse enable karta hai, $I$ ko ridge ki taraf rightward push karta hai, jo peak ko attainable throughput mein convert karne ka ek aur tarika hai. Edge case to respect: agar cold/too-small cache effective $B$ ghatata hai, toh aap slanted ceiling par neeche slide karte ho aur woh $3$ TOPS bhi kho dete ho — ye ek aur reason hai ki reuse (scratchpad) fix sahi lever hai.

Recall Solution 5.2

Scheme: Mixed precision training. Sensitive parts ko — master weights aur gradient accumulation — higher precision (FP32) mein rakho, jabki bulk matmuls (forward/backward) low precision (bfloat16/FP16) mein karo. Loss scaling chhote gradients ko underflow hone se bachata hai. Ye kyun kaam karta hai (domain tolerance): neural nets statistical hain; forward-pass ke millions of MACs rounding noise ko average kar dete hain, toh wahan low precision almost free hai (inference ki tarah). Lekin tiny gradients ka accumulation ek aisa sum hai jahan rounding errors compound hoti hain — woh step high precision rakhta hai. Aap precision exactly wahan spend karte ho jahan domain intolerant hai, aur use wahan bachate ho jahan domain tolerant hai. Woh selective placement hi domain-specific design ka essence hai.

Recall Solution 5.3

Pollack's Rule: single-core performance $area$ ke saath badhti hai. Area double karne par $2 \approx 1.41 \times$ milta hai, $2 \times$ nahi — monolithic core se diminishing returns. Contrast: DSA instead us area ko bahut saare chhote, identical MAC cells par spend karta hai jinka aggregate throughput roughly area ke saath linearly scale karta hai (data-level parallelism), aur per-op control overhead delete karta hai. Toh same transistors se far more useful ops/mm² milti hain — provided arithmetic intensity unhe fed rakhti hai (Roofline caveat from Exercise 5.1).

Recall Self-check: kaunsa lever kaunsa symptom fix karta hai?

Memory-bound kernel, MACs idle ::: Arithmetic intensity badhao — bade tiles, fused ops, zyaada scratchpad, lower-precision operands. Serial fraction total time dominate kar raha hai ::: Amdahl limit — orchestrate karne ke liye CPU rakho; heterogeneous system, sirf DSA nahi. Gradient INT8 mein diverge karta hai ::: Mixed precision — high precision sirf sensitive accumulation ke liye. "Core area double karna = 2x speed" ::: Pollack's Rule se False; monolithic core ke liye $2 \approx 1.41 \times$ . Kernel already ridge ke right mein hai ( $I ≫ I_{ridge}$ ) ::: Compute-bound — zyaada reuse waste hai; ab sirf zyaada MACs throughput badhate hain.