Neural processing units (NPUs) — Hinglish notes

6.5.8 · Hardware › Advanced & Emerging Architectures

NPUs exist kyun karte hain?

Yahi hai 80/20: matrix-multiply (operation types ka wo 20%) runtime ka 80%+ hai. Use optimise karo, baaki ignore karo.

NPU KYA hai?

Key building blocks:

Processing Element (PE): ek MAC unit — $a \times b + c$ compute karta hai.
Systolic array: PEs ki ek 2-D grid jahan data neighbours ke beech rhythmically flow karta hai (jaise heartbeat — Greek systole), har step mein koi global memory access nahi.
On-chip SRAM / scratchpad: weights aur activations ko compute ke paas rakhta hai.
Low precision datapaths: int8, fp16, bfloat16.

Core math hardware se HOW map karta hai?

Step 1 — Wo operation jo humein accelerate karni hai

Ek fully-connected layer $y = W x$ compute karta hai, aur ek batch ek matrix multiply $Y = X W$ hai. Ek output element hai:

$y_{ij} = \sum_{k = 1}^{K} X_{ik} W_{k j}$

Step 2 — Kaam count karna (cost derive karna)

$(M \times K) \cdot (K \times N)$ ka ek matrix multiply, har output element ke liye, $K$ multiplies aur $K - 1$ adds chahiye. $M N$ outputs hain:

$MACs = M \cdot N \cdot K$

Step 3 — Naïve cost math nahi, memory hai

Isko dumb tarike se karo: har MAC ke liye, $X_{ik}$ aur $W_{k j}$ memory se load karo. Yeh $2 \times M N K$ memory reads hain. WHY bad? Ek DRAM read ≈ 640 pJ; ek MAC ≈ 0.2 pJ. Memory tax 1000× se dominate karta hai.

Step 4 — Systolic array (dataflow ka derivation)

PEs ko $K \times N$ grid mein arrange karo. Weights $W_{k j}$ load hoke PEs ke andar stationary rakhe jaate hain (isko weight-stationary kehte hain). Activations $X_{ik}$ left→right flow karti hain, har clock mein data ka ek column enter karta hai. Partial sums top→bottom flow karte hain, har PE karta hai:

$psum_{out} = psum_{in} + X_{in} \times W_{stored}$

Kyunki har activation, inject hone ke baad, PEs ki poori row par march karta hai, ek memory load bahut saare multiplies feed karta hai. Array fill hone ke baad ("pipeline fill" latency), yeh har cycle mein ek full result column produce karta hai.

Step 5 — Throughput aur efficiency (win derive karna)

$R \times C$ PEs ka ek array up to $R \cdot C$ MACs/cycle perform karta hai:

$Peak throughput = R \cdot C \cdot f_{clock} [MACs/s]$

$TOPS = \frac{2 \cdot R \cdot C \cdot f _{clock}}{1 0 ^{12}} (the 2 = one mult + one add)$

Precision: LOW BITS kyun jeetते hain

$Energy_{mult} \propto (bit-width)^{2}$

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Ek line mein NPU kya hai?

Ek domain-specific accelerator jo MAC units ke spatial array ke around bana hai, neural-network matrix multiply ke liye data reuse aur low precision use karke ops/joule maximize karne ke liye optimised hai.

NN inference mein kaunsa single operation dominate karta hai?

Matrix multiplication (dense multiply-accumulate / dot products).

Ek MAC operation define karo.

Multiply-accumulate:

a \times b + c

compute karo; ek multiply plus ek add running accumulator mein.

Systolic array kya hai?

Processing elements ki ek 2-D grid jahan data neighbours ke beech rhythmically flow karta hai, per-step global memory access ke bina massive data reuse enable karta hai.

Memory, arithmetic nahi, asli cost kyun hai?

Ek DRAM read ek MAC se ~1000× zyaada energy cost karta hai, toh naive loading dominate karta hai; reuse fetches amortise karti hai.

M×K by K×N matmul mein MACs ka formula?

M \cdot N \cdot K

Clock f par R×C PE array ka peak throughput?

R \cdot C \cdot f

MACs/s, yaani

2 R C f

ops/s.

Low precision (int8) zyaada MACs per area kyun deta hai?

Multiplier area/energy ~(bit-width)² ke saath scale karta hai, toh int8 fp32 se ~16× chhota hai.

Utilisation kya hai aur kyun care karna chahiye?

actual MACs ÷ (R·C·cycles); peak TOPS meaningless hai agar array chhoti/mismatched layers ya memory limits ki wajah se idle baitha rahe.

"Weight-stationary" ka matlab kya hai?

Weights ek baar load hoke har PE mein hold kiye jaate hain jabki activations stream through karti hain, weight reuse maximize karta hai.

Peak TOPS asli performance kyun nahi hai?

Yeh 100% utilisation aur unlimited bandwidth assume karta hai; real workloads aksar memory-bound hote hain ya array under-fill karte hain.

Recall Feynman: 12-saal ke bachche ko samjhao

Socho ek badi factory of tiny workers jo ek giant square grid mein arrange hain. Har worker sirf EK trick jaanta hai: "do numbers multiply karo aur answer apne pile mein add karo." Numbers rows ke saath walk karte hain jaise conveyor belt par log, aur har worker ek number paas aate waqt pakadta hai, apni chhoti trick karta hai, aur running total neeche pass karta hai. Kyunki thousands of workers apni trick ek saath karte hain, maths ka ek pahaad jo aapke laptop ko bahut waqt lagata, ek second mein ho jaata hai. PEs ki wo grid hi NPU hai — ek calculator factory sirf us type ki maths ke liye bani jo AI ko chahiye.

Connections

Systolic arrays — zyaadatar NPUs ke andar ka dataflow engine.
GPUs and SIMT — general parallel compute vs fixed-function NPUs.
Tensor cores — GPUs par bolted matmul units (NPUs ki taraf ek hybrid step).
Quantization and int8 inference — low precision kyun kaam karta hai.
Roofline model — compute-bound vs memory-bound decide karna.
Dataflow and data reuse — weight/output/row-stationary strategies.
Energy per operation — specialisation motivate karne wali physics.
Domain-specific architectures — broader design philosophy.