Cost optimization and inference latency — Hinglish notes

5.3.16 · AI-ML › MLOps & Deployment

WHAT — hum actually kya optimize kar rahe hain?

WHY percentiles, mean kyun nahi? Kyunki users tail feel karte hain. Agar mean latency 50 ms hai lekin $p 99$ 2 s hai, toh 100 mein se 1 request painfully slow hai — aur ek page jo 100 backend calls karta hai woh almost always ek baar us slow tail ko hit karega. Tail hi perceived quality ko dominate karta hai.

HOW latency banti hai (first principles se derive karo)

Ek single request ki latency stages ka sum hoti hai:

$L = L_{net} + L_{queue} + L_{compute} + L_{overhead}$

Har ek real, physical delay hai:

$L_{net}$ — network round trip (client ↔ server).
$L_{queue}$ — doosri requests ke peeche free worker ka intezaar.
$L_{compute}$ — actual forward pass.
$L_{overhead}$ — serialization, tokenization, pre/post-processing.

FLOPs se compute time

Derivation: work = (operations); rate = (operations/second). Time = work / rate = $F / (η P)$ . Bas yahi hai — yeh sirf arithmetic ke liye $time = distance / speed$ hai.

Queueing blow-up (servers "cliff se girte" kyun hain)

Server ko M/M/1 queue maano: requests rate $λ$ pe aati hain, har ek mean service time $s$ leti hai, toh service rate $μ = 1/ s$ . Utilization define karo $ρ = λ / μ = λ s$ .

Derivation sketch: M/M/1 mein system mein mean number $ρ / (1 - ρ)$ hota hai. Little's Law $N = λ \cdot E [L]$ se, $E [L] = \frac{N}{λ} = \frac{ρ / ( 1 - ρ )}{λ} = \frac{ρ}{λ ( 1 - ρ )} = \frac{s}{1 - ρ}$ $ρ = λ s \Rightarrow ρ / λ = s$ use karke.

WHY this matters: jab $ρ \to 1$ (server ~100% busy), $E [L] \to \infty$ . Latency nonlinearly explode karti hai. Yeh cost-vs-latency ka sabse important fact hai: hardware ko "hot" run karna paisa bachata hai lekin tail latency barbaad kar deta hai.

Figure — Cost optimization and inference latency

HOW cost banti hai

Derivation: cost/inference = (cost/second) / (inferences/second) = $(H /3600) / X$ . $1 0^{6}$ se multiply karo. Toh cost throughput ke inversely proportional hai — jo bhi $X$ badhaye (batching, quantization, better utilization) woh $$$ ghatata hai.

Bade levers (80/20 — inhi se ~80% wins milti hain)

Lever	$L$ pe effect	$C$ pe effect	Kyun
Dynamic batching	↑ (fill hone ka wait)	↓↓	kernel launch ko many requests pe amortize karo
Quantization (fp16/int8)	↓	↓	kam bytes move, faster arithmetic
Distillation / smaller model	↓	↓	kam FLOPs $F$
Caching (KV cache, response cache)	↓↓	↓	recompute skip karo
Autoscaling	$ρ$ safe rakhta hai	↓	sirf zaroori replicas ke liye pay karo
Right-sizing hardware	–	↓	chhote model ke liye A100 mat rent karo

Worked examples

Common mistakes (Steel-man + fix)

Flashcards

Mean ki jagah

p 99

latency kyun use karte hain?

Tail wahi hai jo users feel karte hain; request fan-out rare slow requests ko almost har page load pe hit karata hai.

M/M/1 mean latency formula aur uske variables batao.

E [L] = s / (1 - ρ)

, jahan

s

=mean service time,

ρ = λ s

=utilization.

Utilization

ρ \to 1

hone par latency kyun explode karti hai?

E [L] = s / (1 - ρ) \to \infty

; jab server almost hamesha busy ho toh queue waiting time bina bound ke badhta hai.

FLOPs se compute-time floor formula kya hai?

L_{compute} \geq F / (η P)

— work ko achievable rate (peak

P

times efficiency

η

) se divide karo.

Cost per inference

\propto 1/ X

kyun hai?

C_{1 M} = 1 0^{6} H / (3600 X)

; zyaada requests per second fixed hourly machine cost ko amortize karti hain.

Batching trade-off kya hai?

Bade batches throughput badhate hain (↓cost) lekin unhe fill hone ka wait karna latency badhata hai; max-wait timeout se cap karo.

Little's Law kya hai?

N = λ E [L]

— system mein average number = arrival rate × average time in system.

LLM inference latency ke liye do sabse saste bade wins kya hain?

KV-cache reuse aur quantization (fp16/int8) — dono compute aur bytes moved kam karte hain.

100% GPU utilization target kyun nahi karte?

ρ = 1

ke paas queueing latency diverge ho jaati hai;

ρ \approx 0.6

–0.8 target karo aur autoscale karo.

Recall Feynman: 12-saal ke bacche ko explain karo

Socho ek dukaan mein ek cashier hai. Agar log dheere aate hain, toh aapko jaldi serve milti hai. Jaise hi queue busy hoti jaati hai aur cashier almost kabhi rest nahi kar raha, wait balloon ho jaata hai — ek extra shopper huge delay add kar deta hai. Isliye hum apna computer 100% busy nahi rakhte: uski line bahut badi ho jaati hai. Zyaada logon ko saste mein serve karne ke liye, cashier kai orders ek saath bundle karti hai (batching) — lekin agar woh bundle karne ke liye zyaada der tak wait kare, toh pehle wale customers irritate ho jaate hain. Toh hum bundle karte hain, lekin sirf ek pal ke liye. Aur chhoti, halki machine use karna (simpler model, chhote numbers) har order ko jaldi khatam karta hai aur kam paisa lagta hai.

Connections

Model Quantization and Pruning
Autoscaling and Horizontal Scaling
Little's Law and Queueing Theory
Batching and Throughput
SLA SLO and Monitoring
Roofline Model of Hardware Performance
Model Distillation