Cost optimization and inference latency

WHAT are we actually optimizing?

WHY percentiles, not the mean? Because users feel the tail. If mean latency is 50 ms but $p99$ is 2 s, then 1 in 100 requests is painfully slow — and a page that makes 100 backend calls will almost always hit that slow tail once. The tail dominates perceived quality.

HOW latency is built up (derive from first principles)

A single request's latency is a sum of stages:

$L = L_{\text{net}} + L_{\text{queue}} + L_{\text{compute}} + L_{\text{overhead}}$

Each is a real, physical delay:

$L_{\text{net}}$ — network round trip (client ↔ server).
$L_{\text{queue}}$ — waiting behind other requests for a free worker.
$L_{\text{compute}}$ — the actual forward pass.
$L_{\text{overhead}}$ — serialization, tokenization, pre/post-processing.

Compute time from FLOPs

Derivation: work = (operations) ; rate = (operations/second). Time = work / rate = $F/(\eta P)$ . That's it — it's just $\text{time}=\text{distance}/\text{speed}$ for arithmetic.

The queueing blow-up (why servers "fall off a cliff")

Model the server as an M/M/1 queue: requests arrive at rate $\lambda$ , each takes mean service time $s$ , so service rate $\mu = 1/s$ . Define utilization $\rho = \lambda/\mu = \lambda s$ .

Derivation sketch: In M/M/1 the mean number in system is $\rho/(1-\rho)$ . By Little's Law $N = \lambda \cdot \mathbb{E}[L]$ , so $\mathbb{E}[L] = \frac{N}{\lambda} = \frac{\rho/(1-\rho)}{\lambda} = \frac{\rho}{\lambda(1-\rho)} = \frac{s}{1-\rho}$ using $\rho = \lambda s \Rightarrow \rho/\lambda = s$ .

WHY this matters: as $\rho \to 1$ (server ~100% busy), $\mathbb{E}[L]\to\infty$ . Latency explodes nonlinearly. This is the single most important cost-vs-latency fact: running hardware "hot" saves money but destroys tail latency.

Figure — Cost optimization and inference latency

HOW cost is built up

Derivation: cost/inference = (cost/second) / (inferences/second) = $(H/3600)/X$ . Multiply by $10^6$ . So cost is inversely proportional to throughput — anything that raises $X$ (batching, quantization, better utilization) lowers $$$.

The big levers (80/20 — these give ~80% of wins)

Lever	Effect on $L$	Effect on $C$	Why
Dynamic batching	↑ (wait to fill)	↓↓	amortize kernel launch over many requests
Quantization (fp16/int8)	↓	↓	fewer bytes moved, faster arithmetic
Distillation / smaller model	↓	↓	fewer FLOPs $F$
Caching (KV cache, response cache)	↓↓	↓	skip recompute
Autoscaling	keeps $\rho$ safe	↓	pay only for needed replicas
Right-sizing hardware	–	↓	don't rent an A100 for a tiny model

Worked examples

Common mistakes (Steel-man + fix)

Flashcards

Why use

p99

latency instead of mean?

The tail is what users feel; request fan-out makes rare slow requests hit almost every page load.

State the M/M/1 mean latency formula and its variables.

\mathbb{E}[L]=s/(1-\rho)

, where

s

=mean service time,

\rho=\lambda s

=utilization.

Why does latency explode as utilization

\rho\to 1

\mathbb{E}[L]=s/(1-\rho)\to\infty

; queue waiting time grows without bound when the server is nearly always busy.

Compute-time floor formula from FLOPs?

L_{\text{compute}}\ge F/(\eta P)

— work divided by achievable rate (peak

P

times efficiency

\eta

Why is cost per inference

\propto 1/X

C_{1M}=10^6 H/(3600 X)

; more requests per second amortize the fixed hourly machine cost.

What is the batching trade-off?

Larger batches raise throughput (↓cost) but waiting to fill them raises latency; cap with a max-wait timeout.

Little's Law?

N=\lambda\,\mathbb{E}[L]

— average number in system = arrival rate × average time in system.

Two cheapest big wins for LLM inference latency?

KV-cache reuse and quantization (fp16/int8) — both cut compute and bytes moved.

Why not target 100% GPU utilization?

Queueing latency diverges near

\rho=1

; target

\rho\approx0.6

–0.8 and autoscale.

Recall Feynman: explain to a 12-year-old

Imagine one cashier at a shop. If people arrive slowly, you're served fast. As the line gets busier and the cashier is almost never resting, the wait balloons — one extra shopper adds huge delay. That's why we don't keep our computer 100% busy: it gets a giant line. To serve more people cheaply, the cashier bundles several orders together (batching) — but if she waits too long to bundle, early customers get annoyed. So we bundle, but only for a moment. And using a smaller, lighter machine (a simpler model, smaller numbers) lets each order finish quicker and costs less.

Connections

Model Quantization and Pruning
Autoscaling and Horizontal Scaling
Little's Law and Queueing Theory
Batching and Throughput
SLA SLO and Monitoring
Roofline Model of Hardware Performance
Model Distillation

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, inference deploy karte waqt teen cheezein aapस ko balance karni padti hain: latency (ek prediction kitni jaldi aati hai), throughput (ek second mein kitni predictions), aur cost (paisa per prediction). Tinon ko ek saath jeetना mushkil hai — ek ko push karo to doosra bigadta hai. Engineer ka kaam hai: sabse sasta setup dhoondhna jo phir bhi latency SLA (jaise " $p99 \le 200$ ms") ko meet kare.

Sabse important baat: server ko 100% busy mat chalao. Formula hai $\mathbb{E}[L]=s/(1-\rho)$ , jahan $\rho$ utilization hai. Jaise-jaise $\rho$ 1 ke paas jaata hai, latency phat jaati hai — 50% busy pe 40 ms, 99% busy pe 2000 ms! Same hardware, sirf zyada bhara hua. Isliye $\rho$ ko lagbhag 0.7 pe rakho aur autoscaling use karo. Ye queueing theory ka Little's Law ( $N=\lambda L$ ) se aata hai.

Cost kam karne ka master lever hai batching. GPU ko akeli request bhejo to zyादातर cores waste ho jaate hain; 32 requests ek saath do to throughput $X$ kai guna badh jaata hai, aur cost $C \propto 1/X$ hoti hai — matlab throughput badha to cost automatically gir gayi. Lekin batch bharne ke liye thoda wait karna padta hai, jisse latency badhti hai — isliye ek max-wait timeout lagao. Baaki bade wins: quantization (fp16/int8 — kam bytes, fast math), KV-cache (dobara compute mat karo), aur distillation (chhota model, kam FLOPs).

Yaad rakho BQ-CACHE: Batch, Quantize, Cache, Autoscale, Cut-model, Hardware-rightsize, Evaluate-tail. Aur hamesha $p95/p99$ dekho, mean nahi — kyunki user tail latency feel karta hai, aur ek page 100 backend calls kare to slow tail lagbhag har baar hit hota hai.