Cost optimization and inference latency: Worked examples

5.3.16 · D3 · AI-ML › MLOps & Deployment › Cost optimization and inference latency

Ye page Cost optimization and inference latency ka drill hall hai. Parent note ne aapko formulas diye; yahan hum unhe har case class se guzarte hain — har sign, har limit, har degenerate input — taaki koi bhi scenario interview ya outage mein surprise na kar sake.

Hum parent se sirf teen formulas use karte hain. Pehle unhe pin karte hain taaki har symbol ka matlab clear ho.

Formula Teen tools jinhe hum drill karte hain

(T1) Queue latency — ek request kitni der wait karti hai aur serve hoti hai jab server $ρ$ -full ho: $E [L] = \frac{s}{1 - ρ}, ρ = λ s, 0 \leq ρ < 1$ Yahan $s$ mean service time hai (seconds mein ek request ko ek free worker par kitna waqt chahiye), $λ$ arrival rate hai (requests per second), aur $ρ$ (Greek letter "rho") utilization hai — server kitne fraction time busy rehta hai. Hum service time ka reciprocal service rate $μ = 1/ s$ (letter "mu") bhi kehte hain — ek free worker ek second mein kitni requests finish kar sakta hai. Isse, $ρ = λ s = λ / μ$ saaf padhta hai "arrivals divided by how fast we can clear them."

(T2) Compute floor — ek forward pass se jo fastest possible ho sakta hai: $L_{compute} \geq \frac{F}{η P}$ $F$ = floating-point operations per inference, $P$ = peak hardware rate in FLOP/s, $η$ (letter "eta", 0 aur 1 ke beech ka fraction) = aap us peak ka kitna actually reach karte hain.

(T3) Cost per million — $1 0^{6}$ predictions serve karne ke dollars: $C_{1 M} = \frac{1 0 ^{6} H}{3600 X}$ $H$ = machine cost per hour, $X$ = sustained throughput (requests per second, RPS). $3600$ hours ko seconds mein convert karta hai.

Scenario matrix

Neeche har cell ek case class hai — mathematically ek qualitatively alag situation. Jo worked examples follow karte hain unhe unke cover kiye gaye cell(s) ke saath label kiya gaya hai.

#	Case class	Isme kya special hai	Covered by
A	Low load $ρ$ small	waiting ≈ 0, latency ≈ service time	Ex 1
B	High load $ρ \to 1^{-}$	utilization cliff, near-divergence	Ex 2
C	Overload $ρ \geq 1$	formula breaks — queue infinite hai	Ex 3
D	Zero / degenerate $λ = 0$ ya $F = 0$	empty inputs, sanity limits	Ex 4
E	Compute-bound bada $F$ , tiny queue	latency arithmetic se set hoti hai, waiting se nahi	Ex 5
F	Cost vs throughput $X$ move karo	$C$ kaise scale karta hai jab aap batch karo	Ex 6
G	Real-world word problem	SLA hold karne ke liye replica count choose karo	Ex 7
H	Exam twist	backwards solve karo max safe $λ$ ke liye	Ex 8
I	Combined limit	batching ek saath help bhi karta hai aur hurt bhi	Ex 9

Worked examples

Worked example Ex 1 — Case A: low load, waiting negligible hai

Service time $s = 20 ms$ (→ $0.020$ s), arrival rate $λ = 5$ RPS.

Forecast: ab mean latency guess karo. Kya yeh 20 ms ke paas hogi ya zyada upar?

Seconds mein convert karo, phir $ρ = λ s = 5 \times 0.020 = 0.10$ compute karo. Yeh step kyun? $ρ$ batata hai server kitna full hai; T1 mein sab kuch isi par depend karta hai. Units: $(req/s) \times (s/req)$ = dimensionless — acha, ek fraction.
T1 seconds mein apply karo: $E [L] = \frac{0.020}{1 - 0.10} = \frac{0.020}{0.90} \approx 0.0222 s$ , phir wapas convert karo: $= 22.2 ms$ . Yeh step kyun? Low load par denominator $1 - ρ \approx 1$ , isliye latency ≈ service time.

Verify: $22.2 ms$ raw $20 ms$ service se sirf $11%$ upar hai — intuition se match karta hai ki almost-empty queue aapko barely delay karti hai. Units: seconds mein compute kiya, ms mein report kiya. ✅

Worked example Ex 2 — Case B: utilization cliff

Same $s = 20 ms$ (→ $0.020$ s). Compare karo $ρ = 0.8, 0.95, 0.99$ .

Forecast: $ρ$ mein 0.8→0.95 ka jump sirf 0.15 hai. Kya latency roughly double hogi, ya bahut zyada?

$ρ = 0.8 : E [L] = 0.020/ (1 - 0.8) = 0.020/0.2 = 0.100 s = 100 ms$ .
$ρ = 0.95 : E [L] = 0.020/0.05 = 0.400 s = 400 ms$ .
$ρ = 0.99 : E [L] = 0.020/0.01 = 2.000 s = 2000 ms$ . Yeh steps kyun? Hum hardware aur model fixed rakhte hain aur sirf kitna full run karte hain woh vary karte hain — queueing effect isolate karne ke liye.

Figure kaise padhen: horizontal axis utilization $ρ$ hai (0 = idle, 1 = hamesha busy); vertical axis mean latency ms mein hai. Teal curve $E [L] = s / (1 - ρ)$ hai. Notice karo ki yeh shaded "safe zone" ( $ρ = 0.6$ – $0.8$ ) mein almost flat hai aur phir jaise $ρ \to 1$ near-vertical wall mein upar uthti hai. Teen coloured dots hamare results hain (100, 400, 2000 ms): left-to-right padho aur dekho ki wall ke paas ek tiny rightward step kitna bada vertical jump cost karta hai.

Verify: $0.8 \to 0.95$ ( $0.15$ rise) ne latency $4 \times$ multiply ki; $0.95 \to 0.99$ (sirf $0.04$ zyada) ne ise aur $5 \times$ multiply kiya. Nonlinear — exactly "cliff." Isliye parent kehta hai $ρ \approx 0.6$ – $0.8$ target karo. ✅

Worked example Ex 3 — Case C: overload, formula answer dene se mana kar deta hai

$s = 20 ms$ (→ $0.020$ s), arrivals surge karte hain $λ = 60$ RPS tak.

Forecast: blindly plug in karo toh negative latency milti hai. Physically iska matlab kya hai?

$ρ = λ s = 60 \times 0.020 = 1.2$ .
Naive T1: $E [L] = 0.020/ (1 - 1.2) = 0.020/ (- 0.2) = - 0.100 s = - 100 ms$ . Yeh step kyun? Trap expose karne ke liye: negative time nonsense hai.
Interpret karo: T1 sirf $0 \leq ρ < 1$ ke liye valid hai. Jab $ρ \geq 1$ hota hai, requests server se faster aate hain jitnaa wo clear kar sakta hai, toh queue grow karta hai without bound — mean latency $\infty$ hai, negative nahi. Yeh step kyun? Har formula ka ek domain hota hai; silently usse bahar jaane par garbage milta hai.

Verify: service rate $μ = 1/ s = 1/0.020 = 50$ RPS lekin arrivals $60$ RPS hain. Backlog $60 - 50 = 10$ req/s ki speed se hamesha badhta hai. Steady-state mean latency exist nahi karti. ✅ (Fix: autoscale karke replicas add karo taaki per-replica $λ$ $50$ se neeche aaye.)

Worked example Ex 5 — Case E: compute-bound, waiting irrelevant hai

Model ko $F = 8 \times 1 0^{9}$ FLOP chahiye. GPU peak $P = 100$ TFLOP/s $= 1 0^{14}$ FLOP/s, efficiency $η = 0.4$ . Queue empty hai ( $ρ \approx 0$ ).

Forecast: kya latency $F / (η P)$ floor se dominate hogi ya queue waiting se?

Achievable rate $= η P = 0.4 \times 1 0^{14} = 4 \times 1 0^{13}$ FLOP/s. Yeh step kyun? Real kernels kabhi peak hit nahi karte; hume achievable rate use karni chahiye, $P$ nahi.
$L_{compute} = \frac{8 \times 1 0 ^{9}}{4 \times 1 0 ^{13}} = 2 \times 1 0^{- 4} s = 0.2 ms$ . Yeh step kyun? Time = work / rate — same $time = distance / speed$ idea arithmetic ke liye.
Ab full latency assemble karo. Empty queue ke saath, service time compute floor ke barabar hai, yaani $s = L_{compute} = 0.2$ ms, aur queue wait $E [L] - s = s (\frac{1}{1 - ρ} - 1) \to 0$ jaise $ρ \to 0$ . Isliye total latency $\approx L_{compute} = 0.2$ ms. Yeh step kyun? $s$ aur $L_{compute}$ alag concepts hain (mean service time vs arithmetic floor); yahan queue empty hai, isliye numerically coincide karte hain. Hum names alag rakhte hain is trap se bachne ke liye ki $s$ hamesha compute floor ke barabar hota hai — batching ya overhead ke saath, $s > L_{compute}$ .

Verify: $F$ ko Ex-parent ke $4 \times 1 0^{9}$ se double karke $8 \times 1 0^{9}$ aur $η$ ko $0.3$ se $0.4$ raise karke: $\frac{8}{4} \times \frac{0.3}{0.4} = 2 \times 0.75 = 1.5 \times$ parent ka $0.13 ms \Rightarrow 0.2 ms$ . Consistent. ✅ (Dekho Roofline Model of Hardware Performance jab $η$ khud bandwidth-capped ho.)

Worked example Ex 7 — Case G: real-world word problem (replica count choose karo)

Ek service ko mean latency $E [L] \leq 150$ ms (→ $0.150$ s) hold karni hai. Ek replica ka $s = 25 ms$ (→ $0.025$ s) hai. Total traffic $λ_{tot} = 100$ RPS hai. Kitne replicas $n$ chahiye taaki per-replica load latency budget ke andar rakhe? Assume karo traffic evenly split hoti hai, isliye har replica $λ = λ_{tot} / n$ dekhta hai.

Forecast: ek akela replica — kya woh overload se bhi neeche hai?

Per-replica service rate $μ = 1/ s = 1/0.025 = 40$ RPS. Ek replica $λ = 100$ par $ρ = λ s = 2.5 > 1$ deta hai → overloaded (Case C). Isliye hume kaafi replicas chahiye. Yeh step kyun? $n$ choose karne se pehle hume confirm karna hai ki ek akela replica cope nahi kar sakta, aur $μ$ woh ceiling hai jo ek worker per second clear kar sakta hai — agar $λ > μ$ hum turant Ex 3 ke overload regime mein hain, isliye answer definitely $n \geq 2$ hai.
Allowed maximum $ρ$ ke liye T1 solve karo: $0.150 = \frac{0.025}{1 - ρ} \Rightarrow 1 - ρ = \frac{0.025}{0.150} = 0.1667 \Rightarrow ρ_{m a x} = 0.8333$ . Yeh step kyun? Latency budget ko utilization budget mein invert karo.
Per replica $ρ = λ s = \frac{100}{n} \times 0.025 = \frac{2.5}{n}$ . Require karo $\frac{2.5}{n} \leq 0.8333 \Rightarrow n \geq 3.0$ . Yeh step kyun? Utilization cap ko replica count mein convert karo.
Isliye $n = 3$ replicas (har ek $λ \approx 33.3$ RPS, $ρ = 0.833$ dekhta hai).

Verify: $n = 3$ par, $ρ = 2.5/3 = 0.8333$ , $E [L] = 0.025/ (1 - 0.8333) = 0.025/0.1667 \approx 0.150 s = 150.0 ms$ — bilkul budget par. $n = 2$ par, $ρ = 1.25 > 1$ (overload). Isliye $3$ minimum hai. ✅ (Yahi cheez Autoscaling and Horizontal Scaling Little's Law and Queueing Theory ke zariye automate karta hai.)

Worked example Ex 8 — Case H: exam twist (max safe

$λ$ ke liye backwards solve karo) $s = 40 ms$ (→ $0.040$ s) aur ek single replica par $E [L] \leq 100$ ms (→ $0.100$ s) ka SLA diya gaya hai, maximum arrival rate $λ_{m a x}$ nikalo jo aap accept kar sakte hain.

Forecast: zyada service time budget mein eat karta hai. Kya $λ_{m a x}$ raw $μ = 25$ RPS se bahut neeche hoga?

Budget → utilization: $0.100 = \frac{0.040}{1 - ρ} \Rightarrow 1 - ρ = 0.4 \Rightarrow ρ_{m a x} = 0.6$ . Yeh step kyun? Time budget ko pehle $ρ$ cap mein badlo — $λ$ directly juggle karne se cleaner hai.
$ρ = λ s \Rightarrow λ_{m a x} = \frac{ρ _{m a x}}{s} = \frac{0.6}{0.040} = 15$ RPS. Yeh step kyun? Arrival rate recover karne ke liye $ρ = λ s$ undo karo.

Verify: $λ = 15$ par, $ρ = 15 \times 0.04 = 0.6$ , $E [L] = 0.040/0.4 = 0.100 s = 100 ms$ exactly. Note karo $λ_{m a x} = 15 < μ = 25$ : aapko tail hold karne ke liye 40% headroom chhodni hai — SLA SLO and Monitoring ka SLA-headroom lesson. ✅

Worked example Ex 9 — Case I: batching ek saath help

bhi karta hai aur hurt bhi Batch size $b$ . Batch per service time $s_{b} = 10 + 2 b$ ms hai (10 ms fixed launch, 2 ms/item). Throughput hai $X = \frac{b}{s _{b}}$ jahan $s_{b}$ seconds mein hai. Hum $b = 1$ aur $b = 16$ compare karte hain. Latency comparison fair rakhne ke liye hum utilization $ρ = 0.5$ par fixed rakhte hain arrivals ko har batch ke service time se match karke: set karo $λ = ρ / s_{b}$ (isliye ek bada, slow batch correspondingly lower arrival rate ko feed ki jaati hai). Yeh pure batch-size effect isolate karta hai.

Forecast: bada $b$ throughput badhata hai (acha, sasta). Lekin per-request latency? Compute karne se pehle guess karo.

$b = 1 : s_{b} = 12$ ms $= 0.012$ s $\Rightarrow X = \frac{1}{0.012} \approx 83.3$ RPS. Yeh step kyun? Hum per-batch service time ko throughput mein convert karte hain ( $X = b / s_{b}$ ) kyunki cost (T3) $X$ se drive hoti hai, $s_{b}$ se nahi — yeh number price set karta hai.
$b = 16 : s_{b} = 10 + 32 = 42$ ms $= 0.042$ s $\Rightarrow X = \frac{16}{0.042} \approx 381.0$ RPS. Yeh step kyun? Fixed 10 ms launch 16 items par amortize ho jaata hai, isliye throughput ~ $4.6 \times$ jump karta hai — yahi batching ke saste hone ki poori wajah hai.
Latency side, $b = 1$ : $ρ = 0.5$ par, $E [L] = s_{b} / (1 - 0.5) = 2 s_{b} = 2 \times 12 = 24$ ms. Yeh step kyun? Ab hum cost se latency par switch karte hain; $ρ$ $0.5$ par pinned hone se T1 factor exactly $2$ hai, isliye $E [L] = 2 s_{b}$ aur latency seedha batch ke service time se track karti hai.
Latency side, $b = 16$ : $E [L] = 2 s_{b} = 2 \times 42 = 84$ ms. Yeh step kyun? Same pinned $ρ$ , lekin bade batch ka bada $s_{b}$ latency upar push karta hai — wahi batching jo $X$ ke zariye cost cut ki usne $s_{b}$ ke zariye latency badhaa di. (Check karo $ρ$ $0.5$ rehta hai: $ρ = λ s_{b} = (ρ / s_{b}) s_{b} = ρ$ , isliye hamara arrival choice self-consistent hai.)

Verify: throughput ratio $381.0/83.3 \approx 4.57 \times$ (cost ~ $4.6 \times$ T3 se girta hai), jabki latency $84/24 = 3.5 \times$ badhi. Dono effects real aur opposed hain — exactly wahi trade-off jiske baare mein parent warn karta hai, aur isliye aap batch size ko max-wait timeout ke saath cap karte hain. ✅

Recall Ek formula se poora matrix rebuild karo

$ρ < 1$ ? Use karo $E [L] = s / (1 - ρ)$ . $ρ \geq 1$ ? Latency infinite hai — replicas add karo. $λ = 0$ ? Latency $= s$ . Bada $F$ , empty queue? Latency $= F / (η P)$ . Sasta chahiye? $X$ badhao; cost $1/ X$ ki tarah girta hai. Replicas size karne hain? $ρ$ cap apne latency budget se set karo, phir $n \geq λ_{tot} s / ρ_{m a x}$ .

"Formula negative number deta hai" — yeh symptom kis cell ka hai?

Case C — overload,

ρ \geq 1

, mean latency actually infinite hai.

Latency SLA hold karne ke liye, pehle

ρ

cap karo ya

λ

Budget se

ρ

cap karo, phir

λ_{m a x} = ρ_{m a x} / s

mein convert karo.

Ek line mein batching ka double effect?

X

badhata hai (cost

↓

via

1/ X

) lekin

s_{b}

badhata hai (latency

↑

via

s / (1 - ρ)

T1 (M/M/1) kaunse teen assumptions karta hai?

Poisson arrivals, exponential service times, aur ek single server.