Cost optimization and inference latency: Worked examples

This page is the drill hall for Cost optimization and inference latency. The parent gave you the formulas; here we run them through every case class — every sign, every limit, every degenerate input — so no scenario can surprise you in an interview or an outage.

We only use three formulas from the parent. Let us pin them down first so every symbol is earned.

Formula The three tools we drill

(T1) Queue latency — how long a request waits plus is served when a server is $ρ$ -full: $E [L] = \frac{s}{1 - ρ}, ρ = λ s, 0 \leq ρ < 1$ Here $s$ is the mean service time (seconds one request needs on a free worker), $λ$ is the arrival rate (requests per second), and $ρ$ (the Greek letter "rho") is the utilization — the fraction of time the server is busy. We also name the reciprocal of service time the service rate $μ = 1/ s$ (the letter "mu") — the number of requests one free worker can finish per second. With this, $ρ = λ s = λ / μ$ reads plainly as "arrivals divided by how fast we can clear them."

(T2) Compute floor — the fastest one forward pass can possibly be: $L_{compute} \geq \frac{F}{η P}$ $F$ = floating-point operations per inference, $P$ = peak hardware rate in FLOP/s, $η$ (the letter "eta", a fraction between 0 and 1) = how much of that peak you actually reach.

(T3) Cost per million — dollars to serve $1 0^{6}$ predictions: $C_{1 M} = \frac{1 0 ^{6} H}{3600 X}$ $H$ = machine cost per hour, $X$ = sustained throughput (requests per second, RPS). The $3600$ turns hours into seconds.

The scenario matrix

Each cell below is a case class — a qualitatively different situation the maths can produce. The worked examples that follow are labelled with the cell(s) they cover.

#	Case class	What is special about it	Covered by
A	Low load $ρ$ small	waiting ≈ 0, latency ≈ service time	Ex 1
B	High load $ρ \to 1^{-}$	the utilization cliff, near-divergence	Ex 2
C	Overload $ρ \geq 1$	formula breaks — queue is infinite	Ex 3
D	Zero / degenerate $λ = 0$ or $F = 0$	empty inputs, sanity limits	Ex 4
E	Compute-bound big $F$ , tiny queue	latency set by arithmetic, not waiting	Ex 5
F	Cost vs throughput move $X$	how $C$ scales when you batch	Ex 6
G	Real-world word problem	pick replica count to hold an SLA	Ex 7
H	Exam twist	solve backwards for the max safe $λ$	Ex 8
I	Combined limit	batching helps and hurts at once	Ex 9

Worked examples

Worked example Ex 2 — Case B: the utilization cliff

Same $s = 20 ms$ (→ $0.020$ s). Compare $ρ = 0.8, 0.95, 0.99$ .

Forecast: the jump from 0.8→0.95 is only 0.15 in $ρ$ . Does latency roughly double, or far more?

$ρ = 0.8 : E [L] = 0.020/ (1 - 0.8) = 0.020/0.2 = 0.100 s = 100 ms$ .
$ρ = 0.95 : E [L] = 0.020/0.05 = 0.400 s = 400 ms$ .
$ρ = 0.99 : E [L] = 0.020/0.01 = 2.000 s = 2000 ms$ . Why these steps? We hold hardware and model fixed and only vary how full we run them — isolating the queueing effect.

How to read the figure: the horizontal axis is utilization $ρ$ (0 = idle, 1 = always busy); the vertical axis is mean latency in ms. The teal curve is $E [L] = s / (1 - ρ)$ . Notice it is nearly flat across the shaded "safe zone" ( $ρ = 0.6$ – $0.8$ ) and then rears up into a near-vertical wall as $ρ \to 1$ . The three coloured dots are our results (100, 400, 2000 ms): read left-to-right and see how a tiny rightward step near the wall costs a huge vertical jump.

Verify: $0.8 \to 0.95$ (a $0.15$ rise) multiplied latency $4 \times$ ; $0.95 \to 0.99$ (only $0.04$ more) multiplied it another $5 \times$ . Nonlinear — exactly the "cliff." This is why the parent says target $ρ \approx 0.6$ – $0.8$ . ✅

Worked example Ex 3 — Case C: overload, the formula refuses to answer

$s = 20 ms$ (→ $0.020$ s), arrivals surge to $λ = 60$ RPS.

Forecast: plug in blindly and you get a negative latency. What does that mean physically?

$ρ = λ s = 60 \times 0.020 = 1.2$ .
Naive T1: $E [L] = 0.020/ (1 - 1.2) = 0.020/ (- 0.2) = - 0.100 s = - 100 ms$ . Why this step? To expose the trap: a negative time is nonsense.
Interpret: T1 is only valid for $0 \leq ρ < 1$ . When $ρ \geq 1$ , requests arrive faster than the server can clear them, so the queue grows without bound — mean latency is $\infty$ , not negative. Why this step? Every formula has a domain; leaving it silently gives garbage.

Verify: service rate $μ = 1/ s = 1/0.020 = 50$ RPS but arrivals are $60$ RPS. Backlog grows at $60 - 50 = 10$ req/s forever. A steady-state mean latency does not exist. ✅ (Fix: autoscale to add replicas so per-replica $λ$ drops below $50$ .)

Worked example Ex 5 — Case E: compute-bound, waiting is irrelevant

Model needs $F = 8 \times 1 0^{9}$ FLOP. GPU peak $P = 100$ TFLOP/s $= 1 0^{14}$ FLOP/s, efficiency $η = 0.4$ . Queue is empty ( $ρ \approx 0$ ).

Forecast: will latency be dominated by the $F / (η P)$ floor or by queue waiting?

Achievable rate $= η P = 0.4 \times 1 0^{14} = 4 \times 1 0^{13}$ FLOP/s. Why this step? Real kernels never hit peak; we must use the achievable rate, not $P$ .
$L_{compute} = \frac{8 \times 1 0 ^{9}}{4 \times 1 0 ^{13}} = 2 \times 1 0^{- 4} s = 0.2 ms$ . Why this step? Time = work / rate — the same $time = distance / speed$ idea for arithmetic.
Now assemble the full latency. With an empty queue, the service time equals the compute floor, i.e. $s = L_{compute} = 0.2$ ms, and the queue wait $E [L] - s = s (\frac{1}{1 - ρ} - 1) \to 0$ as $ρ \to 0$ . So total latency $\approx L_{compute} = 0.2$ ms. Why this step? $s$ and $L_{compute}$ are different concepts (mean service time vs the arithmetic floor); here the queue is empty, so they coincide numerically. We keep the names distinct to avoid the trap of thinking $s$ always equals the compute floor — with batching or overhead, $s > L_{compute}$ .

Verify: doubling $F$ from Ex-parent's $4 \times 1 0^{9}$ to $8 \times 1 0^{9}$ and raising $η$ from $0.3$ to $0.4$ : $\frac{8}{4} \times \frac{0.3}{0.4} = 2 \times 0.75 = 1.5 \times$ the parent's $0.13 ms \Rightarrow 0.2 ms$ . Consistent. ✅ (See Roofline Model of Hardware Performance for when $η$ itself is bandwidth-capped.)

Worked example Ex 7 — Case G: real-world word problem (choose replica count)

A service must hold mean latency $E [L] \leq 150$ ms (→ $0.150$ s). One replica has $s = 25 ms$ (→ $0.025$ s). Total traffic $λ_{tot} = 100$ RPS. How many replicas $n$ so that per-replica load keeps latency under budget? Assume traffic splits evenly, so each replica sees $λ = λ_{tot} / n$ .

Forecast: one replica alone — is it even below overload?

Per-replica service rate $μ = 1/ s = 1/0.025 = 40$ RPS. One replica at $λ = 100$ gives $ρ = λ s = 2.5 > 1$ → overloaded (Case C). So we need several. Why this step? Before choosing $n$ we must confirm a single replica cannot cope, and $μ$ is the ceiling on what one worker can clear per second — if $λ > μ$ we are instantly in the overload regime of Ex 3, so the answer is definitely $n \geq 2$ .
Solve T1 for the largest $ρ$ allowed: $0.150 = \frac{0.025}{1 - ρ} \Rightarrow 1 - ρ = \frac{0.025}{0.150} = 0.1667 \Rightarrow ρ_{m a x} = 0.8333$ . Why this step? Invert the latency budget into a utilization budget.
Per replica $ρ = λ s = \frac{100}{n} \times 0.025 = \frac{2.5}{n}$ . Require $\frac{2.5}{n} \leq 0.8333 \Rightarrow n \geq 3.0$ . Why this step? Convert the utilization cap into a replica count.
So $n = 3$ replicas (each sees $λ \approx 33.3$ RPS, $ρ = 0.833$ ).

Verify: at $n = 3$ , $ρ = 2.5/3 = 0.8333$ , $E [L] = 0.025/ (1 - 0.8333) = 0.025/0.1667 \approx 0.150 s = 150.0 ms$ — right on budget. At $n = 2$ , $ρ = 1.25 > 1$ (overload). So $3$ is the minimum. ✅ (This is exactly what Autoscaling and Horizontal Scaling automates via Little's Law and Queueing Theory.)

Worked example Ex 8 — Case H: exam twist (solve backwards for max safe

$λ$ ) Given $s = 40 ms$ (→ $0.040$ s) and an SLA of $E [L] \leq 100$ ms (→ $0.100$ s) on a single replica, find the maximum arrival rate $λ_{m a x}$ you may accept.

Forecast: more service time eats into the budget. Will $λ_{m a x}$ be well below the raw $μ = 25$ RPS?

Budget → utilization: $0.100 = \frac{0.040}{1 - ρ} \Rightarrow 1 - ρ = 0.4 \Rightarrow ρ_{m a x} = 0.6$ . Why this step? Turn the time budget into a $ρ$ cap first — cleaner than juggling $λ$ directly.
$ρ = λ s \Rightarrow λ_{m a x} = \frac{ρ _{m a x}}{s} = \frac{0.6}{0.040} = 15$ RPS. Why this step? Undo $ρ = λ s$ to recover the arrival rate.

Verify: at $λ = 15$ , $ρ = 15 \times 0.04 = 0.6$ , $E [L] = 0.040/0.4 = 0.100 s = 100 ms$ exactly. Note $λ_{m a x} = 15 < μ = 25$ : you must leave 40% headroom to hold the tail — the SLA-headroom lesson of SLA SLO and Monitoring. ✅

Worked example Ex 9 — Case I: batching helps

and hurts at once Batch size $b$ . Service time per batch is $s_{b} = 10 + 2 b$ ms (10 ms fixed launch, 2 ms/item). Throughput is $X = \frac{b}{s _{b}}$ where $s_{b}$ is in seconds. We compare $b = 1$ and $b = 16$ . To make the latency comparison fair we hold utilization fixed at $ρ = 0.5$ by choosing arrivals to match each batch's service time: set $λ = ρ / s_{b}$ (so a bigger, slower batch is fed a correspondingly lower arrival rate). This isolates the pure batch-size effect.

Forecast: bigger $b$ raises throughput (good, cheaper). But per-request latency? Guess before computing.

$b = 1 : s_{b} = 12$ ms $= 0.012$ s $\Rightarrow X = \frac{1}{0.012} \approx 83.3$ RPS. Why this step? We convert per-batch service time to throughput ( $X = b / s_{b}$ ) because cost (T3) is driven by $X$ , not by $s_{b}$ — this is the number that sets the price.
$b = 16 : s_{b} = 10 + 32 = 42$ ms $= 0.042$ s $\Rightarrow X = \frac{16}{0.042} \approx 381.0$ RPS. Why this step? The fixed 10 ms launch is amortized over 16 items, so throughput jumps ~ $4.6 \times$ — that is the whole reason batching is cheap.
Latency side, $b = 1$ : at $ρ = 0.5$ , $E [L] = s_{b} / (1 - 0.5) = 2 s_{b} = 2 \times 12 = 24$ ms. Why this step? We now switch from cost to latency; with $ρ$ pinned at $0.5$ the T1 factor is exactly $2$ , so $E [L] = 2 s_{b}$ and latency tracks the batch's service time directly.
Latency side, $b = 16$ : $E [L] = 2 s_{b} = 2 \times 42 = 84$ ms. Why this step? Same pinned $ρ$ , but the bigger batch's larger $s_{b}$ pushes latency up — the very batching that cut cost via $X$ raised latency via $s_{b}$ . (Check $ρ$ stays $0.5$ : $ρ = λ s_{b} = (ρ / s_{b}) s_{b} = ρ$ , so our arrival choice is self-consistent.)

Verify: throughput ratio $381.0/83.3 \approx 4.57 \times$ (cost drops ~ $4.6 \times$ by T3), while latency rose $84/24 = 3.5 \times$ . Both effects are real and opposed — precisely the trade-off the parent warns about, and why you cap batch size with a max-wait timeout. ✅

Recall Rebuild the whole matrix from one formula

$ρ < 1$ ? Use $E [L] = s / (1 - ρ)$ . $ρ \geq 1$ ? Latency is infinite — add replicas. $λ = 0$ ? Latency $= s$ . Big $F$ , empty queue? Latency $= F / (η P)$ . Want cheaper? Raise $X$ ; cost falls as $1/ X$ . Want to size replicas? Set a $ρ$ cap from your latency budget, then $n \geq λ_{tot} s / ρ_{m a x}$ .

Which cell is "the formula gives a negative number" a symptom of?

Case C — overload,

ρ \geq 1

, mean latency is actually infinite.

To hold a latency SLA, do you cap

ρ

λ

first?

Cap

ρ

from the budget, then convert to

λ_{m a x} = ρ_{m a x} / s

Batching's double effect in one line?

Raises

X

(cost

↓

via

1/ X

) but raises

s_{b}

(latency

↑

via

s / (1 - ρ)

What three assumptions does T1 (M/M/1) make?

Poisson arrivals, exponential service times, and a single server.