The queueing questions below all lean on one formula. Here is why it is true, so you never treat it as a magic incantation.
Step 1 — Little's Law (a counting identity). Watch the server for a long time. Every request that enters must leave, so the average number inside, N, equals how fast they arrive, λ, times how long each stays, E[L]:
N=λE[L].
This is just "stuff in = flow in × time each stays" — think of people in a shop = arrivals/min × minutes each shops.
Step 2 — where does πn=(1−ρ)ρn come from? (the balance, made visual). Picture the system as a row of numbered boxes: box 0 = "empty", box 1 = "one request in system", box 2 = "two", and so on. Figure 1 draws this ladder. Requests arrive at rate λ, so at every box you step right (up one) at rate λ; the single server finishes at rate μ, so at every box you step left (down one) at rate μ.
In the long run each cut between box n and box n+1 must have equal traffic crossing each way — otherwise probability would pile up on one side forever. Let πn be the long-run chance of being in box n. Balancing the flow across the cut:
up over the cutλπn=down over the cutμπn+1⇒πn+1=μλπn=ρπn.
So each box is ρ times as likely as the one before it — a constant shrink factor ρ. Chaining from box 0: πn=ρnπ0. The chances must sum to 1, and ∑nρn=1/(1−ρ) for ρ<1, forcing π0=1−ρ. That gives the geometric law
πn=(1−ρ)ρn.
Step 3 — average the ladder. The average number in system is the mean of that geometric distribution:
N=∑n≥0n(1−ρ)ρn=1−ρρ.
Look at the red curve in Figure 2: harmless and flat until ρ≈0.8, then it rockets up — that shape is the whole story of the "latency cliff."
Step 4 — combine (Little's Law again). Put Step 3's N into Step 1 and use ρ=λs:
E[L]=λN=λρ/(1−ρ)=λλs/(1−ρ)=1−ρs.What it looks like: at ρ→0 you get s (no line, just the work); at ρ→1 the denominator →0 and E[L]→∞. That divergence is the cliff every trap below circles around.
Doubling replicas is not magic — it is the M/M/c model: c identical servers share one queue. The effective utilization drops to ρ=λs/c, because c workers share the same arrival stream.
Why "double replicas halves queue-delay" is only roughly true: halving ρ moves you down the nonlinears/(1−ρ) curve, so the queueing-delay reduction depends on where you started. From ρ=0.9→0.45 the wait shrinks dramatically; from ρ=0.3→0.15 it barely moves because there was almost no queue to remove. See Autoscaling and Horizontal Scaling and Little's Law and Queueing Theory.
Doubling the number of server replicas always halves the p99 latency.
False. In M/M/c, extra replicas lower ρ=λs/c and so cut queueing delay, but Lnet, Lcompute and Loverhead are floors replicas cannot touch — and how much queue-delay you remove depends nonlinearly on your starting ρ.
Running a GPU at higher utilization ρ always lowers cost per inference.
Mostly true for cost but with a hidden clause: higher ρ raises throughput X so C1M∝1/X falls — untilE[L]=s/(1−ρ) breaks the SLA, at which point you must add hardware anyway, erasing the saving.
If mean latency is well under the SLA, the SLA is safe.
False. The SLA is set on p99, and the mean can be tiny while the tail is huge; a page making 100 backend calls, each with a 1% chance of hitting the slow tail, has a 1−0.99100≈63% chance of at least one slow call — so the mean tells you little about felt quality.
Bigger batches always give more throughput.
False beyond the roofline. Once you saturate memory bandwidth (Roofline Model of Hardware Performance) throughput flattens, yet the wait-to-fill time keeps climbing — so latency rises for zero throughput gain.
Quantization to int8 is a free win because benchmarks show negligible accuracy drop.
False. Aggregate benchmarks hide the long tail; int8 can degrade badly on out-of-distribution or adversarial slices, so it is only free after you validate on those held-out slices.
An M/M/1 queue at ρ=1 has a finite but large mean latency.
False. E[L]=s/(1−ρ) has 1−ρ=0 in the denominator, so the mean diverges — the queue grows without bound, it is not merely "large".
Caching lowers cost but has no effect on latency.
False. A hit skips the entire forward pass, so it slashes Lcomputeand cost — that is why KV-cache and response caches are two-for-one wins.
Distillation and quantization are two names for the same optimization.
False. Model Distillation trains a smaller model (fewer FLOPs F); Model Quantization and Pruning keeps the architecture but uses fewer bits per number. They reduce Lcompute by different mechanisms and can be stacked.
Efficiency η in Lcompute≥F/(ηP) can equal 1 with perfect code.
Effectively false in practice — η<1 always because of memory stalls, kernel-launch overhead and un-fused ops; even flawless code hits the memory wall, so the floor is F/(ηP), not F/P.
Merging two servers into one M/M/1 with a shared queue is always worse than two separate M/M/1 queues.
False — it is the opposite. One shared queue feeding c servers (M/M/c) beats c isolated queues, because no server sits idle while another's private line grows; pooling smooths the load.
The error is optimizing the median. Half the requests are belowp50 and unaffected in feel; user pain lives at p95/p99, which this statement never measured.
"We're at 98% GPU utilization — beautifully efficient, ship it."
At ρ=0.98, E[L]=s/(1−0.98)=50s: latency is 50× the service time s. High utilization here means the SLA is almost certainly blown, not that the system is healthy.
"Set batch size B to 512 to maximize throughput."
Missing the max-wait cap. To fillB=512 you may wait far past the SLA, and past the roofline you gain no throughput — the correct control is a timeout that flushes a partial batch.
"Cost per 1M is high, so buy a faster (pricier) GPU."
Confuses hourly price H with cost-per-million C1M. Since C1M=106H/(3600X), a pricier machine raises H; it only helps if it raises throughput Xmore than it raises H. Often right-sizing down to cheaper hardware for a small model wins.
"Little's Law says more arrivals cause more latency, so throttle arrivals."
Misreads the law. N=λE[L] is a conservation identity, not a causal claim; λ rising raises N only because E[L] rises via ρ. The fix is capacity (lower ρ), not blindly throttling users.
"We cut F in half with a smaller model, so latency halved."
Only Lcompute halved. Total L=Lnet+Lqueue+Lcompute+Loverhead; if compute was a small slice, end-to-end latency barely moves — always ask which term dominates first.
"We added a second replica, so queueing delay must be exactly half."
The reduction is nonlinear, not a clean half. Halving ρ moves you along the curved s/(1−ρ); the drop is huge if you started near the cliff and negligible if you started nearly idle.
"Autoscaling failed us — latency spiked before new pods came up."
Not a design error, a lag. Scaling reacts on a delay while ρ overshoots toward 1; the fix is headroom (scale at ρ≈0.7, not 0.95) and predictive/pre-warmed scaling, per Autoscaling and Horizontal Scaling.
Why measure latency at percentiles instead of reporting one average?
Because perceived quality is set by the worst requests a user actually meets, and fan-out (one page → many backend calls) makes the rare tail likely to appear at least once — with 100 calls each 1%-risky, the chance of a slow one is ≈63%.
Why does the compute floor use ηP and not peak P?
Peak P is a hardware datasheet fantasy reached only by ideal kernels; real workloads stall on memory and overhead, so the achievable rate is ηP with efficiency η<1 — dividing work F by the honest rate gives the honest floor.
Why is cost per inference inversely proportional to throughput?
The machine's hourly price H is fixed whether it serves 1 or 1000 requests; spreading that fixed cost over more completed requests (C1M∝1/X) is exactly why batching and higher utilization save money.
Why does batching lower cost but risk latency?
A GPU processing one request wastes most cores, so grouping B requests costs slightly more compute but ~B× throughput X — the price is the wait to collect those B requests, which adds to Lqueue.
Why target ρ≈0.6–0.8 rather than as high as possible?
That range keeps E[L]=s/(1−ρ) finite and stable while still amortizing cost well; it also leaves headroom for traffic bursts and for autoscaling lag before ρ overshoots toward the cliff.
Why does one shared queue over c servers (M/M/c) beat c private queues?
Because pooling prevents the waste where one server idles while another's private line backs up; a shared queue always hands the next request to whichever worker frees first, smoothing load and shortening waits.
Why can two systems with identical hardware and model have wildly different tail latency?
Because tail latency is dominated by Lqueue, which depends on utilization ρ, batching policy, and neighbor interference — configuration and environment, not silicon, set the tail.
Why validate quantized models on adversarial and long-tail slices specifically?
Because low-bit arithmetic degrades most where the signal is subtle or out-of-distribution; aggregate accuracy averages that damage away, so only slice-level testing exposes it.
What happens to E[L]=s/(1−ρ) as ρ→0 (near-idle server)?
It approaches s, the raw service time — with no one else in line there is no queueing, so latency is just the work itself, the best case.
What is Lqueue for the very first request after a cold start?
Zero queueing, but Loverhead balloons — model load, weight paging and JIT warmup dominate, which is why cold-start latency is a separate, often larger problem than steady-state.
If arrival rate λ exceeds service rate μ (ρ>1), what does the formula say and what really happens?
The formula gives a negative "latency", which is nonsense — physically the queue grows unboundedly forever, so the system is unstable and you must add capacity, not tune it.
For a batch size of exactly B=1, what is the batching benefit?
None — you pay the per-request kernel-launch overhead with nothing to amortize it over, which is precisely the wasteful single-request case batching exists to fix.
In M/M/c, what happens as you add servers c while arrival rate λ stays fixed?
Utilization ρ=λs/c falls toward 0, so queueing delay vanishes and latency bottoms out at the compute floor s; beyond that point extra replicas only add idle cost, so autoscaling should stop.
At the roofline's compute-bound region, what does adding memory bandwidth buy you?
Nothing for throughput — you are limited by arithmetic, not data movement, so the fix is fewer FLOPs (distillation) or faster arithmetic (quantization), per Roofline Model of Hardware Performance.
Why can p99 latency spike on a shared GPU even when yourρ looks moderate?
Multi-tenant "noisy neighbors" — another tenant's job contends for memory bandwidth, cache, and SM occupancy, inflating your effective service time s; the queueing cliff then fires at a lower apparent utilization than you measured in isolation.
What SLA behavior should you expect during a traffic spike before autoscaling catches up?
A temporary ρ overshoot pushes p99 up sharply; a well-set SLO (SLA SLO and Monitoring) tolerates brief breaches within an error budget rather than demanding zero-violation perfection.
If a response cache has a 100% hit rate, what dominates latency?
Lnet plus a tiny lookup — compute vanishes entirely, so the network round trip becomes the irreducible floor, and no model optimization can help further.
Recall One-line summary of the traps
Almost every trap is one of three: (1) confusing mean with tail, (2) forgetting the s/(1−ρ) cliff near full utilization, or (3) treating a single lever's local win as an end-to-end win when another latency term dominates.
