Visual walkthrough — Cost optimization and inference latency

Before we start, three plain-English promises about symbols: every letter gets a picture before it gets an equation, and we walk the safe zone, the danger zone, and the two degenerate ends (empty server, full server).

---

Step 1 — What is a "request", and what is a "server"?

WHAT. Picture a single coffee counter. Customers walk up (these are requests — one prediction someone wants). There is one barista (this is our server — one GPU worker doing one forward pass at a time). If the barista is busy, new customers wait in a line (the queue).

WHY start here. Every symbol later — $\lambda$, $s$, $\rho$, $L$ — is just a number attached to something in this picture. If we don't nail the picture first, the algebra is meaningless.

PICTURE. Look at Figure s01: customers (coral dots) flow in from the left, wait in a lavender queue, get served by the mint barista, and leave on the right.

---

Step 2 — Turn "how slow is the barista" into a rate

WHAT. The service time $s$ is seconds per customer. It is often handier to flip it into customers per second. We call that flipped number the service rate $\mu$ (mu):

$\mu =\underset{\text{customers per second}}{\underbrace{\frac{1}{s}}}\text{e.g. }s=0.02\text{ s}\Rightarrow \mu =50\text{ /s}$

Here $s$ (seconds/customer) sits in the denominator so that $\mu$ comes out as customers/second — the reciprocal literally swaps the units.

WHY. We now have two things measured in the same unit (per second): arrivals $\lambda$ and the counter's top speed $\mu$. Same units means we are allowed to compare them by dividing — which is exactly what Step 3 does.

PICTURE. Figure s02: a slow barista ($\mu$ small, wide gaps between finished cups) vs a fast barista ($\mu$ large, cups flying out).

---

Step 3 — Utilization: what fraction of the time is the barista busy?

WHAT. Divide how fast work arrives by how fast the counter can clear it:

$\rho =\frac{\lambda }{\mu }=\frac{\stackrel{\text{arrive /s}}{\overbrace{\lambda }}}{\underset{\text{max serve /s}}{\underbrace{\mu }}}=\lambda \cdot s$

The last equality uses $\mu =1/s$ from Step 2, so dividing by $\mu$ is the same as multiplying by $s$.

WHY divide, and why is this the key number? Because a ratio strips away scale: whether it's a coffee shop or a GPU farm, $\rho$ answers one universal question — "how close to maxed-out are we?" That single number will turn out to control everything.

Edge check on the definition:

• $\rho =0$ → nobody arrives, barista always idle.
• $\rho =1$ → arrivals exactly match top speed; barista never rests.
• $\rho >1$ → customers arrive faster than they can ever be served → the line grows forever. So we will only ever allow $0\le \rho <1$.

PICTURE. Figure s03: a "busy-meter" gauge sweeping from empty (idle) to full (pinned), with the danger zone near 1 shaded coral.

---

Step 4 — Little's Law: the bridge between "how many waiting" and "how long you wait"

WHAT. Here is a law so simple it feels illegal. Let $N$ be the average number of customers in the shop (waiting + being served), and $L$ the average time each customer spends in the shop (their total latency). Then:

$\boxed{N=\lambda \cdot L}$

Reading it term by term: $N$ = people inside; $\lambda$ = people arriving per second; $L$ = seconds each stays.

WHY it's true (and why we need it). Think of it as filling a bathtub. Water arrives at $\lambda$ liters/second and each liter stays $L$ seconds before draining. The steady water level is arrivals-rate × stay-time = $\lambda L$. Nothing about ML, nothing about coffee — just conservation of stuff flowing through. We need this because it is the only bridge connecting a headcount ($N$, easy to reason about) to a wait time ($L$, what we actually care about). See Little's Law and Queueing Theory.

PICTURE. Figure s04: the bathtub — inflow $\lambda$, water level $N$, each drop labelled with its dwell time $L$.

---

Step 5 — How many are in the shop? The magic $\rho /(1-\rho )$

WHAT. For this "random arrivals, random service times, one server" shop (its formal name is an M/M/1 queue — the M's just mean "memoryless / random"), a standard result says the average number inside is:

$N=\frac{\rho }{1-\rho }$

Here the numerator $\rho$ pushes $N$ up (busier = more people), and the denominator $1-\rho$ (the fraction of time the barista is free) shrinks toward $0$ as $\rho \to 1$, which blows $N$ up.

WHY this shape — where does $1-\rho$ come from? Intuitively, whenever the barista finally has a free moment (probability $1-\rho$ of being free), the backlog gets a chance to drain. As $\rho \to 1$ those free moments vanish, so the backlog almost never drains and piles up. That is why the free fraction $1-\rho$ — not the busy fraction — sits in the denominator.

Edge checks:

• $\rho =0$: $N=0/1=0$ — empty shop. ✓
• $\rho =0.5$: $N=0.5/0.5=1$ — on average one person inside.
• $\rho =0.9$: $N=0.9/0.1=9$ — nine people piled up.
• $\rho \to 1$: $N\to \infty$ — infinite line.

PICTURE. Figure s05: the curve $N=\rho /(1-\rho )$ climbing gently, then rocketing to a vertical wall at $\rho =1$.

---

Step 6 — Combine them: the latency formula appears

WHAT. Now we let Little's Law (Step 4) and the headcount (Step 5) collide. Little's Law rearranged gives the wait:

$L=\frac{N}{\lambda }$

Substitute $N=\frac{\rho }{1-\rho }$:

$L=\frac{1}{\lambda }\cdot \frac{\rho }{1-\rho }=\frac{\rho }{\lambda (1-\rho )}$

Finally recall from Step 3 that $\rho =\lambda s$, so $\frac{\rho }{\lambda }=s$. The $\lambda$ cancels:

$\boxed{\mathbb{E}[L]=\frac{s}{1-\rho }}$

Term by term: $s$ is the raw time one customer would take with nobody ahead; the factor $\frac{1}{1-\rho }$ is the congestion multiplier — how many times worse the wait gets because the shop is busy.

WHY this is the whole point. The multiplier $\frac{1}{1-\rho }$ depends only on how full you run the server. At $\rho =0$ it equals $1$ (no penalty). As $\rho \to 1$ it $\to \infty$. Same barista, same $s$ — busyness alone decides your fate.

PICTURE. Figure s06: the algebra visualized — a base bar of height $s$ getting stretched by the congestion multiplier as $\rho$ rises.

---

Step 7 — Walk the whole curve: safe zone, danger zone, and both degenerate ends

WHAT. Let's plug numbers into $\mathbb{E}[L]=s/(1-\rho )$ with $s=20\text{ ms}$ and read off every regime:

$\rho$ (busyness)	$1-\rho$ (free)	$\mathbb{E}[L]$	regime
$0$	$1$	$20$ ms	degenerate: idle, no penalty
$0.5$	$0.5$	$40$ ms	safe — $2\times$
$0.8$	$0.2$	$100$ ms	edge of safe — $5\times$
$0.9$	$0.1$	$200$ ms	danger — $10\times$
$0.99$	$0.01$	$2000$ ms	cliff — $100\times$
$\to 1^{-}$	$\to 0^{+}$	$\to \infty$	degenerate: unbounded

WHY show all of them. The curve is not a straight line — the jump from $0.9$ to $0.99$ (a tiny $9\%$ more busyness) multiplies latency by $10$. This nonlinearity is the cost-vs-latency fact. Notice: going $0.5\to 0.99$ lets you serve the same traffic on about half the machines (you run each twice as full), but latency multiplies by $50$. That is the trap the parent's "max out the GPU" mistake falls into.

The other degenerate end — $\rho >1$. If arrivals exceed the max service rate, no steady $L$ exists at all: the line grows forever and the formula (which assumed $\rho <1$) simply does not apply. In practice this is where Autoscaling and Horizontal Scaling must add a second barista before $\rho$ reaches $1$.

PICTURE. Figure s07: the full curve with the safe band ($\rho \lesssim 0.8$) shaded mint, the danger band coral, the $\rho =1$ asymptote dashed, and each table row marked.

---

The one-picture summary

WHAT. One figure that compresses all seven steps: the coffee-shop picture feeding into the two rates $\lambda ,\mu$, forming $\rho$, Little's Law bridging to $L$, and the final exploding curve — with the boxed result stamped on it.

Recall Feynman retelling — the whole walkthrough in plain words

Imagine one barista. Customers arrive at some pace ($\lambda$) and each takes some time to serve ($s$). Flip the service time and you get how many the barista can finish per second ($\mu =1/s$). Divide arrivals by that top speed and you get one number, $\rho$ — the fraction of time she's busy. Now the bathtub trick (Little's Law): the number of people hanging around equals how fast they arrive times how long each stays. For a random one-line shop, the crowd size is $\rho /(1-\rho )$ — and that $1-\rho$ on the bottom is the fraction of time she's free to catch up. As she gets closer to never resting, that free time vanishes, the crowd blows up, and so does everyone's wait. Divide the crowd by the arrival rate (Little's Law again) and, after the pace terms cancel, you're left with a beautifully simple truth: your wait is just the raw service time stretched by $1/(1-\rho )$. Run at half-busy? Wait doubles. Run at 99% busy? Wait grows a hundred-fold. So we never chase 100% — we keep some slack, and we add baristas (autoscale) before the line runs away.

Recall Quick self-test

The service time is $s$ but the observed mean latency is $4s$. What is $\rho$? ::: $4s=s/(1-\rho )\Rightarrow 1-\rho =1/4\Rightarrow \rho =0.75$. Why does $1-\rho$, not $\rho$, sit in the denominator? ::: It is the fraction of time the server is free to drain the backlog; as free time $\to 0$ the backlog and wait diverge. What happens if $\rho >1$? ::: Arrivals outrun max service; no steady latency exists, the queue grows forever — must add replicas.

---

See also

Little's Law and Queueing Theory · Autoscaling and Horizontal Scaling · Batching and Throughput · SLA SLO and Monitoring · Model Quantization and Pruning · Roofline Model of Hardware Performance · Model Distillation