Foundations — Cost optimization and inference latency

Before you can read the parent note, you need to earn every symbol it throws at you. This page builds each one from nothing — plain words first, then a picture, then why the topic needs it. Read top to bottom; each block leans on the one above.

0. The two words everything rests on: rate and time

Almost every formula here is one childhood idea in disguise:

$\text{time}=\frac{\text{amount of work}}{\text{speed of working}}$

That's the same as "if a car goes 60 km/h and the trip is 120 km, it takes 2 hours." We will reuse this shape for arithmetic (FLOPs ÷ FLOP-per-second), for money (dollars ÷ dollars-per-second), and for lines of people (customers ÷ customers-served-per-second). If you hold onto this one sentence, the rest is bookkeeping.

1. Latency $L$ — the stopwatch for one request

• Picture: a single arrow's journey — out to the server, through the work, and back.
• Symbol: the letter $L$ (for latency). Units are seconds, or milliseconds ($\text{ms}$, one-thousandth of a second, $1\text{ ms}=10^{-3}\text{ s}$).
• Why the topic needs it: the promise you make to users ("the app feels fast") is a promise about $L$.

The parent note splits $L$ into a sum of stages:

$L=L_{\text{net}}+L_{\text{queue}}+L_{\text{compute}}+L_{\text{overhead}}$

Read the little words under each $L$ as subscripts — labels that say which piece of the journey this delay is. Nothing here is multiplied; a total time is just the sum of the times of each part, exactly like a train journey is the sum of each leg.

2. Percentiles $p50$, $p95$, $p99$ — because "average" lies

Run 100 requests and you get 100 different latencies. How do you summarize them?

• Picture: a line of runners sorted by finish time; $p99$ is the person standing 99% of the way down the line — almost the slowest.
• Why the topic needs it: a webpage that makes 100 backend calls will almost certainly hit its slow 1-in-100 request at least once. So users feel the tail, not the average. That is why the parent measures the SLA on $p99$, not the mean.

See SLA SLO and Monitoring for how these percentiles become promises you watch.

3. Throughput $X$ — the factory's output rate

• Picture: a conveyor belt; $X$ counts boxes leaving the belt per second.
• Key subtlety: low latency and high throughput are not the same thing. A single fast worker has low $L$ but small $X$; a slow machine bundling many requests at once can have high $L$ but huge $X$. This gap is exactly what Batching and Throughput exploits.

4. FLOPs $F$, peak rate $P$, efficiency $\eta$ — the arithmetic bill

The model doesn't finish by magic; it does arithmetic. We count that arithmetic.

• Picture: $F$ is a giant pile of tiny arithmetic bricks; $P$ is how many bricks a machine can lay per second.
• Big-number shorthand: $10^9$ is a billion, $10^{14}$ is a hundred trillion. TFLOP/s means tera-FLOP/s $=10^{12}$ FLOP/s.

But no chip ever hits its top speed, because it stalls waiting for data and wastes time launching work. We shave it down with an efficiency:

Put it into the "time = work ÷ speed" template, using the achievable rate $\eta P$:

Why $\eta$ and not just $P$? Because ignoring it would predict a fantasy speed you'll never see in production. Whether you are compute-bound or memory-bound is the whole subject of the Roofline Model of Hardware Performance. Making $F$ smaller is what Model Quantization and Pruning and Model Distillation do.

5. The queue: $\lambda$, $s$, $\mu$, $\rho$ — the line at the counter

Requests don't arrive one-at-a-time politely; they pile up and wait. To reason about waiting we borrow the language of Little's Law and Queueing Theory.

• Picture: one cashier ($\mu$ = how fast they scan), a stream of shoppers arriving ($\lambda$), and $\rho$ = "how full is the cashier's schedule." $\rho =0.5$ → busy half the time; $\rho =0.99$ → almost never resting.
• Why $\rho =\lambda s$? Multiply "arrivals per second" $\lambda$ by "seconds of work each" $s$ and you get "seconds of work arriving per second" — the busy fraction. If more than one second of work arrives each second ($\rho \ge 1$), the line grows forever.

Now the punchline formula the parent derives (mean latency of an M/M/1 queue):

Read $\mathbb{E}[L]$ as "the expected (average) latency." The bracketed word inside $\mathbb{E}[\cdot ]$ just names what we're averaging. As $\rho$ climbs toward 1, the denominator $1-\rho$ shrinks toward 0, and dividing by a tiny number gives a huge answer — the waiting time explodes. That single fact is why you never run hardware at 100%.

6. Cost $C$ and the machine rent $H$ — turning seconds into dollars

Same "rate" idea, now with money. Cost per inference = (dollars per second) ÷ (inferences per second):

• Where the numbers come from: $H/3600$ turns dollars-per-hour into dollars-per-second (3600 seconds in an hour). Dividing by $X$ (inferences per second) gives dollars per inference. Multiply by $10^6$ for a million.
• The one takeaway: cost is $\propto 1/X$ — the "$\propto$" symbol means "proportional to." Double the throughput $X$ and you halve the cost. That is why Batching and Throughput and Autoscaling and Horizontal Scaling are money levers, not just speed levers.

7. How it all fits together

Every foundation on this page pours into the same triangle: latency, throughput, cost. The parent note is the story of trading between its three corners.

Equipment checklist

Cover the right side and answer before revealing.