Exercises — Cost optimization and inference latency

WHAT tool: the $p99$ definition — "$1\%$ of individual calls exceed $900$ ms" means each call has probability $q=0.01$ of being slow. WHY this and not the mean: the mean hides the tail; the question is about any slow call across many, which is a probability-of-at-least-one problem. A page is fast only if all $100$ calls are fast: $P(\text{all fast})=(1-q{)}^{100}=0.99^{100}\approx 0.366$ So $P(\text{at least one slow})=1-0.366\approx 0.634\approx 63\%.$ Lesson: a "rare" $1\%$ tail becomes the common case under fan-out.

WHAT tool: $L_{\text{compute}}\ge F/(\eta P)$ — work over achievable rate. WHY $\eta P$ not $P$: real kernels never hit peak (memory stalls, launch overhead), so we divide by what the chip actually delivers. $L_{\text{compute}}=\frac{8\times 10^9}{0.4\times 2\times 10^{13}}=\frac{8\times 10^9}{8\times 10^{12}}=10^{-3}\text{ s}=1.0\text{ ms}.$

WHAT tool: $\rho =\lambda s$ — fraction of time the single server is busy. WHY multiply: $s$ is seconds-of-work per request, $\lambda$ is requests per second, so $\lambda s$ is seconds-of-work arriving per second = busy fraction. Convert $s=25\text{ ms}=0.025$ s: $\rho =20\times 0.025=\mathrm{0.5.}$ Server is $50\%$ busy — safely below the danger zone.

WHAT tool: $\mathbb{E}[L]=s/(1-\rho )$ (M/M/1 mean, derived in the intro from Little's Law × the M/M/1 backlog $\rho /(1-\rho )$). At $\rho =0.5$: $\mathbb{E}[L]=\frac{25}{1-0.5}=50\text{ ms}.$ Note the mean is $50$ ms, but the tail $p99$ would be several times larger — the mean is a sizing guide, not the SLA number. Double the load: $\rho =40\times 0.025=1.0$ — that's not allowed. $\rho \ge 1$ means work arrives faster than the server can clear it ($\lambda \ge \mu$); the queue grows without bound and $\mathbb{E}[L]\to \infty$. The server is unstable. You must add a replica or shed load before $\rho$ reaches $1$.

WHAT tool: $C_{1M}=10^{6}H/(3600X)$. WHY: cost per inference = (dollars/second) ÷ (inferences/second) = $(H/3600)/X$; scale to a million. $C_{1M}=\frac{10^6\times 4.5}{3600\times 150}=\frac{4.5\times 10^6}{5.4\times 10^5}\approx \$8.33.$

WHAT tool: invert $\mathbb{E}[L]=s/(1-\rho )$ to solve for $\rho$. WHY invert: we know the target latency and want the allowed load — the unknown moved to the other side. $100=\frac{30}{1-\rho }\Rightarrow 1-\rho =\frac{30}{100}=0.3\Rightarrow \rho =\mathrm{0.7.}$ You may run up to $70\%$ busy and still (in expectation) meet the mean budget. Caveat: if the real SLA is on $p99$, run lower than $0.7$ and measure the tail — the mean is not the percentile.

Apply $\mathbb{E}[L]=40/(1-\rho )$:

• $\rho =0.6$: $40/0.4=100$ ms.
• $\rho =0.8$: $40/0.2=200$ ms.
• $\rho =0.95$: $40/0.05=800$ ms.
• $\rho =0.99$: $40/0.01=4000$ ms.

Growth factor: $4000/100=40\times$. Read the figure (s01) carefully. The horizontal axis is utilization $\rho$ from $0$ to $1$ on a linear scale; the vertical axis is mean latency $\mathbb{E}[L]$ in ms (also linear), fixed at $s=40$ ms. The blue curve is $s/(1-\rho )$. Three features carry the lesson: (1) the green-shaded safe zone $\rho <0.8$ where the curve is nearly flat — adding load barely changes latency; (2) the sharp upward knee near $\rho \approx 0.9$; (3) the near-vertical wall as $\rho \to 1$ where the four marked dots climb $100\to 200\to 800\to 4000$ ms. Toward the far left ($\rho \to 0$) the curve flattens to $\mathbb{E}[L]\to s=40$ ms — the no-queue floor: an almost-idle server just returns the raw service time, you can never go below $s$. (See Little's Law and Queueing Theory.)

WHAT tool: throughput = items ÷ time; per-item compute latency = batch time (each item waits the whole batch). Unit analysis for the throughput conversion (worked through): a batch of $b$ items finishes in $s(b)$ milliseconds. Raw rate is $\frac{b\text{ items}}{s(b)\text{ ms}}$. To convert ms→s we use the identity $1\text{ s}=1000\text{ ms}$, i.e. multiply by $\frac{1000\text{ ms}}{1\text{ s}}$: $X=\frac{b\text{items}}{s(b)\text{ms}}\times \frac{1000\text{ms}}{1\text{s}}=\frac{1000b}{s(b)}\frac{\text{items}}{\text{s}}.$ The "ms" cancels, leaving items/s — the factor $1000$ is not arbitrary, it is this unit conversion. Now:

• $b=1$: $s=20+5=25$ ms, $X=1000\cdot 1/25=40$ items/s. Latency $=25$ ms.
• $b=8$: $s=20+40=60$ ms, $X=1000\cdot 8/60\approx 133.3$ items/s. Latency $=60$ ms.
• $b=32$: $s=20+160=180$ ms, $X=1000\cdot 32/180\approx 177.8$ items/s. Latency $=180$ ms.

Read the figure (s02). Horizontal axis: batch size $b$ (linear, $1$ to $40$). It has two vertical axes: the left one (green) is throughput $X$ in items/s, the right one (red) is per-item latency in ms. The green throughput curve rises steeply then flattens (it saturates toward $1000/5=200$ items/s as the fixed $20$ ms is amortized away); the red latency line climbs straight and never bends ($s(b)=20+5b$ is linear). The yellow dots mark $b=1,8,32$. The takeaway lives in the gap between the two curves: going $b=1\to 32$ multiplies throughput $\approx 4.4\times$ (so cost per item drops $\approx 4.4\times$, since $C\propto 1/X$) but per-item latency grows $25\to 180$ ms ($7.2\times$). Throughput gains saturate while latency keeps climbing — the essence of the batching trade-off. See Batching and Throughput.

Baseline floor: $12\times 10^9/(6\times 10^{12})=2.0$ ms. (a) Quantization: $12\times 10^9/(1.2\times 10^{13})=1.0$ ms — a $2\times$ speedup (see Model Quantization and Pruning). (b) Distillation: $3\times 10^9/(6\times 10^{12})=0.5$ ms — a $4\times$ speedup (see Model Distillation). Distillation wins on raw latency here ($4\times$ vs $2\times$), because it attacks $F$ directly. But it needs retraining and risks accuracy loss; quantization is cheaper to deploy. In practice you often stack both: $12\times 10^9$ FLOPs at $F/4$ and $2\times$ hardware rate $\Rightarrow 3\times 10^9/(1.2\times 10^{13})=0.25$ ms ($8\times$).

WHAT tool: per-replica capacity at target utilization. Recall the service rate $\mu =1/s$ from the intro — the requests one replica clears per second when always busy. Here $\mu =1/s=1/0.02=50$ RPS. At the target $\rho =0.75$ we only let each replica take $\rho \mu =0.75\times 50=37.5$ RPS. WHY the cap: running at $\rho =0.75$ keeps $\mathbb{E}[L]=20/(1-0.75)=80$ ms — inside budget and with headroom to absorb the random bunching of arrivals (short bursts). The $25\%$ idle is deliberate insurance, not waste. $R=\lceil \frac{300}{37.5}\rceil =\lceil 8\rceil =8\text{ replicas}.$

Total hourly spend = R \cdot H = 8 \times 2.40 = \19.20$/hr.Effectiveaggregatethroughput$X = 300 RPS. $$C_{1M} = \frac{10^6 \times 19.20}{3600 \times 300} = \frac{1.92\times10^{7}}{1.08\times10^{6}} \approx \17.78.$$ Sanity note: we paid for $8\times 37.5=300$ RPS of capacity and used all $300$ — the $25\%$ idle headroom is baked into needing $8$ (not $6$) machines, i.e. it's the price of meeting latency.

(a) $R=\lceil 300/93.75\rceil =\lceil 3.2\rceil =4$ replicas. (b) Hourly = 4 \times 2.40 = \9.60$;$X = 300: $$C_{1M} = \frac{10^6 \times 9.60}{3600 \times 300} = \frac{9.6\times10^{6}}{1.08\times10^{6}} \approx \8.89.$$ Cost halved (\17.78 \to $8.89$)—becauseweneedhalfthemachines.(c)A\ast mean\ast budgetcheckgives$\approx 80 + 30 = 110$ms.\ast \ast Butdonotreadthisasa$p99$guarantee.\ast \ast Addingafixedbatch-waittoameanisalower-boundsketch:thetruetailisworsebecause(i)batch-waititselfvaries(somerequestsarrivejustasabatchcloses,otherswaitthefulltimeout),and(ii)queueingvarianceinflates$p99$wellabovethemean.So$110$msmeanis\ast plausibly\ast safeagainsta$150$ms\ast \ast mean\ast \ast target,butfora$p99 \le 150$msSLAyoumust\ast \ast measurethereal$p99$** in load tests, not add means (see SLA SLO and Monitoring). The cost win is real; the latency claim is provisional until measured.

Step 1 — max safe $\rho$ per option (invert $\mathbb{E}[L]=s/(1-\rho )$):

• A: $120=40/(1-\rho )\Rightarrow 1-\rho =40/120=0.333\Rightarrow {\rho }_{\max }=0.6\overline{6}$.
• B: $120=15/(1-\rho )\Rightarrow 1-\rho =15/120=0.125\Rightarrow {\rho }_{\max }=0.875$.

WHY B tolerates higher $\rho$: its service is so fast ($15$ ms) that even a busy queue stays under budget.

Step 2 — per-replica served rate $={\rho }_{\max }\cdot \mu ={\rho }_{\max }/s$:

• A: $0.6\overline{6}/0.040=16.6\overline{6}$ RPS.
• B: $0.875/0.015=58.3\overline{3}$ RPS.

Step 3 — replicas (round up):

• A: $R=\lceil 500/16.6\overline{6}\rceil =\lceil 30.0\rceil =30$.
• B: $R=\lceil 500/58.3\overline{3}\rceil =\lceil 8.57\rceil =9$.

Step 4 — cost per 1M ($X=500$ served, hourly $=R\cdot 3$):

• A: C_{1M} = \frac{10^6 \times (30\times 3)}{3600\times 500} = \frac{9\times10^{7}}{1.8\times10^{6}} = \50.00$.
• B: C_{1M} = \frac{10^6 \times (9\times 3)}{3600\times 500} = \frac{2.7\times10^{7}}{1.8\times10^{6}} = \15.00.$

Verdict: Option B is $3.3\times$ cheaper (\15$vs$$50$)\ast and\ast meetsthemeanbudgetmorecomfortably—becausethesmallermodelbothcuts$s$(morecapacitypermachine)andallowshighersafe$\rho$.\ast \ast Twocatches:\ast \ast (i)the$1.5%$accuracydrop—nevershipitblind,validateonheld-outandadversarialslices(see[[ModelDistillation]]);(ii)theseare\ast mean\ast budgets—beforelaunch,load-testandconfirmthereal$p99$ meets the SLA, since the mean under-states the tail.

Compute floor ::: $L_{\text{compute}}\ge F/(\eta P)$ Service rate ::: $\mu =1/s$ (requests one server clears per second when busy) Queue latency ::: $\mathbb{E}[L]=s/(1-\rho )$, with $\rho =\lambda s=\lambda /\mu$ Max safe $\rho$ from a mean budget ::: $\rho =1-s/\mathbb{E}[L{]}_{\text{target}}$ Replicas needed ::: $R=\lceil \lambda /({\rho }_{\max }\mu )\rceil$ Cost per 1M ::: $C_{1M}=10^{6}H/(3600X)$ Why mean ≠ SLA ::: percentile SLAs ($p99$) run above the mean; size on the mean, then measure the tail.