Model serving frameworks (TorchServe, Triton): Worked examples

This page is the drill hall for Model serving frameworks (TorchServe, Triton). The parent gave you three formulas. Here we push each of them into every corner — every sign of the ratio $c / m$ , the zero cases, the infinite-batch limits, a word problem, and an exam trap.

Before touching a single number, let's re-anchor the three symbols so nobody uses a letter they haven't earned.

The single number that decides everything is the ratio $r = c / m$ . It answers: "how many samples' worth of compute does one door-opening cost?" Large $r$ → door is expensive → batching is a jackpot. Small $r$ → door is cheap → batching is pointless. Watch that ratio in every row below.

The scenario matrix

Every serving-math question you can be asked is one of these cells. The examples that follow are each tagged with the cell they cover.

Cell	Regime / input	What's special	Example
A	$c ≫ m$ (large $r$ )	Overhead-bound → huge speedup	Ex 1
B	$m ≫ c$ (small $r$ )	Compute-bound → batching flops	Ex 2
C	$c \approx m$ (r≈1)	The "typical" middle	Ex 3
D	$B \to \infty$	Limiting ceiling $1 + r$	Ex 4
E	$B = 1$	Degenerate: no batch at all	Ex 5
F	$c = 0$	Zero overhead → speedup is exactly 1	Ex 6
G	$m = 0$	Zero compute → division-by-zero danger	Ex 6
H	Throughput + concurrency $N$	Real capacity planning	Ex 7
I	Word problem: SLA vs delay	$L_{req}$ budget under a p99 limit	Ex 8
J	Exam twist: find $B$ for target $S$	Invert the formula	Ex 9

We visualise the whole $S (B)$ landscape once, so every example lands on a curve you can see.

Figure — Model serving frameworks (TorchServe, Triton)

Look at the figure: three curves, one per regime. The magenta curve (overhead-bound) rockets up and flattens near a high ceiling. The orange curve (compute-bound) barely lifts off $1$ . The violet middle curve sits between. Every dashed horizontal line is that curve's ceiling $1 + r$ . This is the mental picture behind all ten examples.

The worked examples

Worked example Ex 1 — Cell A: overhead-bound, batching is a jackpot

$c = 8$ ms, $m = 2$ ms/sample, $B = 16$ . Find speedup $S$ and its ceiling.

Forecast: guess — will $S$ be closer to $1 \times$ or $5 \times$ ? Jot it down.

Compute the ratio $r = c / m = 8/2 = 4$ . Why this step? $r$ tells us instantly the door costs 4 plates' worth — batching should pay off big.
Serial time $T_{serial} = 16 (8 + 2) = 160$ ms. Why? each of 16 requests pays the full door + one plate.
Batch time $T_{batch} = 8 + 16 (2) = 40$ ms. Why? the $8$ ms door is paid once; only the $16 \times 2 = 32$ ms of plates scales.
$S = 160/40 = 4 \times$ . Why? speedup is just serial ÷ batch.
Ceiling $= 1 + r = 1 + 4 = 5 \times$ . Why? the best you can ever do is spread one door over infinitely many plates.

Verify: $4 < 5$ ✓ (must be below its ceiling). Units: ms/ms → dimensionless ✓. We're at $80%$ of the ceiling — good ROI.

Worked example Ex 6 — Cells F & G: the two zero-inputs

F ( $c = 0$ ): free door. G ( $m = 0$ ): free plates. Use $B = 16$ .

Forecast: one of these breaks the math. Which?

Cell F, $c = 0$ , $m = 2$ : $S = \frac{16 ( 0 + 2 )}{0 + 16 ( 2 )} = \frac{32}{32} = 1 \times$ . Why? with no door cost there's nothing to amortise, so batching gives nothing — $S = 1$ exactly, for any $B$ . Ceiling $1 + 0/ m = 1$ ✓.
Cell G, $c = 8$ , $m = 0$ : throughput $= \frac{B}{c + B m} = \frac{16}{8 + 0} = 2$ req/ms. Why this step? with zero compute, only the door cost limits you — throughput becomes $B / c$ , and rises without bound as $B$ grows. Watch: $S = \frac{16 ( 8 + 0 )}{8 + 0} = 16 = B$ . Why? with $m = 0$ the ceiling $1 + c / m$ is $+ \infty$ , and $S$ equals $B$ exactly.
Danger note: never plug $m = 0$ and $B = 0$ — denominator $c + B m$ would be $c = 8$ (fine here), but $B = 0$ makes throughput $0/8 = 0$ (a batch of nothing does nothing). Guard against $B = 0$ in code.

Verify: F gives $S = 1$ ✓ (matches its ceiling). G gives $S = B = 16$ ✓ and throughput $2$ ✓. Both edge cases handled, no illegal division.

Worked example Ex 8 — Cell I: word problem, SLA-bounded delay

Requests arrive at $λ = 500$ /s. Your p99 SLA says $L_{req} \leq 50$ ms. Model $c = 4$ ms, $m = 1$ ms, $t_{net} = 6$ ms. What's the largest queue-wait $w$ (≈ max_queue_delay) you can afford, and how many requests $B$ does that gather?

Forecast: guess whether you can afford more or less than $30$ ms of waiting.

Budget: $L_{req} = w + (c + B m) + t_{net} \leq 50$ . Why this step? the SLA caps the total a request feels, so solve for the free knob $w$ .
We don't yet know $B$ , but $B$ is set by $w$ : at $λ = 500$ /s $= 0.5$ req/ms, in $w$ ms you gather $B \approx λ w = 0.5 w$ . Why? arrival rate × wait gives expected arrivals (see the parent's note that this is not the throughput formula).
Substitute: $w + (4 + 0.5 w \cdot 1) + 6 \leq 50 \Rightarrow 1.5 w + 10 \leq 50 \Rightarrow 1.5 w \leq 40 \Rightarrow w \leq 26.67$ ms. Why? collect the $w$ terms ( $w$ from waiting, $0.5 w$ from the extra compute those gathered samples add).
At $w = 26.67$ ms, $B = 0.5 \times 26.67 \approx 13.3 \Rightarrow$ round down to $B = 13$ . Why? you can only batch whole requests.

Verify: plug $w = 26.67, B = 13.3$ : $L = 26.67 + (4 + 13.3) + 6 = 26.67 + 17.3 + 6 = 49.97 \approx 50$ ms ✓ — sits right on the SLA. Set max_queue_delay ≈ 26 ms.

Recall Which single number predicts whether batching is worth it?

The ratio $r = c / m$ ::: overhead over marginal compute. Ceiling speedup is $1 + r$ ; large $r$ → batch hard, small $r$ → don't bother (quantize instead).

Recall What is

$S (B)$ when $B = 1$ , and why? Exactly $1 \times$ ::: a batch of one is serial — nothing is amortised, for any $c, m$ .

Recall Concurrency (

$N$ instances) vs batching — how do they combine in throughput? Multiply ::: batching sets per-instance throughput $B / (c + B m)$ ; $N$ independent copies multiply it by $N$ until the GPU saturates.