Visual walkthrough — Model serving frameworks (TorchServe, Triton)

---

Step 1 — What is one prediction, drawn as a bar of time?

WHAT. When a client sends a model one input (say, one photo to classify), the server spends some amount of time before it can reply. Let us just draw that time as a horizontal bar — length = seconds.

WHY. Before we can talk about "faster", we need a picture of time itself. Everything below is just cutting and stacking these bars. If we can see time as length, "speedup" becomes "shorter bar", which anyone can eyeball.

PICTURE. Look at the figure. The bar splits into two coloured chunks, because real inference time is genuinely two different things glued together:

• The orange chunk $c$ = fixed overhead. This is the setup tax the GPU pays every time you press "go": launching the compute kernel, copying data from CPU memory to GPU memory, framework bookkeeping. It does not depend on how much data you send — it is a doorway you must walk through even to compute a single number. (See GPU architecture & kernel launch overhead for why this doorway exists.)
• The blue chunk $m$ = marginal compute per sample. This is the actual arithmetic for one input. Send two inputs and you do this arithmetic twice.

---

Step 2 — Serving $B$ requests the naive way (one taxi each)

WHAT. Now $B$ clients each send one request. "$B$" is just how many requests — a whole number like $1,2,16$. The naive server handles them one after another: finish request 1 completely, then start request 2, and so on.

WHY. This is the Flask + `model.predict()` behaviour the parent warned about. We derive its cost so we have something to beat. You cannot prove batching helps until you know the number it must improve on.

PICTURE. $B$ identical bars laid end to end. Crucially, every bar carries its own orange doorway $c$. You pay the setup tax $B$ separate times — that repeated orange is the waste we will hunt down.

$T_{\text{serial}}=\underset{\text{how many}}{\underbrace{B}}(\underset{\text{doorway, paid each time}}{\underbrace{c}}+\underset{\text{one sample’s work}}{\underbrace{m}})$

Read it as: "$B$ copies of (one doorway + one unit of work)."

---

Step 3 — The trick: one doorway for the whole crowd

WHAT. Instead of $B$ separate launches, we stack all $B$ inputs into a single block and push them through the GPU in one forward pass. This block-of-inputs is called a batch; $B$ is its batch size.

WHY. A GPU has thousands of tiny arithmetic units sitting side by side. Processing 1 sample uses a sliver of them; the rest idle. Feeding it $B$ samples at once puts those idle units to work during the same launch — so we walk through the orange doorway only once for the whole crowd. (The parent's slogan: a GPU is a bus, not a taxi.)

PICTURE. One orange doorway $c$ at the front, then $B$ blue blocks of work stacked after it. Compare directly to Step 2: the extra orange doorways vanished.

$T_{\text{batch}}=\underset{\text{doorway, ONCE}}{\underbrace{c}}+\underset{\text{B units of work}}{\underbrace{Bm}}$

"One doorway, then $B$ units of arithmetic."

---

Step 4 — Line the two bars up and read off the speedup

WHAT. We define speedup $S$ = old time ÷ new time. Bigger $S$ = better.

WHY. We chose division (not subtraction) on purpose: it answers "how many times faster?", a scale-free number you can compare across models. Subtraction would give raw milliseconds, which mean nothing without context.

PICTURE. The two bars from Steps 2 and 3 stacked vertically, same starting edge. The ratio of their lengths is $S$.

$S(B)=\frac{T_{\text{serial}}}{T_{\text{batch}}}=\frac{\stackrel{\text{Step 2 length}}{\overbrace{B(c+m)}}}{\underset{\text{Step 3 length}}{\underbrace{c+Bm}}}$

• Numerator $B(c+m)$: the long naive bar.
• Denominator $c+Bm$: the short batched bar.
• Their gap (the deleted oranges) is what $S$ measures.

---

Step 5 — Push $B$ to infinity: the ceiling

WHAT. What happens if we make the batch enormous? We take the limit as $B\to \infty$.

WHY a limit? A limit answers "where is this heading if I keep going?" We use it because managers ask "so should I just crank batch size forever?" — and the honest answer is a single ceiling number, which only a limit reveals.

PICTURE. A curve of $S$ against $B$: it climbs steeply, then flattens against a horizontal dashed line. To find that line, divide top and bottom by $B$ (letting the tiny $c/B$ terms fade to nothing):

$S(B)=\frac{B(c+m)}{c+Bm}=\frac{c+m}{\frac{c}{B}+m}\underset{B\to \infty }{\to }\frac{c+m}{m}=1+\frac{c}{m}$

• $\frac{c}{B}\to 0$: the shared doorway, split across ever-more samples, becomes negligible per sample.
• What remains, $\frac{c+m}{m}=1+\frac{c}{m}$, is the hard ceiling: no batch size can beat it.

---

Step 6 — The two extreme regimes (why some models love batching and some shrug)

WHAT. The ceiling $1+\frac{c}{m}$ behaves totally differently in two worlds:

• Overhead-bound ($c\gg m$): tiny cheap model, fat doorway. Ceiling $1+\frac{c}{m}$ is huge.
• Compute-bound ($m\gg c$): giant LLM layer, thin doorway. Ceiling $1+\frac{c}{m}\approx 1$.

WHY. These are the degenerate cases the parent listed — the edges where the general formula collapses into a slogan. Covering both means you'll never be surprised in production.

PICTURE. Two speedup curves on one axis: a blue one ($c$ big) shooting up to a high ceiling, an orange one ($m$ big) hugging the value 1 almost flat.

---

Step 7 — Where does the batch come from? The queue-and-timer

WHAT. Real requests don't all arrive together. The server waits a little, collecting arrivals into a batch, then fires. Two rules end the wait: reach max_batch_size, or hit max_queue_delay — whichever comes first.

WHY. Batching only pays off if a batch actually forms. This waiting adds a new time cost we hadn't drawn: the queue wait $w$. Ignoring it is exactly the p99 trap the parent warned about.

PICTURE. A timeline: requests drop in as dots; a timer bar counts up to max_queue_delay; when it fires (or the box fills), the collected dots leave together as one batch. The full per-request latency is now three stacked pieces:

$L_{\text{req}}=\underset{\text{queue wait}}{\underbrace{w}}+\underset{\text{batch compute}}{\underbrace{(c+Bm)}}+\underset{\text{network}}{\underbrace{t_{\text{net}}}}$

---

Step 8 — The throughput view, and stacking instances

WHAT. Throughput = how many requests finish per second. One batched instance clears $B$ requests every $(c+Bm)$ seconds:

$\text{Throughput}\approx \frac{B}{c+Bm}(\text{req/s per instance})$

Run $N$ concurrent instances (separate model copies) and multiply by $N$ until the GPU runs out of memory or compute.

WHY. Latency (Steps 1–7) is "how long one request waits"; throughput is "how many the system clears". They are different questions and the tradeoff between them is the whole art of serving. Concurrency ($N$) and batching ($B$) are orthogonal — different axes you tune together.

PICTURE. Two dials side by side: a batching dial (one copy, wider inputs) and a concurrency dial ($N$ parallel copies). Total throughput $=N\times \frac{B}{c+Bm}$.

---

The one-picture summary

The final figure compresses the whole journey: (top) $B$ serial bars each with its own orange doorway → (middle) one batched bar with a single doorway → (right) the speedup curve rising to its ceiling $1+c/m$, with the queue-wait $w$ tucked in front. Everything you tuned lives on this one canvas.

Recall Feynman retelling — say it back in plain words

Every prediction costs a fixed doorway fee ($c$) plus a bit of actual work per item ($m$). If you serve people one at a time, everyone pays the doorway fee — huge waste. So the server waits a few milliseconds, gathers a crowd of $B$ people, and walks them through the door together: one fee, then $B$ helpings of work. That makes things $\frac{B(c+m)}{c+Bm}$ times faster. But you can't win forever — as the crowd grows, the shared fee-per-person shrinks toward zero and the speedup flattens at $1+\frac{c}{m}$. Cheap models (fat doorway) win big; giant models (tiny doorway, heavy work) barely notice. And gathering the crowd means people wait in line ($w$), which fattens the tail latency — so batch size is a dial, not a free lunch. Want more raw throughput on top? Run several copies of the kitchen at once (concurrency $N$).

Recall

What are the two parts of one request's inference time? ::: A fixed overhead $c$ (paid once per launch) and a marginal compute $m$ (per sample). Why is serial serving wasteful? ::: It pays the fixed overhead $c$ once for every request ($Bc$ total) instead of once per batch. What is the batching speedup formula and its limit? ::: $S(B)=\frac{B(c+m)}{c+Bm}$, with ${\lim }_{B\to \infty }S=1+\frac{c}{m}$. When does batching barely help? ::: When compute dominates ($m\gg c$), so the ceiling $1+c/m\approx 1$. How is batching different from concurrency? ::: Batching = one model copy processing many inputs in one pass; concurrency = $N$ separate copies running in parallel. What third time-cost appears once you form batches from a live stream? ::: The queue wait $w$ while requests accumulate — it inflates p99 latency.

---

See also: ONNX & model interchange · A-B testing & model versioning · back to parent.