Model serving frameworks (TorchServe, Triton): Question bank

Recall How to use this bank

Cover the right side. Say your reasoning. Then reveal. If your reason differs from the reveal even when your verdict matched, you got lucky, not right.

The symbols this page uses (read first)

Every trap below leans on five quantities. Before you use them, here is what each means in plain words and why it enters the maths.

Definition The five quantities

$B$ = batch size: how many requests we stack and push through the model in one forward pass. It is a count, so always an integer $\geq 1$ (you cannot process half a request).
$c$ = fixed overhead time per batch (seconds): the cost paid once per batch no matter how many samples are in it — kernel launch, moving data to the GPU, framework glue. See GPU architecture & kernel launch overhead for where this comes from.
$m$ = marginal compute time per sample (seconds): the extra time each additional sample adds — the actual number-crunching. Real models have $m > 0$ (never negative — a sample can't save time).
$w$ = queue wait: how long a request sits waiting for its batch to fill before compute starts.
$t_{net}$ = network time: time to carry the request over the wire and the reply back — client↔server transport, nothing to do with the model itself.

Figure — Model serving frameworks (TorchServe, Triton)

Deriving the two core formulas (so the symbols mean something)

Serve $B$ requests one-by-one. Each request pays the full fixed cost $c$ and its own compute $m$ , so the total is $B$ copies of $(c + m)$ : $T_{serial} = B (c + m)$

Serve the same $B$ as one batch. Now $c$ is paid once for the whole group; only the compute $m$ scales with the number of samples: $T_{batch} = c + B m$

Why batching wins is exactly the difference: we deleted $B - 1$ copies of the overhead $c$ . The speedup is the ratio of the two times — "how many times faster": $S (B) = \frac{T _{serial}}{T _{batch}} = \frac{B ( c + m )}{c + B m}$

Throughput = work done per second = requests finished ÷ time to finish them. A batch of $B$ finishes every $(c + B m)$ seconds, so: $Throughput = \frac{B}{c + B m} (requests/sec per model instance)$

The curve of $S (B)$ against $B$ (see figure above) rises fast then flattens — that flattening is the whole story of the True/False traps.

True or false — justify

Batching a model always reduces total latency for the batched requests.

False. It reduces per-batch overhead, but each request now also waits in the queue (

w

) for the batch to fill; for a nearly compute-bound model (

m ≫ c

) the compute barely shrinks while the wait is added, so an individual request can end up slower.

Running more worker processes (concurrency) increases throughput the same way batching does.

False in mechanism. Concurrency runs multiple copies of the model in parallel (more memory), each doing its own forward pass; batching packs many inputs into one forward pass on one copy. They are orthogonal — you usually combine them.

If a model's speedup from batching is measured at

4 \times

for

B = 16

, doubling to

B = 32

will give roughly

8 \times

False. Speedup

S (B) = \frac{B ( c + m )}{c + B m}

saturates at the ceiling

1 + c / m

; once you're near it, more batch buys almost nothing while raising p99 latency and GPU memory use.

Triton and TorchServe are interchangeable because both expose an HTTP prediction API.

False. Triton is framework-agnostic (TensorRT/ONNX/PyTorch/TF/Python), supports concurrent instances and ensembles for max GPU utilization; TorchServe is PyTorch-first and simpler. Match the tool to your stack, see ONNX & model interchange.

For a compute-bound giant LLM layer, increasing max_batch_size is the primary lever to hit your latency SLA.

False. When

m ≫ c

the batching ceiling

1 + c / m \approx 1

, so batching is nearly useless; reach for Model quantization & TensorRT (shrink

m

) or more model instances instead.

The v2 inference protocol was invented by KServe and Triton adopted it.

False. Triton defines the v2 inference protocol; KServe and other stacks later adopted it — not the other way around. See Autoscaling & Kubernetes (KServe).

Little's Law is the formula that gives you throughput of a model instance.

False. Throughput is

B / (c + B m)

— a service rate set by how fast one instance chews a batch. Little's Law & queueing theory (

L = λW

) relates average in-flight requests

L

, arrival rate

λ

, and mean sojourn time

W

; it describes the queue's behaviour given demand, not how fast the server itself runs. It can't give throughput because it contains no

c

m

— it never looks inside the model.

A great mean latency means your service meets its latency SLA.

False. SLAs are on tail latency (p95/p99), see Latency SLAs & p99. Queueing and batching fatten the tail, so an excellent mean can hide a terrible p99.

Spot the error

"Set max_queue_delay to 100 ms so we always fill big batches — throughput will be maximal and everyone's happy."

The error is ignoring the SLA. The per-request latency is

L_{req} = w + (c + B m) + t_{net}

— queue wait plus batch compute plus network. A 100 ms floor on

w

alone blows most p99 budgets; you dial delay only as high as

L_{req}

still fits the SLA.

"Our GPU is 100% utilized during a single request's forward pass, so batching can't help."

Momentary 100% utilization inside a kernel ≠ high average utilization. The GPU idles between requests during data transfer and kernel launch (the fixed cost

c

); batching amortizes those gaps, which is exactly what the speedup formula captures via

c

"We'll deploy with a plain Flask + model.predict() server; it returns predictions just fine."

It works functionally but serves one request at a time (GPU idle ~95%), has no batching, no versioning/A-B, no built-in p99/throughput metrics, and can't mix backends — the exact gaps serving frameworks exist to fill. See A-B testing & model versioning.

"To chain tokenizer → BERT → reranker, we call three separate servers and stitch results in the client."

That pays three network hops (

3 \times t_{net}

) per request. Triton's ensemble DAG chains them inside one server call, cutting latency and glue code — a core reason to pick Triton for heterogeneous pipelines.

"Concurrency is free — just crank instance_group count up as high as you want."

Each instance is a full model copy consuming GPU memory and compute; past saturation you OOM or instances contend for the same SMs, so throughput plateaus or drops. See GPU architecture & kernel launch overhead.

"The .mar archive stores only the model weights."

It also bundles the handler (preprocess → inference → postprocess) plus metadata — the code that turns raw bytes/JSON into tensors and back is packaged alongside the weights.

Why questions

Why is a GPU described as "a bus, not a taxi" for serving?

A bus has huge fixed cost per trip but scales cheaply per passenger; sending one passenger wastes it. Likewise the fixed overhead

c

is paid per batch, so grouping many requests into one forward pass drives the per-request cost down.

Why does the speedup ceiling equal

1 + c / m

rather than being unbounded?

B \to \infty

S (B) = \frac{B ( c + m )}{c + B m} \to \frac{c + m}{m} = 1 + \frac{c}{m}

. The marginal compute

m

per sample never disappears — you can only ever recover the wasted fixed overhead

c

, so the best case is "removed all overhead", nothing more.

Why do overhead-bound models batch beautifully but compute-bound LLMs give diminishing returns?

Overhead-bound means

c ≫ m

, so the ceiling

1 + c / m

is large. Compute-bound means

m ≫ c

, so the ceiling collapses toward

1 \times

— there's almost no overhead left to amortize.

Why does raising max_queue_delay increase throughput but hurt individual latency?

A longer delay lets more arrivals accumulate → larger

B

→ more requests per batch-time → higher throughput. But each request now waits longer in queue (

w

rises), inflating its total latency

L_{req}

and the tail.

Why does p99 latency matter more than the mean?

p99 is the latency value that 99% of requests come in under — only the slowest 1% exceed it. It reads off the far-right of the latency distribution, exactly the region queueing and batching inflate, so it exposes the pain a low mean hides. See Latency SLAs & p99.

Why can Triton reach higher GPU utilization than a naive PyTorch server?

It combines dynamic batching (one forward pass for many inputs), multiple concurrent instances (instance_group), and hardware-optimized backends like TensorRT — keeping the GPU's SMs busy instead of idling between one-at-a-time requests.

Edge cases

What happens to the speedup formula when

B = 1

S (1) = \frac{1 ( c + m )}{c + 1 \cdot m} = 1

. A "batch" of one is identical to serial — no overhead is amortized, so no speedup. This is the correct degenerate baseline.

B = 1.5

a valid batch size, and can

m

be negative?

No to both.

B

is a count of requests, so it must be an integer

\geq 1

— the formula is only evaluated at whole numbers even though we draw

S (B)

as a smooth curve for intuition. And

m < 0

is unphysical: a sample cannot make compute finish sooner, so

m > 0

always.

What is the throughput

B / (c + B m)

in the limit

B \to \infty

It approaches

\frac{1}{m}

requests/sec per instance — throughput is capped by the marginal compute, so you cannot batch your way past

1/ m

; to go faster you need smaller

m

(quantization) or more instances

N

What if

c = 0

(zero fixed overhead)?

Then

S (B) = \frac{B \cdot m}{B m} = 1

for all

B

and the ceiling

1 + c / m = 1

. With no overhead to amortize, batching gives exactly nothing — it purely helps by hiding fixed costs.

What happens if max_batch_size exceeds GPU memory for the largest input?

The batched forward pass OOMs and the whole batch fails, taking down several users' requests at once. Batch size must be capped by memory headroom, not just by the throughput you'd like.

Requests arrive at 500/s with max_queue_delay = 5 ms — is the batch exactly 2.5 full?

On average yes:

500 \times 0.005 = 2.5

requests per window. But real arrivals are stochastic (typically modelled as a Poisson process), so any single window sees a random count scattered around 2.5 — sometimes 0, sometimes 6. The 2.5 is a mean, not a guarantee; Little's Law & queueing theory handles this variability properly.

Two model copies (concurrency 2) each with batch size 16 — what's the total throughput?

Roughly

N \times \frac{B}{c + B m}

with

N = 2

: each instance contributes its own batched throughput, so you multiply by

N

— until GPU memory or compute saturates, after which adding instances stops helping. See Autoscaling & Kubernetes (KServe).

Flashcards

State the batching speedup ceiling and when it is small.

The ceiling is

1 + c / m

; it is small (near

1 \times

) when the model is compute-bound (

m ≫ c

), as in large LLM layers.

Define overhead-bound vs compute-bound.

Overhead-bound:

c ≫ m

, fixed per-batch cost dominates, batching helps a lot. Compute-bound:

m ≫ c

, per-sample crunching dominates, batching barely helps.