Model serving frameworks (TorchServe, Triton): Exercises

Before we start, one reminder of the two numbers everything hangs on — because you will earn every symbol here, not memorise it.

The figure below makes this concrete. On the left we serve $B = 4$ requests one at a time: notice the amber "overhead" block of height $c$ is redrawn four separate times — we pay the fixed cost on every single request. On the right we batch the same 4 requests: there is one amber block, then four cyan compute blocks stacked on top. The area we saved — three missing amber blocks — is exactly where the speedup comes from. Keep this picture in mind for every exercise: batching deletes repeated overhead, nothing else.

Figure — Model serving frameworks (TorchServe, Triton)

Two more symbols will show up the moment a request travels over a network, so let us earn them now rather than meet them cold in an exercise:

Definition Queue wait

$w$ and network time $t_{net}$ A real request's wall-clock latency is more than just the batch compute $c + B m$ . It also includes:

$w$ = queue wait — the time the request sits idle in the server's queue waiting for enough friends to arrive so a batch can be dispatched. Picture a bus stop: you board, then wait while the driver lets a few more passengers on. Bigger max_queue_delay → bigger $w$ .
$t_{net}$ = network time — the round-trip time for the request bytes to travel client → server and the response to travel back. It has nothing to do with the model; it is pure "distance on the wire."

Putting all three together gives the full latency budget of one request: $L_{req} = queue wait w + batch compute (c + B m) + network round trip t_{net} .$ This budget is the backbone of Levels 4 and 5, so we defined every one of its terms first.

Level 1 — Recognition

Exercise 1.1 (L1)

Which serving framework is framework-agnostic (runs TensorRT, ONNX, PyTorch, TF, Python behind one server): TorchServe or Triton?

Recall Solution

Triton. TorchServe is PyTorch-first (it consumes .mar archives). Triton was built by NVIDIA to run many backends behind one unified HTTP/gRPC server, which is exactly why you pick it for a mixed stack. See the comparison table in the parent note.

Exercise 1.2 (L1)

In the latency budget $L_{req} = w + (c + B m) + t_{net}$ , which term is the queue wait — the time a request sits waiting for its batch to fill?

Recall Solution

The term $w$ is the queue wait (defined above). It is the time the request spends idle in the queue before the batch is dispatched. It grows when you raise max_queue_delay, and it is the term that quietly fattens your p99 tail (see Latency SLAs & p99).

Exercise 1.3 (L1)

Name the file that packages a TorchServe model, and name the file that configures a Triton model.

Recall Solution

TorchServe: the ==.mar== (Model Archive) — serialized weights + a handler class + metadata, built with torch-model-archiver.
Triton: the ==config.pbtxt== inside the model repository — declares inputs/outputs, max_batch_size, dynamic_batching, instance_group.

Level 2 — Application

Exercise 2.1 (L2)

A model has $c = 6$ ms overhead and $m = 3$ ms/sample. You serve a batch of $B = 10$ . Compute (a) serial time, (b) batched time, (c) the speedup $S$ .

Recall Solution

(a) Serial — each of the 10 requests pays the full $c + m$ : $T_{serial} = 10 (6 + 3) = 10 \times 9 = 90 ms .$ (b) Batched — pay $c$ once, then $m$ per sample: $T_{batch} = 6 + 10 \times 3 = 6 + 30 = 36 ms .$ (c) Speedup: $S = \frac{90}{36} = 2.5 \times .$ What it looks like: the overhead bar of height $c = 6$ is drawn once instead of ten times — that saved area is your speedup (exactly the s01 figure, at $B = 10$ instead of 4).

Exercise 2.2 (L2)

Same model ( $c = 6$ , $m = 3$ ). What is the theoretical ceiling on speedup, no matter how large the batch grows?

Recall Solution

Take the limit as $B \to \infty$ in $S (B) = \frac{B ( c + m )}{c + B m}$ . Divide top and bottom by $B$ : $S (B) = \frac{c + m}{\frac{c}{B} + m} B \to \infty \frac{c + m}{m} = 1 + \frac{c}{m} .$ $S_{m a x} = 1 + \frac{6}{3} = 1 + 2 = 3 \times .$ So at $B = 10$ we already got $2.5 \times$ of a $3 \times$ ceiling — very close, more batching barely helps.

Exercise 2.3 (L2)

For that same model at $B = 10$ , what is the throughput (requests/sec per instance)?

Recall Solution

Throughput = batch size ÷ time to process the batch. The batch time is $c + B m = 36$ ms $= 0.036$ s. $Throughput = \frac{B}{c + B m} = \frac{10}{0.036} \approx 277.8 req/s .$ Why this and not Little's Law? This is just "how many requests come out per second from one instance running back-to-back batches." Little's Law ( $L = λW$ , see Little's Law & queueing theory) is a different question — how deep the queue grows — and we don't need it here.

Level 3 — Analysis

Exercise 3.1 (L3)

Two models, same $B = 8$ :

Model A (cheap CNN): $c = 9$ ms, $m = 1$ ms.
Model B (huge LLM layer): $c = 1$ ms, $m = 40$ ms.

Compute the speedup for each and explain which model batching helps and why.

Recall Solution

Model A: $S_{A} = \frac{8 ( 9 + 1 )}{9 + 8 \times 1} = \frac{80}{17} \approx 4.71 \times .$ Ceiling $1 + c / m = 1 + 9 = 10 \times$ — huge headroom, batching is a big win.

Model B: $S_{B} = \frac{8 ( 1 + 40 )}{1 + 8 \times 40} = \frac{328}{321} \approx 1.022 \times .$ Ceiling $1 + c / m = 1 + 0.025 = 1.025 \times$ — essentially no win.

Why the difference: speedup comes entirely from amortising the fixed $c$ . When $c ≫ m$ (Model A) the fixed cost dominates, so paying it once instead of $B$ times saves a lot. When $m ≫ c$ (Model B) the batch time is basically all per-sample compute, which scales with $B$ anyway — nothing to amortise. For Model B you need Model quantization & TensorRT or more instances, not bigger batches.

Exercise 3.2 (L3)

Requests arrive at $λ = 400$ requests/sec. Your max_queue_delay is set to $d = 5$ ms. (a) Roughly how many requests accumulate in one delay window? (b) What delay would you need to gather a full batch of $B = 16$ ?

Recall Solution

(a) In a window of $d$ seconds, arrivals $\approx λ d$ : $400 \times 0.005 = 2 requests per window.$ (Only ~2 gathered — nowhere near a big batch.)

(b) Solve $λ d = B$ for $d$ : $d = \frac{B}{λ} = \frac{16}{400} = 0.04 s = 40 ms .$ What this means: to fill a batch of 16 at this arrival rate you must let each request wait up to 40 ms in the queue (this is the queue wait $w$ ). Only do that if your Latency SLAs & p99 budget allows a 40 ms queue tax — the delay is a dial trading latency for throughput.

Level 4 — Synthesis

Exercise 4.1 (L4)

Your model: $c = 4$ ms, $m = 2$ ms/sample. Network round trip $t_{net} = 10$ ms. Your SLA says p99 latency must be $\leq 60$ ms. Requests arrive at $λ = 1000$ /s. Assume worst-case a request waits the full queue delay $d$ , and you dispatch batches of size $B = λ d$ (i.e. one full window's arrivals). Find the largest queue delay $d$ (hence largest batch $B$ ) that still meets the SLA, and the resulting throughput.

Recall Solution

Step 0 — fix ONE unit so nothing is ambiguous. Everything below is in milliseconds (ms). The arrival rate is $λ = 1000$ req/s. Convert it to ms: $1000$ req/s $= 1$ req/ms (because 1000 requests spread over 1000 ms is one request every millisecond). Working in ms keeps $c$ , $m$ , $t_{net}$ and $d$ all in the same unit so we never mix seconds and ms.

Step 1 — WHAT is the batch size in terms of the delay? A window of $d$ ms, at 1 req/ms, gathers $B = λ d = (1 req/ms) \times (d ms) = d$ requests. So numerically $B = d$ (a 15 ms window gathers 15 requests). This is why $B$ and $d$ share a number here — the arrival rate is exactly one per ms.

Step 2 — WHAT is the worst-case latency? Assemble the budget defined earlier: $L = w d + compute (c + B m) + t_{net} .$ Substitute $B = d$ , $c = 4$ , $m = 2$ , $t_{net} = 10$ (all in ms): $L = d + (4 + 2 d) + 10 = 3 d + 14.$

Step 3 — WHY set it equal to the SLA? We want the largest $d$ , so push latency right up to the cap of 60 ms: $3 d + 14 \leq 60 \Rightarrow 3 d \leq 46 \Rightarrow d \leq 15.33 ms .$ Take $d = 15.33$ ms $\Rightarrow B = λ d \approx 15.3 \approx 15$ requests per batch (round down to stay under the SLA).

Step 4 — throughput at $B = 15$ , batch time $c + B m = 4 + 30 = 34$ ms $= 0.034$ s: $Throughput = \frac{15}{0.034} \approx 441 req/s per instance.$ Since arrivals are 1000/s, one instance is not enough — you'd run $⌈ 1000/441 ⌉ = 3$ concurrent instances (Triton instance_group count: 3) to keep up. This is the synthesis: batching alone can't absorb the load, concurrency finishes the job.

Exercise 4.2 (L4)

You have a pipeline: Python tokenizer → ONNX BERT → TensorRT reranker, all on one GPU. Which framework, and how do you configure it so the three stages happen in one server call with no extra network hops?

Recall Solution

Framework: Triton. Only Triton runs Python, ONNX (ONNX & model interchange) and TensorRT (Model quantization & TensorRT) backends behind one server. Configuration: define an ensemble — a DAG in config.pbtxt that chains tokenizer → BERT → reranker, wiring each stage's outputs to the next stage's inputs. The client makes one request; Triton runs all three stages internally on the GPU. Why it wins: TorchServe would force all three into PyTorch handlers (losing TensorRT's speed), and a naive microservice split would cost 3 network hops — three times the $t_{net}$ tax. The ensemble collapses that to one hop.

Level 5 — Mastery

Exercise 5.1 (L5)

A team reports: "We switched from batch $B = 4$ to $B = 64$ and throughput barely moved, but our p99 latency doubled and we started getting occasional GPU OOM crashes." The model has $c = 2$ ms, $m = 5$ ms/sample. Diagnose all three symptoms quantitatively and prescribe a fix.

Recall Solution

Symptom 1 — throughput barely moved. Compute throughput at each batch: $B = 4 : \frac{4}{2 + 4 \times 5} = \frac{4}{22} \approx 0.1818 req/ms = 181.8 req/s .$ $B = 64 : \frac{64}{2 + 64 \times 5} = \frac{64}{322} \approx 0.1988 req/ms = 198.8 req/s .$ Gain: only $198.8/181.8 \approx 1.094$ , about +9%. Why? $m ≫ c$ (5 vs 2), so the speedup ceiling is $1 + c / m = 1.4 \times$ — there was almost nothing to amortise. Batching was the wrong lever.

Symptom 2 — p99 doubled. Bigger $B$ needs a longer max_queue_delay to fill, and the batch compute itself grows: at $B = 64$ the batch takes $c + B m = 2 + 320 = 322$ ms versus $2 + 20 = 22$ ms at $B = 4$ . Every request in that big batch waits for the whole 322 ms pass → the tail latency balloons.

Symptom 3 — OOM. Activation memory scales with $B$ ; $16 \times$ the batch $\approx 16 \times$ the activation memory, overflowing GPU RAM on the largest inputs.

Prescription: this is a compute-bound model ( $m ≫ c$ ). The right levers are (i) Model quantization & TensorRT to shrink $m$ (and memory), (ii) more concurrent instances to raise throughput without inflating any single request's latency, and (iii) roll back to a modest batch (say $B = 8$ ) that respects the SLA. Bigger batches were never going to help a model whose ceiling is $1.4 \times$ .

Exercise 5.2 (L5)

Design question. You must serve two models on one GPU: a cheap CNN ( $c = 8$ , $m = 1$ ms) and a heavy LLM ( $c = 1$ , $m = 30$ ms). You have a strict p99 $\leq 50$ ms on the CNN and a lenient p99 $\leq 800$ ms on the LLM. Sketch a Triton config strategy: batch sizes, delays, and instance groups, and justify each choice from the maths.

Recall Solution

CNN (cheap, overhead-dominated). Ceiling $1 + c / m = 1 + 8 = 9 \times$ → batch aggressively. But the SLA is tight (50 ms). Budget: $L = d + (c + B m) + t_{net}$ . If $t_{net} \approx 10$ ms, we have $50 - 10 = 40$ ms for queue delay $d$ + compute. Pick e.g. $B = 16$ : compute $= 8 + 16 = 24$ ms, leaving $40 - 24 = 16$ ms for queue delay $d$ . So max_batch_size: 16, max_queue_delay_microseconds: 15000 (15 ms, just under the 16 ms headroom). One or two instances suffice — batching does the heavy lifting here because the ceiling is $9 \times$ .

LLM (compute-dominated). Ceiling $1 + c / m = 1 + 1/30 \approx 1.033 \times$ → batching is almost useless, so do not chase a huge batch. The lenient 800 ms SLA instead lets you lean on modest batches + concurrency + quantization. Concretely:

Batch size: pick a modest $B = 8$ . Compute $= c + B m = 1 + 8 \times 30 = 241$ ms, comfortably inside 800 ms even after adding a queue delay and $t_{net}$ . Going bigger buys almost no speedup (ceiling $1.033 \times$ ) while eating GPU memory — so max_batch_size: 8.
Queue delay: with a generous 800 ms budget you can afford a longer window to actually fill $B = 8$ , e.g. max_queue_delay_microseconds: 20000 (20 ms). Even $20 + 241 + t_{net}$ stays far under 800 ms.
Instance group: the real throughput lever here is replication, not batching. If GPU memory permits, run instance_group { count: 2, kind: KIND_GPU } so two LLM copies process batches in parallel — doubling throughput without touching any single request's latency.
Quantization: apply Model quantization & TensorRT to cut $m$ (say 30 → 12 ms) and the memory footprint; this is what makes both a bigger effective throughput and the extra instance affordable.

Sharing one GPU. Use Triton's per-model instance_group and let dynamic batching run independently per model. The CNN's tight SLA drives its batch/delay (aggressive batch, short 15 ms delay); the LLM's compute cost drives its instance count (modest batch, replicate + quantize). This is the whole lesson: match the lever to the model's $c$ -vs- $m$ regime, per model, on shared hardware — see also Autoscaling & Kubernetes (KServe) for scaling instances beyond one GPU.

Recall One-line summary to test yourself

Batching helps ::: only when overhead dominates compute, capped at $1 + c / m$ ; when $m ≫ c$ use quantization + more instances instead. The queue delay $d$ ::: is added directly to every request's latency (queue wait $w$ ), so it trades throughput for tail latency. Concurrency vs batching ::: concurrency = many model copies in parallel (more memory); batching = one copy, many inputs in one pass; combine both.