Model serving frameworks (TorchServe, Triton)

WHY do we need a dedicated serving framework?

WHY not just Flask + model.predict()? A naive Flask app:

runs inference one request at a time → GPU sits idle 95% of the time,
has no batching → you pay per-request overhead again and again,
reloads / crashes lose the model; no versioning or A/B,
gives you no metrics (latency p99, throughput, GPU util) out of the box,
can't easily mix CPU/GPU, multiple models, or multiple frameworks.

Serving frameworks solve exactly these. The two dominant ones:

	TorchServe	NVIDIA Triton
Origin	PyTorch (Meta+AWS)	NVIDIA
Native models	PyTorch (`.mar`)	Framework-agnostic: TensorRT, ONNX, PyTorch, TF, Python, custom
Killer feature	Simple PyTorch handlers	Dynamic batching, multi-model, concurrent model instances, GPU-optimal
Protocols	HTTP + gRPC	HTTP + gRPC (Triton's v2 inference protocol, which KServe adopts)
Best when	Pure-PyTorch shop	Heterogeneous / max GPU utilization

The core idea that makes serving fast: DYNAMIC BATCHING

Deriving why batching helps — from first principles

Model total latency per request has two parts:

Fixed overhead $c$ (kernel launch, data transfer, framework glue) — paid per batch.
Marginal compute $m$ per sample.

Serve $B$ requests one-by-one: $T_{\text{serial}} = B\,(c + m)$

Serve them as one batch of size $B$ : $T_{\text{batch}} = c + B\,m$

Why? The fixed cost $c$ is paid once for the whole batch instead of $B$ times. Speedup: $S = \frac{T_{\text{serial}}}{T_{\text{batch}}} = \frac{B(c+m)}{c + Bm}$

HOW dynamic batching actually works (Triton & TorchServe both do this):

Requests arrive asynchronously.
Server holds them in a queue up to max_batch_size OR until max_queue_delay (e.g. 5 ms) elapses.
Whichever hits first → pad/stack into a batch → single forward pass → split results back.

Throughput of a single model instance

Figure — Model serving frameworks (TorchServe, Triton)

TorchServe: how it's built

# my_handler.py  — the 3 lifecycle steps you override
class MyHandler(BaseHandler):
    def preprocess(self, data):        # bytes/JSON -> tensor
        return torch.tensor(...)
    def inference(self, x):            # forward pass (batched!)
        return self.model(x)
    def postprocess(self, y):          # tensor -> JSON response
        return y.argmax(1).tolist()

Batching config lives per model: batch_size, max_batch_delay (ms). Multiple worker processes give concurrency; each holds a model copy.

Triton: how it's built

# config.pbtxt (sketch)
max_batch_size: 32
dynamic_batching { max_queue_delay_microseconds: 5000 }
instance_group [ { count: 2, kind: KIND_GPU } ]

Triton runs many backends (TensorRT for peak GPU speed, ONNX Runtime, PyTorch, Python) behind one unified server. Its HTTP/gRPC API follows the v2 inference protocol that Triton defines (later adopted by KServe and other serving stacks) — not the other way around.

Worked Examples

Common Mistakes (Steel-manned)

Flashcards

What problem does dynamic batching solve on a GPU?

The GPU sits idle serving one request at a time; batching amortizes fixed per-batch overhead

c

across many samples, raising throughput.

Formula for batching speedup and its ceiling

S(B)=\dfrac{B(c+m)}{c+Bm}

, ceiling

=1+\dfrac{c}{m}

B\to\infty

When does batching barely help?

When marginal compute

m

dominates fixed overhead

c

(compute-bound models, big LLMs) → ceiling

1+c/m \approx 1

Two knobs that trigger a batch to flush in dynamic batching

max_batch_size reached OR max_queue_delay timeout elapsed, whichever comes first.

What is a TorchServe .mar file?

A Model Archive bundling serialized weights + a handler (preprocess/inference/postprocess) + metadata.

Concurrency vs batching?

Concurrency = multiple model copies running in parallel (more memory); batching = one copy processing many inputs in one forward pass. Orthogonal, combinable.

Triton's biggest differentiator vs TorchServe?

Framework-agnostic backends (TensorRT/ONNX/TF/PyTorch/Python), concurrent model instances, and ensembles — tuned for max GPU utilization.

What is a Triton ensemble?

A DAG chaining multiple models (e.g. tokenizer→BERT→classifier) served in one request without extra network hops.

Which latency metric matters for SLAs and why?

Tail latency p95/p99, because queueing/batching fatten the tail even when the mean looks fine.

Relationship between queue delay and achievable batch size?

Larger max_queue_delay lets more requests accumulate → bigger

B

→ higher throughput but higher queue-wait latency

w

Who defines the "v2 inference protocol" Triton uses?

Triton (NVIDIA) defines it; KServe and other serving stacks adopt/implement it — not the reverse.

Is Throughput ≈ B/(c+Bm) an application of Little's Law?

No — it's just batch size ÷ batch processing time. Little's Law (

L=\lambda W

) separately relates queue depth, arrival rate, and mean wait.

Recall Feynman: explain to a 12-year-old

A trained model is like a super-fast robot chef. If the chef cooks one plate, walks it to the table, comes back, and cooks the next plate — that's slow because the walking (overhead) happens every time. A serving framework is the smart kitchen manager: it waits a tiny moment, gathers a tray of orders, cooks them all at once, then delivers them together. TorchServe is a kitchen that only cooks pasta (PyTorch) but is easy to run. Triton is a big kitchen that cooks pasta, pizza, and sushi (any model type) and runs many chefs at once. The manager decides how long to wait for the tray — wait too long and hungry customers get grumpy (high latency); don't wait at all and the chef wastes trips (low throughput).

Connections

Model quantization & TensorRT — the real fix when batching hits its $1+c/m$ ceiling.
ONNX & model interchange — the format Triton loves.
Little's Law & queueing theory — governs queue depth vs arrival rate (distinct from the batch-throughput formula here).
Latency SLAs & p99 — why tail latency governs serving config.
Autoscaling & Kubernetes (KServe) — KServe implements Triton's v2 inference protocol; scales instances horizontally.
A-B testing & model versioning — served via TorchServe/Triton version routing.
GPU architecture & kernel launch overhead — origin of the fixed cost $c$ .

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, model ko train karna ek baar ka kaam hai, par usse serve karna matlab restaurant chalane jaisa hai — same prediction ko fast, reliably, bahut saare users ko ek saath dena hota hai. Agar aap simple Flask app banaoge to GPU 95% time khaali baithta hai, kyunki ek time pe sirf ek request process hoti hai. Isiliye serving frameworks — TorchServe aur Triton — use karte hain, jo batching, concurrency, versioning aur monitoring khud handle karte hain.

Sabse important idea hai dynamic batching. GPU ek bus jaisa hai, taxi nahi — ek passenger ke liye poori bus chalana waste hai. Framework thoda wait karta hai (max_queue_delay), kuch requests ikattha karta hai, aur sab ka ek saath forward pass chalata hai. Fixed overhead $c$ (kernel launch, data transfer) ek hi baar pay hota hai poore batch ke liye, isliye speedup milta hai: $S = \frac{B(c+m)}{c+Bm}$ , aur maximum $1+c/m$ tak. Yaad rakho — agar model compute-heavy hai (bada LLM, $m \gg c$ ), to batching zyada help nahi karega; wahan quantization ya TensorRT chahiye. Throughput $= B/(c+Bm)$ bas "kitni requests, kitne time me" ka simple ratio hai — ye Little's Law nahi hai; Little's Law alag cheez hai jo queue kitni gehri banegi wo batati hai.

TorchServe simple hai aur sirf PyTorch models ke liye best hai — aap ek handler likhte ho (preprocess → inference → postprocess) aur .mar file bana dete ho. Triton zyada powerful hai: koi bhi framework (TensorRT, ONNX, TF, PyTorch, Python) chala sakta hai, ek saath multiple model copies (concurrency) aur ensembles (tokenizer→model→postprocess ek hi call me) support karta hai. Ek chhoti si baat: Triton apna v2 inference protocol khud define karta hai, aur KServe usse adopt karta hai — ulta nahi. Rule of thumb: pure PyTorch shop → TorchServe; mixed models + max GPU utilization → Triton. Aur hamesha p99 latency dekhna, sirf average nahi.

Model serving frameworks (TorchServe, Triton)

WHY do we need a dedicated serving framework?

The core idea that makes serving fast: DYNAMIC BATCHING

Deriving why batching helps — from first principles

Throughput of a single model instance

TorchServe: how it's built

Triton: how it's built

Worked Examples

Common Mistakes (Steel-manned)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections