Model serving frameworks (TorchServe, Triton)

5.3.10 · AI-ML › MLOps & Deployment

WHY kyu chahiye ek dedicated serving framework?

WHY sirf Flask + model.predict() kyun nahi? Ek naive Flask app:

• inference ek request at a time chalata hai → GPU 95% time idle rehta hai,
• no batching → aap per-request overhead baar baar pay karte ho,
• reload / crash hone par model chala jaata hai; koi versioning ya A/B nahi,
• koi metrics nahi deta (latency p99, throughput, GPU util) out of the box,
• easily CPU/GPU, multiple models, ya multiple frameworks mix nahi kar sakta.

Serving frameworks exactly yahi solve karte hain. Do dominant ones:

Woh core idea jo serving ko fast banati hai: DYNAMIC BATCHING

Batching kyun help karta hai — first principles se derive karna

Model ki total latency per request ke do parts hain:

• Fixed overhead $c$ (kernel launch, data transfer, framework glue) — per batch pay hota hai.
• Marginal compute $m$ per sample.

$B$ requests one-by-one serve karo: $T_{\text{serial}}=B(c+m)$

Unhe ek batch of size $B$ ke roop mein serve karo: $T_{\text{batch}}=c+Bm$

Kyun? Fixed cost $c$ poori batch ke liye ek baar pay hoti hai, $B$ baar ki jagah. Speedup: $S=\frac{T_{\text{serial}}}{T_{\text{batch}}}=\frac{B(c+m)}{c+Bm}$

Dynamic batching actually kaise kaam karta hai (Triton aur TorchServe dono yahi karte hain):

1. Requests asynchronously arrive hoti hain.
2. Server unhe ek queue mein hold karta hai max_batch_size tak YA jab tak max_queue_delay (jaise 5 ms) expire na ho jaye.
3. Jo pehle ho → pad/stack karke batch banao → single forward pass → results wapas split karo.

Ek single model instance ka throughput

TorchServe: yeh kaise bana hai

# my_handler.py  — the 3 lifecycle steps you override
class MyHandler(BaseHandler):
    def preprocess(self, data):        # bytes/JSON -> tensor
        return torch.tensor(...)
    def inference(self, x):            # forward pass (batched!)
        return self.model(x)
    def postprocess(self, y):          # tensor -> JSON response
        return y.argmax(1).tolist()

Batching config per model rehti hai: batch_size, max_batch_delay (ms). Multiple worker processes concurrency dete hain; har ek ek model copy hold karta hai.

Triton: yeh kaise bana hai

# config.pbtxt (sketch)
max_batch_size: 32
dynamic_batching { max_queue_delay_microseconds: 5000 }
instance_group [ { count: 2, kind: KIND_GPU } ]

Triton kai backends chalata hai (peak GPU speed ke liye TensorRT, ONNX Runtime, PyTorch, Python) ek unified server ke peeche. Iska HTTP/gRPC API v2 inference protocol follow karta hai jo Triton define karta hai (baad mein KServe aur dusre serving stacks ne adopt kiya) — na ki ulta.

Worked Examples

Common Mistakes (Steel-manned)

Flashcards

Connections

• Model quantization & TensorRT — asli fix jab batching apni $1+c/m$ ceiling hit kare.
• ONNX & model interchange — woh format jo Triton pasand karta hai.
• Little's Law & queueing theory — queue depth vs arrival rate govern karta hai (yahan ke batch-throughput formula se alag).
• Latency SLAs & p99 — kyun tail latency serving config govern karti hai.
• Autoscaling & Kubernetes (KServe) — KServe Triton ka v2 inference protocol implement karta hai; instances ko horizontally scale karta hai.
• A-B testing & model versioning — TorchServe/Triton version routing ke zariye serve hota hai.
• GPU architecture & kernel launch overhead — fixed cost $c$ ka origin.

Concept Map