Foundations — Model serving frameworks (TorchServe, Triton)

Before you touch Model serving frameworks (TorchServe, Triton) you must be able to read its language without stumbling. This page builds every symbol from nothing. Nothing here assumes you have seen a GPU, a queue, or a millisecond budget before. We go slow on purpose.

0. What is "a request"? (the atom everything is built on)

Picture a counter at a shop. A customer walks up (a request arrives), hands over a slip (the input), waits, and receives an answer (the prediction). The whole page is about that counter: how long each customer waits, and how many we can serve.

Everything below — latency, throughput, batching — is just careful bookkeeping about this queue of customers.

1. Time and its unit: the millisecond (ms)

Why this unit and not seconds? Serving happens fast. A good model answers in single- or double-digit milliseconds. If we measured in seconds every number would be like $0.008$, hard to read. In milliseconds it becomes $8$ — human-friendly. So the whole topic lives in ms.

2. Latency: how long ONE request takes

Picture a stopwatch started when the customer hands over the slip and stopped when they get the answer. The reading on that stopwatch is latency.

3. Throughput: how MANY requests per second

Picture the shop's daily headcount: not how long one customer waited, but how many customers walked out satisfied per minute. That is throughput.

4. The GPU: why a "bus, not a taxi"

Why does serving care? A GPU is happiest when you hand it a big pile of numbers all at once. Handing it one number wastes almost all of its parallel lanes — like driving an empty bus.

You'll meet the deeper reason in GPU architecture & kernel launch overhead; for now hold this picture: many lanes, best filled full.

5. The two costs of running the model: $c$ and $m$

This is the single most important pair of symbols in the whole topic. We build them from a stopwatch.

Picture baking cookies. Preheating the oven takes a fixed 8 minutes ($c$) whether you bake 1 cookie or 100. Each cookie then adds a little baking time ($m$). The oven-preheat is the fixed overhead; the per-cookie time is the marginal cost.

This one formula, $T=c+Bm$, is the seed from which the parent's batching maths grows. If you understand why $c$ appears once and $m$ appears $B$ times, you understand batching.

6. Batch size $B$: the size of the pile

Picture loading the bus: $B$ is the number of passengers you let on before you drive off. $B=1$ is a taxi (one rider, drives immediately). $B=16$ is a bus that waited to fill up.

7. Queue wait $w$ and delay budget: waiting to fill the bus

To fill a bus you must let earlier passengers wait for later ones. That waiting is $w$. The server caps it with a knob called max queue delay (e.g. 5 ms): "never make anyone wait longer than this before we drive off, even with an empty seat."

8. Concurrency $N$: running several buses at once

Picture opening more counters in the shop. Batching makes each counter serve a group at once; concurrency adds more counters. They are different levers — combine both.

9. Speedup $S$: the payoff number

Why a ratio? A ratio strips away the units and answers the only question you care about: did batching help, and by how much? Same-time = $1$; twice-as-fast = $2$; the bigger the better.

The parent proves this ratio can never exceed $1+\frac{c}{m}$. You don't need that limit yet — just know $S$ is "the batching payoff," a pure number.

10. Symbols you'll see referenced but not derived here

You will meet these in context; this table just stops them from feeling like magic when they appear.

Prerequisite map

Read it top-down: the request atom splits into the two things we measure (latency, throughput); the GPU picture forces the two costs $c$ and $m$; those plus batch size give the time formula, which gives speedup and wait; concurrency multiplies throughput — all feeding the topic.

Equipment checklist

Cover the right side, answer, then reveal. If any answer is fuzzy, re-read that section before the parent note.