Visual walkthrough — Model serving (REST APIs, FastAPI)

This deep-dive extends Model serving (REST APIs, FastAPI). It leans on the throughput/scaling ideas you'll meet again in Load Balancing & Autoscaling and Batching & Inference Optimization.

---

Step 1 — A request is a journey with stages

WHAT. We draw one request as a horizontal timeline. It leaves the client, crosses the network, gets unpacked, gets processed, the model runs, the answer gets packed, and it goes home.

WHY. Before we can add up times, we must agree that time is spent in a sequence, not all at once. A request cannot be deserialized before it arrives; it cannot run inference before it is deserialized. This ordering is the whole reason the total is a sum and not something fancier.

PICTURE. Below, each coloured block is one stage. The width of a block = time spent there. The request cannot be in two blocks at once, so the blocks sit side by side, never overlapping.

---

Step 2 — Name each stage's time, right on the picture

WHAT. We give each block a symbol. Reading left to right:

$T_{\text{network}},T_{\text{deserialize}},T_{\text{preprocess}},T_{\text{inference}},T_{\text{serialize}}$

• $T_{\text{network}}$ — time for the bytes to travel over the wire, client → server. Nothing is computed yet; this is pure distance/speed.
• $T_{\text{deserialize}}$ — time to unpack the raw bytes into a JSON object the code can read.
• $T_{\text{preprocess}}$ — time to shape that object into what the model expects (e.g. build the 2D array [[x1, x2]]).
• $T_{\text{inference}}$ — time for the model itself to compute $f(x)\to y$. The actual thinking.
• $T_{\text{serialize}}$ — time to pack the answer back into JSON bytes to send home.

WHY these five and not more or fewer? Each is a place where a different kind of work happens (moving, unpacking, shaping, computing, packing). Splitting further gives no new insight; merging them hides where the time actually goes. Five is the smallest set that still tells you what to fix.

PICTURE. Same blocks as Step 1, now each labelled with its symbol beneath it.

---

Step 3 — Add the blocks: total time is a sum

WHAT. We slide all five blocks against each other with no gaps and measure the full width. That width is $T_{\text{total}}$ — the latency one client feels, from "I hit send" to "I got a reply."

WHY a plain sum? Because of Step 1: the stages are strictly sequential and never overlap. The total length of blocks laid end to end is just their lengths added. No multiplication, no max — the geometry of "one after another" is addition. This is why we chose the timeline picture: it makes "+" the only possible answer.

PICTURE. The five blocks fused into one bar, with a bracket underneath spanning the whole thing labelled $T_{\text{total}}$.

Term-by-term, the bracket is telling you: travel + unpack + shape + think + pack. Every millisecond a user waits lives in exactly one of those words. This is the first "gift" formula — now fully earned, every symbol defined in Step 2 and the total defined just above.

---

Step 4 — The 80/20 case: one stage dominates

WHAT. We redraw the bar but with realistic widths: for a big neural net, the "think" block is enormous and the others are slivers. For a tiny model behind a slow link, the "travel" block swallows everything.

WHY this matters. Optimizing a sliver is wasted effort. If $T_{\text{inference}}=45$ ms and everything else sums to $5$ ms, shaving the $5$ ms in half saves you $2.5$ ms — a 5% win. Halving inference saves $22.5$ ms — a 45% win. The picture screams which block to attack first.

PICTURE. Two bars stacked: (top) inference-dominated, (bottom) network-dominated. The dominant block is highlighted; a callout points to "fix this one."

---

Step 5 — Degenerate case: what if a stage is zero?

WHAT. We test the formula at its edges — the "does it still make sense?" cases.

• No preprocessing needed ($T_{\text{preprocess}}=0$): the model eats raw JSON directly. The block has zero width and vanishes. Sum still valid: adding $0$ changes nothing.
• Model already in RAM, cached warm-up done: no extra load term appears in $T_{\text{total}}$ at all — that's why the parent loads the model at startup, so model-load time is never a stage inside a request. It happened once, before any request existed.
• Localhost / same machine ($T_{\text{network}}\to 0$): client and server are the same box, so travel time collapses. The bar shrinks to compute + pack only.

WHY show these. A formula you can't stress-test is a formula you don't understand. Setting a term to $0$ and watching the picture stay coherent proves the sum is robust, not a coincidence of one example.

PICTURE. The full bar next to three "degenerate" bars, each with one block collapsed to zero width, and the total bracket shrinking accordingly.

---

Step 6 — Now many requests: parallel workers

WHAT. In Step 1 a single request could only be in one stage at a time. But different requests can be in different workers simultaneously. We draw $N_{\text{workers}}$ horizontal lanes, one per worker, each lane running its own full $T_{\text{total}}$ bar — and crucially, they overlap in wall-clock time.

WHY the leap. This is the exact opposite of Step 1's rule. One request = one lane = sequential (sum). Many requests = many lanes = parallel (they stack in time, not in length). The picture makes the switch from "+" to "count how many finish per second" visible.

PICTURE. Four stacked lanes, each a $T_{\text{total}}$-wide bar, all starting at $t=0$. A vertical dashed line at $t=T_{\text{total}}$ shows that after one $T$, all four have finished.

---

Step 7 — Derive throughput by counting finishes

WHAT. Look at the dashed line in Step 6. In one time-slice of length $T_{\text{total}}$, exactly $N_{\text{workers}}$ requests crossed the finish line. Rate = amount ÷ time, so:

$\text{Throughput}\approx \frac{\stackrel{\text{finished per slice}}{\overbrace{N_{\text{workers}}}}}{\underset{\text{length of one slice, in seconds}}{\underbrace{T_{\text{total}}}}}\left[\frac{\text{requests}}{\text{second}}\right]$

• $N_{\text{workers}}$ (numerator) — how many requests complete in one batch of time (defined in Step 6).
• $T_{\text{total}}$ (denominator) — how long that batch of time is, per request per worker, in seconds (defined in Step 3).

WHY division, and why this tool? "Requests per second" is literally a ratio: a count over a duration. The moment we ask "how fast do results come out?" the only tool that answers it is a rate = count/time. That's why throughput is a fraction, not a sum.

PICTURE. The same lanes with the finish-line, plus a "conveyor belt" arrow showing $N$ boxes dropping off the end every $T$ seconds, and the fraction $N/T$ written beside it.

---

Step 8 — Degenerate & limiting cases of throughput

WHAT & WHY. We again poke the edges:

• Zero workers ($N_{\text{workers}}=0$): $\text{Throughput}=0/T=0$. No servers ⇒ nothing finishes. Sane.
• Adding workers forever ($N_{\text{workers}}\to \infty$): the formula predicts unbounded throughput — but the picture warns us this assumes every worker has its own CPU/GPU. Once workers outnumber cores they queue for hardware, and a hidden waiting-stage appears in $T_{\text{total}}$, which grows. The clean $N/T$ line bends over. This is exactly the wall Load Balancing & Autoscaling manages.
• Requests arriving faster than $N/T$: a queue forms; latency for a waiting request becomes $T_{\text{total}}+T_{\text{queue}}$. The $\approx$ in Step 7 is precisely this: the formula is the no-queue best case.

PICTURE. A graph: throughput (y) vs number of workers (x). A straight ideal line $N/T$, then a real curve that flattens once workers exceed the core count, with the gap shaded and labelled "queueing / contention."

---

The one-picture summary

WHAT. One figure fuses the whole story: a single request's five-stage bar (the sum, Step 3) sitting inside a stack of $N_{\text{workers}}$ overlapping lanes (the ratio, Step 7), with both master equations printed beside their geometry.

Recall Feynman retelling — explain the whole walkthrough to a friend

Picture a single letter travelling to a puzzle-room: it crosses the street (network), gets its envelope opened (deserialize), the numbers get copied onto a worksheet (preprocess), the puzzle gets solved (inference), and the answer gets sealed in a reply envelope (serialize). Because the letter can only be at one desk at a time, its total wait is just those five times added up — that's $T_{\text{total}}$. To go faster, find the fattest step and shrink that one (usually the solving). Now imagine four puzzle-rooms working at once: even though each letter still takes the same total time, four answers pop out every $T$ seconds, so the rate is $4÷T$ — that's throughput, a count over a time, which is why it's a division. Add rooms and the rate climbs — until the rooms run out of desks (cores) and start waiting in line, at which point the neat straight line bends and a new "waiting" delay quietly creeps back into $T_{\text{total}}$.

Recall Rapid self-test

Why is $T_{\text{total}}$ a sum and not a maximum? ::: Because a single request passes through the stages sequentially — one at a time, never overlapping — so the times lay end to end and add. If $T_{\text{preprocess}}=0$, is the formula still valid? ::: Yes; $0$ is the additive identity, so that stage simply drops out and the sum stays correct. Why is throughput a division ($N/T$) rather than a sum? ::: Because it asks "how many finish per second" — a rate, i.e. a count divided by a duration. Why is throughput written with $\approx$ and not $=$? ::: Because it holds exactly only in the ideal no-queue, dedicated-hardware case; real contention and queueing make it a best-case ceiling. With $N=6$ workers at $T=30\text{ms}=0.030\text{s}$, what is the ideal throughput? ::: $6/0.030=200$ req/s. Why does the ideal $N/T$ line eventually flatten? ::: When workers outnumber physical cores they queue for hardware, adding a hidden waiting term to $T$, so the rate stops growing linearly.

---

Related: Model serving (REST APIs, FastAPI) · Batching & Inference Optimization · Load Balancing & Autoscaling · gRPC vs REST · Docker & Containerization · Model Monitoring & Drift