Model serving (REST APIs, FastAPI): Exercises

The two formulas we lean on come straight from the parent note:

Before we start, a one-time reminder of the unit trick we use constantly: milliseconds → seconds means divide by $1000$ , because $1 s = 1000 ms$ . So $50 ms = 0.050 s$ .

Level 1 — Recognition

Goal: can you name the piece and pick the right label? No maths yet.

L1.1 — For each row of a POST /predict conversation, name the piece (Endpoint, Request body, Response, or Status code): (a) /predict (b) {"x1": 2.0, "x2": 5.1} (c) 422 (d) {"pred": 1, "prob": 0.87}

Recall Solution L1.1

(a) Endpoint — it is the URL path the client hits. (b) Request body — the input features, sent as JSON. (c) Status code — the outcome signal ( $422$ = Unprocessable Entity). (d) Response — the prediction returned as JSON.

L1.2 — Match each tool to its ONE job: FastAPI, Uvicorn, Pydantic, joblib. Jobs: (i) owns the socket and speaks HTTP; (ii) validates/parses incoming JSON; (iii) binds URL+method to a Python function; (iv) loads the model file from disk.

Recall Solution L1.2

Uvicorn ::: (i) owns the socket and speaks HTTP — the mail carrier.
Pydantic ::: (ii) validates/parses incoming JSON — the note-checker.
FastAPI ::: (iii) binds URL+method to a Python function via @app.post(...).
joblib ::: (iv) loads the model file (model.pkl) from disk.

Level 2 — Application

Goal: plug numbers into the two formulas.

L2.1 — A server has $3$ Uvicorn workers. Each handles one request in $T = 40 ms$ . Compute the max throughput in req/s.

Recall Solution L2.1

WHAT: convert to seconds, then apply throughput $= N / T$ . WHY: the formula needs $T$ in seconds so the answer comes out in req/s. $T = 40 ms = 0.040 s, Throughput = \frac{N}{T} = \frac{3}{0.040} = 75 req/s .$

L2.2 — A single request breaks down as: $T_{net} = 12$ , $T_{deser} = 3$ , $T_{pre} = 5$ , $T_{inf} = 25$ , $T_{ser} = 5$ (all ms). Find $T_{total}$ and the throughput of one worker.

Recall Solution L2.2

WHAT: sum the pipeline stages (they happen one after another for a single request). $T_{total} = 12 + 3 + 5 + 25 + 5 = 50 ms = 0.050 s .$ One worker: $Throughput = 1/0.050 = 20$ req/s.

L2.3 — With the L2.2 breakdown, which single stage should you optimize first, and by the 80/20 rule roughly how much of total time is it?

Recall Solution L2.3

WHAT: find the dominant term. $T_{inf} = 25$ ms is the biggest. WHY: optimizing a $25/50 = 50%$ chunk pays far more than shaving the $3$ ms deserialize step. $\frac{T _{inf}}{T _{total}} = \frac{25}{50} = 0.5 = 50%.$ Attack inference first (batching, quantization, a smaller model). See Batching & Inference Optimization.

Level 3 — Analysis

Goal: compare designs and explain the trade-off, not just compute.

L3.1 — Design A: one worker, $T = 50$ ms/request. Design B: batching — one call on $100$ rows takes $200$ ms total. Compare throughput (rows/s) and per-row latency.

Recall Solution L3.1

WHAT/WHY: batching amortizes fixed overhead across many rows, but each individual row now waits for the whole batch.

Design A throughput: $1/0.050 = 20$ rows/s. Per-row latency: $50$ ms.
Design B throughput: $100$ rows in $0.200$ s $= 500$ rows/s. Per-row effective latency: $200/100 = 2$ ms of compute, but a row may wait up to $200$ ms for the batch to fill. Trade-off: batching multiplies throughput ( $20 \to 500$ , a $25 \times$ gain) at the cost of tail latency. Great for offline scoring, risky for a real-time UI.

Look at the figure: the flat batch bar towers over the single-request bar for throughput, while the right panel shows latency moving the opposite way.

Figure — Model serving (REST APIs, FastAPI)

L3.2 — You need $300$ req/s. Each worker does $T = 50$ ms/request. How many workers minimum?

Recall Solution L3.2

WHAT: invert the throughput formula for $N$ . Since $Throughput = N / T$ , $N = Throughput \times T = 300 \times 0.050 = 15 workers .$ WHY exactly 15, not 14.x: $14$ workers give $14/0.050 = 280 < 300$ ; $15$ give $300$ . You must round up — you can't run a fractional worker. See Load Balancing & Autoscaling.

L3.3 — Client sends {"x1": "cat", "x2": 5.1}. Compare the outcome with vs without the Pydantic schema, and say which HTTP status class each returns and who it blames.

Recall Solution L3.3

Without Pydantic: the string "cat" reaches model.predict, which throws a ValueError. FastAPI turns an uncaught exception into 500 — a 5xx, which blames the server.
With Pydantic: validation fails before the model runs → 422 Unprocessable Entity, a 4xx, correctly blaming the client's bad input. Analysis: the same bad request produces opposite fault attribution. 4xx = "you sent garbage"; 5xx = "I broke." Pydantic shifts the blame to the correct party and protects the model.

Level 4 — Synthesis

Goal: assemble multiple pieces into a working design and reason about it end-to-end.

L4.1 — Sketch the request lifecycle for POST /predict as an ordered flow, naming which component owns each step, and give the mnemonic from the parent note.

Recall Solution L4.1

Order (L-V-P-R): Listen → Validate → Predict → Return.

Mnemonic: "Little Vans Please Reverse" = Listen, Validate, Predict, Return. Key design choice: the model is loaded once at startup and stays in RAM, so step "M" never touches disk. See Docker & Containerization for shipping this whole stack as one image.

L4.2 — A model needs a GPU and is $2$ GB on disk. You must serve a JavaScript web frontend. Justify (a) using a REST API instead of importing the model in JS, and (b) loading at startup, and (c) computing the throughput if a request costs $80$ ms with $6$ replicas.

Recall Solution L4.2

(a) A JS frontend cannot pickle.load a PyTorch model and has no GPU — but it can send an HTTP request. REST gives language independence and one source of truth for the $2$ GB weights, instead of copying them everywhere. (b) Loading a $2$ GB model from disk is expensive; doing it per request would add hundreds of ms and could exhaust memory. Load once at module/startup scope so all requests reuse the RAM copy. (c) $T = 80 ms = 0.080 s$ , $N = 6$ : $Throughput = \frac{6}{0.080} = 75 req/s .$

Level 5 — Mastery

Goal: multi-step reasoning, degenerate cases, and pushing formulas to their limits.

L5.1 — You currently serve at $T_{total} = 50$ ms ( $T_{inf} = 25$ , rest $= 25$ ). You cut inference in half to $12.5$ ms via quantization. (a) New $T_{total}$ ? (b) New single-worker throughput? (c) By what factor did throughput improve — and why is it NOT $2 \times$ ?

Recall Solution L5.1

(a) $T_{total} = 25 + 12.5 = 37.5 ms = 0.0375 s$ (the non-inference $25$ ms is untouched — Amdahl's law in disguise). (b) $Throughput = 1/0.0375 = 26. \overline{6} \approx 26.67$ req/s. (c) Old throughput $= 20$ req/s, so improvement $= 26.67/20 = 1.33 \times$ , not $2 \times$ . WHY: you only halved the inference half of the pipeline. The fixed $25$ ms floor caps how much any inference speedup can help — the classic diminishing-returns wall.

L5.2 — Degenerate limit. Suppose you make inference free: $T_{inf} \to 0$ . With the $25$ ms of non-inference overhead fixed, what is the maximum possible single-worker throughput? What does this tell you about where to look next?

Recall Solution L5.2

As $T_{inf} \to 0$ , $T_{total} \to 25 ms = 0.025 s$ , so $Throughput_{m a x} = \frac{1}{0.025} = 40 req/s .$ Even with a perfect, instant model you cannot beat $40$ req/s per worker. The ceiling is set by $T_{net} + T_{deser} + T_{pre} + T_{ser}$ . Next optimization must target those (bigger batches, faster serialization / gRPC, less preprocessing), not the model. This is why you always find the dominant term after each fix.

L5.3 — Batch vs many-calls, full comparison. Scoring $1000$ rows two ways: Plan A — $1000$ separate POST /predict calls, each $T_{total} = 50$ ms on one worker. Plan B — one POST /predict_batch of $1000$ rows: fixed overhead $25$ ms + $0.05$ ms per row of inference. Compute total wall-clock time for each and the speedup.

Recall Solution L5.3

Plan A: $1000 \times 50 ms = 50000 ms = 50 s$ (each call pays the full overhead again). Plan B: overhead paid once + vectorized inference: $25 ms + 1000 \times 0.05 ms = 25 + 50 = 75 ms = 0.075 s .$ Speedup: $50000/75 = 666. \overline{6} \approx 666.67 \times$ . WHY so huge: Plan A repeats the fixed per-request overhead $1000$ times; Plan B amortizes it once and lets the model's vectorized predict chew all rows together. This is the core argument for the batch endpoint in the parent note (Example 3). See Batching & Inference Optimization.

Wrap-up recall

Recall One-line takeaways (hide and recite)

Throughput uses $T$ in seconds ::: convert ms → s by dividing by 1000 first.
To hit a target rate, workers $N = rate \times T$ ::: then round up.
Speeding one pipeline stage helps only by its share ::: fixed overhead sets a ceiling.
Batching trades tail latency for huge throughput ::: amortize overhead across rows.
4xx blames the client, 5xx blames the server ::: Pydantic converts crashes into 422.