Model serving (REST APIs, FastAPI): Worked examples

The scenario matrix

Think of an incoming HTTP request as a point that lands somewhere in a grid. The two axes that matter most are: is the input well-formed? and how many items are we predicting? Layered on top are degenerate inputs (empty, zero, extreme) and performance limits (latency, throughput).

#	Case class	What makes it different	Worked in
A	Happy single — valid, one row	The "sunny day" path → `200`	Ex 1
B	Type-invalid input	Wrong type (string where float expected) → `422`	Ex 2
C	Missing / extra field	Schema shape wrong → `422`	Ex 3
D	Degenerate: empty batch	`rows = []` — valid JSON, zero work	Ex 4
E	Numpy leak	Valid predict, but non-JSON return type → `500`	Ex 5
F	Latency budget	Which stage dominates $T_{total}$ ?	Ex 6
G	Throughput / scaling	Word problem: how many workers?	Ex 7
H	Batch vs N singles	Same 1000 rows, two designs → cost tradeoff	Ex 8
I	Exam twist: cold start	Model loaded per-request vs at startup	Ex 9
J	Extreme value / overflow	Valid float but `inf`/`NaN` slips through	Ex 10

Every cell A–J is covered below. Cells B, C, J are the "bad-input quadrants" — the equivalent of the tan-formula failing in quadrants II/III/IV: naive code gives the wrong status code, and we show the sign-based fix (which 4xx vs 5xx).

Example 1 — Case A: the happy single request

Worked example Valid single prediction

Client sends POST /predict with body {"x1": 2.0, "x2": 5.1}. The loaded model is a logistic classifier that outputs class 1 with probability 0.87 for this point. What does the server return, and with which status?

Forecast: Guess the response body and status code before reading on.

Uvicorn reads raw bytes off port 8000 into memory. Why this step? Uvicorn owns the socket; FastAPI never touches the network directly.
FastAPI matches POST /predict to the handler, then parses the JSON body into Features(x1=2.0, x2=5.1). Why this step? The type hint f: Features is the trigger — FastAPI validates before your code runs, so the handler only sees clean data.
Handler builds [[2.0, 5.1]] (a 2-D array) and calls model.predict. Why this step? sklearn expects a 2-D array of shape (n_samples, n_features); a single sample is still one row → double brackets.
Return {"pred": 1, "prob": 0.87}. Why this step? A Python dict of native types serializes to JSON automatically → 200 OK.

Verify: One sample in, one prediction out, no exception raised → status 200. Probability $0.87 \in [0, 1]$ ✓, class $\in {0, 1}$ ✓. Sanity: the model's probabilities are [0.13, 0.87], and the biggest sits at position 1 → argmax = 1, which equals pred. So the returned class and the highest-probability class agree ✓ (see the argmax definition above).

Example 2 — Case B: wrong type (the "quadrant II" of bad input)

Worked example String where a float belongs

Client sends {"x1": "cat", "x2": 5.1}.

Forecast: Which status code — 422, 500, or 200? Who is to blame, client or server?

FastAPI tries to coerce "cat" to float. It fails. Why this step? Pydantic runs first. float("cat") is impossible, so validation stops here.
FastAPI returns 422 Unprocessable Entity with a message pinpointing x1. Why this step? The request was syntactically fine JSON but semantically violates the contract → 4xx (client's fault), not 5xx.
model.predict is never called. Why this step? The garbage was stopped at the boundary — the model stays clean.

Verify: Contrast with no validation: model.predict([["cat", 5.1]]) raises ValueError → handled as 500 (server error). We want 422. Rule of thumb: malformed input the client controls → 4xx; something we broke → 5xx. Here the fix flips a wrong 500 into a correct 422.

Example 3 — Case C: missing / extra field

Worked example Wrong shape, not wrong type

Two clients: (a) sends {"x1": 2.0} (missing x2); (b) sends {"x1": 2.0, "x2": 5.1, "x3": 9}.

Forecast: Does either return 200? Which field does FastAPI complain about?

Case (a): x2 is absent. Pydantic marks x2 as required and missing → 422. Why this step? x2: float has no default, so it is mandatory.
Case (b): x3 is extra. By default Pydantic ignores unknown fields → parses to Features(x1=2.0, x2=5.1) → 200. Why this step? Default config is lenient on extras; only declared fields are enforced.
To reject extras too, tell the schema to forbid unknowns. In Pydantic v2 you add a class attribute model_config = ConfigDict(extra="forbid"), where ConfigDict is Pydantic's built-in settings object (imported with from pydantic import ConfigDict) that tweaks how a schema validates. Setting extra="forbid" makes any undeclared field a 422. Why this step? If a typo like "x2 " (trailing space) silently drops a real value, you'd rather fail loud → 422.

Verify: (a) → 422 (missing) ✓. (b) → 200 by default, or 422 if extra="forbid". Both outcomes are client-attributable → 4xx family, never 5xx.

Example 4 — Case D: the degenerate empty batch

Example 5 — Case E: the numpy leak

Worked example Returning a raw numpy scalar

Handler does return {"pred": model.predict(x)[0]} where model.predict yields np.int64(1).

Forecast: 200, or a surprise 500? The math is correct — so what breaks?

Prediction is correct: value is 1, logically fine. Why this step? The model did its job; the bug is downstream.
FastAPI tries to JSON-serialize np.int64(1). The default encoder doesn't know this type → raises → 500. Why this step? JSON only knows native int, float, str, bool, list, dict.
Fix: cast — int(model.predict(x)[0]) or .tolist() for arrays. Why this step? Casting produces a native type the encoder understands.

Verify: Same logical output pred = 1, but int(np.int64(1)) == 1 and is JSON-safe → 200. This is a "correct-answer-wrong-envelope" failure: the value is right, the type leaks.

Example 6 — Case F: which latency stage dominates?

Figure 1 — The five pipeline stages of one request, stacked into a single bar. Height of each coloured block = time that stage takes (labelled in ms). The two orange blocks are inference and serialize; the tall inference block (70 ms) is called out with an arrow as the "dominant stage." The total bar height reads T_total = 100 ms. Read it top-to-bottom: a request passes through each block in series, so total time is the sum of the block heights.

Worked example Reading the latency budget

Measured per-request: $T_{network} = 20$ ms, $T_{deserialize} = 2$ ms, $T_{preprocess} = 3$ ms, $T_{inference} = 70$ ms, $T_{serialize} = 5$ ms.

Forecast: What is $T_{total}$ , and which single optimization helps most?

Sum the pipeline (from the parent's formula $T_{total} = T_{network} + T_{deserialize} + T_{preprocess} + T_{inference} + T_{serialize}$ ): $20 + 2 + 3 + 70 + 5 = 100$ ms. Why this step? A request flows through each stage in series → total is the sum (look at the stacked bar in Figure 1; each block sits on top of the previous).
Find the dominant term: $T_{inference} = 70$ ms is $70%$ of the total. Why this step? The 80/20 rule — optimizing anything but the tall block barely moves the sum.
Estimate the payoff: halve inference to $35$ ms → total $= 65$ ms (a $35%$ cut). Halving serialize instead ( $5 \to 2.5$ ) saves $2.5$ ms → total $97.5$ ms (a $2.5%$ cut). Why this step? Compare the two candidate savings to prove where to spend effort.

Verify: $20 + 2 + 3 + 70 + 5 = 100$ ✓. New total after halving inference $= 20 + 2 + 3 + 35 + 5 = 65$ ✓. Fraction saved $= (100 - 65) /100 = 0.35 = 35%$ ✓. Inference block is tallest → optimize it first (see Batching & Inference Optimization).

Example 7 — Case G: how many workers? (word problem)

Worked example Sizing for a traffic target

Your service must handle 300 requests/sec at peak. Each worker processes one request in $T = 40$ ms. How many workers do you need? What if traffic doubles?

Forecast: Guess the worker count before computing.

Recall throughput $\approx N_{workers} / T_{per request}$ (parent formula). Solve for $N$ : $N = Throughput \times T$ . Why this step? We know the rate we need and the time per job; invert the relation.
Plug in: $N = 300 \times 0.040 = 12$ workers. Why this step? $0.040$ s is $40$ ms in SI units — keep units consistent (seconds).
Traffic doubles to 600 req/s: $N = 600 \times 0.040 = 24$ workers. Why this step? Throughput scales linearly with workers, so doubling load doubles workers.

Verify: Check the forward direction: $12$ workers $/0.040$ s $= 300$ req/s ✓; $24/0.040 = 600$ req/s ✓. Units: $req = (req/s) \times s$ ✓. Past a point, add machines not just workers → see Load Balancing & Autoscaling.

Example 8 — Case H: batch endpoint vs N single calls

Figure 2 — Two stacked bars comparing designs for scoring 1000 rows. Left bar (Design A: 1000 separate requests) is dominated by a huge teal "overhead" block because the per-request overhead is paid 1000 times; its total is 10200 ms. Right bar (Design B: one batch request) is tiny — teal overhead paid once (10 ms) plus the same orange inference (200 ms), total 210 ms. The plum arrow labels the killed overhead. The visual gap (~48x shorter) is the whole argument for batching.

Worked example 1000 predictions, two designs

You must score 1000 feature rows. Design A: 1000 separate POST /predict calls. Design B: one POST /predict_batch with all 1000 rows. Fixed per-request overhead (network + deserialize) is $10$ ms; per-row inference is $0.2$ ms.

Forecast: Which is faster, and by how much?

Design A cost: each call pays overhead + one inference: $1000 \times (10 + 0.2) = 1000 \times 10.2 = 10200$ ms. Why this step? Overhead is paid per request, so 1000 requests pay it 1000 times (the tall left bar in Figure 2).
Design B cost: overhead paid once, inference vectorized over 1000 rows: $10 + 1000 \times 0.2 = 10 + 200 = 210$ ms. Why this step? One round-trip → overhead once; sklearn's vectorized predict amortizes the per-row cost.
Speedup: $10200/210 \approx 48.6 \times$ . Why this step? Ratio of the two totals quantifies the win.

Verify: $1000 \times 10.2 = 10200$ ✓; $10 + 200 = 210$ ✓; $10200/210 = 48.57 \dots$ ✓. The overhead term is what batching kills — exactly why batch endpoints exist.

Example 9 — Case I: exam twist, the cold-start trap

Worked example Model loaded per-request vs at startup

Model load takes $800$ ms; inference takes $50$ ms. Compare average latency for Design X (load inside the handler, every request) vs Design Y (load once at startup), over 100 requests.

Forecast: How much slower is X per request? Does Y's load time ever "count"?

Design X per-request latency: $800 + 50 = 850$ ms every request. Why this step? Loading inside the handler repeats the disk I/O + deserialization each call.
Design Y per-request latency: $50$ ms; the $800$ ms is paid once at startup, before any request. Why this step? The model sits in RAM; requests reuse it (parent Step 1).
Over 100 requests: X total $= 100 \times 850 = 85000$ ms. Y total $= 800 + 100 \times 50 = 5800$ ms. Amortized per-request for Y $= 5800/100 = 58$ ms. Why this step? Spreading one-time cost over many requests makes it negligible.

Verify: X: $100 \times 850 = 85000$ ✓. Y: $800 + 5000 = 5800$ ✓; amortized $58$ ms ✓. Y is $\approx 14.7 \times$ cheaper in total ( $85000/5800 = 14.65 \dots$ ). The load cost "disappears" as request count grows — the essence of "load once."

Example 10 — Case J: extreme value / `NaN` leak

Worked example A valid float that isn't a valid

feature Client sends {"x1": 1e309, "x2": "NaN"}. Note 1e309 overflows to inf (infinity — a float too big to represent); "NaN" means "Not a Number" and parses to a float NaN in permissive parsers.

Forecast: Does Pydantic stop this? Does the model? What status code results?

Type check passes: inf and NaN are technically floats, so a bare float field lets them through. Why this step? Type validation checks kind, not sanity — the "quadrant IV" trap where the naive check looks fine but the value is pathological. So plain Pydantic does NOT stop it.
The model receives the bad value: many sklearn models raise on inf/NaN → an unhandled exception → 500; others silently return a nonsense probability with status 200. Why this step? Because validation was too loose, the pathological number reaches inference — and now we look broken (500) even though the client sent nonsense.
Fix: tighten the schema so infinities and NaNs are rejected at the boundary, e.g. in Pydantic v2: x1: float = Field(..., allow_inf_nan=False). Now inf/NaN fail validation. Why this step? This shifts the error back to a correct 422 (client's fault) and the model is never touched — the same boundary-guard idea as Example 2, one layer deeper.

Verify: Behaviour flips: without the constraint, NaN/inf → 500 (server blamed) or a nonsense 200; with allow_inf_nan=False, → 422 (client blamed) ✓. Answer to the forecast: plain Pydantic does not stop inf/NaN, the model may crash, and the correct end state after the fix is 422. Same lesson as Case B, deeper: type-valid ≠ semantically valid.

Coverage recap

Recall Did we hit every cell?

A (Ex 1), B (Ex 2), C (Ex 3), D (Ex 4), E (Ex 5), F (Ex 6), G (Ex 7), H (Ex 8), I (Ex 9), J (Ex 10). Every row of the matrix has a worked example. ✓

Recall Status-code decision rule

Bad input the client controls (wrong type, missing field, inf/NaN) ::: 4xx (usually 422) Something our code/model broke (numpy leak, unhandled exception) ::: 5xx (usually 500) Valid but empty input ::: 200 with an empty result

Related depth: Batching & Inference Optimization, Load Balancing & Autoscaling, Model Monitoring & Drift, gRPC vs REST, Docker & Containerization, CI-CD for ML.