Model serving (REST APIs, FastAPI): Question bank

True or false — justify

Answer format is Claim ::: Verdict + the one reason that decides it.

REST is stateless, so the server literally cannot remember anything between requests

False — the protocol is stateless (each request must carry everything the server needs), but the server can still hold state elsewhere, e.g. a loaded model in RAM or a database; statelessness is about not relying on prior requests.

Loading the model once at startup means the server can never serve two requests at the same time

False — one loaded model object can be read by many concurrent handlers; loading once is about not re-reading disk, not about locking. (Thread-safety of predict is a separate concern.)

A 500 status code always means the client sent bad data

False — 500 means the server failed; bad client data should surface as a 4xx (like 422). If bad input reaches 500, your validation leaked and you're blaming yourself for the client's mistake.

Using POST /predict instead of GET makes the model run faster

False — the method choice is about semantics and safety (POST = submit data for processing, no caching, body allowed), not raw speed; inference time is identical.

FastAPI and Uvicorn are two names for the same thing

False — FastAPI is the app logic (routing, validation); Uvicorn is the ASGI web server that owns the socket and speaks HTTP. FastAPI can't listen on a port by itself.

Pydantic validation adds so much overhead that skipping it meaningfully increases throughput

False — validation is microseconds versus milliseconds of inference; skipping it mainly buys you 500s and a security hole, not speed.

Doubling the number of workers always doubles throughput

False — only until you hit a shared bottleneck (CPU cores, GPU, memory, or a downstream DB). Beyond that, adding workers adds contention, not throughput. See Load Balancing & Autoscaling.

JSON can serialize a numpy int64 prediction directly

False — JSON only knows native types; np.int64/np.float32 raise a serialization error. You must cast to int/float or call .tolist().

A batch endpoint is faster per row than 1000 single calls

True — one HTTP round-trip plus vectorized inference beats 1000 round-trips and 1000 un-vectorized calls; the overhead is amortized. See Batching & Inference Optimization.

Because requests are stateless, the model must be reloaded from disk on every request

False — this confuses "no request memory" with "no server memory." The model lives in RAM across requests; reloading per request is the classic latency-killing mistake.

Spot the error

Each line describes a snippet or decision; the reveal names the bug and the fix.

@app.get("/predict") with features packed into the query string

GET has URL length limits, gets cached by proxies, and logs your input data in plain URLs; use POST with a JSON body instead.

def predict(): model = joblib.load("model.pkl"); return model.predict(x)

The model is reloaded inside the handler → disk I/O on every request, hundreds of ms of latency, possible memory blow-up; load it once at startup scope.

return {"pred": model.predict(x)[0]} where predict returns a numpy array

model.predict(x)[0] is an np.int64, which JSON can't serialize → runtime error; wrap it as int(...).

def predict(f): x = [f.x1, f.x2]; model.predict(x)

sklearn expects a 2D array (list of rows); passing a 1D list raises a shape error. Use [[f.x1, f.x2]].

A handler that catches every exception and returns 200 OK with {"error": "..."}

This lies to the client — automated systems that check status codes think it succeeded. Let validation errors be 4xx and real failures be 5xx.

Defining class Features: x1: float as a plain class (no BaseModel)

Without inheriting pydantic.BaseModel, FastAPI won't validate or parse the body; the type hint alone does nothing. Inherit from BaseModel.

Running the server as python main.py and expecting concurrency

FastAPI needs an ASGI server (uvicorn main:app); running the script directly gives you no listening socket and no concurrency.

preds = model.predict(b.rows); return {"preds": preds} in a batch endpoint

preds is a numpy array → JSON fails; call preds.tolist().

Why questions

Reason on the left, the answer on the right.

Why is POST semantically correct for /predict but not GET

/predict submits data for processing and produces a computed result; GET is meant for reading a fixed resource with no side effects and no large body.

Why does Pydantic put errors in the right place

It rejects malformed JSON before the model runs and returns 422, so the blame lands on the client (4xx) instead of surfacing as a server crash (5xx).

Why load the model at module scope instead of inside the function

So the expensive disk read + deserialization happens once, and every request reuses the in-RAM object; per-request loading multiplies latency and memory use.

Why does statelessness make horizontal scaling easy

If no request depends on server memory of a previous one, any replica can handle any request, so a load balancer can freely spread traffic across identical copies. See Load Balancing & Autoscaling.

Why do we serve a model behind a network boundary instead of importing it everywhere

One source of truth (update once), no need to copy GB-sized weights or special hardware to every client, and language independence — a JS frontend can send HTTP but can't pickle.load a PyTorch model.

Why optimize the dominant term in the latency budget first

Total latency is a sum of pipeline stages; shaving 90% off a tiny stage barely moves the total, while cutting the biggest stage gives most of the gain — the 80/20 rule.

Why does a batch endpoint raise throughput without changing the model

It replaces many round-trips and un-vectorized calls with one round-trip and one vectorized predict, so fixed per-call overheads are paid once. See Batching & Inference Optimization.

Why might gRPC be chosen over REST for internal model calls

gRPC uses binary Protobuf (smaller, faster to (de)serialize) and HTTP/2 multiplexing, cutting

T_{serialize}

and

T_{network}

for high-volume service-to-service traffic. See gRPC vs REST.

Edge cases

The reveals here cover the scenarios the happy path skips.

A client sends {"x1": 2.0} and omits x2

Pydantic rejects it with 422 (missing required field) before the model runs — the contract requires both fields, so no garbage reaches inference.

A client sends valid JSON but with an extra field {"x1":2,"x2":5,"hack":1}

By default Pydantic ignores unknown extra fields and parses the rest fine; if you need strictness, configure the model to forbid extras and return 422.

A client sends an empty body to POST /predict

Validation fails immediately (422) since required fields are absent; the handler and model are never invoked.

Two requests arrive at the exact same millisecond with one worker

A single worker processes them in sequence (queued), so the second request's latency includes the first's inference time; add workers/replicas to parallelize. See Load Balancing & Autoscaling.

The batch endpoint receives {"rows": []} (empty batch)

model.predict([]) returns an empty array; you get {"preds": []} with 200 — degenerate but valid, so don't crash on it, just return empty.

Inference time is tiny but network time dominates the latency budget

Optimizing the model won't help; the win is in transport — batching, compression, keep-alive connections, or switching to gRPC. See gRPC vs REST.

The input passes validation but is far outside the training distribution

The API still returns a confident-looking prediction — validation checks types, not plausibility; out-of-distribution inputs are caught by Model Monitoring & Drift, not by Pydantic.

You deploy a new model version and old and new replicas run simultaneously during rollout

Different clients may briefly get answers from different model versions; use versioned endpoints or controlled rollouts so results stay consistent. See CI-CD for ML.

A model needs a GPU but the container has none

Inference either errors or silently falls back to slow CPU, spiking

T_{inference}

; the serving container must be built and scheduled with the right hardware. See Docker & Containerization.

Recall One-line summary of the traps

Statelessness ≠ no server memory; 5xx ≠ client's fault; load once, validate always, cast to native types, and remember validation checks types, not truth.

Question: What is the single most common latency bug in this whole page? ::: Reloading the model inside the request handler instead of once at startup.