Model serving (REST APIs, FastAPI)
WHY serve models at all?
WHY not just import the model in every app?
- Model files can be huge (GBs) and need special hardware (GPU) → you don't want a copy everywhere.
- You want one source of truth: update the model in one place, everyone benefits.
- You want language independence: a JavaScript frontend can't
pickle.loada PyTorch model, but it can send an HTTP request.
So we put the model behind a network boundary and talk to it with a standard protocol.
WHAT is a REST API?
For model serving, the key pieces:
| Piece | Meaning | Example |
|---|---|---|
| Endpoint | URL path a client hits | POST /predict |
| Request body | Input features as JSON | {"x1": 2.0, "x2": 5.1} |
| Response | Prediction as JSON | {"pred": 1, "prob": 0.87} |
| Status code | Outcome signal | 200 OK, 422 Unprocessable, 500 Error |
HOW FastAPI works (derive the flow from scratch)
We build the server piece by piece, asking why at each layer.
Step 1 — Load the model ONCE at startup.
from fastapi import FastAPI
import joblib
app = FastAPI()
model = joblib.load("model.pkl") # loaded at import time → stays in RAMWhy this step? Loading a model is expensive. If we loaded it inside the request handler, every request would re-read the file → latency explodes. Load once, reuse across requests.
Step 2 — Define the input schema with Pydantic.
from pydantic import BaseModel
class Features(BaseModel):
x1: float
x2: floatWhy this step? The outside world sends untrusted JSON. Pydantic validates and parses it: if a
client sends x1: "hello", FastAPI auto-rejects with 422 before your model ever sees garbage.
This is your contract.
Step 3 — Define the endpoint.
@app.post("/predict")
def predict(f: Features):
x = [[f.x1, f.x2]]
y = model.predict(x)[0]
p = model.predict_proba(x)[0].max()
return {"pred": int(y), "prob": float(p)}Why this step? The decorator @app.post("/predict") binds the URL + method to the function.
FastAPI reads the type hint f: Features, so it knows to parse the body into that schema
automatically. We return a dict → FastAPI serializes it to JSON.
Step 4 — Run it with an ASGI server.
uvicorn main:app --host 0.0.0.0 --port 8000Why this step? FastAPI is just the app logic; Uvicorn is the actual web server that listens on a socket and speaks HTTP. FastAPI is built on ASGI (Asynchronous Server Gateway Interface), which lets it handle many concurrent requests without blocking.
Worked examples
Common mistakes (Steel-man → Fix)
Active recall
Recall Quick self-test (hide and answer)
- Why load the model at startup, not per request?
- Which HTTP method for
/predictand why? - What does Pydantic do for you?
- What's the difference between FastAPI and Uvicorn?
- How do you increase throughput without changing the model?
Recall Feynman: explain to a 12-year-old
Imagine you're the only kid who can solve a tricky math puzzle. Instead of everyone crowding your desk, you put a mailbox outside your room. People drop a note with their numbers, you solve it, and put the answer back in the mailbox. The mailbox is the REST API. FastAPI is the set of rules for how notes must be written, and it throws back any note that's scribbled wrong. Uvicorn is the mail carrier who actually walks notes to and from your door. You keep the puzzle-solving book (the model) open on your desk all day so you don't fetch it every time.
Forecast-then-Verify
Recall Predict before you run
You have 4 Uvicorn workers, each handling one request in 50 ms. Forecast the max throughput. ... Verify: requests/sec. If you halve inference to 25 ms, throughput doubles to 160 req/s — confirming latency and throughput are inversely linked.
Flashcards
What is model serving?
Why load the model at startup instead of per request?
Which HTTP method should /predict use and why?
What does Pydantic provide in FastAPI?
What is the difference between FastAPI and Uvicorn?
Formula for serving latency?
Formula for throughput?
Why is a batch endpoint faster than many single requests?
Why cast predictions to native Python types before returning?
What HTTP status code should a malformed input return, and why not 500?
Connections
- Docker & Containerization — package the FastAPI app + model into a portable image.
- Model Monitoring & Drift — the served endpoint is where you log inputs/outputs for drift.
- Batching & Inference Optimization — throughput techniques beyond a single worker.
- gRPC vs REST — a lower-latency binary alternative to REST for internal services.
- CI-CD for ML — automatically test and deploy new model versions behind the same endpoint.
- Load Balancing & Autoscaling — scaling from the throughput formula.
Concept Map
Hinglish (regional understanding)
Intuition Hinglish mein samjho
Socho tumne ek badhiya ML model train kar liya — par woh abhi sirf ek .pkl file hai disk pe pada
hua. Koi bhi use tab tak use nahi kar sakta jab tak tum use ek service na bana do. Model
serving ka matlab yahi hai: model ko ek program ke andar wrap karke network pe expose karna,
taaki client (website, mobile app, doosra service) input bheje aur prediction wapas le — bina model
ko apne paas rakhe.
REST API is baat-cheet ki language hai: client POST /predict pe ek JSON body bhejta hai jaise
{"x1": 2.0, "x2": 5.1}, aur server wapas {"pred": 1, "prob": 0.87} bhejta hai. FastAPI ek
Python framework hai jo yeh sab asaan bana deta hai — Pydantic schema se woh galat input ko
automatically reject kar deta hai (422), aur Uvicorn naam ka server actually port pe listen
karta hai. Yaad rakho: FastAPI = logic, Uvicorn = mail carrier jo request andar-bahar le jaata hai.
Do sabse important practical baatein: (1) Model ko startup pe ek baar load karo, har request pe
nahi — warna har call slow ho jaayegi. (2) Jab bahut saare predictions chahiye ho, toh batch
endpoint banao — 1000 rows ek hi model.predict call mein nikalna 1000 alag HTTP calls se kaafi
tez hai. Throughput badhane ka formula simple hai: workers batao aur per-request time ghatao —
Throughput = N_workers / T_per_request. Yeh cheezein interview aur real production dono mein
puchi jaati hain, isliye lifecycle "Listen, Validate, Predict, Return" rat lo.