MLOps & Deployment

Question 1

(a) [5 marks] $r(b) = R_{\max}/b = 8000/b$ tokens/s per request.

$T_{\text{req}}(b) = 0.040 + \frac{256}{8000/b} = 0.040 + \frac{256\,b}{8000} = 0.040 + 0.032\,b \text{ s}.$ (2 marks derivation)

Set $\le 0.500$: $0.032\,b \le 0.460 \Rightarrow b \le 14.375$. (2) Largest integer batch: $b^\star = 14$. (1) Then $T_{\text{req}}(14) = 0.040 + 0.448 = 0.488$ s.

(b) [7 marks] Service rate $\mu = 1/T_{\text{req}}(b^\star) = 1/0.488 = 2.049$ req/s. (1)

For M/M/1, expected response time (Little/queueing result): $W = \frac{1}{\mu - \lambda}. \quad (2)$

Require $W \le 0.5$: $\frac{1}{\mu-\lambda}\le 0.5 \Rightarrow \mu-\lambda \ge 2 \Rightarrow \lambda \le \mu - 2 = 2.049 - 2 = 0.049$ req/s. (3)

Utilisation $\rho = \lambda/\mu = 0.049/2.049 = 0.024$. (1)

Interpretation credit: Since service alone (0.488 s) nearly consumes the 500 ms budget, almost no queueing headroom remains — hence the tiny per-replica $\lambda$. This is the correct, if uncomfortable, conclusion. (Grader: full marks for correct method even noting the tightness.)

(c) [5 marks] Per-replica capacity $\lambda_{\text{rep}} = 0.049$ req/s. $N = \lceil \lambda_{\text{total}}/\lambda_{\text{rep}} \rceil = \lceil 120/0.049 \rceil = \lceil 2449 \rceil = 2449 \text{ replicas}. \quad (2)$ Hourly cost = 2449 \times \2.60 = $6367.4/\text{h}$. *(1)* Requests/hour at full load$= 120 \times 3600 = 432{,}000$. Cost per million requests$= \dfrac{6367.4}{0.432} = $14{,}739$ per million. (2)

(Discussion note: such a strict p99≈service-time budget makes real-time serving expensive; relaxing the SLO or batching harder trades latency for cost. Recognising this earns full marks.)

(d) [7 marks]

• Memory: FP16 = 2 bytes/param, INT8 = 1 byte/param. Footprint = $P\times$ bytes. Saving factor $= 2\text{ bytes}/1\text{ byte} = 2\times$ (weights halve). (2)
• Latency: LLM decode is memory-bandwidth bound; time to stream weights $\propto$ bytes moved. Halving weight bytes roughly halves the per-token weight-load time, and INT8 tensor-core throughput is higher than FP16, so latency drops (bandwidth + compute). (2)
• Accuracy preservation: e.g. per-channel / group-wise quantization, calibration on representative data, or keeping outlier channels in higher precision (SmoothQuant / AWQ). Naming any valid technique with reason. (1.5)
• Monitoring signal: track output quality / perplexity drift or downstream task metric, plus token-level distribution shift vs FP16 baseline, or user thumbs-down rate. (1.5)

Question 2

(a) [8 marks] KL between two Gaussians $Q\sim\mathcal N(\mu_1,\sigma_1^2)$, $P\sim\mathcal N(\mu_0,\sigma_0^2)$: $D_{\text{KL}}(Q\|P)=\mathbb E_Q\Big[\ln\frac{q(x)}{p(x)}\Big].$ $\ln\frac{q}{p}=\ln\frac{\sigma_0}{\sigma_1} -\frac{(x-\mu_1)^2}{2\sigma_1^2}+\frac{(x-\mu_0)^2}{2\sigma_0^2}.$ Take $\mathbb E_Q$: $\mathbb E_Q[(x-\mu_1)^2]=\sigma_1^2$, and $\mathbb E_Q[(x-\mu_0)^2]=\sigma_1^2+(\mu_1-\mu_0)^2$. (4) $D_{\text{KL}}=\ln\frac{\sigma_0}{\sigma_1}-\frac{\sigma_1^2}{2\sigma_1^2}+\frac{\sigma_1^2+(\mu_1-\mu_0)^2}{2\sigma_0^2}=\ln\frac{\sigma_0}{\sigma_1}+\frac{\sigma_1^2+(\mu_1-\mu_0)^2}{2\sigma_0^2}-\frac12.\ \checkmark \quad (2)$

Numeric with $\mu_0=0,\sigma_0=1,\mu_1=0.5,\sigma_1=1.5$: $\ln(1/1.5)=-0.405465,\quad \frac{2.25+0.25}{2}=1.25,$ $D_{\text{KL}}=-0.405465+1.25-0.5=0.344535\ \text{nats}. \quad (2)$

(b) [6 marks]

• Data drift (covariate shift): input distribution changes, $P_{\text{prod}}(x)\ne P_{\text{train}}(x)$, while $P(y\mid x)$ unchanged. (1.5)
• Concept drift: the mapping changes, $P_{\text{prod}}(y\mid x)\ne P_{\text{train}}(y\mid x)$. (1.5)
• High accuracy despite drift: input distribution shifts into a region where the decision boundary is still correct (e.g. more traffic from a demographic the model already handles well). (1.5)
• Accuracy collapse with no input drift: same $P(x)$ but labels' relationship changed — e.g. fraud patterns evolve, spam wording stays but "spam vs not" definition flips. (1.5)

(c) [6 marks]

• Threshold via null distribution: estimate the drift statistic's distribution under "no drift" (bootstrap over historical stable windows). Set threshold at a quantile controlling per-window false-positive rate $\alpha$; apply multiple-comparison / windowing correction to bound the expected number of false retrains. (2)
• Automated CI/CD chain: drift alert → trigger retraining job on versioned data snapshot → automated eval on held-out + shadow traffic → gate on metric threshold → register new model in registry with lineage → canary deploy → promote. (2)
• Versioning for rollback: DVC pins the exact dataset hash, MLflow/registry pins model artifact + code commit + hyperparams; a rollback = redeploy the previous registry version, guaranteed reproducible because all three (data, code, model) are content-addressed. (2)

Question 3

(a) [4 marks]

Example: batch → nightly recommendation precompute; real-time → fraud scoring at checkout. (4: 1 per row + example)

(b) [6 marks] $p_A=0.04,\ p_B=0.044,\ (z_{\alpha/2}+z_\beta)^2=(1.96+0.84)^2=7.84$. (1) $p_A(1-p_A)=0.0384,\quad p_B(1-p_B)=0.044\cdot0.956=0.042064$. Sum $=0.080464$. (2) $(p_B-p_A)^2=(0.004)^2=1.6\times10^{-5}$. (1) $n=\frac{7.84\times0.080464}{1.6\times10^{-5}}=\frac{0.630838}{1.6\times10^{-5}}\approx 39{,}427 \text{ per arm}. \quad (2)$

(c) [6 marks]

import hashlib
from fastapi import FastAPI, Request
import joblib
 
app = FastAPI()
models = {}
 
@app.on_event("startup")
def load():
    models["A"] = joblib.load("model_a.pkl")   # stable
    models["B"] = joblib.load("model_b.pkl")   # canary
 
def route(req_id: str) -> str:
    h = int(hashlib.md5(req_id.encode()).hexdigest(), 16)
    return "B" if (h % 100) < 10 else "A"      # 10% canary
 
@app.post("/predict")
async def predict(request: Request):
    body = await request.json()
    req_id = body["request_id"]
    version = route(req_id)
    pred = models[version].predict([body["features"]])[0]
    return {"prediction": float(pred), "model_version": version}

(4 marks: startup load, hash routing 10%, version in response)

Why hash the request id: deterministic routing means the same user/request always hits the same model — consistent user experience, valid A/B attribution, and no flip-flop between refreshes. Pure random routing would send repeat requests to different models, contaminating measurement and confusing users. (2)

[
  {"claim":"Q1a largest batch size b*=14 gives Treq=0.488s <=0.5","code":"b=14; T=0.040+0.032*b; result=(T<=0.5) and (0.040+0.032*15>0.5)"},
  {"claim":"Q1b lambda max = 1/0.488 - 2 approx 0.0492","code":"mu=1/0.488; lam=mu-2; result=abs(lam-0.04918)<1e-3"},
  {"claim":"Q1c N=2449 replicas for 120 req/s","code":"import math; lam=1/0.488-2; N=math.ceil(120/lam); result=(N==2449)"},
  {"claim":"Q2a KL(N(0.5,1.5^2)||N(0,1)) approx 0.34453","code":"kl=ln(Rational(1,1)/Rational(3,2))+(Rational(9,4)+Rational(1,4))/2-Rational(1,2); result=abs(float(kl)-0.344535)<1e-4"},
  {"claim":"Q3b per-arm n approx 39427","code":"n=(7.84*(0.04*0.96+0.044*0.956))/(0.004**2); result=abs(n-39427)<50"}
]