Noesis
Level 4 — ApplicationMLOps & Deployment

MLOps & Deployment

⏱ 60 minutes60 marksprintable — key stays hidden on paper

Level: 4 (Application — novel problems, no hints) Time limit: 60 minutes Total marks: 60

Answer all questions. Show all reasoning and calculations. Where formulas are used, state them.

Question 1 — Serving capacity & cost sizing (14 marks)

A team deploys a fraud-scoring model as a real-time REST service (FastAPI + Triton). Measured single-replica performance under load:

  • p50 latency = 18 ms, p99 latency = 45 ms
  • Sustained throughput per replica = 220 requests/sec (RPS) before latency degrades
  • Traffic follows a daily pattern; peak = 1,500 RPS, average = 400 RPS
  • SLO: p99 ≤ 60 ms and no request dropped at peak
  • Each replica runs on a GPU node costing $1.80/hour (billed per running replica-hour)

Autoscaling target utilization is 70% of a replica's max throughput.

(a) Compute the minimum number of replicas needed to serve peak traffic within the utilization target. (3) (b) Compute the number of replicas at average traffic under the same target. (2) (c) The team considers a batch alternative for a non-latency-sensitive downstream reporting job that reuses the same model on 40 million records nightly. If a batched offline job processes 9,000 records/sec/replica, and the maintenance window is 4 hours, how many replicas are required? (3) (d) Compare the monthly GPU cost of a fixed fleet sized for peak (24/7) versus an autoscaled fleet that runs at the average-traffic replica count for 20 h/day and peak count for 4 h/day (30-day month). Show both totals. (4) (e) State two concrete architectural changes (other than adding replicas) that would reduce p99 latency, and briefly justify each. (2)

Question 2 — Data drift detection design (12 marks)

You own a deployed churn model. Over a week you observe: accuracy on labelled feedback dropped from 0.88 to 0.79, while the input feature distributions look visually similar.

(a) Distinguish data drift from concept drift, and state which one is most consistent with the observation above. Justify. (4) (b) For a single numeric feature, you compare the training reference distribution to the live window using the Population Stability Index (PSI): PSI=∑i(pi−qi)ln⁡piqi\text{PSI} = \sum_i (p_i - q_i)\ln\frac{p_i}{q_i} Given the 4-bin table below (reference pip_i, live qiq_i), compute PSI and state whether it crosses the common "significant drift" threshold of 0.25. (5)

Bin pip_i qiq_i
1 0.40 0.25
2 0.30 0.30
3 0.20 0.25
4 0.10 0.20

(c) Give one reason why relying only on input-feature drift metrics can fail to catch the degradation described in the intro. (3)

Question 3 — CI/CD & retraining pipeline (12 marks)

Design a retraining + deployment pipeline for a model whose performance decays over time.

(a) List, in order, the stages of an automated CI/CD pipeline for ML that takes a code/data change through to production. Give at least 6 distinct stages. (6) (b) Explain the role of a model registry and data versioning (DVC) in making a retraining run reproducible. (3) (c) Your retraining is triggered when validation accuracy on a rolling window drops below 0.82. Given the daily accuracy series [0.87, 0.86, 0.84, 0.83, 0.81, 0.80] and a trigger rule "retrain if the 3-day moving average falls below 0.82", determine the first day the trigger fires. Show the moving averages. (3)

Question 4 — Deployment target decision & quantization (12 marks)

A vision model (FP32, 96 MB, 24 GFLOPs/inference) must run on a battery-powered edge device with a 512 MB memory budget and an NPU rated at 40 GFLOPs/sec effective.

(a) Compute theoretical inference latency at FP32 on the NPU. (2) (b) The team exports to ONNX and applies INT8 quantization, reducing model size 4× and giving a 3.2× effective throughput speedup on the NPU (INT8 path). Compute new model size and new latency. (3) (c) INT8 quantization drops top-1 accuracy from 91.4% to 89.9%. State the accuracy loss and give one technique to recover accuracy while keeping INT8 inference. (3) (d) Contrast edge deployment with centralized cloud serving for this use case on two axes: latency and privacy. (2) (e) Why is ONNX (rather than the native training framework format) commonly used as the export target for cross-platform edge deployment? (2)

Question 5 — A/B test for models (10 marks)

You run an A/B test comparing model B (new) against model A (current) on conversion rate.

  • Group A: 12,000 users, 960 conversions
  • Group B: 12,000 users, 1,080 conversions

(a) Compute the conversion rate for each group and the absolute + relative lift of B over A. (4) (b) Compute the pooled standard error and the two-proportion z-statistic. State whether the result is significant at the 5% level (critical |z| = 1.96). (4) (c) State one reason why "ship B because its offline validation AUC was higher" is not a substitute for this online A/B test. (2)

Answer keyMark scheme & solutions

Question 1 (14)

(a) Effective capacity/replica = 70% × 220 = 154 RPS. Replicas = ⌈1500 / 154⌉ = ⌈9.74⌉ = 10 replicas. (formula 1, arithmetic 1, ceiling 1)

(b) ⌈400 / 154⌉ = ⌈2.60⌉ = 3 replicas. (calc 1, ceiling 1)

(c) Records/replica in window = 9000 × 3600 × 4 = 129,600,000. Replicas = ⌈40,000,000 / 129,600,000⌉ = ⌈0.309⌉ = 1 replica. (window throughput 1, division 1, ceiling 1)

(d) Cost/replica-hour = $1.80.

  • Fixed peak fleet: 10 replicas × 24 h × 30 × 1.80 = $12,960/month.
  • Autoscaled: (3 replicas × 20 h + 10 replicas × 4 h) × 30 × 1.80 = (60 + 40) × 30 × 1.80 = 100 × 54 = $5,400/month. Savings ≈ $7,560. (fixed 2, autoscaled 2)

(e) Any two, 1 mark each: dynamic/adaptive request batching on Triton (amortizes per-call overhead, raises GPU utilization); model quantization/INT8 (faster kernels); caching frequent/duplicate scores; smaller/distilled model; moving preprocessing off the critical path / async I/O.

Question 2 (12)

(a) (4) Data drift = change in input distribution P(X)P(X) while the input→output relationship may stay fixed. Concept drift = change in P(Y∣X)P(Y\mid X), i.e., the relationship between features and target changes. Here inputs look unchanged but accuracy fell → most consistent with concept drift (2 for definitions, 2 for correct choice + justification).

(b) (5) Compute each term (pi−qi)ln⁡(pi/qi)(p_i-q_i)\ln(p_i/q_i):

  • Bin1: (0.15)·ln(0.40/0.25)=0.15·0.470004=0.070501
  • Bin2: (0)·… = 0
  • Bin3: (−0.05)·ln(0.20/0.25)=(−0.05)(−0.223144)=0.011157
  • Bin4: (−0.10)·ln(0.10/0.20)=(−0.10)(−0.693147)=0.069315

PSI = 0.070501 + 0 + 0.011157 + 0.069315 = 0.15097 ≈ 0.151. 0.151 < 0.25 → does not cross the significant-drift threshold (per-term work 3, sum 1, conclusion 1).

(c) (3) Input drift metrics only watch P(X)P(X); concept drift changes P(Y∣X)P(Y\mid X) and can leave feature distributions unchanged, so no input-drift alarm fires despite falling accuracy. You must monitor outcome/label-based performance metrics (with feedback) too.

Question 3 (12)

(a) (6, 1 each for ≥6 valid ordered stages) Example:

  1. Code/data commit triggers pipeline (version control + DVC pointer).
  2. Lint / unit tests / data validation (schema checks).
  3. Build training environment / container.
  4. Automated training on versioned data.
  5. Model evaluation against metrics + baseline gate.
  6. Register model in registry (staging).
  7. Integration/serving tests (load a container, hit endpoint).
  8. Deploy to staging → canary/A-B → production (promote in registry).

(b) (3) Model registry stores versioned model artifacts with lineage (which code, data, params, metrics produced them) and manages stage transitions (staging→prod), enabling rollback and audit. DVC versions the datasets and pipeline files so a training run can be re-executed on the exact same data snapshot. Together: same code + same data version + logged params ⇒ reproducible run.

(c) (3) 3-day moving averages:

  • Day3: (0.87+0.86+0.84)/3 = 0.8567
  • Day4: (0.86+0.84+0.83)/3 = 0.8433
  • Day5: (0.84+0.83+0.81)/3 = 0.8267
  • Day6: (0.83+0.81+0.80)/3 = 0.8133 ← first < 0.82. Trigger first fires on Day 6. (averages 2, answer 1)

Question 4 (12)

(a) Latency = 24 GFLOP / 40 GFLOP/s = 0.6 s (600 ms). (2)

(b) Size = 96 / 4 = 24 MB (within 512 MB budget). Latency = 600 ms / 3.2 = 187.5 ms. (size 1, latency 2)

(c) Accuracy loss = 91.4% − 89.9% = 1.5 percentage points. Recovery technique: quantization-aware training (QAT) (or per-channel calibration / fine-tuning with calibration data) — trains the model to be robust to INT8 rounding, recovering accuracy while keeping INT8 inference. (loss 1, technique + why 2)

(d) (2) Latency: edge avoids network round-trip → lower, deterministic latency; cloud adds RTT. Privacy: edge keeps raw data on-device (no upload), better privacy; cloud sends data off-device.

(e) (2) ONNX is a framework-agnostic interchange format supported by many runtimes/hardware backends (ONNX Runtime, TensorRT, mobile NPUs), so one exported model runs across heterogeneous edge targets without shipping the full training framework.

Question 5 (10)

(a) Rate A = 960/12000 = 0.08 (8%); Rate B = 1080/12000 = 0.09 (9%). Absolute lift = 0.01 (1 pp). Relative lift = 0.01/0.08 = 12.5%. (rates 2, lifts 2)

(b) Pooled p = (960+1080)/24000 = 2040/24000 = 0.085. SE = √[ p(1−p)(1/nA + 1/nB) ] = √[0.085·0.915·(1/12000+1/12000)] = √[0.077775 · 0.000166667] = √(1.29625e-5) = 0.0036003. z = (0.09−0.08)/0.0036003 = 2.7776 ≈ 2.78. |z| = 2.78 > 1.96 → statistically significant at 5%. (pooled p 1, SE 1, z 1, conclusion 1)

(c) (2) Offline AUC measures ranking on historical data and can't capture online effects (real traffic distribution, latency-induced timeouts, UI/feedback loops, business metric vs proxy metric mismatch). Only a live A/B test measures the actual causal impact on the target metric (conversions).

[
  {"claim":"Q1a peak replicas = 10","code":"import math; result = math.ceil(1500/(0.70*220))==10"},
  {"claim":"Q1d fixed=12960 and autoscaled=5400","code":"fixed=10*24*30*1.80; auto=(3*20+10*4)*30*1.80; result = (fixed==12960) and (auto==5400)"},
  {"claim":"Q2b PSI approx 0.151 and below 0.25","code":"import sympy as sp; p=[0.40,0.30,0.20,0.10]; q=[0.25,0.30,0.25,0.20]; psi=sum((a-b)*sp.log(a/b) for a,b in zip(p,q)); val=float(psi); result = abs(val-0.15097)<1e-3 and val<0.25"},
  {"claim":"Q3c Day6 first MA below 0.82","code":"s=[0.87,0.86,0.84,0.83,0.81,0.80]; ma=[round(sum(s[i-2:i+1])/3,4) for i in range(2,len(s))]; first=next(i+3 for i,v in enumerate(ma) if v<0.82); result = first==6"},
  {"claim":"Q4b INT8 latency=187.5ms","code":"result = abs((600/3.2)-187.5)<1e-9"},
  {"claim":"Q5b z-stat approx 2.78 and significant","code":"import sympy as sp; p=2040/24000; se=sp.sqrt(p*(1-p)*(1/12000+1/12000)); z=(0.09-0.08)/se; zf=float(z); result = abs(zf-2.7776)<1e-2 and zf>1.96"}
]