5.3.1 · D5MLOps & Deployment
Question bank — ML project lifecycle
Recall
The four drift/quality terms these traps orbit around, so you never use a word before you own it:
- Data drift ::: the input distribution changes over time (new kinds of inputs arrive), while the "correct answer given the input" rule stays the same.
- Concept drift ::: the mapping changes — the same input now has a different correct label than before.
- Human-level performance (HLP) ::: how well a skilled human does the task. We use it as a practical, empirical proxy for the theoretical best-achievable error.
- Baseline ::: a simple reference score (human, heuristic, or trivial model) that tells you if your fancy model is actually adding value and how much room is left.
Two pictures carry the rest of this page — the drift distinction and the lifecycle loop:
True or false — justify
Every line is a claim. State true or false AND why before revealing.
Training error can never drop below human-level performance.
False. HLP is a floor on the best generalization (dev/test) error, not on training error — an overfit model memorizes noise and can post training error below HLP. See Overfitting and generalization gap.
A model with 99% test accuracy is automatically the best choice to ship.
False. The north-star is the business/decision metric under constraints (latency, cost, fairness); a fast 96% model can beat a slow 99% one. Accuracy is a proxy, not the goal.
Once a model is deployed and bug-free, its accuracy stays constant.
False. Even with perfect code, the live distribution drifts away from training — Data drift and concept drift silently rot a frozen model.
The ML lifecycle is a straight line from scoping to monitoring.
False. It is a loop with feedback arrows from every later stage back to earlier ones — monitoring routinely sends you back to Data or Modeling (see the loop figure above).
If avoidable bias is large, collecting more data is the top priority.
False. Large avoidable bias () points at the model/features, not data quantity; more data mainly attacks variance. See Bias-variance decomposition.
Scoping matters less than modeling because that's where the "real ML" happens.
False. Most project failures are scoping failures (wrong metric/problem), not modeling failures — scoping is the highest-leverage 20%.
Data drift and concept drift are two names for the same thing.
False. Data drift changes (new inputs), concept drift changes (same input, new correct answer); they demand different fixes.
More training data always improves generalization.
False. More noisy or mislabelled data can hurt; label consistency often beats raw quantity — this is the data-centric insight.
A shadow deployment can change user-facing outputs.
False. In a shadow deployment the model runs on live traffic but its output is never used — you only compare silently.
HLP and Bayes error are always exactly equal.
False. HLP is an empirical proxy; on noisy tasks humans make avoidable errors, so true Bayes error can sit below HLP. Equating them is a convenient approximation, not a law.
Spot the error
Each line contains a flawed statement; the reveal names the flaw and corrects it.
"Avoidable bias came out to , so my model is doing better than humans."
Error: negative avoidable bias means , which is an overfitting alarm, not superhuman skill — the real signal now lives in the variance/generalization gap.
"Live accuracy dropped, so I'll retrain on the old data to fix it."
Error: if the drop is from new, unseen inputs ( changed), retraining on old data can't help — you must collect new data. This is data drift, fixed back in Stage 2.
"We hit our accuracy target, so scoping was successful."
Error: accuracy is a modeling metric; scoping success is measured by whether the chosen metric maps to real business value under constraints.
"Bayes error is a strict lower bound on training error."
Error: Bayes/HLP lower-bounds the best generalization error; training error can dive below it via memorization.
"Canary and blue-green are the same because both reduce risk."
Error: canary routes a small percentage of traffic and ramps up gradually; blue-green keeps two full environments and switches all traffic instantly with instant rollback. Different mechanisms. See Deployment strategies (shadow, canary, blue-green).
"Variance = ."
Error: variance is (the generalization gap); lumps bias and variance together.
"Error analysis means computing the overall test accuracy."
Error: Error analysis means categorizing mispredictions into buckets and attacking the biggest bucket first (80/20) — a single aggregate number tells you nothing about where to fix.
"Full autonomy is the safe default for any new deployment."
Error: you climb the ladder (Human → Shadow → AI assist → Partial → Full autonomy) precisely to shrink blast radius — jumping straight to full autonomy maximizes risk.
Why questions
Why do we still need a baseline if we already know our model's accuracy?
Because accuracy alone can't tell you if there's room to improve — a baseline (HLP/heuristic) reveals how much of the remaining error is even reducible.
Why is the expected-value formula () useful in scoping if the numbers are just guesses?
It forces you to price feasibility and maintenance together, exposing projects that look attractive on payoff but are killed by ongoing cost — a rough number beats no number.
Why can overfitting make "avoidable bias" meaningless?
Overfitting drives below HLP, so avoidable bias () goes negative; a negative "avoidable" quantity has no actionable meaning and signals you should attack variance instead.
Why does modern practice lean data-centric rather than model-centric?
For a fixed capable model, cleaner and more consistent data usually generalizes better than yet another architecture tweak — the marginal return often lives in the data. See Data-centric vs model-centric AI.
Why does monitoring belong inside the lifecycle rather than after it?
Because monitoring is what closes the loop — it detects drift and triggers the feedback arrow back to Data/Modeling, turning a one-shot pipeline into a living cycle.
Why compare to HLP instead of to zero?
Because zero error is often impossible (irreducible Bayes error); HLP is the realistic floor, so the gap to HLP is what's actually worth chasing.
Why is a slow, ultra-accurate model sometimes a worse ship than a fast, less-accurate one?
If the satisficing constraint (e.g. latency ms) is violated, the accurate model can't be used at all in the decision it was built to drive — the constraint gates the metric.
Edge cases
If , what should you conclude?
You've hit the achievable floor with no variance gap — further gains require lowering HLP itself (better data/features/labels), not more of the same.
— is that possible, and what does it hint?
Yes: e.g. but a tiny 20-example dev set happens to contain only easy cases and scores , making variance . A negative variance gap is a data-split red flag (unrepresentative or leaked dev set), not a triumph — re-sample dev before trusting the number.
What if in the expected-value formula?
Then ; the project only ever loses money — a clear "do not build" even before touching data.
Live inputs are identical to training but labels users assign have shifted — which drift?
Concept drift: is unchanged but changed, so the same input now maps to a different correct answer.
A canary at 5% traffic shows no metric change after a week — safe to ramp to 100%?
Cautiously yes, but confirm the 5% is representative (not just one region/segment); a canary that never saw the risky traffic slice proves nothing about it.
The model's training and live distributions match perfectly, yet accuracy still falls. What's left?
Suspect a pipeline/serving bug (feature skew, versioning, preprocessing mismatch) rather than drift — observability on features, not just outputs, catches this.
Avoidable bias is and variance is — which lever first?
Neither dominates, so pick by cheapness/feasibility: try a quick regularization or data-cleaning pass (variance) and a feature/model change (bias) and re-measure — a tie means experiment, don't guess.