Noesis
5.3.11 · D5MLOps & Deployment

Question bank — CI - CD pipelines for ML

1,435 words7 min read↑ Back to topic

True or false — justify

A green CI test run means the model is safe to ship.
False. CI catches code bugs, but a model can be numerically fine and still be a bad predictor (statistical failure); you need a separate metric-based evaluation gate comparing to the incumbent.
CT (Continuous Training) is just CI/CD run on a schedule.
Mostly false. A schedule can trigger CT, but the point of CT is fighting model decay from Data & Concept Drift — it exists because the artifact rots even when code is frozen, which scheduling alone doesn't address.
Versioning code in git is enough to reproduce an ML model.
False. Same code + different data or environment gives a different model; reproducibility needs code plus data (Data Versioning (DVC)), model (Model Registry), and pinned environment (Docker & Containerization).
If the new model beats production by any amount, you should promote it.
False. Metric noise means a tiny "win" can be luck; you promote only when the gain clears a margin that exceeds the noise floor, ideally backed by Statistical Significance.
The data-validation stage is optional if your unit tests are thorough.
False. Unit tests check code logic; data validation checks the incoming data (schemas, ranges), which unit tests never see — a bad join producing age = -1 slips straight past logic tests.
Larger test sets let you trust smaller improvements as real.
True. Standard error shrinks as grows, so the required margin shrinks too — more evidence, less noise.
You must compare the new and old model on the same held-out set.
True. Otherwise the score gap mixes model-skill differences with dataset-difficulty differences, and you can't tell which caused the change.
Monitoring is the last stage, so it doesn't affect the pipeline.
False. Monitoring feeds drift/degradation signals back into the Trigger, closing the loop — it's what makes the pipeline continuous rather than one-shot.
A model that passed the gate last month is still valid this month with no re-check.
False. The world moves (concept drift), so a once-valid model can silently degrade; that's precisely why Model Monitoring and CT exist.

Spot the error

"We trained a great model on a laptop, so let's just deploy that exact file."
The error is ignoring training/serving skew: the laptop's library versions and hardware differ from prod, so behaviour can shift. Train inside the same containerized pipeline that runs in production.
"Promotion rule: promote if ."
Missing the margin. It should be with ; strict-but-tiny wins are indistinguishable from noise and cause model thrashing.
"We set once using and reuse it forever."
depends on the current test-set size and base rate ; hard-coding from a tiny sample keeps the margin needlessly huge (or wrong) as your data grows — recompute it from the actual eval set.
"PSI fired at 0.25, so the model is broken — roll back immediately."
PSI measures input/prediction distribution shift, not that the model is broken. High PSI is a retrain trigger (fire CT and re-evaluate), not proof of failure or a rollback command.
"The eval gate uses accuracy on the training data to decide promotion."
Evaluating on training data leaks — the model has seen it, so scores are optimistically inflated. The gate must use a held-out set the model never trained on.
"Register the model only after it's live in production for a week."
Backwards. You register before deploy (Validate → Train → Evaluate → Register → Deploy), so the Model Registry holds the versioned artifact you can promote and roll back to.

Why questions

Why does ML need a third letter (CT) that traditional software CI/CD doesn't?
Because an ML model's world changes even when its code is frozen — user behaviour and data distributions drift — so the artifact itself decays and must be periodically re-grown from fresh data.
Why is the promotion decision written as a formula instead of left to human judgment?
To make the yes/no mechanical, repeatable, and noise-aware; a human eyeballing "looks better" will thrash on random fluctuations and can't be automated in a pipeline.
Why require roughly specifically as the margin?
Two standard errors of a proportion corresponds to about 95% confidence that the observed gain is real rather than sampling noise, tying the margin directly to Statistical Significance.
Why validate the data schema before training rather than just handling errors during training?
Because a training run on garbage data often "succeeds" and ships a subtly broken model; validating up front turns a silent statistical bug into a loud, early, cheap failure.
Why version data and models, not just code, for reproducibility?
The model is a function of code and data and environment; freezing only code lets a different data snapshot silently produce a different model, so all three legs must be pinned.
Why prefer a canary or A/B rollout over flipping 100% of traffic to a new model?
A gate score is measured offline on one held-out set; live behaviour can still surprise you, so A-B Testing & Canary Deployment limits blast radius and validates the model on real traffic before full rollout.

Edge cases

New model and old model score exactly equal on the held-out set.
Do not promote: the gain is , so there's no evidence it's better, and swapping in an untested artifact only adds risk for no reward.
The held-out test set is tiny (e.g. ).
is large, so the required margin becomes huge — you effectively can't trust almost any improvement; the fix is to gather more eval data, not to lower the margin.
Base rate is extreme, e.g. accuracy .
is very small so is tiny and small gains look "significant," yet accuracy is a poor metric near class imbalance — prefer precision/recall/AUC, because a naive majority classifier already scores 0.99.
Data validation passes but the meaning of a feature silently changed (e.g. currency switched to cents).
Range checks may still pass while the model degrades; this is concept drift that schema validation misses, which is why live Model Monitoring and drift metrics are needed downstream of the gate.
The pipeline retrains, but new data is worse (mislabeled batch).
The evaluation gate catches it: the new model fails the test and is rejected, so bad data doesn't auto-promote — the gate is the safety net for CT.
Two retrain triggers fire at once (schedule and drift alarm).
The pipeline should be idempotent/queued so it runs once on the latest data rather than launching duplicate training jobs; both triggers converge to the same Validate→…→Monitor flow.
Monitoring shows metrics fine but PSI high (inputs drifted, outputs still good).
This is covariate shift the model tolerates; you may log it and hold off retraining, since retraining without a metric drop risks trading a working model for noise — high PSI is a warning, not a mandate.

Connections