CI - CD pipelines for ML: Worked examples — AI-ML deep dive

The scenario matrix

Every decision this topic throws at you falls into one of these cells. Each row is a distinct kind of situation, with a distinct "sign" or "degenerate" twist.

Cell	The question being asked	The twist / edge case
A. Clear promote	new $\geq$ prod $+ δ$ ?	gain comfortably above noise
B. Clear reject	new $\geq$ prod $+ δ$ ?	new model worse than prod (negative gain)
C. Borderline	new $\geq$ prod $+ δ$ ?	gain sits right at the noise floor
D. Small test set	how does $n$ change the verdict?	limiting case: tiny $n$ makes $δ$ huge
E. Degenerate metric	$p \to 0$ or $p \to 1$	$S E \to 0$ ; near-perfect model
F. CI validation stop	is the data even valid?	zero / out-of-range input → pipeline halts before training
G. CT drift trigger	should we retrain at all?	PSI above/below threshold
H. Word problem	money cost of a wrong deploy	translate a business story into the gate
I. Exam twist	different metrics / different $n$	comparing two models on unequal test sets — the trap

The worked examples below each carry a [Cell X] tag so you can see the whole matrix get covered.

See

— the "number line" mental model: prod sits at a point, we draw a noise band of width

δ

around it, and a new model only wins if it lands to the right of the band.

Worked examples

Worked example 6 — CI validation stop, degenerate input

[Cell F] The feature pipeline promises age ∈ [0,120] and n_purchases ≥ 0. A bad upstream join injects a row with age = -3 and n_purchases = 0. What does the CI stage do, and does training ever run?

Forecast: one impossible value → the schema check fails → the pipeline halts before burning compute on training.

Range check age. Is $- 3 \in [0, 120]$ ? No. Validation flags a violation. Why this step? CI's data-validation gate exists to catch garbage before it poisons a model.
Range check n_purchases. Is $0 \geq 0$ ? Yes — zero is a valid, non-degenerate count. Why? We must not over-reject: legitimate zeros are fine; only the out-of-range field fails.
Pipeline decision. Any failed check ⇒ STOP. Training does not start; a loud error is raised. Why? A silent bad model costs money for weeks (parent's warning); a loud early stop costs seconds.

Verify: exactly one of the two checks fails ( $a g e$ fails, $n_p u r c ha ses$ passes), and one failure is enough to halt. ✅ This is the classic "CI failure that saves you." See Data Versioning (DVC) for pinning the data that should have arrived.

Worked example 7 — CT drift trigger, above/below threshold

[Cell G] Live monitoring reports the PSI of a key feature over two consecutive weeks: week-1 $P S I = 0.14$ , week-2 $P S I = 0.27$ . Which weeks (if any) fire the retrain trigger? Threshold $0.2$ .

Forecast: 0.14 is "moderate", 0.27 is "significant". Guess: week 2 fires, week 1 doesn't.

Week 1. $0.14 > 0.2$ ? No — moderate drift; log it, watch it, don't retrain yet. Why this step? Retraining on every minor wobble thrashes the system (parent's mistake #2 analogue).
Week 2. $0.27 > 0.2$ ? Yes → fire CT trigger. Why? The world moved enough that the model is likely mis-anchored — see Data & Concept Drift.
Downstream. The CT run produces a candidate model that then must pass the same promotion gate (Cells A–E) before deploy. Why? Drift justifies retraining, but the retrained model still has to earn promotion.

Verify: $0.14 < 0.2$ (no) and $0.27 > 0.2$ (yes) ✅. Exactly one week triggers. Monitoring → trigger → gate is the closed loop from Model Monitoring.

Worked example 8 — Word problem: money cost of a wrong deploy

[Cell H] A fraud model runs on $200, 000$ transactions/day. Prod accuracy $p = 0.95$ , $n = 8000$ held-out. A candidate scores $0.955$ . Each additional correct catch is worth $4; deploying a worse model that drops accuracy costs the same per miss. Should we deploy, and what's the expected daily value if we do?

Forecast: gain is 0.005; with $n = 8000$ noise is small, likely promote; then multiply by traffic.

$S E$ . $0.95 \cdot 0.05/8000 = 0.00000594 = 0.00244$ . Why this step? Establish the noise floor before trusting 0.5 points.
$δ$ . $2 \times 0.00244 = 0.00488$ .
Gate. Gain $0.005 \geq 0.00488$ → PROMOTE. Why? Just clears 2 $S E$ ( $0.005/0.00244 = 2.05 S E$ ) — credible.
Value. Extra correct fraction $= 0.005$ . Daily extra catches $= 0.005 \times 200000 = 1000$ . Value $=1000\times\$ 4=$4000/\text{day}$. Why this step? The business decision is the gate's reason to exist — translate accuracy into dollars.

Verify: $0.005/0.00244 \approx 2.05$ (passes gate) and $0.005 \times 200000 \times 4 = 4000$ ✅. Units: (fraction)×(txns/day)×($/txn) = $/day. Because the gain is barely real, ship it as a canary first (A-B Testing & Canary Deployment) to confirm the $4000 in the wild.

Worked example 9 — Exam twist: the unequal test-set trap

[Cell I] Model X scored $0.88$ on its own test set of $n_{X} = 1000$ . Model Y scored $0.86$ on a different test set of $n_{Y} = 1000$ . A teammate says "X wins, promote X." Where's the trap, and what's the correct procedure?

Forecast: the scores came from different data — the comparison is invalid, so the naive "0.88 > 0.86" verdict is unearned.

Spot the violation. The promotion rule requires the same held-out set for both models. Here $X$ and $Y$ were scored on different sets. Why this step? A 2-point difference could be a dataset difference, not a skill difference (parent's rule: apples-to-apples).
Correct procedure. Re-evaluate both models on one frozen test set, then apply the gate. Why? Only then does $s_{X} - s_{Y}$ measure skill.
Suppose the re-eval gives $X = 0.87$ , $Y = 0.86$ on shared $n = 1000$ . $S E = 0.87 \cdot 0.13/1000 = 0.0001131 = 0.01064$ ; $δ = 0.0213$ . Gain $= 0.01 \geq 0.0213$ ? No → do NOT promote X over Y. Why? On equal footing the "win" evaporates into the noise band.

Verify: naive gain $0.88 - 0.86 = 0.02$ but on unequal sets → invalid; fair gain $0.87 - 0.86 = 0.01 < δ = 0.0213$ → reject ✅. The trap: comparing skill to dataset luck. Winner is only chosen after a shared eval — store both in the Model Registry and re-score.

Recall Which cell was which?

A clear promote (big n) ::: Example 1 Clear reject / negative gain ::: Example 2 Borderline exactly at 2 SE ::: Example 3 Limiting: tiny test set flips the verdict ::: Example 4 Degenerate metric p → 1 ::: Example 5 CI data-validation stop with a valid zero ::: Example 6 CT trigger above vs below PSI threshold ::: Example 7 Word problem: dollars per day ::: Example 8 Exam trap: unequal test sets ::: Example 9

Connections

CI - CD pipelines for ML — the parent; this page drills its promotion gate.
Model Registry — store every candidate here before the shared re-eval (Ex. 9).
Data Versioning (DVC) — pin the data that should have arrived (Ex. 6).
Data & Concept Drift — the world-moves cause behind CT triggers (Ex. 7).
Model Monitoring — closes the loop: monitor → trigger → gate.
A-B Testing & Canary Deployment — safety net for "barely promote" cases (Ex. 3, 8).
Statistical Significance — the exact-test upgrade for degenerate $p$ (Ex. 5).
Docker & Containerization — the environment leg that prevents training/serving skew.