Exercises — CI - CD pipelines for ML

This is CT (Continuous Training). It fights model decay from data/concept drift — the world moves even when the code doesn't. See Data & Concept Drift.

Validate → Train → Evaluate → Register → Deploy → Monitor. Mnemonic: "Very Tired Engineers Rarely Deploy Monsters."

Data, model, and environment. Same code + different data ⇒ different model. Version data with Data Versioning (DVC), models in a Model Registry, and pin the environment via a Docker image digest.

Step 1 (WHAT). Plug into $SE=\sqrt{p(1-p)/n}$. $SE=\sqrt{\frac{0.8\cdot 0.2}{1600}}=\sqrt{\frac{0.16}{1600}}=\sqrt{0.0001}=\mathrm{0.01.}$ WHY square-root the fraction? Variance of a proportion is $p(1-p)/n$; the spread (in the same units as accuracy) is its square root, so we take $\sqrt{\cdot }$ to get back to "accuracy points." Step 2 (WHY). $SE=0.01$ says "a random swing of $\pm 1$ accuracy point is normal here." Step 3. Margin $\delta =2SE=0.02$. WHY multiply by 2? Two standard errors ≈ 95% confidence the gain isn't luck (see the boxed rule). A new model must reach at least $0.80+0.02=0.82$ to be promoted.

Step 1 (WHAT). Compute the gain: $s_{\text{new}}-s_{\text{prod}}=0.815-0.800=0.015$. WHY subtract? The gate only cares about the difference $s_{\text{new}}-s_{\text{prod}}$, because a raw high score means nothing unless it beats the incumbent. Step 2 (WHY compare to $\delta$). The required margin is $\delta =0.02$ — the size below which a difference is indistinguishable from random wobble. Step 3. Since $0.015<0.02$, the gain sits inside the noise floor ⇒ Do NOT promote. It could easily be luck.

Step 1 (WHAT). Recompute $SE$ with the bigger $n$: $SE=\sqrt{\frac{0.16}{6400}}=\sqrt{0.000025}=0.005,\delta =2\cdot 0.005=\mathrm{0.01.}$ WHY does $SE$ shrink? $n$ sits in the denominator under the square root, so $SE\propto 1/\sqrt{n}$. Quadrupling $n$ (from $1600$ to $6400$) multiplies $SE$ by $1/\sqrt{4}=1/2$ — it halves. Step 2 (WHY it matters). The gain is unchanged at $0.015$, but the bar $\delta$ dropped to $0.01$. Step 3. $0.015\ge 0.01$ ⇒ Promote. Lesson (WHAT IT MEANS): the same improvement that was untrustworthy at $n=1600$ becomes trustworthy at $n=6400$, purely because more test data shrank the noise floor. Figure s01 below makes this visible.

Step 1 (WHAT). Compare each reading to the threshold $0.2$. WHY a threshold at all? Some drift is always present (noise). PSI turns "the world moved" into a single number so we can set an objective trigger instead of eyeballing it. Step 2. $0.05,0.12,0.18$ are all $\le 0.2$ (no trigger); $0.24>0.2$ on day 4. Step 3. Day 4 fires the retrain. This kicks off drift-driven retraining, feeding the new candidate into the promotion gate.

The difference in scores now mixes two effects: (1) genuine model-skill difference, and (2) difference in dataset difficulty. You cannot tell them apart. The new "win" might just be an easier test set. Fix: score both models on one identical held-out set so only skill varies. This is the apples-to-apples rule behind the promotion gate.

The missing gate is the model-evaluation gate that compares the candidate's score $s_{\text{new}}$ against the incumbent's score $s_{\text{prod}}$ on a held-out set (both defined in the toolkit above). Unit tests check code correctness; schema validation checks data sanity; the smoke run only checks "does training crash?" None of them measure predictive quality. Statistical failure is invisible to all three. See Model Monitoring for the live counterpart.

Training/serving skew. The laptop and production environments differ — library versions, preprocessing defaults, hardware numerics — so the same input is transformed differently at serve time. Fix: train inside the same containerized pipeline (pinned image digest) that serves in prod, eliminating the environment gap.

Step 1 (WHAT). Compute the noise floor once (it depends only on $p,n$, shared by both): $SE=\sqrt{\frac{0.9\cdot 0.1}{2500}}=\sqrt{0.000036}=0.006,\delta =2SE=\mathrm{0.012.}$ WHY one $\delta$ for both? They share the same $p$ and the same test set, so they face the same bar. Step 2 (WHY compare gains). With $s_{\text{prod}}=p=0.90$: X's gain $=0.905-0.900=0.005$; Y's gain $=0.913-0.900=0.013$. Step 3.

• X: $0.005<0.012$ → reject (inside noise).
• Y: $0.013\ge 0.012$ → promote. X isn't "bad code"; its measured edge is simply statistically indistinguishable from noise. Rejecting it prevents model thrash. This is exactly a significance judgement.

1. Trigger: Model Monitoring reports $PSI>0.2$ → drift alarm.
2. Validate: lint, unit tests, data-schema check, tiny smoke train. (Gate 1 — stops on bad code/data.)
3. Train: full run inside the pipeline container → model artifact.
4. Evaluate: score the candidate $s_{\text{new}}$ against the incumbent $s_{\text{prod}}$ on the same held-out set; require $s_{\text{new}}\ge s_{\text{prod}}+\delta$. (Gate 2 — the promotion gate.)
5. Register: push artifact + metadata to the Model Registry (versioned, so rollback is possible).
6. Deploy: package with Docker, roll to staging → integration tests (Gate 3) → production via canary.
7. Monitor: watch live metrics; feeds back to step 1. Loop closed.

Step 1 (WHAT). Treat the canary window as a test set of $n=10000$ points at $p=0.90$: $SE=\sqrt{\frac{0.9\cdot 0.1}{10000}}=\sqrt{0.000009}=0.003,\delta =2SE=\mathrm{0.006.}$ WHY use the same formula as offline? A canary is just an online held-out set; the noise of a proportion obeys the identical $\sqrt{p(1-p)/n}$ law. Step 2 (WHY it constrains duration). The canary can trustworthily detect improvements of $\ge 0.6$ accuracy points only. If your expected gain is smaller than that, run the canary longer to grow $n$ and shrink $SE$. Step 3. This is the A/B/significance logic applied to rollout: pick $n$ so $2SE$ is below the effect you care about.

The remaining suspect is the data (the third leg of the code–data–model triangle). Same code + same environment + different data snapshot ⇒ different model. Pin the exact dataset version with Data Versioning (DVC) and record its hash in the Model Registry alongside the artifact. Reproducibility needs all four: code, data, model, environment.

(a) WHAT/WHY. "95% sure" is exactly the $2SE$ rule, so: $SE=\sqrt{0.85\cdot 0.15/900}=\sqrt{0.0001417}=0.0119,\delta =2\cdot 0.0119=0.0238\approx \mathrm{0.024.}$ (b) Gain $=0.878-0.850=0.028\ge 0.024$ ⇒ Promote. It clears the noise floor. (c) WHY solve an inequality. We want the smallest $n$ making the floor no bigger than the target gain $0.015$, i.e. $2\sqrt{p(1-p)/n}\le 0.015$. Solve step by step: $\sqrt{\frac{0.1275}{n}}\le 0.0075\Rightarrow \frac{0.1275}{n}\le 0.00005625\Rightarrow n\ge \frac{0.1275}{0.00005625}\approx 2267.$ WHY square both sides? To remove the square root; the inequality direction is preserved because both sides are positive. So a test set of about $n\ge 2267$ lets a $+1.5$-point gain clear the gate. Figure s02 shows the curve crossing the target line right there.

Root cause: the promotion gate has no (or too small) margin $\delta$, so it deploys on random $\pm 1$-point noise — a classic "deploy whenever new beats old" bug. Fix 1: enforce $\delta \gtrsim 2SE$ so only real gains pass. Fix 2: add hysteresis / a fixed evaluation cadence and a significance test (e.g. require the gain to hold over multiple evaluations before rollout), plus canary confirmation on live traffic.

1. Does the code pass? → unit tests / lint (CI).
2. Is the data sane? → schema validation (DVC-pinned snapshot).
3. Did training complete in-pipeline? → containerized run (Docker digest, avoids skew).
4. Is it meaningfully better? → $s_{\text{new}}\ge s_{\text{prod}}+\delta$ on the same set (Statistical Significance).
5. Is it stored & rollback-able? → Model Registry (versioned artifact + lineage).
6. Is it safe on live traffic? → canary window sized for significance.
7. Does it stay good? → Model Monitoring + drift watch → loops back to trigger.

Every gate on this page reduces to one habit: compare a change against its own noise floor before letting it move down the belt.