CI - CD pipelines for ML — AI-ML notes

HOW the pipeline flows

The stages, in order:

Trigger — a git push, a schedule (cron), or a drift alarm.
CI: validate — lint code, run unit tests, validate data schema (are columns/types/ranges sane?), run a tiny training run to catch crashes.
Train — full training on the pipeline (not a laptop), producing a model artifact.
Evaluate (gate) — compute metrics on a held-out set; compare to the current production model.
Register — if it passes, push the artifact + metadata to a model registry (versioned).
CD: deploy — package (Docker), roll out to staging, run integration tests, then production.
Monitor — watch live metrics; drift/degradation feeds back to the Trigger → loop closes.

Formula The promotion gate (WHY a formula, not a vibe)

We must automate the yes/no decision to deploy. Define the new model's score $s_{\text{new}}$ and the incumbent's $s_{\text{prod}}$ on the same held-out set. Promote only if it is strictly, meaningfully better: $\text{promote} \iff s_{\text{new}} \ge s_{\text{prod}} + \delta$

Why $\delta$ (a margin)? Metrics have noise. Without a margin you'd churn models on random fluctuations. $\delta$ is your "is this real?" threshold.
Why the same test set? So the comparison is apples-to-apples — otherwise you'd compare skill differences to dataset differences.

Derivation of a safe $\delta$ from noise: if the metric is a proportion (accuracy) over $n$ test points, its standard error is $SE = \sqrt{\frac{p(1-p)}{n}}.$ To be ~95% sure the gain is real, require the gain to exceed roughly $2\,SE$ : $\delta \gtrsim 2\sqrt{\frac{p(1-p)}{n}}.$ How to read it: more test data ( $n\uparrow$ ) → smaller needed margin → you can trust smaller improvements.

Worked examples

Common mistakes

Recall Feynman: explain to a 12-year-old

Imagine a toy factory with a conveyor belt. Every new toy design (code), new batch of plastic (data), and finished toy (model) rides the belt. At each checkpoint a robot inspector checks: Is the design safe? Is the plastic clean? Does the toy actually work better than the one in the shops? Only toys that pass every checkpoint get boxed and sent to stores. And a special sensor watches the shops — if kids stop liking the toy, it rings a bell that starts building a new one automatically. The belt + inspectors + bell together are the ML CI/CD pipeline.

Active recall

What three things change independently in an ML system (unlike normal software)?

Code, data, and model.

What does CT (Continuous Training) add beyond CI/CD?

Automatic retraining triggered by new data or performance drift, to fight model decay.

Why must the promotion gate use the same held-out test set for old and new models?

So the score difference reflects model skill, not differences in the evaluation data.

Why require a margin

\delta

instead of promoting on any improvement?

Metrics are noisy; a small gain can be random luck.

\delta

ensures the improvement exceeds the noise floor.

Give the rule-of-thumb formula for a safe promotion margin on accuracy.

\delta \gtrsim 2\sqrt{p(1-p)/n}

(≈ two standard errors).

What is training/serving skew and how does CI/CD prevent it?

When training and production environments differ, giving different behaviour; prevented by training inside the same containerized pipeline used in prod.

Name the six ordered pipeline stages.

Validate, Train, Evaluate, Register, Deploy, Monitor.

What is a model registry?

A versioned store of trained model artifacts plus metadata (metrics, data version, lineage) used for promotion and rollback.

Which metric commonly triggers a CT retrain, and a rough threshold?

Population Stability Index (PSI); PSI > 0.2 signals significant drift.

Why isn't "all tests pass" enough to ship an ML model?

Tests catch code bugs but not statistical failure; you also need a metric-based evaluation gate.

Connections

Model Registry — where the Register stage stores versioned artifacts.
Data Versioning (DVC) — versioning the "data" leg of the triangle.
Data & Concept Drift — what triggers CT.
Model Monitoring — the feedback loop closing the pipeline.
Docker & Containerization — packaging to avoid training/serving skew.
A-B Testing & Canary Deployment — safer rollout strategies at the Deploy stage.
Statistical Significance — justifies the promotion margin $\delta$ .

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, normal software me sirf code change hota hai. Par ML me teen cheezein alag-alag change hoti hain: code, data, aur model. Isliye ML ka CI/CD ek automatic assembly line jaisa hai — jaise hi koi change aaye (naya git push, fresh data, ya drift alarm), pipeline usko test karti hui production tak le jaati hai. Har stage pe ek "gate" hota hai jahan check hota hai — code sahi hai? data ka schema theek hai? model production wale se behtar hai? Agar koi bhi check fail, toh pipeline waheen ruk jaati hai, aur ganda model users tak nahi pahunchta.

Sabse important ML-specific cheez hai CT (Continuous Training). Kyunki model apne aap "sadta" (decay) hai — duniya badalti hai, users ka behaviour badalta hai, isliye purana model galat hone lagta hai. Isko drift kehte hain (PSI > 0.2 ek common signal hai). Jab drift detect hota hai, CT trigger hota hai aur model automatically retrain hota hai. Yeh feedback loop hi pipeline ko "zinda" rakhti hai.

Ek key idea yaad rakho: model deploy tabhi karo jab naya model production wale se sirf thoda nahi, balki noise se zyada behtar ho. Formula: $\delta \gtrsim 2\sqrt{p(1-p)/n}$ . Iska matlab — agar test data bada hai ( $n$ zyada), toh chhote improvement pe bhi bharosa kar sakte ho; agar test data chhota hai, toh bada improvement chahiye. Isse random luck wale "improvement" pe model baar-baar change nahi hota.

Aur ek galti se bacho: "tests pass ho gaye matlab model achha hai" — yeh galat hai. Tests sirf code bugs pakadte hain, statistical failure (accurate code par bekaar model) nahi. Isliye ek alag evaluation gate zaroori hai. Saath hi, model laptop pe nahi, wahi Docker container ke andar train karo jo production me chalega — warna training/serving skew ho jayega.

CI - CD pipelines for ML

WHAT are the pieces?

HOW the pipeline flows

Worked examples

Common mistakes

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections