Intuition The one core ideaAn ML system is not one thing that changes but three — the code , the data , and the model — and each can rot or break on its own. A CI/CD pipeline is an automated conveyor belt with an inspector at every stop, so that no broken code and no statistically bad model ever reaches real users.
Everything in the parent note is built from a small pile of ideas and symbols. Below we lay each one down from absolute zero — plain words first, a picture second, and only then the notation. Nothing is used before it is earned.
Before any symbol, hold this picture in your head: three sliders that move independently .
Definition Code, Data, Model
Code — the instructions you write (the training script, the feature pipeline). This is what normal software people already version with git.Data — the examples the model learns from. New rows arrive; old rows get relabelled; a column changes meaning.Model — the trained artifact : the actual numbers (weights) produced by running the code on the data.
Intuition Why "independently" is the whole pointIn ordinary software only the code moves. In ML you can freeze the code perfectly and still get a worse system tomorrow, because the data the world feeds you drifts. Look at the red slider in the figure: the model slid even though nobody touched code. That single fact is why ML needs an extra letter — CT — that plain software never needed. See Data & Concept Drift .
The parent draws a chain of boxes with arrows. If you have never read one: a box is a step that does work , and an arrow means "when this finishes, hand the result to the next box." It is a recipe read top to bottom.
An ordered sequence of automatic steps where each step's output becomes the next step's input, and any step is allowed to stop the whole chain if it finds a problem. "Automatic" = a machine runs it on a trigger , not a human clicking buttons.
We need this vocabulary because the entire topic is "what happens between the trigger and the user."
Every quality decision in the pipeline rests on measuring a model. You cannot measure it on the data it studied — that is like grading a student on the exact questions they memorised.
Definition Held-out test setA batch of labelled examples the model never saw during training , kept aside only for grading. Its size — the count of examples — gets the symbol n
Intuition Why hold data out?If you test on training data you measure memory , not skill . The held-out set is the "surprise exam." The symbol n bigger n .
We give a model one number summarising how good it is on the held-out set. Call it a score , symbol s
Definition The score symbols
s new candidate model (just trained).s prod currently live in production (the "incumbent").
Read the subscript as a nametag: new = challenger, prod = the one users hit right now.
Higher s is s new s prod
The most common score is accuracy : out of n p proportion ).
p p = n number correct , 0 ≤ p ≤ 1. p = 0.90 100%
p any yes/no proportion — accuracy, click-rate, fraud-catch-rate. Naming it p
Here is the subtle idea the parent leans on hardest. If you tested the same model on a different random batch of n p standard error , S E
Definition Standard error of a proportionS E = n p ( 1 − p ) . typical distance between your measured p
Let us earn every piece of this formula, because it is used as a black box in the parent:
Intuition Why this exact shape?
Why p ( 1 − p ) A proportion is a coin with bias p uncertain near p = 0.5 certain near p = 0 p = 1 p ( 1 − p ) 0.5 Why divide by n Averaging over more examples cancels flukes. More data → steadier average → smaller wobble.Why the square root? Because averaging cancels noise at the rate of n n halve your uncertainty you need four times the data. The square root is where that "diminishing returns" law lives.
Recall Quick sanity check onS E p = 0.9 n = 2500 S E = 0.9 ⋅ 0.1/2500 = 0.000036 = 0.006 0.90 0.90 ± 0.006
Now we can read the parent's promotion rule symbol by symbol.
Intuition Why a margin at all?If the score wobbles by ± S E hair above prod might just be a lucky wobble. We demand the gain clear the noise. The rule-of-thumb:
δ ≳ 2 S E = 2 n p ( 1 − p ) 2 95% real , not luck. (This "2 × Statistical Significance .) The symbol ≳
δ n not demand a bigger margin — it lets you use a smaller one, because S E
A model can pass every gate today and be wrong next month because the incoming data changed shape . That change is drift , and we need one number to measure it so a machine can raise the alarm.
Definition PSI — Population Stability IndexA single number comparing "the data distribution the model was trained on" against "the data flowing in live." Roughly:
P S I < 0.1 0.1 ≤ P S I < 0.2 P S I ≥ 0.2 trigger .
This is the sensor that closes the loop; details live in Data & Concept Drift and Model Monitoring .
Three more nouns the parent uses without defining:
Model artifact — the saved output of training: a file of weights plus a bit of metadata. The thing that gets deployed. Stored & versioned in a Model Registry .Container — a sealed box (via Docker & Containerization ) holding the code and its exact library versions , so it behaves identically on a laptop, in the pipeline, and in production.Canary / A-B rollout — releasing the new model to a small slice of traffic first to watch it under real load before full rollout (A-B Testing & Canary Deployment ).
Intuition Why containers earn their place"It worked on my machine" happens because your machine's library versions differ from production's — training/serving skew . A container freezes those versions into the artifact, so the environment can't drift out from under you.
Read it top-down: the triangle motivates a pipeline ; a trigger starts it; the test set produces scores ; accuracy feeds noise , noise feeds the margin , and score-plus-margin makes the gate ; passing artifacts go to the registry , ship in a container , get monitored , and monitoring's PSI re-triggers the loop.
Cover the right side; can you answer before revealing?
The three things that change independently in ML Code, data, and model.
What a "trigger" is, with three examples The event that starts the pipeline: a git push, a schedule (cron), or a drift alarm.
Why you grade a model on a held-out set and not training data Training data measures memory, not skill; the held-out set is a surprise exam.
What the symbol n The number of examples in the held-out test set.
Meaning of s new s prod Score of the candidate model vs. the current production model.
What p A proportion (e.g. accuracy) = fraction correct, between 0 and 1.
The standard error formula and why p ( 1 − p ) S E = p ( 1 − p ) / n ;
p ( 1 − p ) measures how jumpy a coin of bias
p is (max at 0.5).
Why S E Averaging cancels flukes; uncertainty falls like
1/ n , so 4× data halves it.
The promotion rule in symbols, and what ⟺ promote ⟺ s new ≥ s prod + δ ⟺
Why the margin uses 2 S E A gain beyond ~2 standard errors is ~95% likely to be real, not noise.
What PSI is and its retrain threshold Population Stability Index measuring data-distribution shift; P S I ≥ 0.2
Difference between a model artifact, a registry, and a container Artifact = saved weights file; registry = versioned store of artifacts; container = sealed code+environment for identical runs.
