CI - CD pipelines for ML: Visual walkthrough — AI-ML deep dive

Prerequisite links you may want open while reading: the parent pipeline, Statistical Significance, A-B Testing & Canary Deployment, Model Monitoring.

Step 1 — What is a "score" really?

WHAT we did: defined the number that ranks two models. WHY: the whole gate is a comparison of two scores, so the score must be pinned down first. PICTURE: below, each test example is one tile. Green tiles = correct, amber = wrong. The score is just the green fraction.

Step 2 — The trap: a score is a sample, not the truth

WHAT we did: separated the fixed truth $p$ from the wobbling measurement $s$ . WHY: if we forget $s$ wobbles, we'll mistake a wobble for a real improvement — the exact bug the margin fixes. PICTURE: the same model tested three times on three different held-out sets gives three different scores scattered around the true $p$ .

Step 3 — How big is the wobble? Enter the standard error

Each test example is like flipping a biased coin: heads (correct) with probability $p$ , tails (wrong) with probability $1 - p$ . We flip it $n$ times and take the fraction of heads. Statisticians long ago worked out how much such a fraction wanders.

PICTURE: the wobble-width $S E$ gets narrower as $n$ grows. Watch the bell squeeze inward.

Step 4 — Turning wobble into a decision distance (both models wobble!)

We have two models: incumbent score $s_{prod}$ and challenger score $s_{new}$ . Their gap is $g = s_{new} - s_{prod} .$

The subtle point the naive version misses: both scores wobble, so their difference wobbles more than either one alone. If two independent measurements each have variance $S E^{2}$ , their difference has variance $S E_{new}^{2} + S E_{prod}^{2}$ (variances of independent quantities add). Taking the square root: $S E_{g} = S E_{new}^{2} + S E_{prod}^{2} .$

$S E_{g}$ ::: the wobble of the gap, not of one score.
When the two scores are close (so $S E_{new} \approx S E_{prod} = S E$ ), this becomes $S E_{g} \approx 2 S E$ — the difference is about $2 \approx 1.41$ times noisier than a single score.

PICTURE: the null wobble of the gap is a bell centred at $0$ with width $S E_{g}$ . The amber gain $g$ must land in the far tail.

Step 5 — Naming the margin $δ$ (and how to actually compute it)

Rearrange $g \geq z S E_{g}$ back into the promotion form. Recall $g = s_{new} - s_{prod}$ : $s_{new} - s_{prod} \geq z S E_{g} ⟺ s_{new} \geq s_{prod} + δ z S E_{g} .$

Simplified rule of thumb. When both models sit at roughly the same rate $p$ on the same test set, $S E_{new} \approx S E_{prod} = p (1 - p) / n$ , so $δ \approx z 2 \frac{p ( 1 - p )}{n} \approx 2.8 \frac{p ( 1 - p )}{n} (using z = 2) .$ The parent note's headline $δ ≳ 2 p (1 - p) / n$ is the loose, conservative-in-the-other-direction version — fine as a first cut, but the $2$ factor above is the honest noise of a difference.

WHAT we did: defined $δ$ from the gap's noise and showed how to fill in $p$ from data. WHY: only now is every symbol in the boxed gate grounded in a picture and a computation. PICTURE: the incumbent score is a line; $δ$ is a shaded amber "safety cushion" stacked on top; only scores above the cushion promote.

Step 6 — Reading the knobs: what makes $δ$ big or small?

PICTURE: a curve of $δ$ against $n$ (falling) and a second against $p$ (a dome peaking at $0.5$ ).

Step 7 — Edge & degenerate cases (never leave a gap)

PICTURE: four mini-panels, one per case, each showing what the gate does.

Step 8 — Plugging in the worked numbers

Parent Example 1: $p = 0.90$ , $n = 2500$ , challenger $s_{new} = 0.912$ . First check the CLT condition: $n p = 2250 \geq 10$ and $n (1 - p) = 250 \geq 10$ ✅ — the bell-curve ruler is valid.

Single-score noise (plug in $\overset{p}{^} = s_{prod} = 0.9$ ): $S E = \frac{0.9 \times 0.1}{2500} = 0.000036 = 0.006.$

Parent's loose rule ( $δ = 2 S E$ ): $δ_{loose} = 2 \times 0.006 = 0.012, g = 0.912 - 0.900 = 0.012 \geq 0.012 \Rightarrow promote (just barely) .$

Honest difference-noise rule ( $δ = 22 S E$ , since both models wobble on the same set, treated as independent for safety): $δ_{diff} = 22 \times 0.006 \approx 0.017, g = 0.012 < 0.017 \Rightarrow do NOT promote yet .$

$0.006$ ::: one single-score noise-width on 2500 points.
$0.012$ vs $0.017$ ::: the parent's optimistic cushion vs the correct difference-aware cushion.
Lesson: the naive $2 S E$ gate would have shipped a model whose gain is within the true noise of a difference. A proper paired McNemar test (or a canary rollout, see A-B Testing & Canary Deployment) is the safe call here.

PICTURE: the number line with $s_{prod}$ , both cushions ( $0.012$ and $0.017$ ), and the challenger landing between them.

The one-picture summary

This single diagram compresses the whole walkthrough: a wobbling measurement $s$ around a hidden truth $p$ , a noise ruler $S E$ built from that wobble, the difference ruler $S E_{g} = S E_{new}^{2} + S E_{prod}^{2}$ , a cushion $δ = z S E_{g}$ stacked on the incumbent, and the automatic switch that fires only when the challenger clears the cushion.

Recall Feynman retelling — the whole story in plain words

A model's "score" is like a batting average measured over a handful of games. Even a fielder of fixed skill won't hit the same average every season — luck jiggles it. So when a new player scores a touch higher than your current star, you can't just swap them: the higher number might be a lucky streak. We measure how much luck can jiggle a single score — that's the standard error, biggest when the player is a coin-flip (around 50/50) and smaller the more games we watch. But we're comparing two jiggling numbers, and the gap between two jiggling numbers jiggles about 1.4 times more than either alone — so the cushion has to be a bit bigger than the naive guess. We insist the newcomer beat the star by that cushion $δ$ . Since we never know a player's true rate, we plug in their observed average (or, to be safe, assume the noisiest 50/50 case, or just reshuffle the games many times and watch the spread). We also make sure we have enough games — at least ten wins and ten losses — before trusting the bell-curve maths at all, and we score both players on the exact same games so a paired comparison is possible. Clear the cushion and it's almost surely real skill, so we promote automatically; fall short and we keep the current star. That cushioned, automatic swap is the promotion gate at the heart of an ML CI/CD pipeline.

Active recall

Question

Answer

What is the difference between

p

and

s

p

is the model's true, unknown correct-rate;

s

is the measured score on one held-out set, which wobbles around

p

Why does

p (1 - p)

appear in the standard error?

It is the spread of a single correct/wrong "coin flip"; maximal at

p = 0.5

, zero at

p = 0, 1

Why is the noise of the gap bigger than the noise of one score?

Variances of the two independent scores add, so

S E_{g} = S E_{new}^{2} + S E_{prod}^{2} \approx 2 S E

What CLT condition must hold before using the bell-curve

S E

trick?

Both

n p \geq 10

and

n (1 - p) \geq 10

(enough correct and enough wrong examples).

How do you fill in the unknown

p

inside

δ

in practice?

Plug in the observed score

\overset{p}{^} = s_{prod}

, or use the worst case

\overset{p}{^} = 0.5

, or bootstrap the test set.

What does the "2" in

δ = 2 S E

approximate, and what are the exact values?

It's a rule of thumb;

z = 1.96

for a 95% two-sided test,

z = 1.645

for a 95% one-sided test.

Which exact test should you use when both models share the same test set?

A paired test — McNemar's test — on the examples where the two models disagree.

Why must both models use the same held-out set?

Otherwise the gap mixes model-skill difference with dataset difference, and a tighter paired test becomes impossible.

Connections

CI - CD pipelines for ML — the parent pipeline whose promotion gate this page derives.
Statistical Significance — the source of the $z$ -values and the exact/binomial tests.
A-B Testing & Canary Deployment — what to do for a barely promoting model.
Model Registry — stores each model with its eval-dataset version so comparisons stay fair.
Model Monitoring — watches live scores and feeds drift back to retrain triggers.