Model retraining pipelines

WHAT is a retraining pipeline?

The key word is automated & repeatable. Manually re-running a notebook is not a pipeline; it isn't versioned, testable, or triggerable.

WHY do we retrain at all? (Drift — first principles)

A model $f_\theta$ is fit to minimise error over a training distribution $P_{\text{train}}(X, Y)$ . In production it sees $P_{\text{serve}}(X, Y)$ . Performance degrades when these diverge. There are two flavours:

WHY this distinction matters: if only $P(X)$ shifted, sometimes reweighting/collecting more data fixes it. If $P(Y\mid X)$ shifted, you MUST retrain — the old labels no longer describe reality.

Deriving the "should I retrain?" signal

We want a scalar that says "the distributions differ." One classic is Population Stability Index (PSI), derived from the same idea as KL divergence.

Split a feature into $k$ bins. Let $a_i$ = fraction of expected (training) samples in bin $i$ , $b_i$ = fraction of actual (recent production) samples in bin $i$ .

Why this form? We want a term that is $0$ when $a_i=b_i$ and grows as they differ, symmetrically. Start from KL: $\sum b_i \ln(b_i/a_i)$ . That is asymmetric. PSI symmetrises it:

$\text{PSI} = \sum_{i=1}^{k} (b_i - a_i)\,\ln\!\frac{b_i}{a_i}$

HOW is a pipeline structured?

Trigger types (WHY each exists):

Trigger	When to use	Why
Scheduled (cron: nightly/weekly)	Steady, predictable drift	Simple, cheap to reason about
Performance-based	You have fast ground-truth labels	Retrain only when needed → saves compute
Drift-based (PSI/KS test)	Labels are delayed	Detect input shift before accuracy drops
Data-volume-based	Enough new data accumulated	New data = new information worth learning

The critical gate: never promote a worse model

The fix is a champion–challenger evaluation:

$\text{promote} \iff \big(\text{metric}(\text{challenger}) - \text{metric}(\text{champion})\big) > \delta$

where $\delta$ absorbs noise (so you don't flip-flop on random $\pm 0.1\%$ wiggles).

Worked examples

Worked example Example 1 — Compute PSI for a shifted feature

Training vs production, 3 bins. Expected $a = [0.5,\,0.3,\,0.2]$ , Actual $b = [0.4,\,0.35,\,0.25]$ .

Step — per-bin term. Why? Each bin's contribution measures local shift.

Bin1: $(0.4-0.5)\ln(0.4/0.5) = (-0.1)(-0.2231)=0.02231$
Bin2: $(0.35-0.3)\ln(0.35/0.3) = (0.05)(0.1542)=0.00771$
Bin3: $(0.25-0.2)\ln(0.25/0.2) = (0.05)(0.2231)=0.01116$

Step — sum. Why? Total drift = sum of local contributions. $\text{PSI} = 0.02231+0.00771+0.01116 = 0.0412$ .

Interpretation: $0.0412 < 0.1$ → stable, do NOT trigger retrain. Notice every term was positive — that's the sign-matching property in action.

Common mistakes

Active recall

Recall Test yourself (hide the answers)

What are the two types of drift, and which forces retraining?
Why is every PSI term guaranteed $\ge 0$ ?
Why can retraining too often be a net loss?
What does the champion–challenger gate protect against?
Why must the eval set be temporally held out?

Recall Feynman: explain to a 12-year-old

Imagine you learned all the popular songs in your class last year, so you're great at guessing what your friends like. But this year new kids joined and everyone likes new songs — your old guesses start missing! A retraining pipeline is like a helper robot that keeps an eye on the class, notices when the songs have changed a lot, and quietly re-learns the new favorites. But it's careful: before trusting its new guesses, it plays a little game against its old self, and only switches if the new version really guesses better. That way it never gets worse by accident.

Flashcards

What is a model retraining pipeline?

An automated, repeatable workflow that ingests fresh data, validates it, retrains, evaluates the candidate vs production, and promotes only if it passes quality gates (else rolls back).

Covariate (data) drift vs concept drift?

Covariate:

P(X)

changes,

P(Y\mid X)

same. Concept:

P(Y\mid X)

itself changes — this one forces retraining.

Write the PSI formula.

\text{PSI}=\sum_i (b_i-a_i)\ln(b_i/a_i)

, over binned expected fractions

a_i

and actual fractions

b_i

Why is every PSI term non-negative?

(b_i-a_i)

and

\ln(b_i/a_i)

always share the same sign, so their product

\ge 0

; the sum is

0

iff distributions match.

PSI interpretation thresholds?

<0.1 stable, 0.1–0.25 moderate (watch), >0.25 significant drift (trigger retrain).

What is champion–challenger?

Champion = current prod model; challenger = freshly retrained candidate. Promote challenger only if it beats champion by margin

\delta

on recent held-out data.

Why not retrain every hour?

Retraining has fixed cost

C_r

; if drift (staleness cost) is small, benefit < cost → net loss. Match frequency to drift rate.

Four common retraining triggers?

Scheduled (cron), performance-based, drift-based (PSI/KS), and data-volume-based.

Why must the evaluation set be temporally held out?

To avoid train–serve leakage and to mimic real production; reusing seen data inflates metrics and hides rot.

Does high PSI mean the model is broken?

No — PSI measures input drift, not accuracy. Confirm with actual performance before deciding.

Why version data + code + params together?

To reproduce and debug a model; the artifact alone can't tell you what produced it (lineage).

Connections

Data drift and concept drift
Model monitoring and observability
Champion-Challenger and A-B testing
Feature stores
KL divergence (PSI's parent idea)
CI-CD for ML (CI-CD-CT)
Data versioning and lineage
Shadow deployment and canary release

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek trained model basically duniya ka ek photo hai jo training ke waqt liya gaya tha. Par duniya toh badalti rehti hai — users badalte hain, fraud ke tareeke badalte hain, prices badalte hain. Isko hum drift kehte hain. Do type ke hote hain: covariate drift (input $P(X)$ badla par rule same) aur concept drift (asli rule $P(Y\mid X)$ hi badal gaya — yahan retrain karna zaroori hai). Retraining pipeline ek automatic machine hai jo yeh drift detect karti hai aur fresh model safely banati hai, bina baar-baar haath se notebook chalaye.

Drift ko measure karne ka ek simple tool hai PSI ( $\sum (b_i-a_i)\ln(b_i/a_i)$ ). Iska mazaa yeh hai ki har term hamesha positive ya zero hota hai (kyunki $(b_i-a_i)$ aur $\ln(b_i/a_i)$ ka sign hamesha same rehta hai), isliye jab distributions match karti hain toh PSI $=0$ . Rule: $<0.1$ theek hai, $>0.25$ matlab "bhai retrain socho".

Sabse important cheez: kabhi bhi naya model bina check kiye deploy mat karo. Naya data corrupt ya seasonal noise ho sakta hai. Isliye champion vs challenger game khelo — purana model (champion) aur naya model (challenger) dono ko recent held-out data pe compare karo, aur naya tabhi promote karo jab woh purane se meaningfully behtar ho (margin $\delta$ se). Aur retrain har ghante mat karo — retraining ka cost hota hai; jab drift ka nuksaan retraining ke kharche se zyada ho, tabhi retrain karo. Yehi 80/20 hai: drift detect karo, gate lagao, lineage save karo — baaki sab detail baad mein.

WHAT is a retraining pipeline?

WHY do we retrain at all? (Drift — first principles)

Deriving the "should I retrain?" signal

HOW is a pipeline structured?

The critical gate: never promote a worse model

Worked examples

Common mistakes

Active recall

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — MLOps & Deployment

Connections