Model retraining pipelines: Worked examples — AI-ML deep dive

Before we start, one piece of notation the parent used that we will lean on here:

The scenario matrix

Every case class this topic can produce, and the example that covers it:

Cell	Case class	What makes it tricky	Example
A	PSI, stable feature ( $b_{i} \approx a_{i}$ )	Must return $\approx 0$ ; do NOT retrain	Ex 1
B	PSI, significant drift ( $> 0.25$ )	Trigger fires	Ex 2
C	PSI, degenerate empty bin ( $b_{i} = 0$ OR $a_{i} = 0$ )	$ln$ blows up on either side	Ex 3
D	PSI limiting case, $b_{i} = a_{i}$ everywhere	The exact-zero boundary	Ex 4
E	Cost logic — retrain is worth it	Benefit $>$ cost	Ex 5
F	Cost logic — retrain too often (net loss)	Benefit $<$ cost	Ex 5
G	Champion–challenger, inside noise band	Do NOT promote	Ex 6
H	Champion–challenger, beats margin	Promote	Ex 6
I	Drift high but accuracy fine (concept-stable)	Trap: don't ship blindly	Ex 7
J	Word problem — pick the trigger type	Reasoning, not arithmetic	Ex 8
K	Exam twist — data drift vs concept drift diagnosis	Which $P$ moved?	Ex 9

We build the machinery once, then reuse it. Two tools appear repeatedly, so let us pin them down in plain words first.

Definition The two scalars we keep computing

PSI (Population Stability Index): one number that says how differently a feature is distributed now versus at training time. Split the feature's range into $k$ bins (buckets). Let $a_{i}$ = fraction of training points in bin $i$ , $b_{i}$ = fraction of recent production points in bin $i$ . Then $PSI = \sum_{i = 1}^{k} (b_{i} - a_{i}) ln \frac{b _{i}}{a _{i}} .$ Here $ln$ is the natural logarithm — the "how many times must $e \approx 2.718$ be multiplied to reach this value?" function. We use $ln$ (and not raw subtraction) because a ratio $b_{i} / a_{i}$ measures relative change, and $ln$ turns that ratio into an additive, symmetric-friendly quantity — the same reason KL divergence uses it.
The promotion gate: promote the new model only if $metric(challenger) - metric(champion) > δ$ , where $δ$ is a noise band — the wiggle we expect from pure luck on a finite test set. Typical values are small: $δ \approx 0.005$ to $0.02$ for a metric like F1, chosen from how much your holdout score bounces between runs.

Read the figure above like this. The lavender bars are the training fractions $a_{i}$ , the coral bars are the recent production fractions $b_{i}$ , one pair per bin along the horizontal axis; the vertical axis is "fraction of samples". Above each bin, the mint number is that bin's PSI contribution $(b_{i} - a_{i}) ln (b_{i} / a_{i})$ — notice every one is positive. That is the key visual: because $(b_{i} - a_{i})$ and $ln (b_{i} / a_{i})$ always share a sign, no bin can ever push the total downward, so PSI can never be negative. Keep that picture in mind for every example.

Worked examples

Cell A — stable feature, do NOT retrain

Worked example Example 1

Training $a = [0.5, 0.3, 0.2]$ , production $b = [0.4, 0.35, 0.25]$ . Forecast: the numbers barely moved. Guess: PSI small, below $0.1$ , no retrain. Will every term be positive?

Step 1 — per-bin terms. Why this step? PSI is a sum of local shifts; we compute each bin's contribution before adding.

Bin 1: $(0.4 - 0.5) ln \frac{0.4}{0.5} = (- 0.1) (- 0.22314) = 0.022314$
Bin 2: $(0.35 - 0.3) ln \frac{0.35}{0.3} = (0.05) (0.15415) = 0.007708$
Bin 3: $(0.25 - 0.2) ln \frac{0.25}{0.2} = (0.05) (0.22314) = 0.011157$

Step 2 — sum. Why this step? Total drift is the sum of local drifts. $PSI = 0.022314 + 0.007708 + 0.011157 = 0.041179.$

Step 3 — decide. Why? Apply the rule of thumb $< 0.1 =$ stable. $0.0412 < 0.1 \Rightarrow$ stable, do NOT trigger.

Verify: every term came out positive (sign-matching property confirmed), and the total is well under $0.1$ . Sanity check: bins summed to $0.5 + 0.3 + 0.2 = 1$ and $0.4 + 0.35 + 0.25 = 1$ — both are valid probability distributions, so PSI is meaningful.

Cell B — significant drift, trigger fires

Cell C — degenerate: an empty bin (either side)

Worked example Example 3

Two degenerate sub-cases share the same cure. (a) Empty production bin. Training $a = [0.4, 0.4, 0.2]$ , production $b = [0.5, 0.5, 0.0]$ — nothing landed in bin 3 recently. (b) Empty training bin. Training $a = [0.5, 0.5, 0.0]$ , production $b = [0.4, 0.4, 0.2]$ — bin 3 had no training data but now sees traffic. Forecast: (a) gives $ln (0/0.2) = ln (0) = - \infty$ ; (b) gives $ln (0.2/0) = ln (\infty) = + \infty$ . Both blow up. What do we do?

Step 1 — spot the singularity, on either side. Why? $ln$ is undefined at $0$ and diverges at $\infty$ . A zero in the numerator ( $b_{i} = 0$ ) sends the term to $- \infty$ ; a zero in the denominator ( $a_{i} = 0$ ) sends it to $+ \infty$ . Either way we must not feed it to a threshold check.

Step 2 — apply the standard fix: $ε$ -smoothing to every bin. Why? Smoothing both $a$ and $b$ (not just the empty side) guarantees no zero appears in numerator or denominator. Nudge every bin by $ε = 0.001$ and renormalise so each stays a valid distribution. For (a): $b \to [0.5, 0.5, 0.001] /1.001 \approx [0.499500, 0.499500, 0.000999]$ . For (b): $a \to [0.5, 0.5, 0.001] /1.001 \approx [0.499500, 0.499500, 0.000999]$ .

Step 3 — recompute the dangerous bin. Why? Bin 3 is the only one whose term changed dramatically.

(a) $(0.000999 - 0.2) ln \frac{0.000999}{0.2} = (- 0.199001) (- 5.29963) = 1.054573$ . Bins 1 & 2: $(0.4995 - 0.4) ln \frac{0.4995}{0.4} = (0.0995) (0.22201) = 0.022090$ each. $PSI_{(a)} \approx 0.022090 + 0.022090 + 1.054573 = 1.098753.$
(b) $(0.2 - 0.000999) ln \frac{0.2}{0.000999} = (0.199001) (5.29963) = 1.054573$ . Bins 1 & 2: $(0.4 - 0.4995) ln \frac{0.4}{0.4995} = (- 0.0995) (- 0.22201) = 0.022090$ each. $PSI_{(b)} \approx 0.022090 + 0.022090 + 1.054573 = 1.098753.$

Step 4 — decide. Both give $1.099 ≫ 0.25 \Rightarrow$ strong drift, investigate. A bin that was empty on either side is itself a red flag worth a human look.

Verify: both results are finite (no $\pm \infty$ ), positive, and equal — the sign-matching product $(b_{i} - a_{i}) ln (b_{i} / a_{i})$ is symmetric under swapping $a \leftrightarrow b$ for these mirrored inputs, so (a) and (b) landing on the same number is exactly right. Always smooth zeros on both sides; never let $ln (0)$ or $ln (\infty)$ reach the comparator.

Cell D — limiting case: identical distributions

Worked example Example 4

Training $a = [0.25, 0.25, 0.25, 0.25]$ , production $b = [0.25, 0.25, 0.25, 0.25]$ . Forecast: nothing changed at all. PSI should be exactly $0$ — the boundary of the whole scale.

Step 1 — evaluate one term. Why? If one term is $0$ , all four are by symmetry. $(0.25 - 0.25) ln \frac{0.25}{0.25} = (0) \cdot ln (1) = (0) (0) = 0.$

Step 2 — sum. Why? Total drift is the sum of the per-bin terms. $PSI = 0 + 0 + 0 + 0 = 0.$

Verify: this is the theoretical floor. Recall the PSI definition at the top of this page: the product $(b_{i} - a_{i}) ln (b_{i} / a_{i})$ is $\geq 0$ in every bin, and it is $0$ in a bin exactly when $a_{i} = b_{i}$ . So the whole sum is $0$ iff $a_{i} = b_{i}$ for every $i$ — which is precisely this case. Any real production PSI you ever see should be $> 0$ ; a literal $0$ almost always means you accidentally compared a dataset against itself (a common lineage bug — see Data versioning and lineage).

Cells E & F — cost logic: worth it vs net loss

Worked example Example 5

Model loses $0.2%$ accuracy per week to drift. Each $1%$ accuracy $\approx \$ 5{,}000/\text{week} $r e v e n u e . A r e t r ainj o b cos t s$ $800$, and each retrain resets accuracy to its fresh (zero-drift) level. Forecast: waiting 4 weeks accumulates real loss — retrain should win. Retraining daily should lose.

Step 1 — staleness cost after 4 weeks (Cell E). Why? Compare accumulated loss to the one-off retrain cost. Accuracy lost after 4 weeks $= 4 \times 0.2% = 0.8%$ . That current week's revenue loss $= 0.8 \times \$ 5000 = $4000$.

Step 2 — decide (Cell E). Why? Benefit vs cost of one retrain now. Retraining resets that $0.8%$ loss $\Rightarrow$ benefit $\approx \$ 4000 $r eco v er e d$ \gg $800 $cos t$ \Rightarrow$ retrain.

Step 3 — set up the daily-cadence comparison (Cell F). Why? We need an apples-to-apples weekly figure for retraining every day. Weekly retrain cost $= 7 \times \$ 800 = $5600$.

Step 4 — quantify the benefit of daily retraining (Cell F). Why? We must show the recovered loss with an explicit formula, not a hand-wave. Drift is $0.2%$ /week $= \frac{0.2%}{7} \approx 0.02857%$ per day. With daily resets, the model is at most one day stale, so per-day loss avoided is $0.02857% \times $5000 = $1.43 per day \Rightarrow 7 \times $1.43 = $10 per week saved versus never retraining after the first reset.$ Even generously counting the entire first-week avoided loss ( $0.2\% \times \$ 5000 = $10$/week of drift build-up), the recovered value is on the order of tens of dollars per week.

Step 5 — decide (Cell F). Why? Compare weekly cost to weekly benefit. $$5600 cost ≫ \sim $10 - $14 benefit \Rightarrow net loss .$

Verify: units all reduce to dollars-per-week, consistent. $\$ 4000 > $800 $(S t e p 2) an d$ $5600 > $14 $(S t e p 5) b o t hh o l d . T h e l esso n : r e t r ain * * f r e q u e n cy m u s t ma t c h d r i f t r a t e * * — w ee k l y i s t h es w ee t s p o t h er e, b ec a u seo n e w ee k a cc u m u l a t es$ $4000/4 = $1000 $o f l oss, co m f or t ab l y c l e a r in g t h e$ $800$ job cost, while daily wastes money resetting a model that has barely moved.

Cells G & H — the promotion gate

Worked example Example 6

Noise band $δ = 0.01$ . Two candidates: (G) Champion F1 $= 0.810$ , Challenger F1 $= 0.815$ . (H) Champion F1 $= 0.810$ , Challenger F1 $= 0.825$ . Forecast: G's gain ( $0.005$ ) is inside the noise; H's gain ( $0.015$ ) clears it. Guess: reject G, promote H.

Step 1 — case G difference. Why? The gate compares the gap to $δ$ . $0.815 - 0.810 = 0.005$ . Since $0.005 < 0.01 = δ$ : do NOT promote. The lift is indistinguishable from luck.

Step 2 — case H difference. Why? Same test, larger gap. $0.825 - 0.810 = 0.015$ . Since $0.015 > 0.01$ : promote. The improvement is beyond the noise band.

Verify: the gate is a simple inequality, so re-plug: G gives $0.005 > 0.01$ ? False → keep champion. H gives $0.015 > 0.01$ ? True → promote. This is exactly the Champion-Challenger and A-B testing logic, and pairs with a Shadow deployment and canary release to de-risk H's rollout.

Cell I — high drift, but accuracy fine (the trap)

Worked example Example 7

A feature's PSI jumps to $0.31$ (above the $0.25$ trigger). You pull last week's labelled holdout: F1 is still $0.812$ , essentially unchanged from the champion's $0.813$ . Forecast: drift alarm rang — but should you ship a new model? Guess: no, because accuracy did not move.

Step 1 — separate what PSI measures from what F1 measures. Why? PSI watches $P (X)$ (the inputs — see the notation box at the top). F1 watches actual correctness, which depends on $P (Y ∣ X)$ (the rule). They are different questions.

Step 2 — read the evidence. Why? $P (X)$ shifted (PSI $= 0.31$ ) but $P (Y ∣ X)$ apparently held (F1 flat). This is pure data drift, and a robust model can tolerate it.

Step 3 — decide. Treat the PSI spike as an early-warning to investigate, not a ship order. Confirm with the performance metric — which says fine — so keep the champion and keep monitoring.

Verify: gap $= 0.813 - 0.812 = 0.001$ , well inside the typical noise band ( $δ \approx 0.005$ – $0.02$ for F1, as defined above), so even if you had trained a challenger, it would fail the gate. Two independent signals agree: don't ship. See Model monitoring and observability for wiring both signals into one dashboard.

Cell J — word problem: choose the trigger

Worked example Example 8

A fraud team's true labels arrive 90 days late (chargebacks settle slowly). Input volume is steady. Fraud tactics evolve unpredictably — no fixed schedule. Which trigger type should fire retraining? Forecast: if labels are 90 days late, performance-based triggering is 90 days blind. Guess: use a drift-based trigger.

Step 1 — eliminate performance-based. Why? It needs fast ground truth. Here truth lags 3 months — you would detect a rotten model a quarter too late.

Step 2 — eliminate scheduled. Why? Scheduled suits predictable drift. "Unpredictable tactics" breaks that assumption; a nightly cron wastes compute on quiet nights and misses sudden attacks.

Step 3 — choose drift-based. Why? Drift-based watches $P (X)$ directly and fires before accuracy visibly drops — exactly what you need when labels are delayed. Compute PSI (or a KS test) on incoming features nightly; retrain when PSI $> 0.25$ .

Verify: cross-check against the parent's trigger table — "Drift-based · use when labels are delayed." Match. Pair it with a feature store so training and serving read identical feature definitions, avoiding false drift alarms from pipeline skew.

Cell K — exam twist: diagnose the drift type

Worked example Example 9

Two incidents in a housing-price model. Diagnose each as data drift or concept drift, and say which forces a retrain.

(a) A new city is added to the app. Input distribution of "square footage" and "zip code" changes, but a 2000-sq-ft house at a given location still costs what the model predicts.
(b) A sudden interest-rate hike. Inputs (house features) look identical to last month, yet the same house now sells for 8% less. Forecast: (a) inputs moved but rule held → data drift. (b) inputs same but rule moved → concept drift, must retrain.

Step 1 — test which $P$ moved in (a). Why? The definitions (top of page) hinge on this. Inputs changed $\Rightarrow P (X)$ moved. The price-given-features rule held $\Rightarrow P (Y ∣ X)$ unchanged. That is the definition of data drift (covariate shift).

Step 2 — response to (a). Why? Only $P (X)$ shifted, so the old labels still describe reality. Verdict: do NOT retrain to fix correctness — the model is still right. You may collect more data from the new region to sharpen inputs, but retraining is not forced.

Step 3 — test which $P$ moved in (b). Why? Inputs identical $\Rightarrow P (X)$ unchanged. Same house, new price $\Rightarrow P (Y ∣ X)$ changed. That is concept drift.

Step 4 — response to (b). Why? When $P (Y ∣ X)$ changes, the old labels no longer describe reality — you MUST retrain. No amount of reweighting fixes a rule that literally changed.

Verify: matches the parent's rule — "if $P (Y ∣ X)$ shifted, you MUST retrain." Conclusion: (a) = data drift → do not retrain (optional data top-up only); (b) = concept drift → retrain forced. See Data drift and concept drift for the full taxonomy.

Active recall

Recall One line per cell (hide the answers)

Cell A — stable PSI value, decision? ::: PSI $\approx 0.041 < 0.1$ ; do NOT retrain. Cell C — production bin OR training bin is $0$ , what breaks and the fix? ::: $ln (0) = - \infty$ or $ln (\infty) = + \infty$ ; $ε$ -smooth both sides then renormalise. Cell D — identical distributions give PSI = ? ::: Exactly $0$ (the floor; $a_{i} = b_{i}$ everywhere). Cell F — why can daily retraining lose money? ::: $7\times\$ 800=$5600 $cos t d w a r f s t h e$ \sim$10 $-$ $14 $d r i f t - l ossr eco v er e d . C e l l H — c ha l l e n g er b e a t sc ham p i o nb y$ 0.015 $w i t h$ \delta=0.01 $; p r o m o t e ? ::: Y es,$ 0.015>0.01 $. C e l l I — P S I hi g hb u tF 1 f l a t; s hi p ? ::: N o — P S I w a t c h es$ P(X) $, F 1 w a t c h escor r ec t n ess; co n f i r m w i t h p er f or man ce f i r s t . C e l l K (b) — s am e in p u t s, n e w p r i ce; d r i f tt y p e ? ::: C o n ce pt d r i f t ($ P(Y\mid X)$ changed) → must retrain.