Exercises — Model retraining pipelines

Think of the two probabilities from the parent note:

• $P(X)$ = "what do inputs look like?"
• $P(Y\mid X)$ = "given this input, what is the correct label?"

The inputs changed drastically (new language = new word distribution), so $P(X)$ shifted. But "unwanted mail is spam" — the rule $P(Y\mid X)$ — is unchanged. Answer: data drift (covariate shift). ==$P(X)$ changed, $P(Y\mid X)$ stayed.==

See Data drift and concept drift for the full taxonomy.

Rule of thumb: $<0.1$ stable, $0.1$–$0.25$ moderate, $>0.25$ significant. $0.18$ sits in $[0.1,0.25]$ → moderate shift — "watch". Action: investigate / monitor, do not auto-trigger a retrain yet. PSI is an input-drift signal, not a performance signal (Model monitoring and observability).

Performance-based triggers need fast labels to measure accuracy — you don't have them (6-month delay). By the time accuracy drops, you're 6 months too late. Answer: drift-based (PSI or the KS test on inputs, both defined above). It catches an input shift before accuracy visibly falls, which is exactly the case when labels are delayed.

Recall $\text{PSI}={\sum }_i(b_i-a_i)\ln (b_i/a_i)$.

Why this exact form? We want a number that is $0$ when the two distributions match and grows as they differ. The log ratio $\ln (b_i/a_i)$ measures how many times bigger production bin $i$ is than training bin $i$ — it is $0$ when they are equal, positive when the bin grew, negative when it shrank. Multiplying by $(b_i-a_i)$ does two jobs at once: it weights the log by how much probability mass actually moved, and it guarantees a non-negative term, because $(b_i-a_i)$ and $\ln (b_i/a_i)$ always share a sign (both positive if the bin grew, both negative if it shrank). Their product is therefore $\ge 0$ in every bin, so the whole sum is $0$ iff $a_i=b_i$ everywhere — exactly what a distance-like drift score must do.

Bin 1: $(0.45-0.6)\ln (0.45/0.6)=(-0.15)\ln (0.75)=(-0.15)(-0.28768)=0.043153$ Bin 2: $(0.55-0.4)\ln (0.55/0.4)=(0.15)\ln (1.375)=(0.15)(0.31845)=0.047768$

Sum: $\text{PSI}=0.043153+0.047768=0.09092$. $0.0909<0.1$ → stable (barely). Both terms positive, as required.

To compare a one-time cost with a recurring benefit fairly, we must fix a horizon. Here we compare over one week going forward: the retrain costs $C_r$ once, and it recovers the accumulated accuracy loss for (at least) the next week of running.

Step — accumulated accuracy lost by week 5. Why? This is the staleness cost $C_s(t)$ made concrete — it grew over the $5$ weeks since the last retrain. Lost $=5\times 0.15\%=0.75\%$.

Step — one-week revenue benefit of recovering it. Why? Restoring $0.75\%$ accuracy buys back $0.75\%$ worth of weekly revenue, and we count it over our one-week horizon. 0.75 \times \8{,}000 = $6{,}000$ recovered over the week.

Step — compare over the same horizon. Benefit \6{,}000$(oneweek)$\gg C_r=$1{,}200$ (one-time). Answer: retrain. Even a single week's recovered revenue dwarfs the one-time job cost — and every additional week the fresh model runs only widens the gap.

Gate: promote $⟺\text{metric}(\text{challenger})-\text{metric}(\text{champion})>\delta$. Difference $=0.909-0.902=0.007$. $0.007>0.005$ → promote the challenger. The improvement clears the noise band, so it is unlikely to be random wiggle. See Champion-Challenger and A-B testing.

We compare cost and benefit over one week so the units match.

Step — weekly cost of daily retraining. 7 \times C_r = 7 \times \1{,}200 = $8{,}400$ per week.

Step — weekly benefit. Daily retraining keeps accuracy near-perfect, so it recovers at most one week's worth of drift: 0.15\% \times \8{,}000 = $1{,}200$ recovered per week.

Step — net over the week. \1{,}200 - $8{,}400 = -$7{,}200 per week. **Net loss of \7,200 per week.**

General rule: retraining frequency must match the drift rate. Recall $C_r$ = one-time retrain cost and $C_s(t)$ = staleness cost accumulated over time $t$. Paying $C_r$ over and over to recover a tiny $C_s(t)$ (because $t$ was kept small) is wasteful. This is the parent's cost inequality $C_s(t)\cdot (\text{accuracy recovered})>C_r$ read in reverse: when $t$ is forced small, $C_s(t)$ never climbs above $C_r$, so the retrain never pays for itself.

PSI measures input drift — a change in $P(X)$ only. It says nothing directly about $P(Y\mid X)$, the input→output rule. Here inputs shifted ($P(X)$ moved, PSI $=0.40$) but the rule is stable ($P(Y\mid X)$ unchanged), and the model is robust to the new input region — hence F1 held at $0.88$. Answer: do NOT ship purely on PSI. Treat PSI as an early-warning trigger to investigate, then confirm with actual performance before promoting. Shipping a retrain here risks replacing a working champion with an unproven challenger.

(a) Empty production bin, $b_i=0$. Term $=(b_i-a_i)\ln (b_i/a_i)=(0-0.1)\ln (0/0.1)=(-0.1)\ln (0)$. $\ln (0)=-\infty$, so the term is $(-0.1)\times (-\infty )=+\infty$. PSI blows up.

(b) Empty training bin, $a_i=0$. Term $=(b_i-a_i)\ln (b_i/a_i)=(0.1-0)\ln (0.1/0)=(0.1)\ln (\infty )$. $\ln (0.1/0)=\ln (+\infty )=+\infty$, so the term is $(0.1)\times (+\infty )=+\infty$. PSI blows up here too — the symmetric failure: a division by zero inside the log instead of a $\ln (0)$.

Standard fix — floor BOTH sides. Replace any zero (whether in $a_i$ or $b_i$) with a tiny $\epsilon$, e.g. $\epsilon =10^{-4}$, before computing the ratio; or merge sparse bins so no bin is empty. Flooring only one side (a common bug) still leaves case (b) — or case (a) — divergent. Empty bins on either distribution are the degenerate case you must guard against.

Correct order: B → D → E → C → A.

• B ingest first — no data, nothing to do.
• D validate before training — the parent's rule: never train on garbage. If validation fails you stop here.
• E re-train on validated data → produces the challenger.
• C evaluate challenger vs champion on a temporally held-out recent slice.
• A promote only if the gate passes, else roll back.

The whole point of ordering is that each stage can fail independently; a failure early stops the flow before you waste compute or ship a bad model.

• Product X → performance-based trigger. Labels are instant, so you can watch live accuracy and retrain exactly when it drops — saving compute (retrain only when needed).
• Product Y → drift-based trigger (PSI or KS on inputs). Labels are 6 months late, so you cannot wait for accuracy; detect the input shift early as a proxy. Optionally add a scheduled safety net (e.g. quarterly).

This matches the parent's trigger table: the label delay is the deciding variable.

(a) Threshold. $25\%$ of $200,000=0.25\times 200,000=50,000$ new validated transactions.

(b) Days to fire. $50,000÷5,000\text{ per day}=10\text{ days}$.

(c) Why it works. New data = new information worth learning. Once a meaningful fraction of fresh, validated examples has accumulated, retraining lets the model absorb patterns that simply did not exist in the old snapshot — even if PSI has not yet crossed a drift threshold. It is a volume proxy for "there is now enough signal to be worth the retrain cost $C_r$." Note the word validated: the trigger counts only data that already passed stage D, so it never fires on garbage.

PSI, original direction ($a\to b$): $\text{PSI}={\sum }_i(b_i-a_i)\ln (b_i/a_i)$. Bin1 $(0.7-0.5)\ln (0.7/0.5)=(0.2)\ln (1.4)=(0.2)(0.33647)=0.067295$ Bin2 $(0.3-0.5)\ln (0.3/0.5)=(-0.2)\ln (0.6)=(-0.2)(-0.51083)=0.102165$ $\text{PSI}=0.067295+0.102165=0.169460$.

PSI, swapped ($b\to a$): each term becomes $(a_i-b_i)\ln (a_i/b_i)$. Bin1 $(0.5-0.7)\ln (0.5/0.7)=(-0.2)(-0.33647)=0.067295$ Bin2 $(0.5-0.3)\ln (0.5/0.3)=(0.2)(0.51083)=0.102165$ Swapped $\text{PSI}=0.067295+0.102165=0.169460$ — identical. Why identical algebraically: $(a-b)=-(b-a)$ and $\ln (a/b)=-\ln (b/a)$, so the two minus signs cancel in every bin. PSI is symmetric. ✓

KL divergence, $\text{KL}(b\parallel a)=\sum b_i\ln (b_i/a_i)$: $=0.7\ln (0.7/0.5)+0.3\ln (0.3/0.5)=0.7\ln (1.4)+0.3\ln (0.6)$ $=0.7(0.33647)+0.3(-0.51083)=0.235529-0.153249=0.082280$.

KL the other way, $\text{KL}(a\parallel b)=\sum a_i\ln (a_i/b_i)$: $=0.5\ln (0.5/0.7)+0.5\ln (0.5/0.3)=0.5(-0.33647)+0.5(0.51083)$ $=-0.168235+0.255415=0.087180$.

$0.082280\ne 0.087180$ → KL is asymmetric. That asymmetry (only $b_i$, or only $a_i$, weights the terms) is exactly why PSI multiplies by the symmetric factor $(b_i-a_i)$: it forces both symmetry and per-bin non-negativity. See KL divergence.

Process each signal in order:

1. PSI $=0.31>0.25$ → significant input drift. This is an investigate trigger, not a ship order (Ex 3.2 logic).
2. Evaluation set overlaps training by 20% → train–serve leakage! The F1 numbers are inflated and untrustworthy. Fix first: re-evaluate on a temporally held-out, unseen slice.
3. Gate on trustworthy numbers: even taking $0.842-0.840=0.002$ at face value, $0.002<\delta =0.01$ → inside the noise band.

Decision: do NOT promote. The leaked eval invalidates the comparison, and even the leaked gap fails the gate. Correct action: fix the eval split, re-run on unseen recent data, then re-apply the gate. Log everything (lineage).

Retrain when benefit $>$ cost: $C_s(t)=r\cdot t>C_r$. $t^{\star }=\frac{C_r}{r}=\frac{3000}{1500}=2\text{ weeks}.$ Interpretation:

• $t\to 0$ (just retrained): $C_s(t)=r\cdot t\to 0<C_r$ → you should never retrain immediately; you'd pay the full one-time $C_r$ to recover almost nothing (this is precisely the daily-retrain trap of Ex 3.1).
• $t\to \infty$: $C_s(t)=r\cdot t\to \infty \gg C_r$ → drift always eventually justifies a retrain. Staleness cost grows without bound, so waiting forever is never optimal either. The only real question is when, and the answer is the crossover $t^{\star }=C_r/r$.

This closes the loop: the whole "how often to retrain" debate reduces to comparing the drift slope $r$ against the fixed job cost $C_r$. Steeper drift (larger $r$) pushes $t^{\star }$ earlier; a cheaper job (smaller $C_r$) does the same.

• Language flood on a spam filter ::: data drift ($P(X)$ moved, rule stable)
• PSI status at $0.18$ ::: moderate — watch, don't auto-retrain
• Trigger for 6-month-delayed labels ::: drift-based
• What the KS statistic measures ::: the biggest vertical gap between the two input CDFs
• What $C_r$ and $C_s(t)$ mean ::: one-time retrain cost; staleness cost accumulated over time $t$
• Days for a $25\%$-of-$200\text{k}$ volume trigger at $5\text{k}$ per day ::: $10$ days
• Break-even week when $C_r=3000,r=1500$ ::: $t^{\star }=2$ weeks
• Why symmetrise KL into PSI ::: KL is asymmetric; $(b_i-a_i)$ factor forces symmetry and per-bin non-negativity
• Two degenerate PSI cases ::: $b_i=0$ gives $\ln (0)$; $a_i=0$ gives $\ln (\infty )$ — floor BOTH sides
• Correct move under high PSI but stable F1 ::: investigate, don't ship
• What kills the champion–challenger gate ::: train–serve leakage in the eval set