Visual walkthrough — Model retraining pipelines

We are answering one question the whole way: "Has the world my model was trained on quietly changed?" — and turning that vague worry into a single number a machine can watch.

---

Step 1 — Two piles of data, drawn as histograms

WHAT. We have two collections of numbers for one feature (say, customer age). One collection is the data the model was trained on ("expected"). The other is recent production data ("actual"). We chop the number line into $k$ equal slices called bins, and count how many samples fall in each.

WHY. Comparing raw clouds of thousands of points is hopeless. Binning turns each cloud into a small bar chart — a shape we can compare bar-by-bar. A bin is just a bucket: "ages 20–30", "ages 30–40", and so on.

PICTURE. Two bar charts, same bins, side by side. The heights are fractions (each chart sums to $1$) so pile sizes don't matter — only shape.

---

Step 2 — What "drift" looks like, bin by bin

WHAT. For each bin, look at the gap $b_i-a_i$: how much more (or less) of production sits in this bin compared to training.

WHY. If nothing drifted, both charts have the same shape, so every gap is $0$. A non-zero gap in any bin is the raw fingerprint of change. This subtraction is our first candidate ingredient.

PICTURE. The two charts overlaid; a coloured arrow in each bin shows the gap. Some arrows point up ($b_i>a_i$, production over-represented), some down ($b_i<a_i$, under-represented).

$\underset{\text{prod fraction}}{\underbrace{b_i}}-\underset{\text{train fraction}}{\underbrace{a_i}}=\underset{\text{signed gap in bin }i}{\underbrace{(b_i-a_i)}}$

---

Step 3 — Killing the cancellation with a ratio

WHAT. Instead of asking "how big is the gap?", ask "how many times bigger is production than training in this bin?" — the ratio $b_i/a_i$.

WHY. A ratio is scale-aware. Going from $0.01$ to $0.02$ is a $2\times$ blow-up (a big relative deal, possibly a new customer segment) even though the gap $0.01$ looks tiny. Absolute gaps miss this; ratios catch it. This is why drift measures speak in ratios, not differences alone.

PICTURE. For one bin, two stacked bars whose ratio is highlighted: $b_i/a_i=2$ means "twice as tall".

---

Step 4 — Why we wrap the ratio in a logarithm

WHAT. We don't use the ratio raw — we take its natural log, $\ln (b_i/a_i)$.

WHY — this is the key tool choice. A bare ratio is lopsided. "Doubling" is $b_i/a_i=2$; "halving" is $b_i/a_i=0.5$. These are equally severe drifts, but $2$ is $1$ unit above the no-change point while $0.5$ is only $0.5$ below it — the raw ratio punishes growth more than shrinkage. The logarithm is exactly the tool that fixes this, because $\ln$ turns "multiply" into "add" and turns reciprocals into sign-flips: $\ln \frac{1}{r}=-\ln r.$ So doubling gives $\ln 2=+0.693$ and halving gives $\ln 0.5=-0.693$ — equal magnitude, opposite sign. That symmetry is what a fair drift score needs, and no other simple function delivers it. (This is the same logic behind KL divergence, which also lives on $\ln$ of a ratio.)

PICTURE. The curve $y=\ln x$ through the point $(1,0)$: flat-crossing at the no-change ratio, symmetric in the log sense about it, negative left of $1$, positive right of $1$.

---

Step 5 — The sign-matching miracle: why every term is $\ge 0$

WHAT. Multiply the two ingredients we built: the signed gap $(b_i-a_i)$ from Step 2 and the log-ratio $\ln (b_i/a_i)$ from Step 4. This product is the per-bin PSI term.

WHY. Look at the signs together. Both ingredients read the same comparison ($b_i$ vs $a_i$), so they always agree in sign — and a product of two same-sign numbers is never negative:

PICTURE. A sign table drawn as two coloured regions. Left region: $b_i<a_i$ (both factors negative → product positive). Right region: $b_i>a_i$ (both factors positive → product positive). The middle line $b_i=a_i$ gives exactly $0$.

$\underset{\begin{array}{c}\text{sign}=\text{sign of drift}\end{array}}{\underbrace{(b_i-a_i)}}\cdot \underset{\begin{array}{c}\text{same sign}\\ \text{(Step 4)}\end{array}}{\underbrace{\ln \frac{b_i}{a_i}}}=\underset{\text{always}}{\underbrace{{\text{term}}_i\ge 0}}$

---

Step 6 — Summing the bins into ONE drift number

WHAT. Add every bin's term. That sum is the Population Stability Index.

WHY. Each term is a local drift measurement; total drift of the whole feature is their sum. Because each piece is $\ge 0$, nothing cancels — a big shift anywhere survives into the total.

$\text{PSI}={\sum }_{i=1}^k\underset{\text{signed gap}}{\underbrace{(b_i-a_i)}}\underset{\text{log-ratio}}{\underbrace{\ln \frac{b_i}{a_i}}}\ge 0$

PICTURE. The per-bin terms drawn as a small ascending stack that accumulates into one bar labelled PSI, next to the traffic-light thresholds.

---

Step 7 — Degenerate cases we must not ignore

WHAT. What happens at the edges: an empty bin, and the perfect-match bin.

WHY. The contract says cover every case. If some bin has $a_i=0$ (training never saw this range) but $b_i>0$ (production does), then $b_i/a_i$ divides by zero → $\ln (\infty )=+\infty$. The term explodes. Likewise $a_i>0,b_i=0$ gives $\ln 0=-\infty$, but the gap is negative, so the product still blows up positively.

PICTURE. A bin with a zero training bar and a tall production bar; a warning label showing the term shooting to infinity, and the standard fix (a tiny floor $\epsilon$ added to empty bins).

---

Step 8 — Watching the numbers walk (worked, term by term)

WHAT. Recompute the parent note's Example 1 so you can see each term land as a bar.

WHY. A worked walk anchors the abstract sum to real arithmetic and confirms the $\ge 0$ property visually — no term dips below the axis.

Expected $a=[0.5,0.3,0.2]$, Actual $b=[0.4,0.35,0.25]$:

Bin	$b_i-a_i$	$\ln (b_i/a_i)$	term
1	$-0.10$	$\ln 0.8=-0.2231$	$+0.02231$
2	$+0.05$	$\ln 1.1\overline{6}=+0.1542$	$+0.00771$
3	$+0.05$	$\ln 1.25=+0.2231$	$+0.01116$

$\text{PSI}=0.02231+0.00771+0.01116=0.0412$

PICTURE. Three positive bars (one from a negative gap, two from positive gaps) all above the axis, summing to the PSI marker below the $0.1$ green line.

---

The one-picture summary

The whole derivation in a single frame: two histograms → per-bin gap $(b_i-a_i)$ and log-ratio $\ln (b_i/a_i)$ → same-sign product $\ge 0$ → sum → PSI → traffic light → retrain trigger.

Recall Feynman: retell the whole walkthrough to a friend

We had two piles of the same measurement — old (training) and new (live). We turned each pile into a little bar chart of fractions. For each bar we asked two things: "how big is the gap?" ($b_i-a_i$) and "how many times did it grow or shrink?" ($b_i/a_i$). Adding raw gaps was useless — pluses and minuses cancel to zero. So we took the log of the ratio, because log makes "double" and "halve" equally sized but opposite in sign, and it's $0$ exactly when nothing changed. The magic: the gap and the log-ratio always point the same way, so their product is never negative. Add all those non-negative pieces and you get one number, PSI. Zero means the world hasn't moved; big means it has. We floor empty bins so nothing blows up. Then a traffic light: green ignore, amber watch, red — wake the retraining robot.

Recall Quick self-test

Why can't a plain sum of gaps detect drift? ::: Because $\sum (b_i-a_i)=1-1=0$ always — pluses and minuses cancel. Why the logarithm and not the raw ratio? ::: $\ln$ makes doubling and halving equal-magnitude, opposite-sign, and is $0$ exactly at ratio $1$. Why is every PSI term $\ge 0$? ::: The gap and the log-ratio always share a sign, and same-sign product is non-negative. What breaks the formula and how do we fix it? ::: An empty bin ($a_i=0$) makes the log-ratio infinite; floor fractions with a tiny $\epsilon$ or merge bins. Does PSI measure accuracy? ::: No — only input drift; confirm with real performance before retraining.

Related: Data drift and concept drift · KL divergence · Model monitoring and observability · CI-CD for ML (CI-CD-CT) · Champion-Challenger and A-B testing · Feature stores · Data versioning and lineage · Shadow deployment and canary release