Model monitoring and observability: Worked examples

Before any symbol appears, let us re-earn the two words we lean on constantly.

The scenario matrix

Every drift computation this topic can throw at you lands in one of these cells. The examples below are labelled by cell so you can see the coverage.

Cell	What makes it tricky	Covered by
A. All bins grow/shrink cleanly	ordinary case, mixed signs	Ex 1
B. A bin stays identical	one term must vanish ( $ln 1 = 0$ )	Ex 2
C. Perfect match, PSI = 0	the degenerate "no drift" floor	Ex 2
D1. Empty actual bin ( $a_{i} = 0$ )	$ln 0 = - \infty$ — needs a fix	Ex 3
D2. Empty reference bin ( $e_{i} = 0$ )	divide-by-zero $a_{i} / e_{i}$ — needs a fix	Ex 3b
E. KS, shift right	offset CDFs, sup at the step	Ex 4
F. KS, identical samples	$D = 0$ degenerate	Ex 5
G. KS, spread change (same mean)	mean unchanged, shape changed	Ex 6
H. Real-world word problem	which tool? interpret the number	Ex 7
I. Exam twist — drift ≠ retrain	reasoning trap, not arithmetic	Ex 8

PSI examples

Worked example

Ex 1 — Cell A: clean mixed drift Reference $e = [0.5, 0.3, 0.2]$ ; new $a = [0.3, 0.3, 0.4]$ .

Forecast: Bin 1 shrank, bin 2 stayed, bin 3 grew. Guess: a moderate PSI, somewhere near the $0.1$ – $0.25$ band. Write your guess down.

Step 1 — bin 1: $(0.3 - 0.5) ln (0.3/0.5) = (- 0.2) (ln 0.6) = (- 0.2) (- 0.5108) = 0.1022$ . Why this step? Bin shrank, so both factors are negative; their product is positive — drift adds, never cancels.

Step 2 — bin 2: $(0.3 - 0.3) ln (0.3/0.3) = (0) (ln 1) = (0) (0) = 0$ . Why this step? Identical bin: the difference factor is $0$ . It contributes nothing — exactly what "no movement here" should mean.

Step 3 — bin 3: $(0.4 - 0.2) ln (0.4/0.2) = (0.2) (ln 2) = (0.2) (0.6931) = 0.1386$ . Why this step? Bin grew, both factors positive → positive term. The biggest mover dominates the score.

Step 4 — sum: $0.1022 + 0 + 0.1386 = 0.2408$ . Why this conclusion? $0.2408$ sits in the $0.1$ – $0.25$ band → moderate shift, investigate. Forecast confirmed.

Verify: Both fraction lists sum to $1.0$ (sanity: they are valid distributions). Every term was $\geq 0$ , matching PSI $\geq 0$ .

Worked example

Ex 3 — Cell D1: empty actual bin ( $a_{i} = 0$ , $ln 0 = - \infty$ ) Reference $e = [0.5, 0.5]$ ; new $a = [1.0, 0.0]$ — the second bin emptied out in production.

Forecast: Bin 2 went to zero. Naively $ln (a_{2} / e_{2}) = ln (0/0.5) = ln (0) = - \infty$ , so $(a_{2} - e_{2}) ln (\dots) = (- 0.5) (- \infty) = + \infty$ . A raw PSI would be infinite — useless. So we must patch the zero.

Step 1 — spot the singularity: the $ln 0 = - \infty$ term appears because the actual bin is empty. Why this step? $ln$ is undefined at $0$ ; one empty actual bin would crash the metric.

Step 2 — apply the standard fix (floor + renormalise): replace every zero fraction with a tiny floor $ε = 0.0001$ , then renormalise each side so its fractions sum back to $1$ . Here new becomes $[1.0, 0.0001]$ with sum $1.0001$ ; dividing each entry by $1.0001$ renormalises to $[0.99990, 0.00009998]$ . Reference has no zeros, so it stays $[0.5, 0.5]$ . Why this step? Flooring keeps the log finite; renormalising restores the "fractions sum to 1" property so the two sides remain comparable distributions. The shift is negligible, so the answer barely moves.

Step 3 — bin 1: $(0.9999 - 0.5) ln (0.9999/0.5) = (0.4999) (ln 1.9998) = (0.4999) (0.6930) = 0.3465$ . Why this step? The healthy, over-full bin contributes a modest positive term — the "normal" part of the drift.

Step 4 — bin 2: $(0.00009998 - 0.5) ln (0.00009998/0.5) = (- 0.4999) (ln 0.00019996) = (- 0.4999) (- 8.517) = 4.258$ . Why this step? Emptying a bin is massive drift, and the smoothed term correctly reports a huge (but finite) contribution.

Step 5 — sum: $0.3465 + 4.258 = 4.604$ . Why this conclusion? $≫ 0.25$ → major drift. A vanished bin is one of the strongest possible signals — often a broken pipeline, not organic drift. (See the Ex 8 trap.)

Verify: Finite because of $ε$ ; the value grows without bound as $ε \to 0$ , matching the " $- \infty$ " forecast. The renormalised new fractions sum to $1$ , and the healthy bin still gave a modest $\approx 0.3465$ .

Worked example

Ex 3b — Cell D2: empty reference bin ( $e_{i} = 0$ , divide-by-zero) Reference $e = [0.5, 0.5, 0.0]$ ; new $a = [0.4, 0.3, 0.3]$ — a new category appeared live that never existed in training (its reference fraction is $0$ ).

Forecast: Bin 3 has $e_{3} = 0$ , so the raw ratio $a_{3} / e_{3} = 0.3/0 = + \infty$ and $ln (+ \infty) = + \infty$ — again infinite. The mirror of Ex 3: the singularity now comes from the denominator. Same floor-and-renormalise fix. Expect a large PSI, because a brand-new category is major drift.

Step 1 — spot the singularity: the divide-by-zero $a_{3} / e_{3}$ appears because the reference bin is empty. Why this step? Division by $0$ is undefined; one empty reference bin would crash the metric — the opposite failure mode to Ex 3.

Step 2 — floor + renormalise the reference: replace $e_{3} = 0$ with $ε = 0.0001$ : reference $[0.5, 0.5, 0.0001]$ sums to $1.0001$ ; renormalise to $[0.499950, 0.499950, 0.00009999]$ . New has no zeros, so it stays $[0.4, 0.3, 0.3]$ . Why this step? Flooring the denominator keeps the ratio finite; renormalising keeps reference a valid distribution.

Step 3 — bin 1: $(0.4 - 0.49995) ln (0.4/0.49995) = (- 0.09995) (ln 0.80008) = (- 0.09995) (- 0.22304) = 0.02229$ . Why this step? Bin 1 shrank a little; both factors negative → small positive contribution.

Step 4 — bin 2: $(0.3 - 0.49995) ln (0.3/0.49995) = (- 0.19995) (ln 0.60006) = (- 0.19995) (- 0.51073) = 0.10212$ . Why this step? Bin 2 shrank more → a bigger positive term.

Step 5 — bin 3: $(0.3 - 0.00009999) ln (0.3/0.00009999) = (0.29990) (ln 3000.3) = (0.29990) (8.00664) = 2.40119$ . Why this step? The brand-new category exploded from ~0 to $0.3$ — the smoothed term reports a huge (finite) contribution, exactly as forecast.

Step 6 — sum: $0.02229 + 0.10212 + 2.40119 = 2.5256$ . Why this conclusion? $≫ 0.25$ → major drift, dominated by the new category. In practice a never-before-seen category usually means a schema/pipeline change or a genuine new segment — investigate before retraining (Ex 8).

Verify: Both sides sum to $1$ after renormalising; the value is finite thanks to $ε$ , and grows without bound as $ε \to 0$ , matching the " $+ \infty$ " forecast.

KS-statistic examples

To read these, we only need the CDF ("cumulative distribution function").

Figure 1 below draws the Ex 4 CDFs so you can see the sup gap. The cyan staircase is $F_{ref}$ (samples $1, 2, 3, 4$ ); the white staircase is $F_{new}$ (samples $3, 4, 5, 6$ , shifted right by 2). The amber arrow marks the largest vertical gap at $x = 3$ : $F_{ref} = 0.75$ , $F_{new} = 0.25$ , so $D = 0.5$ .

Figure — Model monitoring and observability

Worked example

Ex 4 — Cell E: shift-right (the classic drift) Reference [1,2,3,4], new [3,4,5,6]. Both size $n = 4$ .

Forecast: The new sample is the old one slid right by $2$ . The staircases run parallel but offset — expect a big gap around the middle, roughly $0.5$ .

Step 1 — build $F_{ref}$ : at $x = 1, 2, 3, 4$ it reads $0.25, 0.5, 0.75, 1.0$ . Why this step? Each of 4 points adds $1/4 = 0.25$ to the staircase.

Step 2 — build $F_{new}$ : the values $3, 4, 5, 6$ give $F_{new} = 0.25$ at $x = 3$ , $0.5$ at $4$ , $0.75$ at $5$ , $1.0$ at $6$ ; and $F_{new} = 0$ for $x < 3$ . Why this step? We need both staircases on the same axis before we can subtract them; $F_{new}$ stays flat at $0$ until its first data point at $x = 3$ .

Step 3 — scan the combined data points ${1, 2, 3, 4, 5, 6}$ : the widest gap is at $x = 3$ , where $F_{ref} = 0.75$ but $F_{new} = 0.25$ , gap $= 0.5$ . (At $x = 1, 2$ the gap is $0.25, 0.5$ ; from $x = 4$ onward it shrinks back.) Why this step? $sup$ sits at a data point of either sample, so scanning the combined set guarantees we don't miss it.

Step 4: $D = 0.5$ . Why this conclusion? A gap of half the total height is a clear distribution shift — matches the forecast.

Verify: $D \in [0, 1]$ ✓. Recompute over the combined points gives max $= 0.5$ . See Figure 1 above (amber arrow = the sup gap).

Worked example

Ex 6 — Cell G: same mean, different spread Reference [3,4,5,6] (mean $4.5$ ), new [1,4,5,8] (mean $4.5$ ). The mean is unchanged, but the new data is more spread out.

Forecast: A mean-only check would miss this. But KS reads the whole shape, so it should still catch the spread change — expect a small-to-moderate $D$ .

Step 1 — $F_{ref}$ : at $x = 3, 4, 5, 6$ : $0.25, 0.5, 0.75, 1.0$ . Why this step? Building the reference staircase first gives us the baseline shape (four tightly-packed points from 3 to 6) against which the spread-out new data is measured. Step 2 — $F_{new}$ : points $1, 4, 5, 8$ → $F_{new} = 0.25$ at $x = 1$ , $0.5$ at $4$ , $0.75$ at $5$ , $1.0$ at $8$ . Why this step? We put $F_{new}$ on the same axis; its first jump is far left (at $1$ ) and its last far right (at $8$ ) — the wide tails are exactly where a spread change will reveal itself. Step 3 — scan the combined data points ${1, 3, 4, 5, 6, 8}$ (both samples, not just reference):

$x = 1$ : $F_{ref} = 0$ (no ref point $\leq 1$ ), $F_{new} = 0.25$ → gap $0.25$ .
$x = 3$ : $F_{ref} = 0.25$ , $F_{new} = 0.25$ → gap $0$ .
$x = 4$ : $0.5$ vs $0.5$ → gap $0$ .
$x = 5$ : $0.75$ vs $0.75$ → gap $0$ .
$x = 6$ : $F_{ref} = 1.0$ , $F_{new} = 0.75$ (the $8$ not yet counted) → gap $0.25$ .
$x = 8$ : $1.0$ vs $1.0$ → gap $0$ . Why this step? We must include the new-only points $x = 1$ and $x = 8$ : at $x = 1$ the new staircase has already jumped while the reference hasn't, giving a gap of $0.25$ — tied with the largest. Scanning only the reference points would still find $0.25$ here, but checking $x = 1$ confirms no larger gap hides in the left tail. Step 4: $D = 0.25$ . Why this conclusion? Non-zero → KS detects the variance shift a mean-check would miss. This is why we prefer full-distribution tools.

Verify: Means equal ( $4.5 = 4.5$ ) yet $D = 0.25 > 0$ — confirms KS sees shape, not just centre. Max over the full combined set is $0.25$ .

Applied & exam scenarios

Worked example

Ex 7 — Cell H: real-world word problem (which tool?) A fraud model's transaction_amount feature is monitored. Reference vs live proportions across three bins (low / mid / high): $e = [0.7, 0.2, 0.1]$ , $a = [0.5, 0.2, 0.3]$ . Labels (was it fraud?) take 60 days to confirm. Which tool, and what do you do today?

Forecast: Labels are delayed, so live accuracy is unavailable now → we need an input-drift proxy. Bins are discrete fractions → PSI fits. "High amount" tripled its share; guess: moderate-to-major PSI.

Step 1 — pick the tool: PSI on the binned feature, because ground truth is 60 days away (delayed-label problem from the parent note). Why this step? Input drift is an early proxy when performance can't yet be measured.

Step 2 — bin low: $(0.5 - 0.7) ln (0.5/0.7) = (- 0.2) (ln 0.7143) = (- 0.2) (- 0.3365) = 0.0673$ . Why this step? The low-amount bin shrank; both factors negative → a small positive contribution. We compute per bin so we can later see which bin drives the alert.

Step 3 — bin mid: $(0.2 - 0.2) ln (0.2/0.2) = (0) (ln 1) = 0$ . Why this step? The mid bin did not move, so its term vanishes ( $ln 1 = 0$ ) — it contributes nothing and is not the cause.

Step 4 — bin high: $(0.3 - 0.1) ln (0.3/0.1) = (0.2) (ln 3) = (0.2) (1.0986) = 0.2197$ . Why this step? The high-amount bin tripled; both factors positive → the dominant term, flagging where the drift lives.

Step 5 — sum: $0.0673 + 0 + 0.2197 = 0.2870$ . Why this conclusion? $0.287 > 0.25$ → major input drift, driven by the "high amount" bin.

Step 6 — decide: Raise an alert and investigate the high-amount bin — do not auto-retrain, because we have no evidence yet that $P (Y ∣ X)$ changed (concept drift). Could be a new product, or a broken currency conversion.

Verify: Fractions sum to $1.0$ both sides ✓. PSI $= 0.287$ ✓. See Data drift and concept drift and Feature stores and data validation for the null/units checks.

Worked example

Ex 8 — Cell I: the exam twist (drift ≠ retrain) Prompt: "Feature income shows PSI $= 4.6$ overnight (like Ex 3, a bin emptied). A junior engineer schedules an immediate retrain. Justify accept or reject in two sentences."

Forecast: A PSI of $4.6$ is astronomically high — too clean for organic behaviour. Suspect a data bug, not a real world-shift.

Step 1 — read the magnitude: organic drift rarely empties a whole bin overnight; PSI $≫ 0.25$ instantly screams pipeline failure (nulls, unit change, schema break). Why this step? The parent's mistake box "data drift always means retrain" applies exactly here.

Step 2 — the correct action: reject the auto-retrain. First confirm the feed: are income values now null or in the wrong currency? Retraining on garbage bakes the bug in. Why this conclusion? Retraining only helps genuine concept drift ( $P (Y ∣ X)$ change confirmed by dropped labels/performance) — not a broken input pipe.

Verify: Consistent with Ex 3's finite-but-huge PSI and with Model retraining and CI-CD pipelines: retrain triggers should gate on performance, not raw input PSI alone.

Recall

Recall Cover the answers

What does PSI stand for, and what does it measure? ::: Population Stability Index; how much a feature's distribution moved between reference and live data.
Why does a "stayed the same" bin add $0$ to PSI? ::: Both factors are $0$ : $(a_{i} - e_{i}) = 0$ and $ln (a_{i} / e_{i}) = ln 1 = 0$ .
What two singularities break PSI on an empty bin, and the fix? ::: $ln 0 = - \infty$ (empty actual bin, Ex 3) and divide-by-zero $a_{i} /0$ (empty reference bin, Ex 3b); fix by flooring zeros with $ε$ then renormalising.
Which combined set must you scan to find the KS sup gap? ::: All data points of both samples (including new-only points), not just the reference points.
Which drift did Ex 6 catch that a mean-check misses? ::: A spread/variance change with the mean unchanged.
When labels are delayed, which family of metric do you rely on? ::: Input/prediction drift proxies (e.g. PSI), not live accuracy.
Does a huge overnight PSI justify auto-retrain? ::: No — suspect a broken pipeline first; retrain only on confirmed concept drift.

Connections

Data drift and concept drift
KL divergence and cross-entropy
Feature stores and data validation
Model retraining and CI-CD pipelines
A-B testing and shadow deployment
Logging, metrics and tracing
5.3.12 Model monitoring and observability (Hinglish)