Data drift and concept drift detection: Worked examples

Before we start, one word we'll use constantly: a bin is a bucket. We chop a numeric feature's range into a few intervals (say "0–20 years old", "20–40", "40+") and count how many data points land in each. The proportion of a bin is (points in that bin) ÷ (total points). All the $r_{i}$ and $c_{i}$ symbols below are just these proportions — reference vs current.

The scenario matrix

Every drift computation you'll ever do is one of these cells. The worked examples that follow are tagged with the cell they cover.

#	Case class	What's tricky	Example
A	No drift ( $c = r$ )	Must return exactly 0	Ex 1
B	Mass shifts up in one bin ( $c_{i} > r_{i}$ )	Sign check on log-ratio	Ex 2
C	Mass shifts down ( $c_{i} < r_{i}$ )	Term still positive	Ex 2
D	Zero-count bin ( $c_{i} = 0$ or $r_{i} = 0$ )	$ln 0 = - \infty$ — degenerate!	Ex 3
E	Continuous shift, KS	Where is the max gap?	Ex 4 (figure)
F	Categorical mix, chi-square	Observed vs expected counts	Ex 5
G	Concept drift alarm (DDM)	2σ vs 3σ crossing	Ex 6 (figure)
H	Limiting behaviour	PSI as bins → identical/opposite	Ex 7
I	Real-world word problem	Which test? which fix?	Ex 8
J	Exam twist — data drift but NO accuracy loss	Don't retrain reflexively	Ex 9

We use these tools from the parent: PSI/KL, the KS statistic, chi-square, and DDM.

Example 1 — Cell A: the no-drift baseline

Example 2 — Cells B & C: one bin up, one bin down

Worked example Mass moves from bin 1 into bin 3

$r = [0.6, 0.3, 0.1]$ , $c = [0.4, 0.3, 0.3]$ . Compute PSI and confirm each term is positive.

Forecast: two bins change — will the decreasing bin give a negative term?

Bin 1 (down, Cell C): $c_{1} - r_{1} = 0.4 - 0.6 = - 0.2$ ; $ln \frac{0.4}{0.6} = ln 0.6667 = - 0.4055$ . Product $= (- 0.2) (- 0.4055) = 0.0811$ . Why this step? When mass leaves a bin, both factors are negative → product positive. This is why PSI is symmetric: leaving and arriving hurt equally.
Bin 2 (unchanged): $0 \times ln 1 = 0$ . Why this step? No movement, no contribution.
Bin 3 (up, Cell B): $c_{3} - r_{3} = 0.3 - 0.1 = 0.2$ ; $ln \frac{0.3}{0.1} = ln 3 = 1.0986$ . Product $= (0.2) (1.0986) = 0.2197$ . Why this step? Mass arriving: both factors positive → positive term.
$PSI = 0.0811 + 0 + 0.2197 = 0.3008$ .

Verify: Both nonzero terms came out positive (Cells B and C both add, never subtract), so PSI $> 0$ as required. $0.3008 > 0.25$ → significant drift, investigate. ✓

Example 3 — Cell D: the zero-count degenerate case

Worked example A bin the current window never fills

$r = [0.5, 0.5]$ , but the live data landed entirely in bin 1: $c = [1.0, 0.0]$ . What does raw PSI do, and how do we fix it?

Forecast: try to compute the bin-2 term literally. What breaks?

Bin 2 raw: $(0.0 - 0.5) ln \frac{0.0}{0.5} = (- 0.5) ln 0 = (- 0.5) (- \infty) = + \infty$ . Why this step? $ln 0 = - \infty$ . So a fully-empty bin makes PSI blow up to infinity — mathematically a divide-by-nothing degeneracy, not a bug in your intuition.
The standard fix: replace any zero proportion with a tiny $ϵ = 0.0001$ and renormalise. Use $c = [0.9999, 0.0001]$ . Why this step? $ϵ$ -smoothing keeps $ln$ finite while barely perturbing the answer. It encodes "we can't have literally seen zero — the true probability is small, not exactly nothing."
Bin 1: $(0.9999 - 0.5) ln \frac{0.9999}{0.5} = (0.4999) (0.6929) = 0.3464$ .
Bin 2: $(0.0001 - 0.5) ln \frac{0.0001}{0.5} = (- 0.4999) (- 8.5172) = 4.2578$ .
$PSI \approx 0.3464 + 4.2578 = 4.604$ .

Verify: Finite now, and huge ( $≫ 0.25$ ) — correct, because "current window abandoned half the reference support" is catastrophic drift. The $ϵ$ choice only affects the magnitude of an already-enormous score, not the decision. ✓

Example 4 — Cell E: KS statistic, finding the max gap

Figure — Data drift and concept drift detection

The empirical CDF $F (x)$ is just "what fraction of my sample is $\leq x$ ." It's a staircase that steps up by $1/ n$ at each data point.

Build $F_{r e f}$ : steps at $1, 2, 3, 4, 5$ , each $+ 0.2$ . So $F_{r e f} (x) = 0, 0.2, 0.4, 0.6, 0.8, 1.0$ as $x$ passes each point. Why this step? KS compares the two staircases; we need their heights.

Build $F_{c u r}$ : steps at $3, 4, 5, 6, 7$ . $F_{c u r} (3) = 0.2$ , but just before 3 it is $0$ .

Scan for the biggest vertical gap $∣ F_{r e f} (x) - F_{c u r} (x) ∣$ . Evaluate at each reference point:

At $x = 2$ : $F_{r e f} = 0.4$ , $F_{c u r} = 0$ → gap $0.4$ .

At $x = 4$ : $F_{r e f} = 0.8$ , $F_{c u r} = 0.4$ → gap $0.4$ .

At $x = 5$ (just after): $F_{r e f} = 1.0$ , $F_{c u r} = 0.6$ → gap $0.4$ . Why this step? " $sup$ " means the worst-case gap; on staircases the max always occurs at a jump point, so we only check those.

$D = max = 0.4$ .

Verify: A pure +2 shift of a 5-point sample leaves $2$ of $5$ points "unmatched" at the extreme, i.e. gap $2/5 = 0.4$ . Matches. Large $D$ ⇒ KS flags a clear right-shift. ✓

Example 5 — Cell F: chi-square on a categorical feature

Worked example Payment-method mix has shifted

Reference proportions over {Card, Cash, Wallet} $= [0.5, 0.3, 0.2]$ . Current window has $N = 200$ transactions, observed counts $O = [80, 70, 50]$ . Compute $χ^{2}$ .

Forecast: if nothing shifted we'd expect $[100, 60, 40]$ . How far off are we?

Expected counts $E_{i} = N \times r_{i} = 200 \times [0.5, 0.3, 0.2] = [100, 60, 40]$ . Why this step? $χ^{2}$ compares what we saw against what the reference proportions predict at this sample size.
Per-category $\frac{( O _{i} - E _{i} ) ^{2}}{E _{i}}$ :
- Card: $\frac{( 80 - 100 ) ^{2}}{100} = \frac{400}{100} = 4.0$
- Cash: $\frac{( 70 - 60 ) ^{2}}{60} = \frac{100}{60} = 1.6667$
- Wallet: $\frac{( 50 - 40 ) ^{2}}{40} = \frac{100}{40} = 2.5$ Why this step? Squaring makes every deviation positive; dividing by $E_{i}$ scales it — a gap of 10 matters more for a rare category than a common one.
$χ^{2} = 4.0 + 1.6667 + 2.5 = 8.1667$ .

Verify: Degrees of freedom $= k - 1 = 2$ . The critical value at $p = 0.05$ for df $= 2$ is $5.99$ . Our $8.17 > 5.99$ ⇒ reject "same mix" → categorical drift detected. Sanity: $\sum (O_{i} - E_{i}) = - 20 + 10 + 10 = 0$ , as it must (both sum to 200). ✓

Example 6 — Cell G: DDM concept-drift alarm

Current spread $s_{i} = \frac{p _{i} ( 1 - p _{i} )}{n _{i}} = \frac{0.16 \times 0.84}{400} = \frac{0.1344}{400} = 0.000336 = 0.01833$ . Why this step? Each prediction is right/wrong — a Bernoulli. The error rate over $n$ of them wobbles with the standard deviation of a proportion. We need this wobble to know if $0.16$ is "far".

The statistic: $p_{i} + s_{i} = 0.16 + 0.01833 = 0.17833$ . Why this step? DDM tracks the pessimistic edge (rate plus its own noise), not just the rate.

Warning threshold: $p_{min} + 2 s_{min} = 0.10 + 2 (0.02) = 0.14$ . Is $0.17833 \geq 0.14$ ? Yes → past Warning.

Drift threshold: $p_{min} + 3 s_{min} = 0.10 + 3 (0.02) = 0.16$ . Is $0.17833 \geq 0.16$ ? Yes → DRIFT. Why 3σ? If errors were just noise around $p_{min}$ , exceeding 3 standard deviations is a ~0.1% event — too rare to be luck. So we call it real degradation and trigger retraining.

Verify: $0.17833 > 0.16 > 0.14$ , so both thresholds are crossed → Drift state (the stronger). Consistent: a jump from 10% to ~18% pessimistic error is well beyond 3σ. ✓

Example 7 — Cell H: limiting behaviour of PSI

Worked example Two extreme two-bin cases

(a) Distributions become identical: $r = [0.5, 0.5]$ , $c = [0.5, 0.5]$ . (b) Distributions become maximally opposite (with smoothing): $r = [0.5, 0.5]$ , $c = [0.9999, 0.0001]$ .

Forecast: what's the smallest possible PSI? Does the largest blow up?

(a) Identical limit: every $c_{i} - r_{i} = 0$ → PSI $= 0$ . This is the floor. Why this step? Confirms the lower bound — PSI can't go negative (proved via Jensen's inequality in the parent).
(b) Opposite limit: Bin 1: $(0.9999 - 0.5) ln \frac{0.9999}{0.5} = 0.4999 \times 0.6929 = 0.3464$ . Bin 2: $(0.0001 - 0.5) ln \frac{0.0001}{0.5} = (- 0.4999) (- 8.5172) = 4.2578$ . PSI $\approx 4.604$ . Why this step? As one bin empties toward 0, its $ln$ term drives PSI toward $+ \infty$ — there is no upper bound. That's why PSI needs thresholds (0.1, 0.25), not a max-scale normalization.

Verify: (a) hits the theoretical minimum 0; (b) matches Example 3's 4.604 (same numbers) — PSI is unbounded above. The 0.25 "significant" line sits far below this ceiling, which is why real drift comfortably clears it. ✓

Example 8 — Cell I: real-world word problem

Worked example Streaming fraud model, no labels yet

Your fraud model scores transactions live. Fraud labels arrive 60 days late (chargebacks). This week, the transaction-amount distribution PSI jumped to $0.31$ , but you have no fresh labels. What's happening, and what do you do?

Forecast: is this data drift, concept drift, or can't-tell-yet? Which lever do you pull?

Identify the factor. PSI on $P (X)$ (amounts) $= 0.31 > 0.25$ → input distribution moved = data drift confirmed. Why this step? PSI only sees $P (X)$ ; it says nothing about whether the rule $P (Y ∣ X)$ changed. See covariate shift.
Can we see concept drift now? No — DDM needs $Y$ , and labels are 60 days out (label delay). Why this step? Concept drift is invisible until error can be measured. Don't pretend the PSI alarm is concept drift.
Use a proxy. Monitor the model's output-score distribution and predicted-fraud rate. If the fraud-flag rate spikes far beyond history, that's a proxy concept-drift signal. Why this step? When labels lag, prediction distributions are the only live signal (shadow a candidate model too).
Decide. Data drift alone → watch + reweight, don't retrain blindly. Set a delayed-label evaluation to fire in 60 days; if error then exceeds the DDM 3σ line, retrain.

Verify: Decision chain is internally consistent: $0.31 > 0.25$ (drift level) but the action is bounded by what we can measure. We did not trigger a costly retrain on covariate shift alone — matching the parent's steel-manned mistake #1. ✓

Example 9 — Cell J: exam twist (drift without accuracy loss)

Worked example The trap question

Feature age shows PSI $= 0.28$ (significant). But shadow-evaluated accuracy on this week's labelled sample is unchanged at 0.94, same as training. Do you retrain?

Forecast: the PSI alarm is red. Is that enough?

Read both signals. PSI high → $P (X)$ moved (data drift). Accuracy flat → $P (Y ∣ X)$ intact → no concept drift. Why this step? Recall $P (X, Y) = P (Y ∣ X) P (X)$ . Only the $P (X)$ factor moved. If the model already generalizes to the new age region, predictions stay correct.
Cost–benefit. Retraining costs compute, review, and risk. A stable-accuracy covariate shift doesn't justify it yet. Why this step? "Statistical significance ≠ practical harm." The 0.28 PSI is real but harmless here.
Action: Do not retrain. Keep monitoring — if accuracy later dips (concept drift arrives), then trigger via the pipeline.

Verify: Accuracy $0.94 = 0.94$ (unchanged) is the decisive metric; PSI $0.28$ is context. Correct answer: monitor, don't retrain. This is exactly why the parent warns "data drift always means retrain" is a fallacy. ✓

Active recall

Cover the answers first.

Every empty bin makes raw PSI equal to what value, and what's the fix?

+ \infty

; replace zero proportions with a tiny

ϵ

and renormalise.

On staircase (empirical) CDFs, the KS max gap always occurs where?

at a jump point (a data value) — so you only check those.

In chi-square,

E_{i}

comes from where?

reference proportions scaled by the current sample size

N

E_{i} = N r_{i}

DDM's Drift threshold is which multiple of

s_{min}

above

p_{min}

3 s_{min}

(Warning is

2 s_{min}

If PSI is high but shadow accuracy is unchanged, do you retrain?

No — pure covariate shift with intact

P (Y ∣ X)

; monitor instead.

Recall Scenario-matrix self-quiz

Which cell is " $ln 0$ danger"? (Answer: D) 2. Which two cells prove PSI terms stay positive whether mass leaves or arrives? (B and C) 3. Which cell shows PSI has no upper bound? (H) 4. Which cell fires the 3σ retrain trigger? (G)

Connections

Parent topic
Model Monitoring in Production
Retraining Pipelines and Triggers
KL Divergence and Cross-Entropy
Kolmogorov-Smirnov Test
Covariate Shift and Importance Weighting
Feature Stores and Reference Windows
A-B Testing and Shadow Deployment
Class Imbalance and Label Delay