Exercises — Data drift and concept drift detection
Level 1 — Recognition
L1.1 — Name the drift
For each scenario, say whether it is data drift (change in ) or concept drift (change in ). Recall the split:
Here = model inputs (features), = the true label, = "the rule mapping inputs to the answer."
a) A weather app suddenly gets many users from a new country; temperature readings sit in a colder range than training. b) A spam filter: spammers invent a brand-new trick, so emails that used to be "safe" are now spam. c) A store's checkout camera is upgraded to 4K; pixel brightness histograms shift, but a photo of a banana is still a banana. d) A credit model: a new law caps interest, so the same borrower profile now repays more reliably.
Recall Solution
Split rule: ask "did only the inputs move (), or did the input→label rule move ()?"
- a) Data drift. The distribution of temperature inputs shifted colder, but cold-vs-hot still means the same thing. Inputs moved, rule intact.
- b) Concept drift. The same email content now maps to a different label — changed. The rule moved.
- c) Data drift. Pixel statistics () shifted with the new camera, but "banana → banana" is unchanged. Classic covariate shift (Covariate Shift and Importance Weighting).
- d) Concept drift. Same borrower features , but repayment behaviour changed because of the law. The mapping is broken.
Why this ordering matters: the fix differs. Data drift → maybe reweight/monitor. Concept drift → almost always retrain with fresh labels (Retraining Pipelines and Triggers).
L1.2 — Which test fits which feature?
Match the tool to the situation:
- A continuous feature
income. 2. A categorical featurecountry. 3. Detecting that model error is rising with no input change.
Choices: KS test, Chi-square, DDM (Drift Detection Method — the label-based error monitor from the parent note).
Recall Solution
income(continuous) ::: KS test — compares empirical CDFs, no assumed shape.country(categorical) ::: Chi-square — compares observed vs expected category counts.- rising error, inputs normal ::: DDM (Drift Detection Method) — this is concept drift; only visible through labels/error, not through inputs.
Why: KS needs an ordering to build a CDF (continuous). Categories have no order, so we count buckets (chi-square). Error-monitoring needs , which is exactly what concept drift requires.
Chi-square fine print: the chi-square approximation assumes each category's expected count is not too small — the standard rule of thumb is for every category. If a rare category (e.g. a country with 2 expected users) violates this, merge it into an "other" bucket first, otherwise the value is inflated and its p-value is unreliable.
Level 2 — Application
L2.1 — Compute a PSI
A feature is binned into 3 buckets. Reference proportions , current .
Recall the Population Stability Index:
Compute PSI and classify (thresholds: none, – moderate, significant). The figure below shows each bin's contribution as a bar so you can see which bin dominates.
Recall Solution
Term by term (each term is ):
- Bin 1: . Positive because both factors are negative.
- Bin 2: . Unchanged bin contributes nothing.
- Bin 3: .
→ significant drift. As the figure's tallest bar shows, Bin 3 (the rare bucket that tripled) dominates — small rare bins are PSI-sensitive.
L2.2 — KS statistic by hand
Reference sample (sorted) . Current sample . Each has 4 points, so each point adds to its empirical CDF.
The KS statistic is the largest vertical gap between the two step-CDFs:
Find and the where it occurs.
Read the figure first. The figure below draws both empirical CDFs as staircases: the blue staircase is , climbing early because the reference values start low; the pink staircase is , which stays flat at until because the current sample has no values below . The yellow double arrow marks the tallest vertical gap between the two staircases — that arrow's height is the KS statistic . Notice it sits at , not at the largest raw value: that is the exact geometric point the solution locates numerically.
Recall Solution
Build both CDFs and evaluate at every "step" location . = fraction of that sample .
| gap | |||
|---|---|---|---|
| 2 | 0.25 | 0.00 | 0.25 |
| 4 | 0.50 | 0.00 | 0.50 |
| 5 | 0.50 | 0.25 | 0.25 |
| 6 | 0.75 | 0.50 | 0.25 |
| 7 | 0.75 | 0.75 | 0.00 |
| 8 | 1.00 | 1.00 | 0.00 |
The maximum gap is at : reference has already accumulated half its mass while current has accumulated none. This is exactly the yellow arrow in the figure.
Geometric reading: at the two staircases are farthest apart vertically — that is the single number that most cleanly exposes the rightward shift of the current sample.
Level 3 — Analysis
L3.1 — DDM alarm arithmetic
A model logs a running-minimum error rate with . Recall DDM's proportion standard deviation and its thresholds:
\text{Warning: } p_i+s_i \ge p_{min}+2s_{min}, \quad \text{Drift: } p_i+s_i \ge p_{min}+3s_{min}$$ Today $p_i = 0.15$ measured over $n_i = 400$ predictions. Is it OK / Warning / Drift? > [!recall]- Solution > Step 1 — current spread: > $$s_i = \sqrt{\frac{0.15 \cdot 0.85}{400}} = \sqrt{\frac{0.1275}{400}} = \sqrt{0.00031875} = 0.01785.$$ > > Step 2 — current signal: $p_i + s_i = 0.15 + 0.01785 = 0.16785$. > > Step 3 — thresholds: > - Warning line: $p_{min}+2s_{min} = 0.10 + 0.04 = 0.14$. > - Drift line: $p_{min}+3s_{min} = 0.10 + 0.06 = 0.16$. > > Step 4 — compare: $0.16785 \ge 0.16$ → **DRIFT**. (It also clears the warning line, but the stricter alarm wins.) > > **Why the $3s$ line means "real":** under a roughly Gaussian model of error fluctuation, being $3$ standard deviations above the best-ever error is a ~$0.1\%$ fluke — so this is degradation, not noise. ### L3.2 — KL vs symmetrized PSI For a 2-bin feature, $r=[0.5,0.5]$, $c=[0.9,0.1]$. Compute (natural log): (a) $D_{KL}(c\Vert r)=\sum_i c_i\ln\frac{c_i}{r_i}$, (b) $D_{KL}(r\Vert c)$, (c) PSI, and (d) verify PSI $= D_{KL}(c\Vert r)+D_{KL}(r\Vert c)$. > [!recall]- Solution > (a) $D_{KL}(c\Vert r) = 0.9\ln\frac{0.9}{0.5} + 0.1\ln\frac{0.1}{0.5}$ > $= 0.9(0.5878) + 0.1(-1.6094) = 0.5290 - 0.1609 = 0.3681$. > > (b) $D_{KL}(r\Vert c) = 0.5\ln\frac{0.5}{0.9} + 0.5\ln\frac{0.5}{0.1}$ > $= 0.5(-0.5878) + 0.5(1.6094) = -0.2939 + 0.8047 = 0.5108$. > > (c) $\text{PSI} = (0.9-0.5)\ln\frac{0.9}{0.5} + (0.1-0.5)\ln\frac{0.1}{0.5}$ > $= 0.4(0.5878) + (-0.4)(-1.6094) = 0.2351 + 0.6438 = 0.8789$. > > (d) $D_{KL}(c\Vert r)+D_{KL}(r\Vert c) = 0.3681 + 0.5108 = 0.8789 = \text{PSI}$. ✓ > > **Insight:** PSI is exactly the **symmetrized KL** — that is *why* it doesn't care whether you call $r$ or $c$ the "true" distribution, and why each term stays non-negative. See [[KL Divergence and Cross-Entropy]]. > [!mistake] L3 trap: "PSI is just $D_{KL}(c\Vert r)$." > **Why it feels right:** both are sums of $(\cdot)\ln\frac{c}{r}$-shaped terms, so they look identical at a glance. **The catch:** KL weights the log-ratio by $c_i$ (one distribution), PSI weights it by the *difference* $(c_i - r_i)$. Numerically here $0.3681 \ne 0.8789$. **Fix:** remember PSI $= D_{KL}(c\Vert r) + D_{KL}(r\Vert c)$ — the *sum of both directions*, which is why it's symmetric and always larger. --- ## Level 4 — Synthesis ### L4.1 — Design a monitor for a labels-delayed system A fraud model scores transactions live, but the true label ("was it fraud?") arrives **30 days later** (chargeback window). You have a [[Feature Stores and Reference Windows|feature store]] with the training reference window. Design a **two-track** monitoring plan. Specify: what you monitor **daily** (no labels yet) and what you monitor **monthly** (labels arrive), which statistic on each track, and what each track's alarm should **trigger**. > [!recall]- Solution > **Track A — daily, label-free (data-drift proxy for concept drift):** > - Compute **PSI / KS** per feature: current day vs reference window. Continuous features → KS; categorical → chi-square. > - Also monitor the **prediction-score distribution** and **predicted-positive rate** (a cheap proxy — if the model suddenly flags 3× more fraud with unchanged inputs, something moved). See the "no labels" mistake in the parent note. > - **Trigger:** if PSI $> 0.25$ on key features → raise a *warning ticket* and start [[A-B Testing and Shadow Deployment|shadow-deploy]] a candidate; do **not** auto-retrain (data drift alone may be harmless). > > **Track B — monthly, labels arrived (true concept drift):** > - Run **DDM** on the now-labelled 30-day-old cohort: error rate $p_i$, spread $s_i$, and the two DDM thresholds — **Warning** when $p_i+s_i \ge p_{min}+2\cdot s_{min}$, **Drift** when $p_i+s_i \ge p_{min}+3\cdot s_{min}$. > - **Trigger:** if DDM hits the **Drift** line ($3\cdot s_{min}$ above $p_{min}$) → fire the [[Retraining Pipelines and Triggers|retraining pipeline]] with the fresh labels. > > **Why two tracks:** Track A is fast but blind to $P(Y\mid X)$; Track B is authoritative but 30 days late. Track A buys you *early warning*; Track B gives *confirmation and the labels needed to retrain*. This directly implements the "monitor prediction distributions **and** set up delayed-label evaluation" fix. ([[Model Monitoring in Production]]) ### L4.2 — Threshold choice as a cost trade-off You must pick the PSI alarm threshold $\tau$. A false alarm costs a wasted retrain = \$500. A missed real drift costs \$5000 in bad predictions. Given historical rates: at $\tau=0.1$, expected 20 false alarms/yr and 1 missed drift/yr; at $\tau=0.25$, expected 4 false alarms/yr and 3 missed drifts/yr. Which $\tau$ minimizes expected annual cost? > [!recall]- Solution > Expected cost $=$ (false alarms $\times$ \$500) $+$ (missed drifts $\times$ \$5000). > > - $\tau=0.10$: $20(500) + 1(5000) = 10000 + 5000 = \$15{,}000$. > - $\tau=0.25$: $4(500) + 3(5000) = 2000 + 15000 = \$17{,}000$. > > **Choose $\tau=0.10$** ($\$15{,}000 < \$17{,}000$). > > **Why:** even though the sensitive threshold triggers many false alarms, each is cheap; missing an expensive real drift dominates the cost. When misses are $10\times$ costlier than false alarms, err toward **sensitivity**. > [!mistake] L4 trap: "A lower threshold is always safer, so pick the most sensitive one." > **Why it feels right:** "catch more drift" sounds strictly good. **The catch:** in L4.2 the sensitive threshold *did* win — but only because misses cost $10\times$ more. Flip the costs (cheap misses, expensive retrains — e.g. a huge dataset flagging trivial shifts, per the parent's "statistical significance ≠ importance" mistake) and $\tau=0.25$ wins. **Fix:** never pick a threshold from "sensitivity feels safe" — plug in the actual **cost of false alarm vs miss** and compute both expected costs. --- ## Level 5 — Mastery ### L5.1 — The degenerate empty bin You compute PSI and one bin has $r_i = 0.2$ but $c_i = 0.0$ (that category vanished from live traffic). The term is $(0-0.2)\ln\frac{0}{0.2} = (-0.2)\ln(0)$. What happens, why is it a problem, and what is the standard fix? > [!recall]- Solution > **What happens:** $\ln(0) = -\infty$, so the term is $(-0.2)(-\infty) = +\infty$. PSI **blows up to infinity**. The same explosion happens in KL whenever $Q_i=0$ while $P_i>0$ (division by zero inside the log). > > **Why it's a problem:** an infinite score is uninformative — you can't rank features or apply thresholds. It over-punishes a single empty bin that may just reflect small-sample noise. > > **Standard fix — Laplace / epsilon smoothing:** add a tiny $\epsilon$ (e.g. $10^{-4}$) to every bin's proportion and renormalize, so no bin is ever exactly zero. Concretely replace $c_i \leftarrow \frac{c_i + \epsilon}{1 + k\epsilon}$. Now $\ln$ is finite everywhere. Alternatively, **merge** tiny bins so every bin has enough count. > > **Sanity check with $\epsilon = 10^{-4}$:** the vanished bin becomes $\approx (0.0001-0.2)\ln\frac{0.0001}{0.2} = (-0.19993)\ln(0.0005) = (-0.19993)(-7.601) = 1.5197$ — large (correctly flagging a real disappearance) but **finite**. ([[Kolmogorov-Smirnov Test|KS avoids this]] entirely — it's binning-free.) ### L5.2 — Why smoothing beats "just drop the bin" A colleague says: "If a bin is empty, drop it from the PSI sum." Give one concrete failure of this and contrast it with smoothing, using $r=[0.2,0.8]$, $c=[0.0,1.0]$. > [!recall]- Solution > **Drop-the-bin approach:** delete bin 1, keep only bin 2. But bin 2 has $r_2=0.8$, $c_2=1.0$: > $$\text{PSI}_{drop} = (1.0-0.8)\ln\tfrac{1.0}{0.8} = 0.2\,(0.2231) = 0.0446.$$ > This says **"no drift"** ($<0.1$) — yet an entire 20\% category *vanished*! Dropping the empty bin **hides the exact event that mattered most**. > > **Smoothing approach** ($\epsilon=10^{-4}$, renormalize $c \approx [0.0001, 0.9999]$, $r\approx[0.2000,0.8000]$): > - Bin 1: $(0.0001-0.2)\ln\frac{0.0001}{0.2} = (-0.1999)(-7.601) = 1.5195$. > - Bin 2: $(0.9999-0.8)\ln\frac{0.9999}{0.8} = (0.1999)(0.2232) = 0.0446$. > - $\text{PSI}_{smooth} \approx 1.564$ → **significant drift, correctly flagged.** > > **Lesson:** the empty bin *is the signal*. Dropping it gives $0.0446$ (a false "all clear"); smoothing keeps that bin in the sum — finite but *loud* at $\approx 1.564$ — so the detector actually sees the category that disappeared. In short: **never delete the evidence you were hired to detect; keep it, just tame the infinity.** > [!mistake] L5 trap: "Zero counts are just missing data, so ignoring them is harmless." > **Why it feels right:** in most statistics, dropping empties or NaNs is standard hygiene. **The catch:** for *drift*, a bin going to zero (or appearing from zero) is often the **most drifted thing possible** — it's not missing data, it's a distribution change. Ignoring it makes your detector blind to total category disappearance. **Fix:** never drop; **smooth** with $\epsilon$ (finite, still loud) or use a **binning-free** test (KS) that has no zero-bin problem at all. --- ## Active recall > [!recall] Rapid self-check > 1. Which statistic is binning-free and thus immune to the empty-bin blow-up? ::: The KS test (works on CDFs directly). > 2. In L4.2, why did the *sensitive* threshold win? ::: Because a missed drift ($\$5000$) cost $10\times$ a false alarm ($\$500$). > 3. PSI equals the sum of *which two* KL divergences? ::: $D_{KL}(c\Vert r) + D_{KL}(r\Vert c)$. > 4. DDM crosses the Drift line at how many $s_{min}$ above $p_{min}$? ::: $3\,s_{min}$. > 5. What does DDM stand for, and why does it need labels? ::: Drift Detection Method; it monitors prediction *error*, which requires the true label $Y$. --- ## Connections - [[5.3.13 Data drift and concept drift detection (Hinglish)|Parent topic]] - [[Model Monitoring in Production]] - [[Retraining Pipelines and Triggers]] - [[KL Divergence and Cross-Entropy]] - [[Kolmogorov-Smirnov Test]] - [[Covariate Shift and Importance Weighting]] - [[Feature Stores and Reference Windows]] - [[A-B Testing and Shadow Deployment]] - [[Class Imbalance and Label Delay]]