Data drift and concept drift detection: Question bank

This bank drills the boundary between covariate shift, divergence measures, and the practical alarms you set up in Model Monitoring in Production. If a term feels unfamiliar, jump back to the parent topic first.

Terms and symbols used on this page

Before the questions, we nail down every symbol so nothing appears "for free."

True or false — justify

Data drift always causes a drop in model accuracy.

False. Pure covariate shift moves

P (X)

but leaves

P (Y ∣ X)

intact; if the model already generalizes to the new input region, accuracy can stay flat. Drift is a warning, not proof of harm.

P (Y ∣ X)

changes, then

P (X, Y)

must change too.

True. Since

P (X, Y) = P (Y ∣ X) P (X)

, changing the first factor changes the product even if

P (X)

is frozen. Concept drift is always a form of joint-distribution drift.

P (X)

changes, the conditional

P (Y ∣ X)

must also change.

False. They are separate factors of the chain rule. The inputs can shift wholesale while the input→output rule stays perfectly fixed — that is exactly the definition of pure data drift.

Label shift means the same thing as data drift.

False. Data drift moves

P (X)

; label shift moves the label prior

P (Y)

while

P (X ∣ Y)

stays fixed. They are different quadrants of distributional change and are detected differently (input distribution vs. predicted-class mix).

PSI can be negative if the current distribution shrinks in a bin.

False. In each bin

(c_{i} - r_{i})

and

ln \frac{c _{i}}{r _{i}}

always share the same sign (see the "Why" section), so every term is

\geq 0

. PSI is a sum of non-negative pieces and can never go below zero.

KL divergence is a distance, so

D_{K L} (P ∥ Q) = D_{K L} (Q ∥ P)

False. KL is not symmetric — swapping the arguments weights the log-ratio by a different distribution. This asymmetry is precisely why PSI symmetrizes it as

D_{K L} (c ∥ r) + D_{K L} (r ∥ c)

A statistically significant KS p-value means the drift matters for the business.

False. With millions of rows, the KS test flags microscopic shifts as "significant." Significance answers "is there any difference?"; you still need effect size (like PSI magnitude) to know if it matters.

You cannot detect concept drift without any labels.

Mostly true, but softened. Confirming a changed

P (Y ∣ X)

needs

Y

, but proxy signals — confidence-distribution shifts, output-class ratio changes, delayed labels — can hint at it before labels arrive.

A larger current window always gives more reliable drift detection.

False. A big window averages over the moment drift began, diluting and delaying the alarm. Adaptive windows (e.g. ADWIN) that shrink on change react faster.

DDM's 3σ rule assumes the error rate is roughly Gaussian around its running minimum.

True. Treating each prediction as Bernoulli, the error-rate estimate

p

has spread

s = p (1 - p) / n

; the normal approximation lets "beyond

3 s

" mean a ~0.1% fluke, i.e. real degradation.

If accuracy is stable, there can be no drift of any kind.

False. Data drift (moved

P (X)

) can occur with unchanged accuracy. Also two concept drifts can cancel in aggregate accuracy while harming specific subgroups — stable top-line accuracy is not an all-clear.

Spot the error

"We saw

P (X)

shift, so we immediately triggered a full retrain." — what's wrong?

The retrain may be wasted: covariate shift alone need not hurt performance. Check concept drift / live error first before paying the retrain cost.

"KL divergence is zero, so the two samples came from exactly the same underlying distribution." — what's wrong?

KL zero means the two estimated distributions match on the bins/support you measured; sampling noise and binning can hide real differences. Zero KL on samples is evidence, not proof.

"We used the KS test on a categorical feature." — what's wrong?

KS compares empirical CDFs and needs an ordered continuous variable. For unordered categories the ordering is meaningless; use chi-square on category counts instead.

"Our PSI is 0.05, so this feature is fine forever." — what's wrong?

PSI is a snapshot against one reference window; drift is a moving target. A low value today says nothing about tomorrow — monitoring must be continuous, not a one-off check.

"We set the reference window to all historical data so it's maximally representative." — what's wrong?

An ever-growing reference blends old regimes together, so genuine recent shifts get diluted and never cross the threshold. A fixed good-period reference window is usually better.

"DDM never fired, so the model has definitely not degraded." — what's wrong?

DDM needs labels

Y

to compute the error rate

p

. If labels are delayed or absent (see Class Imbalance and Label Delay), the error signal is stale or missing — silence is not safety.

"PSI and KL gave different numbers on the same bins, so one is buggy." — what's wrong?

They measure different things: KL is one-directional, PSI is the symmetrized sum

(c_{i} - r_{i}) ln (c_{i} / r_{i})

. Different values are expected, not a bug.

Why questions

Why factor

P (X, Y) = P (Y ∣ X) P (X)

instead of

P (X ∣ Y) P (Y)

Because inputs

X

arrive first at inference and the model computes

Y

from them; asking "which factor moved" this way separates input drift from rule drift, which map to different fixes.

Why, in PSI, does

(c_{i} - r_{i})

always have the same sign as

ln \frac{c _{i}}{r _{i}}

If bin

i

grew,

c_{i} > r_{i}

(c_{i} - r_{i}) > 0

and the ratio

\frac{c _{i}}{r _{i}} > 1

, whose log is positive. If it shrank,

c_{i} < r_{i}

(c_{i} - r_{i}) < 0

and

\frac{c _{i}}{r _{i}} < 1

, whose log is negative. Same sign either way, so their product

\geq 0

Why does the KS statistic take the supremum of the CDF gap rather than an average gap?

A distribution shift often concentrates the discrepancy at one place (e.g. a shift-right piles the gap where mass moved). The worst-case vertical gap catches this cleanly and yields a distribution-free p-value.

Why weight the log-ratio by

(c_{i} - r_{i})

in PSI instead of using the raw relative change?

The raw relative change

\frac{c _{i} - r _{i}}{r _{i}}

is asymmetric and blows up when

r_{i}

is tiny. Multiplying the log-ratio by the actual difference makes each term non-negative and yields the symmetric KL sum.

Why does concept drift require monitoring error, not just inputs?

A changed

P (Y ∣ X)

can occur with identical inputs

P (X)

; the only visible fingerprint is that the same inputs now produce more wrong predictions — i.e. rising error over time.

Why pair p-values with effect sizes when deploying drift alarms?

p-values shrink with sample size and flag trivial shifts at scale, causing alert fatigue. Effect sizes (PSI, KS

D

magnitude) tell you how big the shift is, so you alert only on business-relevant change.

Why can shadow deployment help with concept drift specifically?

Running a new/candidate model in shadow on live traffic lets you compare error signals under the current

P (Y ∣ X)

before committing, distinguishing genuine rule-change from noise.

Edge cases

If a feature's reference bin has

r_{i} = 0

but current

c_{i} > 0

, what happens to PSI?

The term

(c_{i} - r_{i}) ln \frac{c _{i}}{r _{i}}

divides by zero and

ln \frac{c _{i}}{0} \to \infty

. In practice you add a small

ϵ

(Laplace smoothing) to every bin so the score stays finite.

What is PSI when the current distribution equals the reference exactly?

Every term has

c_{i} - r_{i} = 0

and

ln \frac{c _{i}}{r _{i}} = ln 1 = 0

, so PSI

= 0

— the minimum, meaning "no detectable drift."

Label shift with no change in

P (X ∣ Y)

— can a PSI check on inputs miss it?

Yes if the classes overlap in feature space: the label prior

P (Y)

moved but the marginal

P (X)

barely budged, so PSI stays low. Watch the predicted-class proportions to catch it.

What does

D_{K L} (P ∥ Q)

become if

Q

assigns zero probability where

P

is positive?

It diverges to

+ \infty

: the term

P_{i} ln \frac{P _{i}}{0}

is infinite. This is why practical drift scoring smooths or shares support across bins.

Two drifts push the error rate up and down by equal amounts at once — will DDM fire?

Possibly not: DDM tracks aggregate error, so cancelling effects can mask each other. Per-segment monitoring is needed to catch drift hidden inside a stable average.

Labels arrive weeks late (label delay). What can you still monitor today?

Input distributions (PSI/KS/chi-square) and prediction-side proxies — confidence and output-class ratios. You detect data drift now and reserve concept confirmation for when delayed labels land.

The current window is very small (few live samples). What breaks?

All statistics get noisy: PSI bins are sparse, the KS gap is jumpy, and DDM's

s = p (1 - p) / n

inflates because

n

is small. Detection becomes unreliable until enough samples accumulate.

Sudden vs gradual drift — why might a fixed threshold miss the gradual kind?

Gradual drift creeps below the threshold for a long stretch while slowly eroding performance, so a static cutoff never trips. Cumulative or trend-based detectors catch the slow slide a fixed line misses.

Active recall

Recall One-line self-check

Name the three drift quadrants and which factor moves in each. ::: Data = $P (X)$ ; concept = $P (Y ∣ X)$ ; label shift = $P (Y)$ .
Can drift exist with stable accuracy? ::: Yes — covariate shift or cancelling effects.
Why is KL not a distance? ::: It's asymmetric: $D_{K L} (P ∥ Q) \neq = D_{K L} (Q ∥ P)$ .
What blows PSI/KL up? ::: An empty bin ( $r_{i} = 0$ ) where the other has mass; fix with smoothing.
In DDM, what do $p$ and $n$ mean? ::: $p$ = current error rate, $n$ = number of predictions in the run.