Intuition The one-sentence ideaA model is trained on a snapshot of the world. The world keeps moving. Drift detection is the alarm system that tells you when the world has moved far enough that your model's predictions can no longer be trusted — before your business metrics quietly rot.
Intuition Why this mattersML models assume the future looks like the past. Formally, training assumes the joint distribution P ( X , Y ) P(X, Y) P ( X , Y ) stationary . In reality, user behaviour, prices, sensors, fraud tactics, and language all evolve. When P ( X , Y ) P(X,Y) P ( X , Y )
We can split the joint distribution using the chain rule:
P ( X , Y ) = P ( Y ∣ X ) P ( X ) P(X, Y) = P(Y \mid X)\, P(X) P ( X , Y ) = P ( Y ∣ X ) P ( X )
This factorization is the whole conceptual key — everything below falls out of asking which factor changed .
Definition Two kinds of drift
Data drift (covariate shift) : P ( X ) P(X) P ( X ) P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) inputs look different; the input→output rule is intact.Concept drift : P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) rule itself changed — same input now implies a different label.
Worked example Feel the difference (loan default model)
Data drift: A marketing campaign brings in younger applicants. The age distribution P ( X ) P(X) P ( X ) I I I P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) Concept drift: A recession hits. Now the same 25-year-old with income I I I P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) Why this step? Because the fix differs: data drift may need reweighting/monitoring; concept drift usually needs retraining with new labels .
You never observe P P P samples . So detection = comparing a reference window (training/recent-good) against a current window (live) and asking "are these two samples from the same distribution?"
We compare the reference feature distribution P r e f ( X ) P_{ref}(X) P r e f ( X ) current P c u r ( X ) P_{cur}(X) P c u r ( X )
To see P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) Y Y Y rising prediction error over time , even when inputs look normal.
Worked example A single numeric feature, 3 binsReference proportions r = [ 0.5 , 0.3 , 0.2 ] r = [0.5, 0.3, 0.2] r = [ 0.5 , 0.3 , 0.2 ] c = [ 0.3 , 0.3 , 0.4 ] c = [0.3, 0.3, 0.4] c = [ 0.3 , 0.3 , 0.4 ]
PSI = ( 0.3 − 0.5 ) ln 0.3 0.5 + ( 0.3 − 0.3 ) ln 0.3 0.3 + ( 0.4 − 0.2 ) ln 0.4 0.2 \text{PSI} = (0.3-0.5)\ln\tfrac{0.3}{0.5} + (0.3-0.3)\ln\tfrac{0.3}{0.3} + (0.4-0.2)\ln\tfrac{0.4}{0.2} PSI = ( 0.3 − 0.5 ) ln 0.5 0.3 + ( 0.3 − 0.3 ) ln 0.3 0.3 + ( 0.4 − 0.2 ) ln 0.2 0.4
Term 1: ( − 0.2 ) ln ( 0.6 ) = ( − 0.2 ) ( − 0.5108 ) = 0.1022 (-0.2)\ln(0.6) = (-0.2)(-0.5108) = 0.1022 ( − 0.2 ) ln ( 0.6 ) = ( − 0.2 ) ( − 0.5108 ) = 0.1022 Why positive? both factors negative.
Term 2: 0 × ln 1 = 0 0 \times \ln 1 = 0 0 × ln 1 = 0 Why? bin unchanged contributes nothing.
Term 3: ( 0.2 ) ln ( 2 ) = ( 0.2 ) ( 0.6931 ) = 0.1386 (0.2)\ln(2) = (0.2)(0.6931) = 0.1386 ( 0.2 ) ln ( 2 ) = ( 0.2 ) ( 0.6931 ) = 0.1386
Total PSI ≈ 0.241 \approx 0.241 ≈ 0.241 → moderate-to-significant drift, investigate.
Worked example KS statistic by hand (tiny)Reference sorted = [ 1 , 2 , 3 , 4 ] = [1,2,3,4] = [ 1 , 2 , 3 , 4 ] = [ 3 , 4 , 5 , 6 ] =[3,4,5,6] = [ 3 , 4 , 5 , 6 ] x = 3 x=3 x = 3 F r e f = 3 / 4 = 0.75 F_{ref}=3/4=0.75 F r e f = 3/4 = 0.75 F c u r = 1 / 4 = 0.25 F_{cur}=1/4=0.25 F c u r = 1/4 = 0.25 0.5 0.5 0.5 D = 0.5 D=0.5 D = 0.5 Why x = 3 x=3 x = 3 it's where the two step-functions are most separated.
Common mistake "Data drift always means retrain."Why it feels right: any alarm feels like "do something drastic." The catch: pure covariate shift (P ( X ) P(X) P ( X ) P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) Fix: check model performance / concept drift too before triggering a costly retrain.
Common mistake "No labels? Then I can't detect concept drift, so I'll ignore it."Why it feels right: DDM needs Y Y Y The catch: you can use proxy signals — prediction-confidence distribution shifts, output-class ratio changes, or delayed/partial labels. Fix: monitor prediction distributions and set up delayed-label evaluation.
Common mistake "Bigger window = more reliable detection."Why it feels right: more data = less noise. The catch: a huge current window averages over the moment drift started, delaying and diluting the alarm. Fix: use sliding/adaptive windows (e.g. ADWIN) that shrink when change is detected.
Common mistake "Statistical significance = practical importance."Why it feels right: small p-value looks decisive. The catch: with millions of rows KS/chi-square flags tiny shifts. Fix: pair p-values with effect sizes (PSI magnitude) and business thresholds.
Recall Test yourself before scrolling
Which factor of P ( X , Y ) P(X,Y) P ( X , Y )
Why is PSI always non-negative?
What does the KS statistic's "sup" represent geometrically?
Why does DDM use a 3σ rule?
Why can't you detect concept drift without (some) labels?
Recall Feynman: explain it to a 12-year-oldImagine you learned to catch a ball by watching how your friend throws. Data drift is when a different friend starts throwing — the balls come from new angles (the inputs changed), but a ball still flies the same way, so you can still catch. Concept drift is spookier: the rules of physics seem to change — the same throw now curves differently, so your old catching habit fails. Drift detectors are like a coach constantly asking "are these throws still like the ones you practiced on?" (data drift) and "are you still catching them?" (concept drift). If either answer is "no," time to practice again.
#flashcards/ai-ml
Chain rule factorization used to define drift P ( X , Y ) = P ( Y ∣ X ) P ( X ) P(X,Y)=P(Y\mid X)P(X) P ( X , Y ) = P ( Y ∣ X ) P ( X ) ; data drift changes
P ( X ) P(X) P ( X ) , concept drift changes
P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) .
Data drift (covariate shift) definition P ( X ) P(X) P ( X ) changes while
P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) stays fixed — inputs shift but the input→output rule is intact.
Concept drift definition P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) changes — the same input now maps to different labels; the underlying rule changed.
PSI formula PSI = ∑ i ( c i − r i ) ln c i r i \text{PSI}=\sum_i (c_i - r_i)\ln\frac{c_i}{r_i} PSI = ∑ i ( c i − r i ) ln r i c i over binned proportions; symmetrized KL.
Why PSI is always ≥ 0 sign ( c i − r i ) = sign ( ln c i r i ) \text{sign}(c_i-r_i)=\text{sign}(\ln\frac{c_i}{r_i}) sign ( c i − r i ) = sign ( ln r i c i ) , so every term is non-negative (equivalently it's a symmetrized KL divergence).
PSI thresholds <0.1 no drift, 0.1–0.25 moderate, >0.25 significant.
KL divergence formula D K L ( P ∥ Q ) = ∑ i P i ln P i Q i D_{KL}(P\|Q)=\sum_i P_i\ln\frac{P_i}{Q_i} D K L ( P ∥ Q ) = ∑ i P i ln Q i P i , always ≥0 by Jensen, =0 iff P=Q.
KS statistic D = sup x ∣ F r e f ( x ) − F c u r ( x ) ∣ D=\sup_x|F_{ref}(x)-F_{cur}(x)| D = sup x ∣ F r e f ( x ) − F c u r ( x ) ∣ , the maximum vertical gap between empirical CDFs.
Test for categorical feature drift Chi-square:
χ 2 = ∑ i ( O i − E i ) 2 / E i \chi^2=\sum_i (O_i-E_i)^2/E_i χ 2 = ∑ i ( O i − E i ) 2 / E i comparing observed vs reference-expected counts.
DDM warning/drift rule warning at
p i + s i ≥ p m i n + 2 s m i n p_i+s_i\ge p_{min}+2s_{min} p i + s i ≥ p min + 2 s min , drift at
≥ p m i n + 3 s m i n \ge p_{min}+3s_{min} ≥ p min + 3 s min , with
s i = p i ( 1 − p i ) / n i s_i=\sqrt{p_i(1-p_i)/n_i} s i = p i ( 1 − p i ) / n i .
Why concept drift needs labels It's a change in
P ( Y ∣ X ) P(Y\mid X) P ( Y ∣ X ) ; you can only see the rule break by comparing predictions to true
Y Y Y (or proxies like confidence/output ratios).
Why "bigger current window" hurts detection A large window averages over the change point, delaying and diluting the drift signal; use adaptive/sliding windows (ADWIN).
inputs differ, rule intact
Stationary assumption P of X,Y
P of X,Y = P of Y given X times P of X
Data drift / covariate shift
Fix: retrain with new labels
Reference vs current window
Population Stability Index
Intuition Hinglish mein samjhoDekho, jab hum model train karte hain to hum maan lete hain ki future bhi past jaisa hi rahega — yaani data ka distribution P ( X , Y ) P(X,Y) P ( X , Y ) P ( X , Y ) = P ( Y ∣ X ) P ( X ) P(X,Y) = P(Y|X)\,P(X) P ( X , Y ) = P ( Y ∣ X ) P ( X )
Agar sirf inputs badle (P ( X ) P(X) P ( X ) data drift / covariate shift hai. Agar rule khud badal gaya (P ( Y ∣ X ) P(Y|X) P ( Y ∣ X ) concept drift hai. Fix bhi alag hai: data drift me monitoring/reweighting kaafi ho sakta hai, par concept drift me aksar naye labels ke saath retrain karna padta hai.
Detect kaise karein? Input side pe hum reference window aur current window ko compare karte hain. PSI (∑ ( c i − r i ) ln ( c i / r i ) \sum (c_i-r_i)\ln(c_i/r_i) ∑ ( c i − r i ) ln ( c i / r i ) KS test (do CDF ke beech ka sabse bada gap), categorical ke liye chi-square . Concept drift ke liye labels chahiye — hum error rate track karte hain (DDM me 2σ pe warning, 3σ pe drift alarm). Yaad rakho: statistical significance aur practical importance alag cheez hain — millions rows me chhoti si shift bhi p-value flag kar degi, isliye hamesha effect size (PSI magnitude) aur business threshold dono dekho.
