Data drift and concept drift detection — Hinglish notes

5.3.13 · AI-ML › MLOps & Deployment

Drift exist hi KYU karta hai?

Hum joint distribution ko chain rule se split kar sakte hain:

$P (X, Y) = P (Y ∣ X) P (X)$

Yeh factorization poori conceptual key hai — neeche jo bhi hai woh isi sawaal se nikalti hai ki kaun sa factor badla.

Figure — Data drift and concept drift detection

Hum actually KIYA measure kar rahe hain?

Tum $P$ directly kabhi observe nahi karte — sirf samples. Toh detection = ek reference window (training/recent-good) ko ek current window (live) se compare karna aur poochna "kya yeh dono samples ek hi distribution se hain?"

1. Data drift — input distributions compare karo

Hum reference feature distribution $P_{r e f} (X)$ ko current $P_{c u r} (X)$ se compare karte hain.

Formula Population Stability Index (PSI) — derived

Ek feature ko $k$ bins mein daalo. $r_{i}$ = reference points ka fraction bin $i$ mein, $c_{i}$ = current points ka fraction bin $i$ mein.

Yeh form kyun? Hum ek symmetric, per-bin "surprise" chahte hain jo tab badhti hai jab proportions diverge karein. Ek bin mein relative change $\frac{c _{i} - r _{i}}{r _{i}}$ se shuru karo... lekin yeh asymmetric hai. Iske bajaye log-ratio ko actual difference se weight karo: $PSI = \sum_{i = 1}^{k} (c_{i} - r_{i}) ln \frac{c _{i}}{r _{i}}$ Kaise padhein: har term positive hai chahe $c_{i} > r_{i}$ ho ya $c_{i} < r_{i}$ ( $(c_{i} - r_{i})$ ka sign $ln \frac{c _{i}}{r _{i}}$ ke sign se match karta hai), toh PSI $\geq 0$ . Yeh symmetrized KL divergence hai ( $D_{K L} (c ∥ r) + D_{K L} (r ∥ c)$ ).

Rules of thumb: $< 0.1$ = koi drift nahi, $0.1$ – $0.25$ = moderate, $> 0.25$ = significant drift.

2. Concept drift — tumhe error monitor karna hoga

Worked example: PSI compute karna

Common mistakes (steel-manned)

Active recall

Recall Feynman: ek 12-year-old ko explain karo

Imagine karo tumne ball pakadna seekha apne dost ko throw karte dekhke. Data drift tab hota hai jab ek alag dost throw karne lagta hai — balls naye angles se aate hain (inputs badal gaye), lekin ball abhi bhi usi tarah udhti hai, toh tum phir bhi pakad sakte ho. Concept drift zyada spooky hai: physics ke rules khud change ho jaate hain — wahi throw ab alag curve karti hai, toh tumhari purani pakadne ki habit fail ho jaati hai. Drift detectors ek coach ki tarah hain jo constantly poochta hai "kya yeh throws abhi bhi waise hi hain jaise tumne practice kiye?" (data drift) aur "kya tum abhi bhi unhe pakad rahe ho?" (concept drift). Agar kisi bhi sawaal ka jawab "nahi" hai, toh phir se practice karne ka time hai.

Connections

Model Monitoring in Production
Retraining Pipelines and Triggers
KL Divergence and Cross-Entropy
Kolmogorov-Smirnov Test
Covariate Shift and Importance Weighting
Feature Stores and Reference Windows
A-B Testing and Shadow Deployment
Class Imbalance and Label Delay

#flashcards/ai-ml

Chain rule factorization jo drift define karne ke liye use hoti hai

P (X, Y) = P (Y ∣ X) P (X)

; data drift

P (X)

badalta hai, concept drift

P (Y ∣ X)

badalta hai.

Data drift (covariate shift) ki definition

P (X)

badalta hai jabki

P (Y ∣ X)

fixed rehta hai — inputs shift hote hain lekin input→output rule intact rehta hai.

Concept drift ki definition

P (Y ∣ X)

badalta hai — same input ab different labels pe map karta hai; underlying rule badal gaya.

PSI formula

PSI = \sum_{i} (c_{i} - r_{i}) ln \frac{c _{i}}{r _{i}}

binned proportions pe; symmetrized KL.

PSI hamesha ≥ 0 kyun hota hai

sign (c_{i} - r_{i}) = sign (ln \frac{c _{i}}{r _{i}})

, toh har term non-negative hai (equivalently yeh ek symmetrized KL divergence hai).

PSI thresholds

<0.1 koi drift nahi, 0.1–0.25 moderate, >0.25 significant.

KL divergence formula

D_{K L} (P ∥ Q) = \sum_{i} P_{i} ln \frac{P _{i}}{Q _{i}}

, Jensen se hamesha ≥0, =0 tabhi jab P=Q.

KS statistic

D = sup_{x} ∣ F_{r e f} (x) - F_{c u r} (x) ∣

, empirical CDFs ke beech maximum vertical gap.

Categorical feature drift ke liye test

Chi-square:

χ^{2} = \sum_{i} (O_{i} - E_{i})^{2} / E_{i}

observed vs reference-expected counts compare karta hai.

DDM warning/drift rule

warning

p_{i} + s_{i} \geq p_{min} + 2 s_{min}

pe, drift

\geq p_{min} + 3 s_{min}

pe, jahaan

s_{i} = p_{i} (1 - p_{i}) / n_{i}

Concept drift ko labels kyun chahiye

Yeh

P (Y ∣ X)

mein change hai; rule toot-ta tabhi dikh sakta hai jab predictions ko true

Y

se compare karo (ya proxies jaise confidence/output ratios se).

"Badi current window" detection kyun hurt karti hai

Badi window change point ke upar average kar leti hai, drift signal ko delay aur dilute karti hai; adaptive/sliding windows (ADWIN) use karo.