Model monitoring and observability — Hinglish notes

5.3.12 · AI-ML › MLOps & Deployment

WHY does a deployed model need monitoring?

WHAT can go wrong (the four failure families):

Failure	What changes	Example
Data drift (covariate shift)	$P (X)$	Users ka ek naya age group sign up karta hai
Concept drift	$P (Y ∣ X)$	Same features, lekin "spam" ki definition badal jaati hai
Label/prediction drift	$P (\hat{Y})$	Model suddenly "fraud" 5× zyada predict karne lagta hai
System/operational	latency, errors, throughput	Ek feature pipeline nulls return karne lagti hai

HOW do we detect drift? (Derivation from first principles)

Hum true nai distribution nahi dekh sakte, sirf samples dekh sakte hain. Toh drift detection = ek two-sample test: kya recent batch training/reference batch ke same distribution se draw ki gayi hai?

1. Population Stability Index (PSI) — derive it

Hum chahte hain ki "distribution kitni move hui?" ke liye ek single number ho. Shuruaat karte hain do probability distributions ko bin-by-bin compare karne ke idea se. Feature range ko $B$ bins mein split karo. $e_{i}$ = expected (reference) data ke bin $i$ mein fraction, $a_{i}$ = actual (new) data ka fraction.

Do probabilities compare karne ka ek natural asymmetry-tolerant tarika hai difference ko log-ratio se weight karna (yeh per-bin relative entropy ka symmetrised version hai):

$PSI = \sum_{i = 1}^{B} (a_{i} - e_{i}) ln \frac{a _{i}}{e _{i}}$

Rules of thumb: PSI $< 0.1$ = stable; $0.1$ – $0.25$ = moderate shift, investigate karo; $> 0.25$ = major shift, action lo.

2. Kolmogorov–Smirnov (KS) statistic

Continuous features ke liye, empirical CDFs compare karo:

$D = sup_{x} F_{ref} (x) - F_{new} (x)$

WHY CDF gap ka sup? CDF poora distribution shape capture karta hai; sabse bada vertical gap worst-case disagreement hai. KS test ka p-value batata hai ki woh gap sampling noise se zyada bada hai ya nahi.

3. Jab labels ho: directly performance track karo

Agar ground truth aata hai (chahe delayed ho), toh gold standard bas metric ko sliding window par recompute karna hai:

$Accuracy_{t} = \frac{1}{∣ W _{t} ∣} \sum_{(x, y) \in W_{t}} 1 [\overset{y}{^} (x) = y]$

Ground truth often delayed hota hai (kya loan default hoga? — mahine tak wait karo) ya expensive hota hai (human labels). Exactly isliye input/prediction drift ko early proxies ke roop mein use kiya jaata hai.

Figure — Model monitoring and observability

Worked Examples

Common Mistakes (Steel-manned)

Recall

Recall Active recall — cover the answers

Woh two-sample question kya hai jo saari drift detection ke neeche hai? ⇒ Kya recent samples reference ke same distribution se hain?
PSI exactly 0 tab hota hai jab…? ⇒ dono distributions identical hain.
Monitoring vs observability ek line mein? ⇒ Monitoring = "kya yeh toot gaya?"; observability = "kyun?"
Input/prediction drift ko proxies ke roop mein kyun use karte hain? ⇒ Ground truth delayed/expensive hota hai.
Data drift vs concept drift? ⇒ $P (X)$ change hoti hai vs $P (Y ∣ X)$ change hoti hai.

Recall Feynman: explain to a 12-year-old

Maano tumne apne dog ko laal ball fetch karne ki training di. Ab tum ek park mein jaate ho jahan sab neeli balls throw karte hain. Tumhara dog abhi bhi daudta hai, abhi bhi khush lagta hai — lekin hamesha galat cheez lekar aata hai, aur woh kabhi tumhe nahi batata ki woh confused hai. Model monitoring exactly yahi hai jaise dog par ek chhota camera lagana: tum dekhte ho kaunsi balls throw ho rahi hain (inputs), dog kya lekar aata hai (predictions), aur kabhi-kabhi check karte ho ki woh actually sahi hai ya nahi (labels). Agar balls neeli ho gayi, ek alarm bajta hai taki tum dog ko re-train kar sako isse pehle ki tumhara poora game kharaab ho.

Connections

Data drift aur concept drift mein kya farq hai?

Data drift = input distribution

P (X)

mein change; concept drift = input→output relationship

P (Y ∣ X)

mein change.

Population Stability Index formula define karo.

PSI = \sum_{i} (a_{i} - e_{i}) ln (a_{i} / e_{i})

bins ke over; 0 tabhi jab distributions identical hoon.

Kaunsa PSI value major drift signal karta hai?

PSI > 0.25 (0.1–0.25 = moderate, <0.1 = stable).

KS statistic kya measure karta hai?

D = sup_{x} ∣ F_{r e f} (x) - F_{n e w} (x) ∣

, do empirical CDFs ke beech ka sabse bada gap.

Monitoring vs observability?

Monitoring known metrics track karta hai ("kya yeh toot gaya?"); observability logs/metrics/traces se arbitrary nayi questions poochne deta hai ("kyun toot gaya?").

Input/prediction drift ko early warnings ke roop mein kyun use karte hain?

Kyunki ground-truth labels often delayed ya expensive hote hain, isliye real time mein direct performance measure nahi ho sakti.

Ek single global accuracy kyun misleading hai?

Yeh ek failing subgroup ko hide kar sakti hai; segment/feature bin ke hisaab se slice metrics dekhni chahiye.

Ek reason batao kyun drift ka matlab hamesha retrain nahi hota.

Drift ek broken data pipeline (nulls/wrong units) ho sakti hai ya ek harmless covariate shift jahan

P (Y ∣ X)

unchanged hai.

PSI hamesha non-negative kyun hai?

Har term

(a_{i} - e_{i}) ln (a_{i} / e_{i})

mein same sign ke factors hote hain, isliye product ≥0 hota hai.