Model monitoring and observability: Question bank

Before you start, three words you must not blur:

Data drift = the inputs $P (X)$ change (what the world sends you).
Concept drift = the rule $P (Y ∣ X)$ change (the right answer for the same input changes).
Prediction drift = the outputs $P (\hat{Y})$ change (what the model emits).

The three drifts on one map (so the questions make sense)

The whole question bank keeps asking "which drift is this, and do you need labels?" — so pin the map down first, before any trap.

Figure — Model monitoring and observability

The PSI number — defined once, used everywhere below

Many traps quote "PSI", " $e_{i}$ ", " $a_{i}$ " and thresholds like $0.1$ and $0.25$ . Here is the single source of truth.

Why the KS statistic takes the largest gap — see it once

True or false — justify

A deployed model that returns HTTP 200 for every request is healthy.

False — the whole point of monitoring is that a model silently degrades while still returning 200s; correct HTTP status says nothing about correct predictions.

If PSI

= 0

, the two distributions are identical.

True — every term is

(a_{i} - e_{i}) ln (a_{i} / e_{i})

, and both factors vanish exactly when

a_{i} = e_{i}

for every bin, so the only way the sum is 0 is bin-by-bin equality.

PSI can be negative if a bin shrinks a lot.

False — when a bin shrinks,

(a_{i} - e_{i}) < 0

and

ln (a_{i} / e_{i}) < 0

, so their product is positive; PSI is a sum of non-negative terms and is always

\geq 0

Data drift always causes a drop in accuracy.

False —

P (X)

can move into a region where the learned mapping

P (Y ∣ X)

still holds (harmless covariate shift); accuracy only suffers if the new inputs land where the model was wrong or the true rule also changed.

Concept drift can occur even when the input distribution

P (X)

is perfectly stable.

True — the same features can arrive at the same frequencies while the correct label for them changes (e.g. the definition of "spam" evolves); that is drift in

P (Y ∣ X)

with

P (X)

frozen.

Prediction drift proves concept drift has happened.

False — the model's outputs shift whenever inputs shift, even with the mapping unchanged; distinguishing the two requires labels to measure

P (Y ∣ X)

The KS statistic

D = sup_{x} ∣ F_{ref} (x) - F_{new} (x) ∣

is a whole-distribution comparison, not just a mean comparison.

True — it is built on the cumulative distribution

F

, so a shift in spread, skew or shape (not only the mean) produces a gap; two distributions with equal means but different shape still give

D > 0

A high offline test score guarantees the model is safe in production.

False — the test set is sampled from yesterday's distribution; it certifies the past pattern was learned, not that tomorrow's inputs obey the same distribution.

Monitoring and observability are the same thing with different names.

False — monitoring tracks pre-chosen metrics against thresholds ("is something wrong?"); observability is the property that lets you ask new, unplanned questions from logs/metrics/traces ("why is it wrong?") without shipping code.

A single global accuracy metric is sufficient to catch model failures.

False — a healthy average can hide a failing subgroup (Simpson's-paradox style); you must monitor sliced metrics per segment and per feature bin.

KS as defined applies naturally to a categorical feature like "country".

False — KS needs an ordered variable to build a CDF; categories have no natural order, so use PSI over category bins or a chi-square-style comparison instead.

Retraining is the correct response to any detected drift.

False — drift may be a broken pipeline (nulls, wrong units); retraining on corrupted inputs makes the model worse. Confirm real performance loss and rule out data-quality bugs first.

Spot the error

"I compute PSI, get

0.05

, so I retrain immediately."

PSI

0.05 < 0.1

is the stable band — no action needed. Retraining here wastes effort and risks introducing regressions for no benefit.

"Ground truth is delayed for months, so I can't monitor anything until it arrives."

You can monitor input drift and prediction drift right now as early proxies; they exist precisely because labels are delayed or expensive.

"Predictions for 'fraud' jumped 5×, therefore the fraud concept changed — I'll relabel data."

A 5× jump in

P (\hat{Y})

can come purely from an input shift with the mapping intact; before relabelling, check whether the input distribution moved and whether labelled performance actually dropped.

"KS gave a tiny

D

, so the batch is fine even though its size is only 8 samples."

With very few samples the empirical CDF is coarse and noisy; a small

D

may just mean the test lacks power. The KS p-value (which accounts for sample size), not

D

alone, decides significance.

"Data quality is a separate team's job, not part of model monitoring."

Nulls, unit changes and pipeline breaks show up as input drift and directly poison predictions; data validation (Feature stores and data validation) is a first-class part of the monitoring surface, not external to it.

"Accuracy is stable overall, so no segment can be broken."

A rising segment can mask a collapsing one within the same average; only sliced metrics reveal a subgroup failing while the global number looks calm.

"I'll set my PSI alert threshold to

0.5

to avoid false alarms."

The action band starts at

0.25

(major drift); waiting for

0.5

means the distribution has already moved catastrophically before you're warned — you've traded false alarms for missed real failures.

Why questions

Why is PSI a sum of a product

(a_{i} - e_{i}) ln (a_{i} / e_{i})

rather than just

\sum ∣ a_{i} - e_{i} ∣

The product makes every deviation contribute a positive amount (both factors share sign) and weights it by the log-ratio, so a bin doubling counts more than a bin nudging — it encodes relative, not just absolute, movement, echoing KL divergence and cross-entropy. Numerically, a bin going

0.2 \to 0.4

gives

(0.2) ln 2 = 0.139

while a

0.02 \to 0.04

bin gives only

(0.02) ln 2 = 0.014

— ten times smaller, though both "doubled".

Why take the supremum (largest) gap in the KS statistic instead of an average gap over

x

The sup captures the worst-case disagreement between the two shapes (see the tallest red bar in figure s03); an average could dilute a sharp local mismatch, whereas the largest vertical CDF gap is the single most damning evidence of a shift.

Why are input and prediction drift used as proxies for performance rather than the real thing?

True performance needs ground-truth labels, which are often delayed (did the loan default?) or expensive (human annotation); input/prediction drift are cheap, immediate early-warning signals available before labels arrive.

Why does observability need logs, metrics and traces rather than just an accuracy dashboard?

A dashboard answers only pre-planned questions; when accuracy drops you must slice by a new dimension (e.g. country=DE) and follow a request through the pipeline — that arbitrary drill-down needs the raw logged and traced signals.

Why can retraining make a model worse when the "drift" is actually a broken pipeline?

The pipeline is feeding corrupted features (nulls, wrong units); retraining fits the model to that garbage, baking the fault into the weights instead of fixing the data source.

Why separate

P (\hat{Y})

drift from

P (Y ∣ X)

drift at all?

They demand different fixes: output-only drift from shifted inputs may need no change or just calibration, whereas a genuine

P (Y ∣ X)

shift means the true rule changed and the model must be retrained — and you can only distinguish them with labels.

Edge cases

What does PSI do if an actual bin has

a_{i} = 0

while the reference

e_{i} > 0

ln (0/ e_{i}) = ln 0 = - \infty

, so the raw term blows up; in practice bins are smoothed (add a tiny

ϵ

) or merged so an empty bin doesn't produce an infinite, meaningless PSI.

What is KS

D

if the reference and new samples are identical?

D = 0

— the two empirical CDFs coincide everywhere, so the largest gap is zero, signalling no detectable shift.

What is the maximum possible value of the KS statistic

D

D = 1

, reached when the two distributions have completely disjoint supports (one entirely below the other), so the CDF gap hits the full height of 1 somewhere.

P (X)

and

P (Y ∣ X)

are both perfectly stable but latency triples, is that a monitoring concern?

Yes — operational/system health (latency, errors, throughput) is one of the four failure families; a correct-but-slow model can still breach SLAs and harm users, so it must be alerted on.

Two distributions have the same mean but very different variance — does KS catch it?

Yes — the CDFs cross but diverge in the tails, producing a nonzero sup gap; KS is shape-sensitive, which is exactly why it's not just a mean test.

Concept drift arrives but you have no labels at all, ever — can you detect it?

Not directly, since

P (Y ∣ X)

needs labels; you can only watch input/prediction drift as weak proxies and lean on A-B testing and shadow deployment or manual audits to gather even sparse ground truth.

Drift is detected but performance on the freshly-labelled window is unchanged — do you act?

No retraining is warranted; harmless covariate shift moved the inputs into a region the model still handles well. Log it, keep watching, but changing a working model risks regression.

Recall

Recall One-line self-test

PSI sign guarantee? ⇒ Always $\geq 0$ ; each term is a product of same-signed factors.
PSI thresholds? ⇒ $< 0.1$ stable, $0.1$ – $0.25$ moderate, $\geq 0.25$ act.
KS range? ⇒ $0$ (identical) to $1$ (disjoint supports).
Prediction drift ⇒ concept drift? ⇒ No — needs labels to confirm $P (Y ∣ X)$ changed.
First response to detected drift? ⇒ Rule out data-quality bugs and confirm real performance loss, not blind retrain.
Why proxies at all? ⇒ Labels are delayed/expensive.

Connections

What is the difference between data drift and concept drift?

Data drift = change in input distribution

P (X)

; concept drift = change in the input→output relationship

P (Y ∣ X)

Which drifts can be detected without labels?

Data drift (

P (X)

) and prediction drift (

P (\hat{Y})

); concept drift (

P (Y ∣ X)

) always needs ground-truth labels.

What PSI value signals major drift?

PSI

\geq 0.25

(0.1–0.25 = moderate, <0.1 = stable).