Exercises — Model monitoring and observability

The inputs $X$ look the same, but the mapping from inputs to the correct label changed: the same features now map to a different truth. That is a change in $P(Y\mid X)$ = concept drift. Why this and not data drift? Data drift is $P(X)$ moving — the features themselves shifting. Here the features are unchanged; only the input→output relationship moved. See Data drift and concept drift.

$0.1\le 0.18\le 0.25$, which is the moderate band. Verdict: moderate shift — investigate, do not panic-retrain yet. The bands are: $<0.1$ stable, $0.1$–$0.25$ moderate, $>0.25$ major.

First = monitoring (a known metric against a known threshold — is it broken?). Second = observability (asking a new question of the logged data without shipping code — why is it broken?). See Logging, metrics and tracing.

Term by term, each is $(a_i-e_i)\ln (a_i/e_i)$:

• Bin 1: $(0.3-0.5)\ln (0.3/0.5)=(-0.2)\ln (0.6)=(-0.2)(-0.5108)=0.10216$.
• Bin 2: $(0.4-0.3)\ln (0.4/0.3)=(0.1)\ln (1.3333)=(0.1)(0.28768)=0.028768$.
• Bin 3: $(0.3-0.2)\ln (0.3/0.2)=(0.1)\ln (1.5)=(0.1)(0.405465)=0.0405465$.

Sum $=0.10216+0.028768+0.0405465=0.17147$. Why every term is positive: a bin that shrank has both factors negative (product positive); a bin that grew has both positive. Deviations can only add. Verdict: $0.171$ is in $[0.1,0.25]$ → moderate.

Each sample has 4 points, so each point contributes $1/4=0.25$ to its CDF. Evaluate the gap at every point where either CDF steps up (the sup of a step-function gap always lands at a data point):

The largest gap is at $x=4$: $D=0.5$. Why check only step points: between two data points both CDFs are flat, so the gap cannot grow there — the maximum must occur where one curve jumps. See figure below.

${\text{Accuracy}}_t=\frac{1}{|W_t|}\sum 1[\hat{y}=y]=\frac{17}{20}=0.85$. It exactly meets 0.85, so it is on the boundary — not failing, but with zero margin it deserves a closer watch.

• Feature A (continuous): KS statistic is natural — it compares the full empirical CDFs, and CDFs are defined for continuous, ordered values. PSI also works if you bin A, but binning throws away resolution.
• Feature B (categorical, 6 levels): PSI fits cleanly — the 6 categories are the bins, no ordering needed. KS needs an ordering to define a CDF, so it is awkward for unordered categories.
• Neither replaces performance tracking once labels arrive — drift tests are input-side proxies, not proof of harm. See KL divergence and cross-entropy for why PSI is a symmetrised relative-entropy cousin.

First case: $a_i=e_i$ in every bin, so every $(a_i-e_i)=0$ and every $\ln (a_i/e_i)=\ln 1=0$. Product is 0 in both bins → $\text{PSI}=0$ exactly. That is the only time PSI is 0. Second case: bins now differ, both terms become nonzero-positive products, so $\text{PSI}>0$ — it rises from 0. Concretely $\text{PSI}=(0.1)\ln 1.2+(-0.1)\ln 0.8=0.1(0.18232)+(-0.1)(-0.22314)=0.018232+0.022314=0.040546$.

$P(X)$ moved a lot (high PSI) but the model's accuracy held — strong evidence that $P(Y\mid X)$ is unchanged. This is harmless covariate shift: the input distribution shifted into a region the model still handles correctly. Do not retrain on the drift alone — accuracy is the ground truth and it is fine. Log the shift, keep watching in case labels later degrade. Retraining would spend effort (and risk) for no measured gain. See Model retraining and CI-CD pipelines.

Chain the clues. The global accuracy drop is real (b confirms it is concentrated, not spread). High PSI on income (a) is explained entirely by (c): a pipeline change made income null for DE, which both (i) makes the distribution look wildly shifted and (ii) breaks predictions for exactly those rows. Root cause = data quality / pipeline bug, not concept drift. Prescription: roll back or fix the pipeline so income populates again; do not retrain — training on null-poisoned data would bake the bug in. This mirrors the parent note's observability example. See Logging, metrics and tracing.

Layered proxies, fastest first:

1. Operational (seconds): latency, error rate, null-rate on features — catches pipeline breakage. Proves system health, not model correctness.
2. Input drift (daily): PSI / KS on each feature vs reference — catches $P(X)$ shifts. Proves inputs moved, not that predictions are wrong.
3. Prediction drift (daily): PSI on $\hat{Y}$ (e.g. default-rate the model predicts) — catches output shifts. Suggests something changed; cannot separate input-driven from concept drift.
4. Delayed performance (as labels trickle in): sliding-window accuracy on the 9-month-matured cohort — the only signal that proves real degradation, but arrives late. Rationale: layers 1–3 are cheap early warnings because ground truth (layer 4) is delayed and expensive. Consider A-B testing and shadow deployment to validate a candidate fix before full rollout.

(a) Each independent test raises a false alarm with probability $\alpha =0.05$. Expected false alarms per day $=40\times 0.05=2$. So 2 spurious "drift" alerts every single day, even in a perfectly stable system. (b) This is the multiple-comparisons problem: testing many features inflates the chance of some false positive. The probability of at least one false alarm is $1-(1-0.05{)}^{40}=1-0.95^{40}=0.8715$ — about 87% every day. Fixes: (i) Bonferroni-style tighten to $\alpha /40=0.00125$ per test; (ii) require drift to persist across several windows before paging; (iii) alert on effect size (KS $D$ or PSI magnitude) rather than p-value alone. This keeps engineers trusting the alarms instead of muting them.

Consider the two factors of a bin's term. The logarithm $\ln (a_i/e_i)$ is positive when $a_i>e_i$ (ratio $>1$), negative when $a_i<e_i$ (ratio $<1$), and zero when $a_i=e_i$. Crucially, $(a_i-e_i)$ carries the same sign in each case: positive when the bin grew, negative when it shrank, zero when equal. A product of two same-signed numbers is $\ge 0$. Therefore every term is $\ge 0$, and their sum PSI $\ge 0$. PSI $=0$ iff every term is 0, which requires $a_i=e_i$ in every bin — i.e. the two distributions are identical. Any single bin differing makes PSI strictly positive. (This is why PSI, like KL divergence, is a valid "distance-ish" score — see KL divergence and cross-entropy.)

• PSI is exactly 0 when? ⇒ every bin matches: $a_i=e_i$ for all $i$ (identical distributions).
• Where does the sup in the KS statistic always occur? ⇒ at a data point where one CDF steps up.
• High input PSI but flat accuracy means? ⇒ harmless covariate shift — don't retrain on drift alone.
• 40 features at $\alpha =0.05$ each gives how many false alarms/day? ⇒ about 2 expected (~87% chance of ≥1).
• Why layer proxies before performance? ⇒ ground truth is delayed/expensive; proxies warn early.