Foundations — Model monitoring and observability

Before you can read a formula like $\text{PSI}={\sum }_i(a_i-e_i)\ln (a_i/e_i)$, you need to earn every piece of it: what a distribution is, what a proportion is, what $\ln$ does, what "sum over bins" means. We build them one at a time, each on top of the last. A smart 12-year-old should never hit a symbol we didn't first draw.

0. The raw ingredients: data, features, labels

The picture: imagine a spreadsheet. Each row is a data point. The coloured columns are features $X$. The final column is the true label $Y$. The model reads the coloured columns and scribbles its guess $\hat{Y}$ in the margin.

Why the topic needs it: monitoring watches three different things — the incoming $X$ (data drift), the outgoing $\hat{Y}$ (prediction drift), and the eventual truth $Y$ (performance). If you can't tell these three apart, every later sentence collapses.

1. Distribution — the shape of "how often"

If you don't know what a distribution is, the words "data drift" mean nothing. So we build it visually first.

Look at the figure. The horizontal axis is the feature value (income). Each vertical bar is a bin — one bucket. The taller the bar, the more common that range of values.

2. Proportion — turning counts into fractions

Raw counts are unfair: a reference batch of 10,000 rows and a new batch of 500 rows can't be compared bar-height to bar-height. So we convert counts into proportions.

The picture: take the bar chart from figure s01 and re-label every bar height as a fraction of the whole. Now the reference chart and the new chart are on the same footing — you can lay one over the other.

Why the topic needs it: PSI compares $a_i$ against $e_i$ bin by bin. Without proportions the comparison would be dominated by whichever batch is bigger.

3. The subtraction $a_i-e_i$ — "did this bin grow or shrink?"

Why not stop here? You might think "just add up $|a_i-e_i|$ and call it drift." That's a real measure (it's called total variation). But it treats a shift from 40% to 20% the same as 4% to 2% — yet the second is a proportionally bigger upheaval. We need a tool that notices ratios, not just gaps. Enter the logarithm.

4. The logarithm $\ln$ — measuring how many times bigger

We chose the log-ratio because it answers a specific question: "by what multiplicative factor did this bin change?" A gap subtracts; a ratio divides; the log turns that ratio into a clean, sign-aware number.

5. The summation $\sum$ — collecting every bin into one score

We have one number per bin. We want one number for the whole feature. The tool that "adds up a list" is the summation sign.

See Data drift and concept drift for how this number is read in practice, and KL divergence and cross-entropy for why this exact log-ratio form is a symmetrised relative entropy.

6. The cumulative distribution $F(x)$ and the supremum $\sup$

PSI needs you to choose bins. For a smooth continuous feature, choosing bins is arbitrary. The KS statistic dodges that by comparing cumulative curves instead.

In the figure, two staircases climb the same box. Where the new data sits further right, its staircase lags behind — and the biggest vertical gap between the two curves is marked by the yellow arrow.

7. The indicator $1[\cdot ]$ and windowed accuracy

When true labels $Y$ eventually arrive, the gold-standard check is simply "is the model still right?" — measured on a recent window of data.

8. The probability symbols $P(X)$, $P(Y\mid X)$, $P(\hat{Y})$

These three appear in the parent's failure table. They look scary but say simple things.

Prerequisite map

Equipment checklist

Cover the right side and test yourself — if any answer is fuzzy, re-read that section before tackling the parent topic.

Connections

• Data drift and concept drift
• KL divergence and cross-entropy
• Model retraining and CI-CD pipelines
• Feature stores and data validation
• Logging, metrics and tracing
• A-B testing and shadow deployment