Visual walkthrough — Model monitoring and observability

We build one running example the whole way down: yesterday's users vs today's users, sorted into three age buckets.

Step 1 — Two piles of data become two bar charts

WHAT. We have two batches of a single number (say user age): the reference batch (the data the model trained on) and the new batch (data arriving live today). We chop the number-line into some number of side-by-side bins (buckets), and count how many points fall in each bin. We call that number of bins $B$.

WHY. We cannot compare two clouds of raw numbers directly — there is no single "distance" between two heaps. But if we turn each heap into a bar chart (a histogram), we get $B$ heights we can compare bin-by-bin. Binning is the trick that converts "compare two distributions" into "compare $B$ pairs of numbers."

PICTURE. Two histograms over the same three age bins (so here $B=3$). The pale-yellow bars are yesterday (reference), the blue bars are today (new). Notice they disagree — the young bin shrank, the old bin grew.

Step 2 — Turn counts into fractions so size doesn't fool us

WHAT. Divide every bin's count by the total number of points in its batch. Now each bar is a fraction between $0$ and $1$, and the bars of one batch sum to $1$.

$e_i=\underset{\text{fraction of reference in bin }i}{\underbrace{\frac{\text{count in bin }i}{\text{total reference points}}}},a_i=\underset{\text{fraction of new in bin }i}{\underbrace{\frac{\text{count in bin }i}{\text{total new points}}}}$

• $e_i$ — the expected fraction (reference), the height of bin $i$ in yesterday's chart.
• $a_i$ — the actual fraction (new), the height of bin $i$ in today's chart.
• the subscript $i$ — "which bin", running $1,2,\dots ,B$.

WHY. Yesterday we might have had $10,000$ users and today only $500$. Raw counts would scream "everything shrank!" even if the shape is identical. Dividing by the total strips away batch size and leaves pure shape — the thing drift actually cares about. After this step both charts are on the same $0$-to-$1$ scale.

PICTURE. The same two histograms, now rescaled: each colour's bars add up to a full block of height $1$. Only the shape difference remains.

Step 3 — The first honest attempt: the raw gap $a_i-e_i$

WHAT. For each bin, subtract yesterday's fraction from today's fraction: $d_i=a_i-e_i$

• $d_i>0$ — the bin grew (today has a bigger share here).
• $d_i<0$ — the bin shrank.
• $d_i=0$ — the bin is unchanged.

WHY. This is the most obvious "how different?" measure — literally the height difference of the two bars. It is our starting sketch. We will see it has a flaw, and fixing that flaw builds the rest of PSI.

PICTURE. For each bin, an arrow from the yellow bar top to the blue bar top: up-arrows (grew, pink) and down-arrows (shrank). The lengths are the raw gaps $d_i$.

Step 4 — A relative gap: why we reach for the logarithm

WHAT. Look not at the plain gap but at the ratio $a_i/e_i$, then take its natural logarithm: $r_i=\ln \frac{a_i}{e_i}$

• $\frac{a_i}{e_i}$ — "how many times bigger is today's share than yesterday's?" A ratio of $2$ means doubled; $0.5$ means halved.
• $\ln$ — the natural logarithm, the tool that turns "times bigger" into a symmetric, additive scale.

WHY this tool and not another? A plain gap treats $0.01\to 0.02$ (a share that doubled) the same as $0.50\to 0.51$ (barely nudged) — both have gap $0.01$. But doubling a tiny bin is a big relative event. We want a measure of "how many-fold did this move", and that is exactly what a ratio answers. We then wrap it in $\ln$ because:

• $\ln (1)=0$ — no change ($a_i=e_i$) gives zero, as it should.
• $\ln$ of "doubled" ($=+0.69$) and $\ln$ of "halved" ($=-0.69$) are equal in size, opposite in sign — growth and shrinkage by the same factor are treated even-handedly. Plain ratios don't do this ($2$ vs $0.5$ look lopsided).

This is the same log-of-a-ratio idea behind KL divergence and cross-entropy — it is the natural currency for "distance between two probability shapes."

PICTURE. The log-ratio curve $\ln (a_i/e_i)$ plotted against the ratio $a_i/e_i$: crossing zero at ratio $1$, symmetric on a log x-axis, one bin marked as "doubled" (positive) and one as "halved" (negative).

Step 5 — Multiply gap × log-ratio: the sign trick that stops cancellation

WHAT. Combine the two pieces we built — the raw gap $d_i=(a_i-e_i)$ and the log-ratio $r_i=\ln (a_i/e_i)$ — by multiplying them per bin: ${\text{PSI}}_i=(a_i-e_i)\ln \frac{a_i}{e_i}$

• $(a_i-e_i)$ — how much the share moved (the size).
• $\ln (a_i/e_i)$ — how many-fold it moved (the direction & relative scale).

WHY multiply, and why this kills cancellation. Watch the two factors move together:

Both factors always share the same sign (a bin that grew has a bigger share AND a ratio above $1$; a bin that shrank has both below). So their product is never negative. This is the elegant fix for Step 3's cancellation problem: instead of adding signed gaps that cancel, we add per-bin products that are all $\ge 0$. Every deviation now contributes, none cancel.

PICTURE. For the grown bin and the shrunk bin, a little "sign multiplication" box: $(-)\times (-)=+$ and $(+)\times (+)=+$, both arrows pointing up into a positive pile.

Step 6 — Sum over all bins: one number for the whole shift

WHAT. Add up every bin's non-negative contribution: $\boxed{\text{PSI}=\sum _{i=1}^B(a_i-e_i)\ln \frac{a_i}{e_i}}$

• ${\sum }_{i=1}^B$ — "walk across all $B$ bins and add", where $B$ is the number of bins we defined in Step 1.
• the boxed result is a single number $\ge 0$, equal to $0$ only when every bin matches ($a_i=e_i$ for all $i$).

WHY sum. Each bin told us its own drift. Drift of the whole feature is the total, because a distribution moved a lot if it moved a lot somewhere. Summing non-negative pieces means drift in one bin can never be hidden by "un-drift" elsewhere.

PICTURE. A stacked bar: each bin's positive contribution stacked on top of the last, the final height labelled PSI. Threshold lines at $0.1$ (green, stable) and $0.25$ (pink, act) drawn across.

Step 7 — The degenerate case: an empty bin blows the log up

WHAT. Suppose a bin has $a_i>0$ today but $e_i=0$ in the reference (a brand-new region never seen in training). Then $\ln \frac{a_i}{e_i}=\ln \frac{a_i}{0}=\ln (\infty )=+\infty$ and PSI explodes to infinity. The mirror case $a_i=0$, $e_i>0$ gives $\ln (0)=-\infty$.

WHY it matters and the fix. A zero in a denominator is a division-by-zero — mathematically undefined, numerically a inf or nan. But an empty bin is real: rare categories, first-day-of-a-launch data. The standard fix is to add a tiny floor $\epsilon$ (e.g. $0.0001$) to every fraction, so no bin is ever exactly $0$: $e_i\leftarrow \max (e_i,\epsilon ),a_i\leftarrow \max (a_i,\epsilon )$ Because we nudged the shares, they no longer add up to exactly $1$. So we re-normalise afterwards — divide every floored fraction by its (new) batch total — restoring ${\sum }_i{e}_i=1$ and ${\sum }_i{a}_i=1$. The distributions are not corrupted; the floor only prevents an exact zero, and the re-normalisation keeps both bars honest probability shapes. This keeps PSI finite while still reporting a large value (a genuinely new bin should scream drift). This is exactly the "data validation" concern of Feature stores and data validation.

The both-empty sub-case. If a bin is empty in both batches ($a_i=e_i=0$), after flooring both become $\epsilon$, so $(a_i-e_i)\ln \frac{a_i}{e_i}=(\epsilon -\epsilon )\ln \frac{\epsilon }{\epsilon }=0\cdot \ln (1)=0.$ Good: a bin that never had anyone, before or after, contributes nothing — exactly the "no drift here" answer we want.

PICTURE. Left panel: the log-ratio curve shooting to $+\infty$ as $e_i\to 0$. Right panel: the same curve with the $\epsilon$ floor, capped at a large-but-finite value.

The one-picture summary

Everything at once: raw counts → fractions → per-bin (gap × log-ratio) → summed to one PSI, with the threshold verdict.

Connections

• Data drift and concept drift
• KL divergence and cross-entropy
• Feature stores and data validation
• Model retraining and CI-CD pipelines
• A-B testing and shadow deployment
• Logging, metrics and tracing