Worked examples — Data leakage identification and prevention

This page is the exhaustive drill. The parent topic note told you the three kinds of leakage and why they break a model. Here we hunt down every distinct shape that leakage can take — every "cell" — and work one example per cell until you have seen the whole landscape and can never be surprised again.

Before we start, two plain-words reminders so no symbol arrives unearned:

---

The scenario matrix

Every leakage bug is one of these cells. The examples below are labelled with the cell they cover, and together they touch all of them.

#	Cell class	The trap in one line	Example
A	Target leakage — direct	A feature is literally computed from $y$.	Ex 1
B	Target leakage — downstream	Feature is caused by an event that $y$ causes.	Ex 2
C	Temporal — future window	A rolling/aggregate feature peeks past time $t$.	Ex 3 (figure)
D	Preprocessing contamination	Scaler / imputer fit on train + test together.	Ex 4
E	Split contamination — duplicates/groups	Same entity lands in both train and test.	Ex 5
F	CV contamination	Leakage happening inside cross-validation folds.	Ex 6
G	Degenerate / zero case	Feature is constant, or split leaves a fold empty — silent failure.	Ex 7
H	Limiting case	Leak is tiny ($ϵ\to 0$) or total ($\to$ perfect proxy) — what score do you expect?	Ex 8
I	Real-world word problem	A messy business scenario mixing several cells.	Ex 9
J	Exam twist	Looks leaky but is clean; or looks clean but leaks.	Ex 10
K	Selection / tuning contamination	Feature selection or hyperparameter search run on the full dataset.	Ex 11

Prerequisite links you may want open while reading: Train-testvalidation split strategies, Cross-validation techniques, Feature scaling and normalization, Time series forecasting fundamentals, Causal inference basics.

---

Cell A — Target leakage, direct

Steps.

1. Ask the one rule: at prediction time (loan approval), does loss_amount_usd exist? Why this step? Legality is decided by timing, not by intuition. A loan that has not defaulted yet has no loss recorded.
2. Notice the deterministic link: loss_amount_usd $>0⟺y=1$. So the feature is $X_j=h(y)$ (recall $h(\cdot )$ = "a rule from one number to another") with $h(0)=0,h(1)>0$. Why this step? This is the direct form: the feature is a function of the label itself, not just correlated with it.
3. The model learns the trivial rule $\hat{y}=[X_j>0]$. With a perfect proxy present it copies it and scores near-perfectly on training data. Why this step? Any learner minimises empirical risk by copying a feature that already equals the answer; the ceiling is $100\%$, and any real dataset with a whisker of noise lands just below it — say $99\%$.
4. In production loss_amount_usd is always $0$ (loan just approved), so the model always predicts $\hat{y}=0$. Accuracy collapses to the base rate of the majority class.

Verify: Suppose $8\%$ of loans default. Always-predict-repaid accuracy $=1-0.08=0.92$. So a training model that scored $0.99$ (essentially the $100\%$ ceiling minus noise) becomes a $0.92$ dummy — impressive-looking but learning nothing. The gap $0.99-0.92=0.07$ is the leakage tax made visible.

---

Cell B — Target leakage, downstream (caused by an event $y$ causes)

Steps.

1. Draw the chain: $y(\text{fraud})\to \text{customer notices}\to \text{calls bank}$. So the feature is downstream of $y$. Why this step? Causal inference basics tells us: features on the effect side of the target are unavailable at the instant we must predict, because the effect hasn't happened yet.
2. Timing test: at swipe time (prediction time $t$), has the customer called about this transaction? No — the transaction is happening now. Why this step? Same one rule. Downstream leakage is sneaky because the feature isn't literally $y$, it's a consequence, so it survives a naive "is this the label?" check.
3. Fix: only use features whose value is fixed by time $t$ — amount, merchant_category, hour, historical_fraud_rate_of_merchant. Why this step? Tie it back to the one rule through causality: the causal arrow runs $y\to \text{call}$, so the call can only take its value after $y$ is decided. "After $y$" means "after prediction time," and anything that only exists after prediction time cannot be computed at prediction time — hence it is illegal. Cause-side and contemporaneous features carry no such arrow from $y$, so they stay legal.

Verify (sanity): Ask "could a fraudster's model compute this feature before the swipe?" num_customer_calls_this_week — no, the call is triggered by the fraud. Rejected. hour_of_day — yes, known instantly. Accepted. The test is decidable and consistent.

---

Cell C — Temporal leakage, future window

Figure, in words (so you can follow without seeing it): the horizontal axis is day relative to prediction time, labelled $t-4,t-3,\dots ,t+3$; the vertical axis is the sales value. Each day is a dot. Blue dots ($t-4$ to $t-1$) are known past. The red dot at $t$ is today — the value we are trying to predict — and a red dashed vertical line marks the "now" boundary through it. Orange dots ($t+1$ to $t+3$) are the future, which does not exist yet. A grey double-headed arrow across the top spans $t-2\dots t+2$ and is labelled "centred window (peeks future!)" — notice it crosses the red line into orange territory. A separate green double-headed arrow near the bottom spans $t-4\dots t-1$, labelled "trailing legal window" — it stays entirely left of the red line.

Steps.

1. Look at where the grey (centred) arrow sits relative to the red "now" line: it crosses to the right, into the orange dots. Why this step? Temporal leakage is inherently geometric — it's about which side of the "now" line a data point sits on. Crossing the line = using the future.
2. A centred window $[t-2,t+2]$ reaches into $t+1,t+2$ (orange). Those values do not exist yet at time $t$. Why this step? The average ${\bar{X}}_t=\frac{1}{5}{\sum }_{i=t-2}^{t+2}{X}_i$ literally contains future terms.
3. Split the sum to expose the leak: ${\bar{X}}_t=\underset{\text{legal, }\le t}{\underbrace{\frac{1}{5}(X_{t-2}+X_{t-1}+X_t)}}+\underset{\text{illegal, future}}{\underbrace{\frac{1}{5}(X_{t+1}+X_{t+2})}}$ Why this step? Making the illegal terms explicit shows exactly what to delete.
4. Fix: use a trailing window that touches only the past (the green arrow). To keep the arithmetic honest below, we take the trailing window over the same left-shoulder days we can actually see, $t-2,t-1,t$ (i.e. history up to and including today). See Time series forecasting fundamentals.

Verify (numbers): Take sales $[10,12,14,16,18]$ for days $t-2\dots t+2$. Centred 5-day mean $=(10+12+14+16+18)/5=14$. The legal trailing mean over exactly the three past-and-present days $t-2,t-1,t$ is $(10+12+14)/3=12$. Both are 3-vs-5-day means computed on the same number series, so the comparison is apples-to-apples: they differ ($14\ne 12$), and the difference $14-12=2$ is purely the future's influence. That non-zero gap is the leak.

---

Cell D — Preprocessing contamination

Let $n$ = the total number of rows the scaler is fitted on, $n_{\text{tr}}$ = training rows, $n_{\text{te}}$ = test rows, so $n=n_{\text{tr}}+n_{\text{te}}$.

Steps.

1. The scaler computes $\mu =\frac{1}{n}{\sum }_{i=1}^n{X}_i$ and $\sigma$, then outputs $X^{\prime }=\frac{X-\mu }{\sigma }$. Why this step? $\mu$ and $\sigma$ are learned parameters. Learning them from test rows lets test information touch training features.
2. Fit-on-all mean mixes both groups: ${\mu }_{\text{all}}=\frac{n_{\text{tr}}{\mu }_{\text{tr}}+n_{\text{te}}{\mu }_{\text{te}}}{n_{\text{tr}}+n_{\text{te}}}$ Why this step? It shows ${\mu }_{\text{all}}$ depends on test rows — the contamination in one line.
3. Fix: scaler.fit(X_train) only, then scaler.transform(X_test). Even cleaner: put the scaler inside a pipeline so it can never accidentally see test data.

Verify (numbers): Train sqft $[1000,1500,2000]$ (${\mu }_{\text{tr}}=1500$), test $[3000,3500,4000]$ (${\mu }_{\text{te}}=3500$), equal sizes. Fit-on-all mean $=(1500+3500)/2=2500\ne 1500$. A legal train point $1000$ scaled with the contaminated mean uses $2500$ instead of $1500$ — the numerator $1000-2500=-1500$ instead of the honest $1000-1500=-500$. Wrong by a factor of $3$.

---

Cell E — Split contamination via duplicates / groups

Steps.

1. Rows from one patient are near-duplicates (same physiology). Random split can put visit 1 in train and visit 2 in test. Why this step? The independence assumption behind Train-testvalidation split strategies is between entities, not rows.
2. The model effectively memorises the patient during training and is graded on them in test — an unfair peek. Why this step? Memorising an entity then scoring on that same entity inflates the test score, exactly the "study with the exam answers" failure the parent note warns about.
3. Fix: grouped split — every row of a patient goes entirely to one side (GroupShuffleSplit). Why this step? Forcing whole entities to one side restores true independence between train and test, so the test score measures generalisation to new patients.

Verify (count): With $100$ patients $\times 3$ visits $=300$ rows: under a row-level $80/20$ split, the chance that a given patient has all three of its rows land on the same side is $0.8^3+0.2^3=0.512+0.008=0.52$, so the chance it is split across both sides is $1-0.52=0.48$. Nearly half of all patients leak. A grouped split makes this probability exactly $0$.

---

Cell F — Contamination inside cross-validation

Steps.

1. In $k$-fold CV, each fold takes turns being the validation slice. If scaling happened before the loop, every validation fold already influenced $\mu ,\sigma$. Why this step? CV simulates deployment $k$ times; any pre-loop fitting quietly contaminates all $k$ simulations.
2. Correct order: inside each fold, fit preprocessing on the $k-1$ training folds only, transform the held-out fold. A pipeline passed to cross_val_score does this automatically. Why this step? Re-fitting per fold means the held-out slice never touches the statistics used to transform it — each fold becomes an honest miniature of production.
3. Consequence of getting it wrong: CV scores are optimistic, so your model looks better than it is. Why this step? Naming the concrete symptom (inflated CV score) tells you what to watch for — a CV score you cannot reproduce on truly fresh data is the fingerprint of this bug.

Verify (magnitude): With $k=5$, each validation fold is $20\%$ of data. Pre-loop fitting means $\mu$ is computed from $100\%$ instead of the honest $80\%$ — every fold's held-out $20\%$ leaks into its own $\mu$. Fraction of contaminated statistic per fold $=0.20\ne 0$.

---

Cell G — Degenerate / zero case

Steps.

1. (a) Constant feature. $\sigma =0$, so scaling $\frac{X-\mu }{\sigma }$ divides by zero. Phenomenon: zero-variance no-op — the feature carries no information ($\text{Var}=0$) and the transform is undefined. Not leakage: no external data enters training. Why this step? Separating the two phenomena is the whole point of this cell — a divide-by-zero is a numeric crash, categorically different from information flowing across the now-line.
2. (b) Singleton class. With $1$ positive example, a stratified split cannot place it in both train and validation. Phenomenon: empty-fold instability — one side ends up with $0$ positives, making recall $=\text{TP}/(\text{TP}+\text{FN})=0/0$ undefined. Not leakage: again, no future/test information reaches training; the metric is simply undefined. See Model evaluation metrics. Why this step? Distinguishing "undefined metric" from "inflated metric" matters: leakage inflates scores, degeneracy makes them undefined or crashing — opposite symptoms.
3. Fix: drop zero-variance columns; for tiny classes, resample or report the metric's instability honestly rather than trusting one undefined number. Why this step? Removing information-free columns kills the divide-by-zero, and flagging singleton-class instability stops you from believing a $0/0$ metric.

Verify (numbers): Constant column $[7,7,7,7]$: variance $=\frac{1}{4}\sum (7-7{)}^2=0$ (transform undefined). Recall with $0$ true positives available: denominator $\text{TP}+\text{FN}=0\Rightarrow$ undefined. Both are degenerate, neither is leakage.

---

Cell H — Limiting cases

Steps.

1. $ϵ\to 0$ (total leak): the noise vanishes, so $X_j=h(y)$ is a perfect encoding of $y$. Training accuracy $\to 100\%$; the model is just reading the answer. Why this step? This is the upper limit — the worst, most flattering case, and it names the ceiling every leak approaches.
2. $ϵ\to \infty$ (leak drowned in noise): the $h(y)$ signal is swamped, so $X_j$ is effectively pure noise, carries no information about $y$, and accuracy $\to$ base rate (predict the majority class). Why this step? The lower limit shows leakage severity is continuous, not binary — real bugs live between these extremes.
3. Interpolation: severity rises smoothly as the encoding $h(y)$ dominates the noise $ϵ$, matching the parent note's "$\Delta$ is unbounded" claim — bigger signal-to-noise ratio $\Rightarrow$ accuracy nearer the $100\%$ ceiling. Why this step? Tying severity to the same symbols $h$ and $ϵ$ makes the middle predictable: you can estimate how bad a partial leak is from how cleanly the feature tracks $y$.

Verify (numbers): Base rate $30\%$ positives. Perfect proxy ($ϵ\to 0$) $\Rightarrow$ acc $=1.00$. Pure noise ($ϵ\to \infty$) $\Rightarrow$ acc $=\max (0.30,0.70)=0.70$. So the observed accuracy must lie in $[0.70,1.00]$; anything above $0.70$ that vanishes in production is exactly the leak signature.

---

Cell I — Real-world word problem (mixes several cells)

Steps.

1. cancellation_ticket_opened — churning causes the ticket. Cell B, downstream target leak. Drop it. Why this step? The causal arrow $y\to \text{ticket}$ means the ticket only exists after $y$, i.e. after prediction time (start of month) — so it is unavailable and illegal.
2. avg_monthly_spend averaged over the whole year includes the churn month itself. Cell C, future window. Recompute using only months $<t$. Why this step? Any average reaching into or past the prediction month contains values that do not exist yet at the month's start.
3. Whole-data MinMax scaling. Cell D, preprocessing contamination. Fit the scaler on train only, inside a pipeline. Why this step? MinMax's min and max are fitted statistics; fitting them on test rows leaks test information into training features exactly as in Cell D.
4. Same customer across months → use a grouped, time-ordered split (Cell E): earlier months train, later months test, grouped by customer. Why this step? Grouping by customer stops one person straddling both sides, and ordering by time stops future months training a model scored on the past.

Verify (severity estimate): Base churn rate $=5\%$, so an always-predict-stay dummy scores $1-0.05=0.95$. The direct-ish ticket feature alone can push training accuracy to $\approx 0.99$; after removing it and the future-average, an honest model must still beat $0.95$ to be worth anything, and its production accuracy should sit within a few points of its CV accuracy. That closing gap $|R_{\text{prod}}-\hat{R}|\to \text{small}$ is the proof the fixes worked.

---

Cell J — Exam twist (looks leaky but isn't / vice versa)

Steps.

1. (a) Clean. age_at_signup is fixed at signup, fully known at prediction time $t$. Correlation with $y$ is fine — leakage is about timing/causality, not about being predictive. Why this step? The classic exam trap: students reject useful features just because they're correlated. Correlation is the goal, not the crime.
2. (b) Leaky. The median used for imputation is a fitted statistic; computing it over train + test is Cell D contamination in disguise. Why this step? Imputation is preprocessing, so it obeys the same fit-on-train-only rule as scaling. See Production model deployment — in production you impute with the stored training median.

Verify (numbers): Train credit_score $=[600,650,700]$, median $=650$. Full-data (with test $[800,850,900]$) median of $[600,650,700,800,850,900]=(700+800)/2=750\ne 650$. Imputing a missing training value with $750$ instead of the honest $650$ leaks $750-650=100$ points of test information. Bug confirmed.

---

Cell K — Selection / tuning contamination

Steps.

1. Feature selection on all data. Ranking features by correlation with $y$ uses the labels of the test rows to decide which columns survive. Why this step? The selection decision is a fitted quantity — it depends on test $y$. That decision then shapes the training features, so test information flows backward exactly like a contaminated $\mu$ in Cell D.
2. Hyperparameter search on all data. Picking the setting with the best full-dataset score lets the model config be tuned to fit the test rows. Why this step? The chosen hyperparameters are also fitted quantities; optimising them against data that includes the test set is a subtler train-test contamination — the "study with the exam answers" failure applied to model configuration.
3. Fix: split first. Do feature selection and hyperparameter search only inside training folds — nest them inside CV so the held-out fold never influences which features or settings are chosen. Why this step? Nesting restores the one rule at the level of decisions, not just feature values: nothing derived from the held-out data may touch what the model is or which columns it reads.

Verify (magnitude): With a single $80/20$ split, selecting features on all data means the test rows contribute a fraction $20\%=0.20\ne 0$ of the correlations that decide the feature set — a non-zero, illegal influence. Nested CV drives that contribution to exactly $0$.

---

Recall Self-test

Which cell: avg_of_5_days_centred_on_today ::: C — temporal future window. Which cell: scaler fit on train+test ::: D — preprocessing contamination. Which cell: same user's rows split randomly ::: E — group/duplicate split contamination. Which cell: loss_amount when predicting default ::: A — direct target leakage. Which cell: feature caused by an event the label triggers ::: B — downstream target leakage. Which cell: top-10 features chosen by correlation before splitting ::: K — selection/tuning contamination. Is a highly correlated but time-legal feature leaky? ::: No — correlation is allowed; only bad timing/causality leaks. Constant column and singleton-class split — are these leakage? ::: No — they are degenerate failures (undefined/crashing), not future peeking. How does perfect-proxy accuracy compare to base rate? ::: Perfect proxy $\to 100\%$; base rate is the floor when the leak is pure noise. What does $h(y)$ mean? ::: A rule that manufactures a feature value out of the answer $y$ — the hallmark of a target leak.