Exercises — Data leakage identification and prevention

Answer: (c) chargeback_amount — target leakage.

WHY: A chargeback only exists because a fraud dispute was filed, and disputes are caused by fraud. The causal chain is $\text{is\_fraud}\to \text{dispute}\to \text{chargeback\_amount}.$ The feature is downstream of the label — it is computed only after you already know the answer. At prediction time (the moment the transaction happens) this number does not exist yet.

The other three (amount, merchant_category, time_of_day) are all known at transaction time, so they are legitimate.

1. Leak (train-test contamination). The scaler's mean/std are learned from test rows too — test statistics leak into training features. Breaks Rule S.
2. No leak. The statistic (median) is learned on train only, then applied to test. This is exactly the correct pattern. Obeys Rule S.
3. No leak — for the encoding itself. (Test categories unseen in training must be handled, but that's a robustness issue, not leakage.)
4. Leak (target leakage in disguise). If IDs were assigned in diagnosis order, the ID number secretly encodes when/whether a diagnosis happened — a proxy for the label. Breaks Rule T/target.
5. Leak (train-test contamination via feature selection). SelectKBest scores each feature against the label $y$ using all rows, so the choice of which columns survive is influenced by test-set labels. The test set has already voted on your feature list. Breaks Rule S. The fix: select features inside the training fold only (or inside a Pipeline).

Step 1 — learn statistics on TRAIN only (Rule S). Here $n=3$. ${\mu }_{\text{train}}=\frac{1000+1500+2000}{3}=1500.$ ${\sigma }_{\text{train}}=\sqrt{\frac{(1000-1500{)}^2+(1500-1500{)}^2+(2000-1500{)}^2}{3}}=\sqrt{\frac{250000+0+250000}{3}}=\sqrt{166666.\overline{6}}\approx \mathrm{408.25.}$

Step 2 — apply to the test point $3500$: $X^{\prime }=\frac{3500-1500}{408.25}\approx \boxed{4.90}.$

What it looks like: the test point sits ~4.9 standard-deviations above the training mean — a huge extrapolation. This is honest: it truthfully warns you that the test houses are far outside the training range. If you had (wrongly) fit on all six houses, ${\mu }_{\text{all}}=2500$ and the same point would look like a tame $\approx 0.93\sigma$ — a comforting lie that hides the distribution shift.

The window is $[T+5-29,T+5]=[T-24,T+5]$.

• Days $T-24\dots T$ are training days — fine to use, they are in the past.
• Days $T+1\dots T+5$ are test days.

But we're computing the feature for $t=T+5$. Under Rule T we may only use data strictly before $T+5$, i.e. up to $T+4$. So the mean is contaminated by the future point at day $T+5$ itself, and any test-day values in $[T+1,T+4]$ are still legitimate history relative to $T+5$.

The real leak: using day $T+5$'s own value inside its own feature (a future point predicting itself). More broadly, the classic bug is computing the window on the whole frame so that, for a test-day feature, values are pooled together globally rather than causally. Leaked days: $\{T+5\}$ for this point (and, in the naive global recompute, all later days that shift earlier windows). See Time series forecasting fundamentals.

Leaky statistics over all six points $[1000,1500,2000,3000,3500,4000]$ (here $n=6$): ${\mu }_{\text{all}}=\frac{15000}{6}=2500.$ Deviations squared: $(1500^2)+(1000^2)+(500^2)+(500^2)+(1000^2)+(1500^2)=2250000+1000000+250000+250000+1000000+2250000=7000000.$ ${\sigma }_{\text{all}}=\sqrt{\frac{7000000}{6}}=\sqrt{1166666.\overline{6}}\approx \mathrm{1080.12.}$ $X_{\text{leaky}}^{\prime }=\frac{3500-2500}{1080.12}\approx \mathrm{0.926.}$

Comparison: honest $=4.90$, leaky $=0.93$.

Sanity filter ($|X^{\prime }|>3$ flags): the honest value 4.90 is correctly flagged as out-of-distribution; the leaky value 0.93 passes silently. The leak literally disables your own safety check — it dissolves the test point into a merged distribution so the extrapolation becomes invisible. That is the quiet danger: leakage doesn't just inflate accuracy, it hides the very evidence that would have warned you.

In 5-fold CV, each row serves as a validation row in exactly one fold and a training row in the other four. If you scale once globally, then for every fold the validation rows contributed their values to $\mu ,\sigma$ — so all 5 folds are contaminated (5 out of 5).

Why biased upward: each fold's validation features were normalized using knowledge of those very validation points, so the model never faces a truly unseen distribution. Averaged over folds, the reported score overstates real-world performance. The correct pattern refits the scaler inside each fold on that fold's training portion only — most cleanly done with a Pipeline passed to the CV routine. The identical warning applies to feature selection and hyperparameter tuning: if you pick features or tune $k$ once on all the data before CV, every fold is contaminated in exactly the same way. See Cross-validation techniques.

Correct order: (ii) split → then, fit on train only: (i) impute → (v) encode → (iii) scale → (vi) select features → (vii) tune hyperparameters (via CV inside the training data) → (iv) fit final model → finally apply every learned transform/selection to the test set → evaluate once.

Justification (Rule S): the split must come first so every subsequent statistic (imputation value, encoding table, scaler mean/std), plus every decision (which $k$ features survive, which hyperparameters win) is learned only from training rows. Feature selection and tuning are learning steps too — they consult the label — so they must never see test rows. In practice wrap (i,v,iii,vi,iv) in a single Pipeline and run the tuning search with cross-validation on the training set only — this makes leakage structurally impossible. See Pipeline and workflow automation and Feature scaling and normalization.

Use expanding-window (walk-forward) validation:

• Fold 1: train on days $[1,100]$, validate on $[101,130]$.
• Fold 2: train on $[1,130]$, validate on $[131,160]$.
• Fold $k$: train on all days up to a cutoff, validate on the next contiguous block — never a random shuffle.

Rule T enforcement: every validation day $t$ is predicted using only data with date $<t$. All rolling features (means, lags) for a validation day are computed from the training history only. No future row ever enters a past feature. Never use plain shuffled k-fold on ordered time data — it randomly mixes future into training folds. See Time series forecasting fundamentals and Cross-validation techniques.

Yes — this is a subtle target/temporal leak (a proxy). Follow-up appointments are typically scheduled because a clinician already anticipates readmission risk, or are booked at/after discharge decisions that correlate with the outcome. The causal question (see Causal inference basics): is this feature a cause of readmission, or an effect of the same process that drives readmission? If it's downstream of the outcome decision, it leaks.

Proposed test procedure: for each patient, attach a timestamp to the appointment count and freeze the feature to a snapshot taken at the exact prediction moment (e.g., the instant of admission). Re-count using only appointments scheduled strictly before that cutoff. Then retrain and compare:

• If the dramatic accuracy boost survives the point-in-time snapshot, the feature carries genuine, legitimately-available signal — keep it.
• If the boost collapses once you restrict to pre-cutoff appointments, the signal came from information that only existed after the outcome was effectively decided — it is leakage, and the column must be dropped or replaced by its pre-cutoff version.

Conclusion: a "too good to be true" jump in accuracy from a single feature is itself the diagnostic symptom. Treat such a feature as guilty until the point-in-time reconstruction proves it innocent.

Why it leaks (target leakage into a feature): for a given row, its own label $y_i$ is included in the average that becomes its feature value. In the extreme (a zip that appears once), the "encoded" feature equals that row's own label — the model reads the answer directly off the feature. Even with several rows per zip, each row's feature is partly built from its own $y$, so the causal arrow $y\to X^{\prime }$ (the forbidden yellow arrow in the opening diagram) is present. Rare categories leak the most, because the mean is dominated by very few labels.

The fix — leave-one-out / out-of-fold encoding:

• Leave-one-out: when encoding row $i$, average the labels of all other rows with the same zip, excluding row $i$ itself. Now the feature can never contain $y_i$.
• Out-of-fold (K-fold target encoding): split training data into folds; encode each fold using target means computed only from the other folds. Then encode the test set using the means learned on the full training set.
• Smoothing: blend the category mean toward the global mean, weighted by how many rows the category has, so rare zips don't over-fit to a handful of labels.

What it looks like: exactly the arrow-of-time picture — target-mean encoding secretly draws the $y\to X^{\prime }$ arrow; leave-one-out and out-of-fold erase that arrow by making sure a row's own label never feeds its own feature. The same danger applies to target-frequency or count encodings computed with the label in scope. See Feature scaling and normalization.

1. Point-in-time reconstruction (catches temporal + target/proxy leak): for a random sample of training rows, reconstruct exactly what was knowable at prediction time. Any feature not reproducible from that snapshot is dropped.
2. Fit-on-train-only enforcement (catches contamination, encoding, selection & tuning leaks): all transforms — scaling, imputation, target encoding, feature selection, hyperparameter search — live inside a single Pipeline; CI fails if any .fit() or selection/tuning step touches validation/test data. See Pipeline and workflow automation.
3. Train-then-production metric gap (catches everything): monitor $\Delta =|R_{\text{prod}}-\hat{R}|$. A large gap after deploy is the empirical fingerprint of leakage; see Model evaluation metrics.
4. Feature-importance red-flag review (catches proxies & target encodings): any single feature with implausibly high importance gets a causal audit — is it downstream of the label, or built using the label? See Causal inference basics.

A feature computed after the label is known is target leakage. Learning $\mu ,\sigma$ on the full dataset before splitting is train-test contamination. The rule "a feature at time $t$ uses only data before $t$" is Rule T (temporal). Replacing a category by the average label of its rows (including the row's own label) is target-mean encoding leakage. Choosing features or tuning hyperparameters on the full dataset before splitting is selection/tuning leakage (a form of train-test contamination). The honest test-point z-score in L2.1 was ::: about 4.90 The leaky test-point z-score in L3.1 was ::: about 0.93 The safe order is split ::: first, then fit-on-train-only for every transform, selection and tuning step