Question bank — Data leakage identification and prevention

A high validation score is proof that a model has no leakage. ::: False — leakage inflates validation scores, so a suspiciously perfect score is a symptom of leakage, not proof of its absence. Leakage always shows up as an obvious bug in the code. ::: False — the most dangerous leakage is a legitimate-looking feature or a scaler fit on all data; the code runs fine and only production performance reveals the problem. If a feature is strongly correlated with the target, you should keep it because it is predictive. ::: False — a near-perfect correlation is the classic fingerprint of target leakage; ask why it correlates, because a feature computed after the target is decided will not exist at prediction time. Fitting a scaler on the full dataset is harmless because scaling is "just preprocessing," not model training. ::: False — the scaler learns the mean and variance from the data, and using test rows to compute them leaks test-set statistics into the training features (see Feature scaling and normalization). Shuffling rows before a train-test split fixes temporal leakage. ::: False — shuffling makes it worse for time-ordered data, because future rows land in training and let the model peek forward across the split boundary. Data leakage and overfitting are the same phenomenon. ::: False — overfitting is memorising noise within legitimately-available features; leakage is using information that would not exist at prediction time at all, so more regularisation cannot cure it. Removing the target column from the feature matrix guarantees no target leakage. ::: False — the leak often hides in a different column that is causally downstream of the target (a dispute flag, a chargeback amount), not in the target itself. If train and test come from the same distribution, leakage is impossible. ::: False — leakage can happen even with identical distributions, e.g. a global scaler or a rolling feature that shares information across the split. A model with 100% training accuracy is a sign of good feature engineering. ::: False — near-perfect training accuracy usually means a feature is a re-encoding of the label, i.e. target leakage, and it will collapse in production. Cross-validation automatically protects you from leakage. ::: False — if preprocessing (imputation, scaling, encoding) is done before the CV loop, every fold is contaminated; the fix is to put preprocessing inside the pipeline that is refit per fold (see Cross-validation techniques).

"I fit StandardScaler on X, then called train_test_split." ::: The scaler saw the test rows before the split, so test statistics leak into every training feature; split first, then fit the scaler on training data only and transform the test set. "I filled missing values with the column mean computed over the whole dataset." ::: The mean absorbs test-set values, leaking them into training rows; compute the imputation statistic on training data only and apply it to the test set. "For fraud prediction I used account_balance_after_dispute as a feature." ::: That column is filled in after the fraud investigation, so it is causally downstream of the label; it cannot exist at transaction time and is textbook target leakage. "I computed a 30-day rolling mean on the full series, then split by date into train and test." ::: Test-period windows reach back across the split into training days, and training windows can reach into test days after a shuffle; compute the rolling feature using only data available before each point (see Time series forecasting fundamentals). "I one-hot encoded categories using the full dataset so train and test have matching columns." ::: Building the vocabulary from test data leaks which categories appear in the test set; fit the encoder on training data and handle unseen categories gracefully at transform time. "I selected the top-20 features by correlation on all data before cross-validating." ::: Feature selection that peeks at the whole dataset (including validation folds) leaks the labels through the selection step; selection must live inside the pipeline refit on each fold. "I upsampled the minority class with SMOTE, then split into train and test." ::: Synthetic points generated from the whole dataset can place near-duplicates of test rows into training; resample after splitting and only on the training portion. "My user ID was numeric, so I left it in as a feature and got great scores." ::: If IDs correlate with the label ordering (e.g. later IDs are labelled later), the model exploits the ID as a proxy for information not available in production.

Why does leakage make the production–validation gap potentially unbounded rather than just a few percent? ::: Because a leaked feature can be a near-perfect encoding of the target, training error can approach zero while production error stays large, and the gap grows with how strongly the feature depends on the label. Why is the causal direction between a feature and the target the key test for target leakage? ::: If the arrow points target → feature, the feature only exists once the target is known, so it will be missing or blind at prediction time; causal reasoning (Causal inference basics) tells you which features are "downstream" and forbidden. Why must preprocessing statistics be learned inside the training split, not the whole dataset? ::: Because those statistics (mean, variance, category vocabulary, imputation value) are part of the learned model, and letting them see test data means the test set can no longer honestly estimate production performance. Why does temporal leakage violate the machine-learning assumption that "the future looks like the past"? ::: The model is meant to learn only from the past to predict the future, but temporal leakage feeds future values into past feature rows, so the training setup no longer mirrors the real forward-in-time flow of data. Why can a leaked model score highly on every evaluation metric yet still be worthless? ::: Every metric is computed on the same contaminated validation data, so they all reward the same illusion; the failure only surfaces on genuinely unseen production data. Why is grouping or time-based splitting sometimes required instead of a plain random split? ::: When rows are correlated within a group (same patient, same user, same day), a random split can put near-identical rows on both sides, letting the model "recognise" test rows through their siblings in training.

Is it leakage to use a feature that is available at prediction time but was expensive to collect during training? ::: No — availability at prediction time is the only test; cost or effort of collection is a separate engineering concern, not leakage. If a rolling window at time $t$ uses only data strictly before $t$, is the feature safe? ::: Yes — using only $X_{t-1},X_{t-2},\dots$ mirrors what you truly know at prediction time, so no future information leaks (this is the correct fix for temporal leakage). A feature is derived from the target but the derivation is also possible without the target in production — safe or not? ::: Safe only if your production pipeline actually computes it from available inputs, not from the label; you must replicate the exact production computation path, or it silently becomes leakage. Zero-variance edge case: a scaler's $\sigma =0$ on a constant training column — how does this connect to leakage discipline? ::: The scaler must handle $\sigma =0$ (leave the column unscaled) using training statistics; if you sneakily switch to full-data statistics to avoid the zero, you reintroduce contamination. First few rows of a time series have no full history for a rolling feature — is dropping them or back-filling from the future acceptable? ::: Drop them or mark them missing; back-filling from later dates injects future information and is temporal leakage, the very thing you are trying to prevent. A duplicate row appears in both train and test after a random split — is this leakage? ::: Yes — the identical row makes the test evaluation trivially easy and no longer measures generalisation; de-duplicate before splitting or use group-aware splitting (Train-testvalidation split strategies). In k-fold cross-validation, is it safe to impute missing values once on the whole dataset before the loop? ::: No — that imputation sees the held-out fold each time; the imputer must be refit on the training portion inside every fold via a pipeline. Is a feature that leaks only tiny information (weak correlation with the target through the split) worth fixing? ::: Yes in principle, because even weak leakage biases your estimate optimistically; the disciplined default is to eliminate all cross-split information flow rather than judge its size.