Data leakage identification and prevention

2.1.13 · AI-ML › Data Preprocessing & Feature Engineering

[!definition] What IS Data Leakage?

Data leakage woh hai jab model training process mein aise information ka use hota hai jo real-world deployment scenario mein prediction time par available nahi hogi.

Teen main types:

1. Target leakage: Woh features jo target variable se influenced ya derived hain
2. Train-test contamination: Validation/test set ki information ka training ko influence karna
3. Temporal leakage: Future information ka past predictions mein aana

WHY These Are Different

• Target leakage = woh variables use karna jo tum jo predict kar rahe ho usse causally downstream hain
• Train-test contamination = training aur validation ke beech independence assumption ko violate karna
• Temporal leakage = real-world data flow ki temporal ordering ko violate karna

[!formula] Leakage ka Mathematical Framework

Chalte hain derive karte hain ki leakage learning process ko first principles se kyun todta hai.

ML assumption: $P_{\text{train}}(X,y)=P_{\text{test}}(X,y)=P_{\text{production}}(X,y)$

Hum expected risk minimize karna chahte hain: $R(f)={\mathbb{E}}_{(X,y)\sim P_{\text{production}}}[\mathcal{L}(f(X),y)]$

Lekin hamare paas sirf training data ka access hai, isliye hum empirical risk minimize karte hain: $\hat{R}(f)=\frac{1}{n}{\sum }_{i=1}^n\mathcal{L}(f(X_i),y_i)$

Derivation ki leakage yeh kyun todti hai:

Leakage ke saath, hamare features $X^{\prime }=g(X,y,\text{future info})$ ban jaate hain, sirf $X$ nahi.

Training ke dauran: $\hat{R}(f)=\frac{1}{n}{\sum }_{i=1}^n\mathcal{L}(f(g(X_i,y_i,{\text{leak}}_i)),y_i)$

YEH step kyun? Model $f\circ g$ seekhta hai jahan $g$, $y$ ya unavailable information par depend karta hai.

Lekin production mein, hamare paas sirf $X$ hai, $g(X,y,\text{leak})$ nahi: $R_{\text{production}}(f)=\mathbb{E}[\mathcal{L}(f(X),y)]$

Divergence: $\Delta =|R_{\text{production}}(f)-\hat{R}(f)|>ϵ$

Yeh divergence unbounded hai jab $g$ strongly $y$ ya future information par depend karta hai.

Target leakage ke liye concrete derivation:

Maano feature $X_j=h(y)+ϵ$ jahan $h$ deterministic hai aur $ϵ$ small noise hai.

Model seekhta hai: $\hat{y}=f(X)\approx f(h(y))\approx h^{-1}(h(y))=y$

WHY? Feature target ka near-perfect encoding hai.

Training error: $\hat{R}=\frac{1}{n}{\sum }_{i=1}^n|h^{-1}(h(y_i)+{ϵ}_i)-y_i{|}^2\approx 0$

Lekin production mein, $X_j$ available nahi hai ya $y$ jaane bina compute hota hai: $R_{\text{prod}}=\frac{1}{n}{\sum }_{i=1}^n|f(X_{\text{available}})-y_i{|}^2\gg 0$

[!example] Worked Example 1: Credit Card Fraud (Target Leakage)

Scenario: Transaction data use karke fraudulent transactions predict karna.

Dataset:

transaction_id, amount, merchant, is_fraud, account_balance_after_dispute
1001, 500, Amazon, 0, 5000
1002, 2000, Unknown, 1, 3000  ← disputed, money returned
1003, 100, Walmart, 0, 4900

Leakage: account_balance_after_dispute ko feature ke roop mein include karna.

Kyun sahi lagta hai: "Balance changes fraud patterns indicate kar sakte hain!"

Kyun GALAT hai:

1. account_balance_after_dispute fraud investigation ke BAAD calculate hota hai
2. Fraud label is_fraud disputes cause karta hai, jo balance changes cause karte hain
3. Causal direction: is_fraud → dispute → account_balance_after_dispute

Step-by-step impact:

Training phase:

# Model dekhta hai:
X = [500, 'Amazon', 5000]  → y = 0 (koi dispute nahi, balance unchanged)
X = [2000, 'Unknown', 3000] → y = 1 (dispute filed, money returned!)

YEH step kyun? Balance column near-perfect predictor hai kyunki yeh dispute outcome encode karta hai.

Model seekhta hai: "Agar balance transaction amount se kam hua, toh fraud hai!" (99% accuracy)

Production phase:

# Model receive karta hai:
X = [1500, 'Shady Site', ???]  ← account_balance_after_dispute abhi exist nahi karta!

Accuracy collapse ho jaati hai kyunki woh feature jo 99% training accuracy deta tha woh unavailable hai.

Fix: Sirf woh features use karo jo transaction time par available hain:

• amount, merchant, time_of_day, location, historical_fraud_rate
• Kabhi nahi: dispute_filed, chargeback_amount, investigation_outcome

[!example] Worked Example 2: Time Series (Temporal Leakage)

Scenario: Daily stock returns predict karna.

Dataset (2024-01-01 to 2024-12-31):

date, open, close, volume, rolling_mean_30d, return_next_day
2024-01-15, 100, 102, 1M, 101.5, 0.02
2024-01-16, 102, 104, 1.2M, 102.0, 0.015

Leakage: rolling_mean_30d ko poore dataset par split se pehle compute karna.

Problem ki step-by-step derivation:

# GALAT: Global calculation
df['rolling_mean_30d'] = df['close'].rolling(30).mean()
train = df[df['date'] < '2024-07-01']  # Pehle 6 months
test = df[df['date'] >= '2024-07-01']  # Aakhri 6 months

YEH kyun galat hai: 2024-07-15 ke test sample ke liye:

• rolling_mean_30d 2024-06-15 to 2024-07-15 ka data use karta hai
• Lekin 2024-06-15 to 2024-06-30 training set mein hai!
• Information backward flow karti hai: test set → training features

Mathematical formalization:

Test point $t=T+k$ ke liye (jahan $T$ train/test split hai):

Feature calculation: ${\bar{X}}_{T+k}=\frac{1}{30}{\sum }_{i=T+k-29}^{T+k}{X}_i$

YEH step kyun? Rolling window test point se 30 days peeche dekhta hai.

Isme yeh include hota hai: ${\bar{X}}_{T+k}=\frac{1}{30}\left({\sum }_{i=T+k-29}^T{X}_i+{\sum }_{i=T+1}^{T+k}{X}_i\right)$ ↑ training data ↑ test data

Model train aur test sets ke beech spurious correlations seekhta hai.

Correct approach:

# SAHI: Pehle temporal split, phir calculate
train = df[df['date'] < '2024-07-01']
test = df[df['date'] >= '2024-07-01']
 
# Rolling mean SIRF training data par us point tak calculate karo
train['rolling_mean_30d'] = train['close'].rolling(30).mean()
 
# Test ke liye: calculation mein sirf training data use karo
for i in test.index:
    relevant_history = df[(df['date'] < df.loc[i, 'date']) & 
                          (df['date'] >= df.loc[i, 'date'] - timedelta(days=30))]
    test.loc[i, 'rolling_mean_30d'] = relevant_history['close'].mean()

YEH step kyun? Hum real-world scenario simulate kar rahe hain: prediction time $t$ par, tum sirf $t-1$ tak ki history jaante ho.

[!example] Worked Example 3: Train-Test Contamination (Scaling Leakage)

Scenario: Alag-alag scales ke features ke saath house prices predict karna.

Dataset:

house_id, sqft, bedrooms, price
1, 2000, 3, 300000
2, 1500, 2, 250000
.., .., ...
100, 3000, 4, 500000

Leakage: Scaler ko poore dataset par fit karna.

# GALAT approach
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # X mein train aur test dono hain
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

Kyun sahi lagta hai: "Standardization toh sirf preprocessing hai, model fitting nahi!"

Derivation ki kyun galat hai:

StandardScaler compute karta hai: $\mu =\frac{1}{n}{\sum }_{i=1}^n{X}_i,{\sigma }^2=\frac{1}{n}{\sum }_{i=1}^n(X_i-\mu {)}^2$

YEH step kyun? Hum poore data par population statistics compute kar rahe hain.

Phir scale karta hai: $X_i^{\prime }=\frac{X_i-\mu }{\sigma }$

Agar hum saare data par fit karein ($n$ = train + test samples): ${\mu }_{\text{all}}=\frac{1}{n_{\text{train}}+n_{\text{test}}}\left({\sum }_{i\in \text{train}}{X}_i+{\sum }_{j\in \text{test}}{X}_j\right)$

Contamination: Test set statistics training features ko influence karti hain!

Model par impact:

Model training data par seekhta hai jahan features test set information use karke scale ki gayi hain: $X_{\text{train}}^{\prime }=\frac{X_{\text{train}}-{\mu }_{\text{all}}}{{\sigma }_{\text{all}}}$

Production mein, naya data alag distribution ka hoga: ${\mu }_{\text{prod}}\ne {\mu }_{\text{all}},{\sigma }_{\text{prod}}\ne {\sigma }_{\text{all}}$

Numerical example:

Training houses: sqft = [1000, 1500, 2000], mean = 1500
Test houses: sqft = [3000, 3500, 4000], mean = 3500

Sabpar fit karein toh: ${\mu }_{\text{all}}=2500$

Training sample (1000 sqft): $X^{\prime }=\frac{1000-2500}{\sigma }=\text{large negative}$

YEH step kyun? Test set ki high values mean ko upar kheench deti hain, jisse training samples zyada extreme lagte hain.

Model seekhta hai ki "large negative scaled values" low prices predict karte hain, lekin yeh relationship scaling ka artifact hai, true pattern nahi.

Correct approach:

# SAHI: Train par fit karo, dono par transform karo
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Sirf train se stats seekho
X_test_scaled = scaler.transform(X_test)  # Train stats test par apply karo

Yeh production simulate karta hai: tum historical data se ${\mu }_{\text{train}}$ aur ${\sigma }_{\text{train}}$ compute karoge, phir naye samples par apply karoge.

[!mistake] Common Mistakes & Steelmanning

Mistake 1: "Data augmentation leakage cause nahi kar sakta"

Kyun sahi lagta hai: "Main sirf zyada training samples bana raha hoon, test data ko touch nahi kar raha!"

Steel-man argument: Data augmentation training set size ko naya data collect kiye bina badhata hai. Yeh pure preprocessing step lagta hai.

Kyun leakage CAN cause karta hai:

# Image classification example
X_train, X_test, y_train, y_test = train_test_split(X, y)
 
# GALAT: Split ke baad augment karo, lekin augmented data par normalize karo
X_train_augmented = augment(X_train)  # Ab 10x bada
scaler.fit(X_train_augmented)  # Stats augmentation se influenced
X_train_final = scaler.transform(X_train_augmented)
X_test_final = scaler.transform(X_test)  # Augmented stats use karta hai!

Problem: Augmentation distribution change kar deta hai. Agar tum statistics (mean, std) augmented data par compute karo, tum ek aise distribution use kar rahe ho jo production mein exist nahi karta.

Mathematical view:

Augmentation $X^{\prime }\sim P^{\prime }(X)$ create karta hai jahan $P^{\prime }(X)\ne P(X)$

Agar tum ${\mu }^{\prime },{\sigma }^{\prime }$ se $P^{\prime }$ se scale karo, lekin test/production data $P$ se aata hai: $\text{train distribution: }{P}^{\prime }(X),\text{test/prod distribution: }P(X)$

YEH step kyun? Tumne training aur evaluation ke beech distribution shift create kar diya hai.

Fix: Statistics original training data par compute karo, phir augment karo:

scaler.fit(X_train)  # Original training distribution
X_train_scaled = scaler.transform(X_train)
X_train_augmented = augment(X_train_scaled)  # Scaling ke BAAD augment karo

Mistake 2: "Cross-validation saari leakage prevent kar leta hai"

Kyun sahi lagta hai: "CV data ko multiple times split karta hai, toh information leak nahi ho sakti!"

Steel-man: Cross-validation ek single train/test split par overfitting se protect karta hai. Yeh ek robust validation strategy hai.

Kyun INCOMPLETE hai:

CV har fold ke andar train-test contamination se protect karta hai, lekin in cheez se NAHI:

1. Global preprocessing leakage:

# GALAT: CV se pehle scale karo
X_scaled = scaler.fit_transform(X)
cross_val_score(model, X_scaled, y, cv=5)  # Har fold global stats use karta hai!

Kyun galat? Har fold ka validation set apne training set ki scaling ko influence karta hai.

2. Time series mein temporal leakage:

# GALAT: Time series par random CV
cross_val_score(model, X, y, cv=5)  # Random splits temporal order violate karte hain!

Time series ke liye fix: TimeSeriesSplit use karo:

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
# Ensure karta hai ki fold i sirf fold i+1 se pehle ka data use kare

Derivation ki time series ke liye random CV kyun fail hota hai:

Random CV mein, fold $k$ mein ho sakta hai:

• Training: $\{t_1,t_5,t_{10},t_{15}\}$
• Validation: $\{t_3,t_7,t_{12}\}$

YEH kyun galat hai? $t_{15}$ par train karna jabki $t_{12}$ par validate karna matlab future use karke past predict karna!

Model seekhta hai: ${\hat{y}}_{t_3}=f(X_{t_3},\underset{\text{future information}}{\underbrace{X_{t_5},X_{t_{10}},X_{t_{15}}}})$

Mistake 3: "Feature selection leakage cause nahi kar sakta"

Kyun sahi lagta hai: "Main sirf relevant features choose kar raha hoon, test labels use nahi kar raha!"

Steel-man: Feature selection noise hataakar model performance improve karta hai. Safe lagta hai.

Kyun leakage KARTA HAI:

# GALAT: Full dataset par features select karo
from sklearn.feature_selection import SelectKBest
selector = SelectKBest(k=10)
X_selected = selector.fit_transform(X, y)  # Saare y values use karta hai, test bhi!
X_train, X_test, y_train, y_test = train_test_split(X_selected, y)

Leakage mechanism:

Feature selection correlation/mutual information compute karta hai: $I(X_j;y)={\sum }_{x,y}P(x,y)\log \frac{P(x,y)}{P(x)P(y)}$

YEH step kyun? Hum measure kar rahe hain ki $X_j$ $y$ ke baare mein kitna batata hai.

Agar saare data par compute karein: $I_{\text{all}}(X_j;y)=f(X_{j,\text{train}},y_{\text{train}},X_{j,\text{test}},y_{\text{test}})$

Test set labels influence karte hain ki training set ke liye kaun se features select honge!

Concrete example:

Dataset: 100 samples, 1000 features

• Feature $X_{42}$ randomly $y_{\text{test}}$ se correlated hai (spurious)
• Feature $X_{73}$ truly predictive hai

Agar hum saare data se features select karein, $X_{42}$ select ho sakta hai kyunki yeh test labels se correlate karta hai. Model yeh spurious correlation seekhta hai, lekin generalize nahi karega.

Fix:

# SAHI: Feature selection CV ke andar
pipe = Pipeline([
    ('selector', SelectKBest(k=10)),
    ('model', RandomForestClassifier())
])
cross_val_score(pipe, X, y, cv=5)  # Selector har fold mein refit hota hai

Systematic Leakage Detection Framework

Step 1: Causal Graph Analysis

Causal relationships draw karo:

[Transaction] → [Fraud Occurs] → [Investigation] → [Dispute Filed] → [Balance Change]
     ↑ use karna OK   ↑ Target        ↗ LEAKED features ↗

Rule: Sirf woh features use karo jo target ke causally upstream hain ya usse independent hain.

Step 2: Timeline Analysis

Har feature ke liye poochho: "Yeh value kab pata hoti hai?"

Prediction time: t
Feature X_j available at: t_j

Agar t_j > t → TEMPORAL LEAKAGE

Step 3: Validation Distribution Test

Feature distributions compare karo:

# Leakage signal check karo
from scipy.stats import ks_2samp
 
for col in X_train.columns:
    stat, p_value = ks_2samp(X_train[col], X_test[col])
    if p_value < 0.05:
        print(f"Warning: {col} distributions differ - possible leakage")

YEH kyun kaam karta hai? Agar preprocessing ne information leak ki, toh train/test distributions artificially similar honge.

Step 4: Ablation Testing

Suspected leaky features hataao aur retrain karo: ${\Delta }_{\text{accuracy}}=\text{Acc}(\text{all features})-\text{Acc}(\text{without suspicious feature})$

Agar ${\Delta }_{\text{accuracy}}>10\%$, deeply investigate karo—ek feature itna powerful nahi hona chahiye jab tak woh leak na kar raha ho.

[!recall]- Feynman Explanation

Socho tum exam ki preparation kar rahe ho, aur tumhara dost tumhe ek study guide deta hai. Baad mein pata chalta hai ki us study guide mein actual test ke questions the! Tumne bahut acha kiya, lekin tumne actually material nahi seekha.

Data leakage tab hota hai jab tumhara AI model training ke dauran "answers peek" kar leta hai. Yeh teen main tarike se hota hai:

1. Answer question mein hi hai: Jaise agar main poochhun "Paris jis country mein hai uski capital kya hai?"—answer (France) basically question mein hi hai. Woh target leakage hai—aisi information use karna jo sirf answer ki wajah se exist karti hai.

2. Kal ke newspaper se study karna: Socho Monday test ke liye study kar rahe ho lekin tumhare paas Tuesday ka newspaper hai. Tum dekhte ho Monday ke baad kya hota hai! Woh temporal leakage hai—past predict karne ke liye future information use karna.

3. Test se pehle answers share karna: Agar tumhare practice tests aur real test ne kuch answers share kiye, tum test day mein acha karte lekin actually material nahi jaante. Woh train-test contamination hai—training aur testing information mix karna.

Kyun bura hai: Tumhara model genius lagta hai (99% accuracy!) lekin real world mein fail ho jaata hai kyunki usne galat cheez seekhi. Yeh test answers memorize karne aur concepts samajhne mein fark hai.

Prevent kaise karein: Hamesha poochho "Kya real life mein prediction time par yeh information mere paas hogi?" Agar nahi, toh training mein use mat karo!

[!mnemonic] T.T.T. Leakage Check

Target dependency → Kya yeh feature target par depend karta hai? Temporal order → Kya yeh information future se hai? Training isolation → Kya maine yeh sirf training data par fit/compute kiya?

Koi bhi feature use karne se pehle, teeno T's check karo. Agar koi bhi fail ho, leakage ke liye investigate karo.

Memory aid: "Test The Timeline" — feature creation se prediction tak information flow trace karo.

Key Prevention Strategies (80/20 Rule)

Yeh 20% practices 80% leakage prevent karti hain:

1. Pipeline everything — sklearn.pipeline.Pipeline use karo taaki saari preprocessing har CV fold mein refit ho
2. Split first, preprocess second — HAMESHA koi bhi statistics/scaling/feature selection se pehle split karo
3. Temporal awareness — Time series ke liye TimeSeriesSplit use karo aur kabhi shuffle mat karo
4. Causal reasoning — Causal graph draw karo; sirf target ke upstream features use karo
5. Production simulation — Poochho "Kya prediction time par yeh available hoga?"

Active Recall Practice

Connections

• Train-test-validation split strategies - Proper splitting contamination prevent karta hai
• Cross-validation techniques - CV ko temporal order aur preprocessing isolation respect karna chahiye
• Feature scaling and normalization - Subtle leakage ka common source
• Time series forecasting fundamentals - Temporal leakage yahan critical hai
• Model evaluation metrics - Leakage metrics ko deceptively inflate karta hai
• Pipeline and workflow automation - Pipelines preprocessing leakage prevent karte hain
• Causal inference basics - Causality samajhna target leakage prevent karta hai
• Production model deployment - Train/prod distribution mismatch leakage reveal karta hai

#flashcards/ai-ml

Data leakage kya hai? :: Woh information jo training dataset ke bahar se hai aur model training ke dauran leak hoti hai—prediction time par available nahi hogi—jisse validation performance artificially high hoti hai jo production mein generalize nahi karti.

Poore dataset par scaling kyun leakage cause karta hai? :: Scaler ka mean aur std test data use karke compute hota hai, isliye training features test set statistics se transform hoti hain. Yeh train aur test ke beech ek artificial relationship banata hai jo production mein exist nahi karegi jahan naye data ki alag statistics hongi.

Target leakage kya hai aur ek example do :: Aise features use karna jo target variable se causally downstream ya directly influenced hain. Example: fraud predict karne ke liye "account_balance_after_dispute" use karna, jab disputes sirf fraud identify hone ke BAAD hote hain—feature target outcome encode karta hai.

Target leakage ML assumptions ko todne ka mathematical reason bataao :: ML assume karta hai ki hum $\mathbb{E}[(f(X)-y{)}^2]$ minimize karte hain jahan $X$, $y$ se independent hai. Target leakage ke saath, features $X^{\prime }=g(X,y)$ ban jaati hain, isliye hum distribution $P(X^{\prime },y)$ with dependence par train karte hain, lekin $P(X,y)$ par deploy karte hain, unbounded generalization error create karta hai.

Concept Map