Handling imbalanced datasets (SMOTE, undersampling)

2.1.11 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Classification tasks mein, imbalanced datasets tab hote hain jab ek class doosri class(es) se bahut zyada hoti hai. Isse ek biased model banta hai jo majority class ko toh achhe se predict karta hai, lekin minority class par fail ho jaata hai—aur minority class hi aksar woh class hoti hai jis par humein sabse zyada dhyaan dena chahiye (fraud, disease, rare events).

Real-world impact: Ek cancer diagnosis miss karna 1000 false alarms se zyada matter karta hai. Raw accuracy meaningless ho jaati hai—humein aisi techniques chahiye jo model ko minority patterns seekhne par majboor karein.

Core Problem: Models Imbalance Par Kyun Fail Hote Hain

KYUN hota hai:

• Naturally rare events (equipment failure, disease)
• Biased collection (sirf positives record karna)
• Time-based drift (fraud patterns data collection se tez badal jaate hain)

KYA break hota hai:

1. Loss function bias: Cross-entropy loss errors ko equally average karta hai. 100:1 imbalance ke saath, majority class errors gradient par dominate karte hain—model "hamesha majority predict karo" seekh leta hai.
2. Decision boundary skew: Algorithms aisi boundaries dhundh'te hain jo overall error minimize karein. Optimal boundary minority class territory ki taraf shift ho jaati hai, uska decision region shrink karta hai.
3. Feature irrelevance: Minority class patterns majority noise mein dab jaate hain. Features jo minorities ko perfectly separate karte hain unhe ignore kar diya jaata hai agar woh majority accuracy ko thoda hurt karte hain.

KAISE detect karein:

• Class distribution analysis: pd.Series(y).value_counts()
• Performance metrics: High accuracy lekin minority class par low recall/precision
• Confusion matrix: Model almost exclusively majority class predict karta hai

Solution 1: Random Undersampling

Expected Information Loss ki Derivation:

Dataset $D$ se shuru karo jisme $N_{\text{maj}}$ majority aur $N_{\text{min}}$ minority samples hain.

Hum majority ko ek target tak downsample karte hain jo majority:minority ratio $r$ deta hai (jaise $r=1$ perfect balance ke liye): $N_{\text{maj}}^{\prime }=r\cdot N_{\text{min}}$

Information loss majority data ke fraction ke roop mein (denominator $N_{\text{maj}}$ hai, kyunki hum sirf majority samples discard karte hain): $\text{Loss}=\frac{N_{\text{maj}}-N_{\text{maj}}^{\prime }}{N_{\text{maj}}}=1-\frac{r\cdot N_{\text{min}}}{N_{\text{maj}}}$

100:1 imbalance ke liye ($N_{\text{maj}}=100\cdot N_{\text{min}}$), $r=1$ (1:1) par balance karne ka matlab hai: $\text{Loss}=1-\frac{1\cdot N_{\text{min}}}{100\cdot N_{\text{min}}}=1-0.01=99\%$

YEH step kyun? Hum cost quantify kar rahe hain—99% majority data kho dene ka matlab hai 99% learned patterns gayab. Agar majority class ke diverse subgroups hain, toh hum unme se zyada miss kar lenge.

Step 1: Target 1:1 balance ($r=1$) → 100 legitimate transactions rakho

import numpy as np
from sklearn.utils import resample
 
# WHY? Independent sampling ke liye classes alag karo
legit = X[y == 0]  # 9,900 samples
fraud = X[y == 1]  # 100 samples
 
# WHY? Sirf r * N_minority majority samples rakho (r = 1 yahan)
legit_downsampled = resample(legit, 
                             n_samples=len(fraud),
                             random_state=42)
 
# WHY? Ordering bias rokne ke liye combine aur shuffle karo
X_balanced = np.vstack([legit_downsampled, fraud])
y_balanced = np.hstack([np.zeros(len(fraud)), np.ones(len(fraud))])

Result: 200 total samples (100 each class). Model ab equal representation dekhta hai, lekin humne 9,800 legitimate transactions phek diye—possibly kuch rare legitimate patterns bhi jo fraud jaisi lagti hain.

KAB use karein: Bade datasets mein jahan majority data lose karne ke baad bhi sufficient samples bachte hain (jaise 1M → 100K abhi bhi rich hai). Fast prototyping ke liye.

Solution 2: SMOTE (Synthetic Minority Oversampling Technique)

First Principles se Derivation:

Goal: Ek naya minority sample ${\mathbf{x}}_{\text{new}}$ create karo jo existing samples ke "beech" ho—minority class count badhate hue minority class distribution preserve karo.

Step 1: Nearest Neighbor Foundation

Minority sample ${\mathbf{x}}_i$ ke liye, Euclidean distance use karke $k$ nearest minority neighbors dhundho: $d({\mathbf{x}}_i,{\mathbf{x}}_j)=\sqrt{{\sum }_{f=1}^F(x_{i,f}-x_{j,f}{)}^2}$

Euclidean kyun? Hum feature space locality assume karte hain—kareeb ke points class characteristics share karte hain. Doosre metrics (Manhattan, Mahalanobis) bhi kaam karte hain lekin tuning chahiye.

Step 2: Linear Interpolation

$k$ neighbors mein se random neighbor ${\mathbf{x}}_{\text{neighbor}}$ select karo. Synthetic sample generate karo: ${\mathbf{x}}_{\text{new}}={\mathbf{x}}_i+\lambda \cdot ({\mathbf{x}}_{\text{neighbor}}-{\mathbf{x}}_i)$

jahan $\lambda \sim \text{Uniform}(0,1)$ ek random interpolation factor hai.

YEH step kyun?

• $\lambda =0$: ${\mathbf{x}}_{\text{new}}={\mathbf{x}}_i$ (duplicate)
• $\lambda =1$: ${\mathbf{x}}_{\text{new}}={\mathbf{x}}_{\text{neighbor}}$ (duplicate neighbor)
• $0<\lambda <1$: Naya point unke beech ki line segment par—ek plausible "in-between" sample

Geometric intuition: Agar do frauds ke patterns hain [high_amount, foreign_location] aur [high_amount, night_time], toh SMOTE [high_amount, foreign_location_AND_night_time] create karta hai—ek synthetic lekin realistic fraud.

Step 3: Balance Hone Tak Repeat Karo

Zaroori synthetic samples ki sankhya: $N_{\text{synthetic}}=N_{\text{maj}}-N_{\text{min}}$

Har synthesis ke liye, randomly ek minority sample pick karo aur Step 2 apply karo.

Step 1: Malignant sample pick karo

x_i = [2.5, 0.8, 0.6]  # size=2.5cm, density=0.8, irregularity=0.6

Step 2: k=5 nearest malignant neighbors dhundho (KNN use karke)

neighbors = [[2.3, 0.75, 0.65],
             [2.7, 0.82, 0.58],
             [2.4, 0.79, 0.63],
             [2.6, 0.81, 0.59],
             [2.5, 0.80, 0.61]]

Kyun? Hum sirf malignant samples dekhte hain—ensures karta hai ki synthetic malignant territory mein rahe.

Step 3: Ek neighbor randomly pick karo aur interpolate karo

x_neighbor = [2.7, 0.82, 0.58]
lambda_ = 0.4  # random
 
x_new = x_i + lambda_ * (x_neighbor - x_i)
      = [2.5, 0.8, 0.6] + 0.4 * ([2.7, 0.82, 0.58] - [2.5, 0.8, 0.6])
      = [2.5, 0.8, 0.6] + 0.4 * [0.2, 0.02, -0.02]
      = [2.5, 0.8, 0.6] + [0.08, 0.008, -0.008]
      = [2.58, 0.808, 0.592]

YEH step kyun? Synthetic tumor original se neighbor ki taraf 40% raaste par hai—thoda bada size (2.58), thodi zyada density (0.808), thodi kam irregularity (0.592). Plausible malignant characteristics!

Step 4: 950 benign samples se match karne ke liye 900 baar repeat karo

Result: Dataset mein ab 950 benign (original) + 50 malignant (original) + 900 malignant (synthetic) = 1,900 samples hain 950:950 balance ke saath.

KAB use karein: Chhota minority class jahan data lose karna hurt karta hai. Best kaam karta hai jab minority class dense clusters banati ho (fraud patterns, random noise nahi).

Comparison: Kab Kya Use Karein

Hybrid approach: Dono combine karo! Majority ko 10:1 tak undersample karo, phir minority ko 5:1 tak SMOTE karo. Dono methods ki extremes reduce hoti hain.

Mistake 1: Train-test split se pehle SMOTE apply karna

# WRONG!
X_smote, y_smote = SMOTE().fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X_smote, y_smote)

Kyun sahi lagta hai: "Mujhe sab cheez ke liye balanced data chahiye."

Problem: Synthetic test samples training samples se interpolate ho sakte hain—tum essentially aisi data par test kar rahe ho jo tumhare training set se "bani" hai. Model artificially correlated patterns dekhta hai, performance overestimate hoti hai.

Steel-man: Instinct training data ke liye sahi hai. Fix yeh hai:

# RIGHT!
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train_smote, y_train_smote = SMOTE().fit_resample(X_train, y_train)
# Test set original rehta hai, imbalanced

Mistake 2: Accuracy ko metric ki tarah use karna

# WRONG!
accuracy = (TP + TN) / (TP + TN + FP + FN)

Kyun sahi lagta hai: "Accuracy standard metric hai."

Problem: Training data balance karne ke baad bhi, test imbalanced ho sakta hai (jaisa hona chahiye—reality reflect karta hai). Ek 99:1 test set 99% accuracy ko trivial bana deta hai.

Fix: Aisi metrics use karo jo minority class ko weight dein:

• Precision: $\frac{TP}{TP+FP}$ (predicted positives mein se kitne correct?)
• Recall: $\frac{TP}{TP+FN}$ (actual positives mein se kitne pakde?)
• F1-Score: $\frac{2\cdot \text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}$ (harmonic mean, balanced view)

Cost-sensitive tasks (fraud, cancer) ke liye, cost-sensitive learning ya PR-AUC (precision-recall curve area) use karo.

Mistake 3: SMOTE mein k ≥ N_min use karna

# WRONG for small minority class
SMOTE(k_neighbors=10)  # but only 8 minority samples exist → error/degenerate

Kyun sahi lagta hai: "Zyada neighbors = zyada information."

Problem: SMOTE ki sirf ek strict requirement hai $k<N_{\text{min}}$ (tumhare paas existing minority samples se zyada neighbors nahi ho sakte). Agar $k$ $N_{\text{min}}$ ke kareeb pahunch jaata hai, toh har synthesis almost poori class use karta hai—global minority distribution par overfit hota hai aur local subclusters blur ho jaate hain (alag fraud types ek mush mein ghul jaate hain).

Fix: Zyaadatar implementations default karte hain k_neighbors = min(5, N_min - 1). $k$ small rakho (typically 5) aur strictly minority samples ki sankhya se kam.

U.N.D.E.R. = Unwanted Neighbors Discarded, Extremely Rapid (Majority neighbors phek deta hai, super fast hai, info lose hoti hai)

Advanced Variants

ADASYN (Adaptive Synthetic Sampling):

• Un minority samples ke liye zyada synthetics generate karta hai jo "learn karna mushkil" hain (decision boundary ke kareeb)
• Density-weighted: Sparse regions mein ${\mathbf{x}}_i$ ko zyada synthetics milte hain
• Complex boundaries ke liye better lekin computationally heavier

Borderline-SMOTE:

• Sirf un minority samples se synthesize karta hai jo class boundary ke kareeb hain (KNN dwara misclassify hue)
• Model ka dhyaan decision boundary par focus karta hai, easy interior samples par nahi
• Best jab classes separable hain lekin boundary fuzzy ho

Edited Nearest Neighbors (ENN):

• Un majority samples ko remove karta hai jinke neighbors mostly minority hain
• Overlap regions clean karta hai—intelligence ke saath undersampling
• SMOTE ke saath combine karo (yahi hai SMOTE-ENN): Pehle SMOTE, phir noisy synthetics aur overlap zones mein majority samples hatane ke liye ENN

SMOTE-Tomek:

• Tomek links opposite-class nearest neighbors ke pairs hote hain (ek majority aur ek minority point jo ek doosre ke closest hain)
• SMOTE ke baad, decision boundary sharpen karne ke liye Tomek links remove karo
• SMOTE akele se cleaner separation

Implementation Checklist

from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline
 
# 1. Pehle Split karo
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
 
# 2. Strategy choose karo
# Option A: Pure SMOTE
smote = SMOTE(k_neighbors=5, random_state=42)
X_train_bal, y_train_bal = smote.fit_resample(X_train, y_train)
 
# Option B: Hybrid pipeline
pipeline = Pipeline([
    ('undersample', RandomUnderSampler(sampling_strategy=0.5)),  # majority:minority = 2:1
    ('oversample', SMOTE(sampling_strategy=1.0))  # then balance to 1:1
])
X_train_bal, y_train_bal = pipeline.fit_resample(X_train, y_train)
 
# 3. Balanced par train karo, original par test karo
model.fit(X_train_bal, y_train_bal)
y_pred = model.predict(X_test)  # X_test resample NAHI hota
 
# 4. Minority-aware metrics se evaluate karo
from sklearn.metrics import classification_report, roc_auc_score
print(classification_report(y_test, y_pred))
print(f"ROC-AUC: {roc_auc_score(y_test, y_pred_proba)}")

Undersampling aisa hai jaise red marbles nikal lo jab tak 50 red aur 50 blue na reh jayein. Ab robot blue par dhyaan deta hai! Lekin tumne 900 red marbles phek diye—shayad kuch ke special patterns the jo tumhe chahiye the.

SMOTE aisa hai jaise magic cloning: Tum do blue marbles dekhte ho jo thodi similar hain, phir ek naya blue marble create karte ho jo unke beech ka hai. Is tarah tum 900 fake blue marbles banate ho taki robot utni hi blues dekhe jitni reds. Robot seekhta hai ki blues matter karti hain! Lekin tum ne woh marbles invent kiye—woh real nahi hain, isliye robot fake patterns seekh sakta hai.

Trick? SMOTE homework ke liye use karo (training), lekin robot ko real jar par test karo (950:50 ke saath) taki ensure ho sake ki woh real world mein kaam karta hai!

Connections

• 2.1.3-Feature-scaling-and-normalization: SMOTE Euclidean distance use karta hai, isliye features pehle scale hone chahiye (unscaled Age aur Income KNN ko break kar dete hain)
• 2.1.8-Cross-validation-and-holdout-methods: Stratified CV ensure karta hai ki minority class har fold mein aaye—imbalanced data ke liye critical
• 3.2.4-Precision-recall-and-F1-score: Woh metrics jo imbalanced evaluation ke liye actually matter karte hain—accuracy misleading hai
• 3.4.7-Cost-sensitive-learning: Resampling ka alternative—directly loss function ko class cost se weight karo
• 4.1.2-Decision-trees-and-random-forests: Tree algorithms imbalance ko linear models se better handle karte hain (minority regions partition kar sakte hain)
• 2.1.10-Dealing-with-missing-data: Imputation SMOTE se PEHLE honi chahiye (KNN interpolation ko complete feature vectors chahiye)

#flashcards/ai-ml

Concept Map