Handling imbalanced datasets (SMOTE, undersampling)

Overview

In classification tasks, imbalanced datasets occur when one class significantly outnumbers the other(s). This creates a biased model that predicts the majority class well but fails on the minority class—often the class we care about most (fraud, disease, rare events).

Real-world impact: Missing one cancer diagnosis matters more than 1000 false alarms. Raw accuracy becomes meaningless—we need techniques that force the model to learn minority patterns.

The Core Problem: Why Models Fail on Imbalance

WHY it happens:

• Naturally rare events (equipment failure, disease)
• Biased collection (only recording positives)
• Time-based drift (fraud patterns change faster than data collection)

WHAT breaks:

1. Loss function bias: Cross-entropy loss averages errors equally. With 100:1 imbalance, majority class errors dominate the gradient—model learns "always predict majority."
2. Decision boundary skew: Algorithms find boundaries that minimize overall error. The optimal boundary shifts toward the minority class territory, shrinking its decision region.
3. Feature irrelevance: Minority class patterns get drowned in majority noise. Features that perfectly separate minorities are ignored if they slightly hurt majority accuracy.

HOW to detect:

• Class distribution analysis: pd.Series(y).value_counts()
• Performance metrics: High accuracy but low recall/precision on minority class
• Confusion matrix: Model predicts majority class almost exclusively

Solution 1: Random Undersampling

Derivation of Expected Information Loss:

Start with dataset $D$ with $N_{\text{maj}}$ majority and $N_{\text{min}}$ minority samples.

We downsample the majority to a target that gives majority:minority ratio $r$ (e.g., $r=1$ for perfect balance): $N'_{\text{maj}} = r \cdot N_{\text{min}}$

Information loss as a fraction of majority data (denominator is $N_{\text{maj}}$, since we only discard majority samples): $\text{Loss} = \frac{N_{\text{maj}} - N'_{\text{maj}}}{N_{\text{maj}}} = 1 - \frac{r \cdot N_{\text{min}}}{N_{\text{maj}}}$

For a 100:1 imbalance ($N_{\text{maj}} = 100 \cdot N_{\text{min}}$), balancing to $r=1$ (1:1) means: $\text{Loss} = 1 - \frac{1 \cdot N_{\text{min}}}{100 \cdot N_{\text{min}}} = 1 - 0.01 = 99\%$

WHY this step? We're quantifying the cost—losing 99% of majority data means 99% of learned patterns disappear. If the majority class has diverse subgroups, we'll miss most of them.

Step 1: Target 1:1 balance ($r=1$) → keep 100 legitimate transactions

import numpy as np
from sklearn.utils import resample
 
# WHY? Separate classes for independent sampling
legit = X[y == 0]  # 9,900 samples
fraud = X[y == 1]  # 100 samples
 
# WHY? Keep only r * N_minority majority samples (r = 1 here)
legit_downsampled = resample(legit, 
                             n_samples=len(fraud),
                             random_state=42)
 
# WHY? Combine and shuffle to prevent ordering bias
X_balanced = np.vstack([legit_downsampled, fraud])
y_balanced = np.hstack([np.zeros(len(fraud)), np.ones(len(fraud))])

Result: 200 total samples (100 each class). Model now sees equal representation, but we threw away 9,800 legitimate transactions—potentially including rare legitimate patterns that look like fraud.

WHEN to use: Large datasets where losing majority data still leaves sufficient samples (e.g., 1M → 100K still rich). Fast prototyping.

Solution 2: SMOTE (Synthetic Minority Oversampling Technique)

Derivation from First Principles:

Goal: Create new minority sample $\mathbf{x}_{\text{new}}$ that lies "between" existing samples—preserving the minority class distribution while increasing count.

Step 1: Nearest Neighbor Foundation

For minority sample $\mathbf{x}_i$, find $k$ nearest minority neighbors using Euclidean distance: $d(\mathbf{x}_i, \mathbf{x}_j) = \sqrt{\sum_{f=1}^{F} (x_{i,f} - x_{j,f})^2}$

WHY Euclidean? We assume feature space locality—nearby points share class characteristics. Other metrics (Manhattan, Mahalanobis) work but require tuning.

Step 2: Linear Interpolation

Select random neighbor $\mathbf{x}_{\text{neighbor}}$ from the $k$ neighbors. Generate synthetic sample: $\mathbf{x}_{\text{new}} = \mathbf{x}_i + \lambda \cdot (\mathbf{x}_{\text{neighbor}} - \mathbf{x}_i)$

where $\lambda \sim \text{Uniform}(0, 1)$ is a random interpolation factor.

WHY this step?

• $\lambda = 0$: $\mathbf{x}_{\text{new}} = \mathbf{x}_i$ (duplicate)
• $\lambda = 1$: $\mathbf{x}_{\text{new}} = \mathbf{x}_{\text{neighbor}}$ (duplicate neighbor)
• $0 < \lambda < 1$: New point on line segment between them—a plausible "in-between" sample

Geometric intuition: If two frauds have patterns [high_amount, foreign_location] and [high_amount, night_time], SMOTE creates [high_amount, foreign_location_AND_night_time]—a synthetic but realistic fraud.

Step 3: Repeat Until Balanced

Number of synthetic samples needed: $N_{\text{synthetic}} = N_{\text{maj}} - N_{\text{min}}$

For each synthesis, randomly pick a minority sample and apply Step 2.

Step 1: Pick malignant sample

x_i = [2.5, 0.8, 0.6]  # size=2.5cm, density=0.8, irregularity=0.6

Step 2: Find k=5 nearest malignant neighbors (using KNN)

neighbors = [[2.3, 0.75, 0.65],
             [2.7, 0.82, 0.58],
             [2.4, 0.79, 0.63],
             [2.6, 0.81, 0.59],
             [2.5, 0.80, 0.61]]

WHY? We only look at malignant samples—ensures synthetic stays in malignant territory.

Step 3: Randomly pick one neighbor and interpolate

x_neighbor = [2.7, 0.82, 0.58]
lambda_ = 0.4  # random
 
x_new = x_i + lambda_ * (x_neighbor - x_i)
      = [2.5, 0.8, 0.6] + 0.4 * ([2.7, 0.82, 0.58] - [2.5, 0.8, 0.6])
      = [2.5, 0.8, 0.6] + 0.4 * [0.2, 0.02, -0.02]
      = [2.5, 0.8, 0.6] + [0.08, 0.008, -0.008]
      = [2.58, 0.808, 0.592]

WHY this step? The synthetic tumor is 40% of the way from the original toward the neighbor—larger size (2.58), slightly higher density (0.808), slightly less irregular (0.592). Plausible malignant characteristics!

Step 4: Repeat 900 times to match the 950 benign samples

Result: Dataset now has 950 benign (original) + 50 malignant (original) + 900 malignant (synthetic) = 1,900 samples with 950:950 balance.

WHEN to use: Small minority class where losing data hurts. Works best when minority class forms dense clusters (fraud patterns, not random noise).

Comparison: When to Use What

Hybrid approach: Combine both! Undersample majority to 10:1, then SMOTE minority to 5:1. Reduces extremes of both methods.

Mistake 1: Applying SMOTE before train-test split

# WRONG!
X_smote, y_smote = SMOTE().fit_resample(X, y)
X_train, X_test, y_train, y_test = train_test_split(X_smote, y_smote)

Why it feels right: "I need balanced data for everything."

The problem: Synthetic test samples may be interpolated from training samples—you're testing on data "created from" your training set. Model sees artificially correlated patterns, overestimates performance.

Steel-man: The instinct is correct for training data. The fix:

# RIGHT!
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train_smote, y_train_smote = SMOTE().fit_resample(X_train, y_train)
# Test set stays original, imbalanced

Mistake 2: Using accuracy as the metric

# WRONG!
accuracy = (TP + TN) / (TP + TN + FP + FN)

Why it feels right: "Accuracy is the standard metric."

The problem: Even after balancing training data, test may be imbalanced (as it should be—reflects reality). A 99:1 test set makes 99% accuracy trivial.

The fix: Use metrics that weight minority class:

• Precision: $\frac{TP}{TP + FP}$ (of predicted positives, how many correct?)
• Recall: $\frac{TP}{TP + FN}$ (of actual positives, how many caught?)
• F1-Score: $\frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$ (harmonic mean, balanced view)

For cost-sensitive tasks (fraud, cancer), use cost-sensitive learning or PR-AUC (precision-recall curve area).

Mistake 3: Using k ≥ N_min in SMOTE

# WRONG for small minority class
SMOTE(k_neighbors=10)  # but only 8 minority samples exist → error/degenerate

Why it feels right: "More neighbors = more information."

The problem: SMOTE's only strict requirement is $k < N_{\text{min}}$ (you cannot have more neighbors than existing minority samples). If $k$ approaches $N_{\text{min}}$, each synthesis uses nearly the whole class—overfits to the global minority distribution and blurs local subclusters (different fraud types blend into mush).

The fix: Most implementations default to k_neighbors = min(5, N_min - 1). Keep $k$ small (typically 5) and strictly less than the number of minority samples.

U.N.D.E.R. = Unwanted Neighbors Discarded, Extremely Rapid
(Throws away majority neighbors, super fast, loses info)

Advanced Variants

ADASYN (Adaptive Synthetic Sampling):

• Generates more synthetics for minority samples that are "hard to learn" (near decision boundary)
• Density-weighted: $\mathbf{x}_i$ in sparse regions gets more synthetics
• Better for complex boundaries but computationally heavier

Borderline-SMOTE:

• Only synthesizes from minority samples near the class boundary (misclassified by KNN)
• Focuses model attention on decision boundary, not easy interior samples
• Best when classes are separable but boundary is fuzzy

Edited Nearest Neighbors (ENN):

• Removes majority samples whose neighbors are mostly minority
• Cleans overlap regions—undersampling with intelligence
• Combine with SMOTE (this is SMOTE-ENN): SMOTE first, then ENN to remove noisy synthetics and majority samples in overlap zones

SMOTE-Tomek:

• Tomek links are pairs of opposite-class nearest neighbors (a majority and minority point that are each other's closest)
• After SMOTE, remove Tomek links to sharpen the decision boundary
• Cleaner separation than SMOTE alone

Implementation Checklist

from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from imblearn.pipeline import Pipeline
 
# 1. Split FIRST
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
 
# 2. Choose strategy
# Option A: Pure SMOTE
smote = SMOTE(k_neighbors=5, random_state=42)
X_train_bal, y_train_bal = smote.fit_resample(X_train, y_train)
 
# Option B: Hybrid pipeline
pipeline = Pipeline([
    ('undersample', RandomUnderSampler(sampling_strategy=0.5)),  # majority:minority = 2:1
    ('oversample', SMOTE(sampling_strategy=1.0))  # then balance to 1:1
])
X_train_bal, y_train_bal = pipeline.fit_resample(X_train, y_train)
 
# 3. Train on balanced, test on original
model.fit(X_train_bal, y_train_bal)
y_pred = model.predict(X_test)  # X_test is NOT resampled
 
# 4. Evaluate with minority-aware metrics
from sklearn.metrics import classification_report, roc_auc_score
print(classification_report(y_test, y_pred))
print(f"ROC-AUC: {roc_auc_score(y_test, y_pred_proba)}")

Undersampling is like taking out red marbles until you have 50 red and 50 blue. Now the robot pays attention to blue! But you threw away 900 red marbles—maybe some had special patterns you needed.

SMOTE is like magic cloning: You look at two blue marbles that are kinda similar, then create a new blue marble that's halfway between them. You make 900 fake blue marbles this way so the robot sees just as many blues as reds. The robot learns blues matter! But you invented those marbles—they're not real, so the robot might learn fake patterns.

The trick? Use SMOTE for homework (training), but test the robot on the real jar (with 950:50) to make sure it works in the real world!

Connections

• 2.1.3-Feature-scaling-and-normalization: SMOTE uses Euclidean distance, so features must be scaled first (unscaled Age and Income break KNN)
• 2.1.8-Cross-validation-and-holdout-methods: Stratified CV ensures minority class appears in every fold—critical for imbalanced data
• 3.2.4-Precision-recall-and-F1-score: The metrics that actually matter for imbalanced evaluation—accuracy is misleading
• 3.4.7-Cost-sensitive-learning: Alternative to resampling—directly weight loss function by class cost
• 4.1.2-Decision-trees-and-random-forests: Tree algorithms handle imbalance better than linear models (can partition minority regions)
• 2.1.10-Dealing-with-missing-data: Imputation must happen BEFORE SMOTE (KNN interpolation needs complete feature vectors)

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

**Step 2:** Combine → 200 total samples (100 legit + 100 fraud)
**Result:** Balanced but we discarded 9,800 legitimate patterns (99% loss)!

**When to use:** Large datasets where even the minority class has thousands of samples, so the undersampled majority still retains enough diversity.

## Solution 2: SMOTE (Synthetic Minority Oversampling)

> [!intuition] The SMOTE Breakthrough
> Instead of deleting majority data, SMOTE ==creates synthetic minority samples== by interpolating between existing ones. Imagine drawing lines between nearby minority points and placing new samples along those lines. We're not copying—we're generating plausible new examples in the minority region.

> [!definition] SMOTE
> **Synthetic Minority Oversampling Technique**: generates synthetic samples by interpolating between a minority sample and its ==k-nearest minority neighbors==.

**Derivation of the SMOTE Interpolation Formula:**

**Goal:** Create a new synthetic point between two real minority points that lies plausibly within the minority class region.

**Step 1 — Pick a base sample.** Select a minority sample $\vec{x}_i$ (a feature vector).

**Step 2 — Find neighbors.** Compute its $k$-nearest minority neighbors using Euclidean distance:
$$d(\vec{x}_i, \vec{x}_j) = \sqrt{\sum_{f=1}^{F} (x_{i,f} - x_{j,f})^2}$$
where $F$ is the number of features. Randomly pick one neighbor $\vec{x}_{zi}$.

**Step 3 — Build the connecting vector.** The difference $\vec{x}_{zi} - \vec{x}_i$ points from our base sample toward the neighbor:
$$\vec{v} = \vec{x}_{zi} - \vec{x}_i$$

**Step 4 — Interpolate.** To land *between* the two points, scale $\vec{v}$ by a random fraction $\lambda \in [0,1]$ and add it to the base:
$$\vec{x}_{\text{new}} = \vec{x}_i + \lambda \cdot (\vec{x}_{zi} - \vec{x}_i), \quad \lambda \sim U(0,1)$$

**Step 5 — Verify the bounds.**
- If $\lambda = 0$: $\vec{x}_{\text{new}} = \vec{x}_i$ (the base sample)
- If $\lambda = 1$: $\vec{x}_{\text{new}} = \vec{x}_{zi}$ (the neighbor)
- If $\lambda = 0.5$: exact midpoint

This guarantees the synthetic point lies **on the line segment** between two real minority samples—inside the minority region, never extrapolating outward.

> [!formula] SMOTE Core Equation
> $$\vec{x}_{\text{new}} = \vec{x}_i + \lambda \cdot (\vec{x}_{zi} - \vec{x}_i)$$
> where $\vec{x}_i$ = minority sample, $\vec{x}_{zi}$ = randomly chosen k-nearest minority neighbor, $\lambda \sim U(0,1)$ = random interpolation factor.

> [!example] SMOTE Step-by-Step (2D)
> **Minority sample:** $\vec{x}_i = (2, 3)$ — e.g., (transaction_amount_scaled, time_scaled)
> **Chosen neighbor:** $\vec{x}_{zi} = (4, 7)$
> **Random $\lambda = 0.6$**
>
> **Compute each feature independently:**
>
> Feature 1: $x_{\text{new},1} = 2 + 0.6 \times (4 - 2) = 2 + 0.6 \times 2 = 2 + 1.2 = 3.2$
>
> Feature 2: $x_{\text{new},2} = 3 + 0.6 \times (7 - 3) = 3 + 0.6 \times 4 = 3 + 2.4 = 5.4$
>
> **New synthetic sample:** $\vec{x}_{\text{new}} = (3.2, 5.4)$
>
> ✓ Lies on the segment between $(2,3)$ and $(4,7)$, at 60% of the way.

> [!formula] SMOTE Algorithm
>