Handling imbalanced datasets (SMOTE, undersampling): Exercises

This page is a self-test ladder. Each problem is stated cleanly, then a fully worked solution hides inside a collapsible callout — try first, then reveal. Difficulty climbs from "do you recognise the idea?" to "can you design a whole pipeline?".

Everything here builds on the parent topic. We reuse two formulas from there. Both are re-explained the moment they appear so you never need to flip back.

Definition The two tools we will keep using

Undersampling target — how many majority samples we KEEP when we downsample: $N_{maj}^{'} = r \cdot N_{min}$ Here $N_{min}$ is the count of the rare class, and $r$ is the majority-to-minority ratio we want after sampling ( $r = 1$ means perfectly balanced). We keep $r$ copies of majority for every minority sample.

SMOTE interpolation — how we build one synthetic minority point: $x_{new} = x_{i} + λ (x_{neighbor} - x_{i}), λ \in [0, 1]$ $x_{i}$ is a real rare sample, $x_{neighbor}$ is one of its near neighbours (same class), and $λ$ is a dial from 0 (sit on $x_{i}$ ) to 1 (sit on the neighbour). Bold letters mean "a list of feature numbers", e.g. $[2.5, 0.8]$ .

How is $λ$ chosen, and where does diversity come from? For each synthetic point SMOTE draws a fresh $λ$ at random, typically from a ==uniform distribution over $[0, 1]$ == (written $λ \sim Uniform (0, 1)$ — every value between 0 and 1 equally likely). So if you build 900 synthetic points from the same pair, each lands at a different, randomly chosen spot along the segment. That randomness — combined with randomly re-picking the seed $x_{i}$ and its neighbour each time — is exactly what spreads the synthetic points out instead of stacking them on one location. Diversity is not magic; it is repeated random draws of $λ$ .

Level 1 — Recognition

Exercise 1.1

A dataset has 9 700 "normal" emails and 300 "spam" emails. State (a) which class is the minority, and (b) the imbalance ratio written as majority : minority, rounded to a whole number.

Recall Solution 1.1

The minority is the smaller class → spam (300 samples). Imbalance ratio $= N_{maj} : N_{min} = 9700 : 300$ . Divide both sides by 300: $\frac{9700}{300} \approx 32.3$ , so about $32 : 1$ . Answer: (a) spam; (b) $\approx 32 : 1$ .

Exercise 1.2

Fill the blank: A model that always predicts the majority class on a $99 : 1$ dataset scores about ___ % accuracy but has a minority-class recall of ___.

Recall Solution 1.2

$99 : 1$ means majority is 99 out of every 100 samples → predicting majority every time is correct 99% of the time → 99% accuracy. Recall = (rare cases caught) / (rare cases that exist). It catches zero rare cases → recall $= 0$ . This is exactly the "lazy model" trap. See 3.2.4-Precision-recall-and-F1-score for why recall exposes it.

Level 2 — Application

Exercise 2.1

You have 8 000 majority and 200 minority samples. You undersample to $r = 3$ (three majority per minority). How many majority samples do you keep, and what is the total dataset size after?

Recall Solution 2.1

Keep $N_{maj}^{'} = r \cdot N_{min} = 3 \times 200 === 600 ==$ majority samples. Minority is untouched → 200 stay. Total $= 600 + 200 === 800 ==$ .

Exercise 2.2

For the same dataset in 2.1, compute the information loss as a fraction of majority data. Recall the loss formula measures discarded majority over original majority: $Loss = 1 - \frac{r \cdot N _{min}}{N _{maj}}$

Recall Solution 2.2

$Loss = 1 - \frac{3 \times 200}{8000} = 1 - \frac{600}{8000} = 1 - 0.075 === 0.925 ==$ . So we threw away 92.5% of majority samples. Even keeping $r = 3$ is brutal here — the majority was 40× the minority.

Exercise 2.3

SMOTE step. A minority point is $x_{i} = [4.0, 1.0]$ , its chosen neighbour is $x_{neighbor} = [5.0, 3.0]$ , and $λ = 0.25$ . Compute $x_{new}$ .

Figure — Handling imbalanced datasets (SMOTE, undersampling)

Recall Solution 2.3

Step 1 — direction vector: $x_{neighbor} - x_{i} = [5.0 - 4.0, 3.0 - 1.0] = [1.0, 2.0]$ . Step 2 — scale by $λ$ : $0.25 \times [1.0, 2.0] = [0.25, 0.5]$ . Step 3 — add to $x_{i}$ : $[4.0, 1.0] + [0.25, 0.5] === [4.25, 1.5] ==$ . Look at the figure: the new point sits one-quarter of the way along the segment (the orange dot), exactly where $λ = 0.25$ places it.

Level 3 — Analysis

Exercise 3.1

A dataset is $500 : 5$ (majority:minority). Compare two plans:

Plan A: undersample majority to $r = 1$ .
Plan B: SMOTE the minority up to match the majority.

State the final size of each class under each plan, and which plan risks overfitting the minority and why.

Recall Solution 3.1

Plan A: keep $r \cdot N_{min} = 1 \times 5 = 5$ majority. Final: 5 majority + 5 minority = 10 samples total. We discarded 495 majority points — catastrophic loss. Plan B: synthesise $N_{maj} - N_{min} = 500 - 5 = 495$ new minority points. Final: 500 majority + 500 minority = 1000 total. Overfitting risk → Plan B. With only 5 real minority seeds, all 495 synthetic points are interpolations among those same 5. They cluster in a tiny region, so the model memorises 5 blobs instead of learning the true minority distribution. Validate this danger with 2.1.8-Cross-validation-and-holdout-methods — always apply SMOTE inside the training fold only.

Exercise 3.2

Explain why applying SMOTE before splitting into train/test corrupts your evaluation. Give the concrete leakage mechanism.

Recall Solution 3.2

SMOTE creates a synthetic point on the line between a real point and its neighbour. If SMOTE runs on the whole dataset first, a synthetic point can be built from a pair where one seed later lands in train and its interpolation lands in test (or vice-versa). The test point is then a near-copy of a training point → the model has effectively "seen" it → test scores are inflated. This is data leakage. Correct order: split first → SMOTE the training set only → evaluate on the untouched real test set.

Level 4 — Synthesis

Exercise 4.1

Design an end-to-end pipeline for a $1000 : 1$ fraud problem (1 000 000 legit, 1 000 fraud). You want (a) manageable size, (b) no leakage, (c) a metric that reflects catching fraud. Combine techniques and justify each choice.

Recall Solution 4.1

A defensible pipeline:

Split first (stratified, so both folds keep the $1000 : 1$ ratio) — prevents leakage, per Exercise 3.2 and 2.1.8-Cross-validation-and-holdout-methods.
Moderate undersample the majority on the train fold to, say, $r = 10$ : keep $10 \times 1000 = 10000$ legit. This drops size from 1M to a workable ~11 000 while a $10 : 1$ ratio still leaves rich majority variety (unlike $r = 1$ ).
SMOTE the minority on the train fold up to match: synthesise $10000 - 1000 = 9000$ fraud points → 10 000 fraud. Now $10000 : 10000$ .
Scale features (2.1.3-Feature-scaling-and-normalization) — SMOTE uses Euclidean distance, so unscaled features would let one large-range feature dominate the neighbour search.
Train a model robust to remaining imbalance, e.g. 4.1.2-Decision-trees-and-random-forests with class weights, or plug in 3.4.7-Cost-sensitive-learning.
Evaluate on the untouched real test fold with F1 / recall / PR-AUC (3.2.4-Precision-recall-and-F1-score), never raw accuracy. The combo (undersample then SMOTE) shrinks the giant majority AND enriches the minority — cheaper and less overfit-prone than SMOTE alone on a 1M-row table.

Exercise 4.2

In Exercise 4.1's pipeline, after step 3 how many total training samples do you have, and what is the memory saving factor versus not undersampling at all (SMOTE-only would balance to $1000000$ each)?

Recall Solution 4.2

After step 3: $10000$ legit $+ 10000$ fraud $=== 20000 ==$ training samples. SMOTE-only balance would give $1000000 + 1000000 = 2000000$ samples. Saving factor $= \frac{2 000 000}{20 000} === 100 \times ==$ smaller training set.

Level 5 — Mastery

Exercise 5.1

SMOTE geometry with a degenerate case. Minority points are $A = [0, 0]$ , $B = [4, 0]$ , and a lone outlier $C = [0, 10]$ . You run SMOTE with $k = 1$ (each point's single nearest neighbour). Using Euclidean distance, determine each point's nearest neighbour, then describe where synthetic points can and cannot appear — and why $C$ is dangerous.

Recall Solution 5.1

Distances (Euclidean, $Δ x^{2} + Δ y^{2}$ ):

$A \to B$ : $4^{2} + 0^{2} = 4$ ; $A \to C$ : $0^{2} + 1 0^{2} = 10$ . Nearest to $A$ is $B$ .
$B \to A$ : $4$ ; $B \to C$ : $4^{2} + 1 0^{2} = 116 \approx 10.77$ . Nearest to $B$ is $A$ .
$C \to A$ : $10$ ; $C \to B$ : $\approx 10.77$ . Nearest to $C$ is $A$ . Where synthetics appear: on segment $A \leftrightarrow B$ (when $A$ or $B$ is seed) and on segment $C \to A$ (when $C$ is seed). No synthetics land near the empty right side or off the segments. Why $C$ is dangerous: synthetics from $C$ stretch along the long $C \to A$ line (see the red segment in the figure), sprinkling minority points through mostly-empty space that may actually be majority territory. SMOTE cannot tell an outlier from a real cluster — it faithfully interpolates noise. Fix: clean outliers first or use a variant (Borderline-SMOTE) that skips isolated points.

Exercise 5.2

Prove that every SMOTE point lies inside the convex hull of the minority samples, so SMOTE can never invent a point beyond the existing data spread.

Recall Solution 5.2

Take any synthetic point $x_{new} = x_{i} + λ (x_{neighbor} - x_{i})$ with $λ \in [0, 1]$ . Rearrange: $x_{new} = (1 - λ) x_{i} + λ x_{neighbor}$ . The two weights $(1 - λ)$ and $λ$ are each $\geq 0$ and sum to $1$ — this is a convex combination of two real minority points. A convex combination of points that are themselves in a set always lies within that set's convex hull. Therefore every synthetic point sits on a line segment between two real minority points, hence inside their convex hull. ∎ Consequence: SMOTE never extrapolates outward — it can only fill inside the minority cloud. That is why it cannot fix a minority class that is simply under-sampled in range (the true minority region is bigger than what you observed).

Recall Quick self-check

Undersample keep-count formula ::: $N_{maj}^{'} = r \cdot N_{min}$ How SMOTE achieves diversity across many synthetic points ::: each point draws a fresh random $λ \sim Uniform (0, 1)$ Why SMOTE runs after the train/test split ::: to prevent synthetic points leaking real neighbours across folds What convex-combination fact bounds SMOTE points ::: they lie inside the convex hull of real minority samples