Visual walkthrough — Handling imbalanced datasets (SMOTE, undersampling)

This page rebuilds the SMOTE result from the parent note — Handling imbalanced datasets — as a sequence of pictures. We start with nothing but dots on a page and end with the interpolation formula

${\mathbf{x}}_{\text{new}}={\mathbf{x}}_i+\lambda ({\mathbf{x}}_{\text{neighbor}}-{\mathbf{x}}_i)$

Every symbol will be earned before it is used. A smart 12-year-old who has never seen a vector should be able to follow from line one.

---

Step 1 — Draw the problem: dots on a map

WHAT. Imagine every data sample is a single dot on a flat map. Each dot's position is decided by two measurements. In our tumour example those two measurements are tumor_size (horizontal) and cell_density (vertical). We colour benign dots cyan and malignant dots amber.

WHY. Before any formula, we must see imbalance. "Imbalance" is an abstract word; on the map it is simply a cloud of cyan dots and only a tiny scatter of amber dots. The model, trying to draw a line that is right most often, will happily sacrifice the few amber dots.

PICTURE. Look at how sparse the amber region is. There is empty space between the amber dots — space where a real malignant case could plausibly live but where we have no data.

---

Step 2 — What "close" means: the Euclidean distance

WHAT. To find dots that are near a given amber dot, we need a number for "how far apart." We use the straight-line distance — stretch a ruler between two dots and read its length.

WHY this tool and not another? We want a single number that grows when points differ in any feature. The straight-line (Euclidean) distance answers exactly the question "how far is this dot from that dot, counting every feature at once?" Other rulers exist (Manhattan distance counts steps along the grid, Mahalanobis stretches axes by their spread), but Euclidean is the plain ruler and the SMOTE default.

PICTURE. The ruler forms the long side of a right triangle whose short sides are the gap in each feature. Pythagoras turns those two gaps into the one ruler-length.

For $F$ features the pattern just continues: sum a squared gap per feature, then take the square root. The parent note's ${\sum }_{f=1}^F$ is only shorthand for "add one such term for each of the $F$ features."

Recall

Why square the gaps before adding? ::: So they can't cancel out (a $+3$ gap and a $-3$ gap are both distance, not zero) and so larger differences count more.

---

Step 3 — Find the k nearest neighbours (amber only)

WHAT. Pick one amber dot, call it ${\mathbf{x}}_i$. Measure its distance to every other amber dot. Keep the $k$ closest ones — here $k=5$. These are its nearest neighbours.

WHY. New synthetic points must land in genuinely-malignant territory. If we let a benign dot be a neighbour, our invented line could cross into cyan land and produce a fake "malignant" that is really benign — a poisoned sample. So we search only among amber dots.

PICTURE. A dashed circle grows out of ${\mathbf{x}}_i$ until it captures exactly 5 amber dots. Everything inside the circle is a candidate partner; the cyan dots are ignored even when they are nearby.

---

Step 4 — Draw the line and slide along it: the interpolation

WHAT. From the 5 neighbours, pick one at random — call it ${\mathbf{x}}_{\text{neighbor}}$. Draw the straight segment from ${\mathbf{x}}_i$ to ${\mathbf{x}}_{\text{neighbor}}$. Now drop a new amber dot somewhere on that segment.

WHY a segment, and why random along it? A point sitting between two real malignant cases shares a blend of their traits, so it is a believable new malignant case. Choosing a random spot each time gives variety — every synthetic dot is slightly different, unlike simply copying an existing dot (which teaches the model nothing new).

PICTURE. The amber segment connects the two real dots; a fresh amber star sits partway along it. Move the star and it traces the whole segment.

We now need a symbol for "how far along the segment." Call it $\lambda$ (the Greek letter lambda), a dial from 0 to 1:

${\mathbf{x}}_{\text{new}}=\underset{\text{start dot}}{\underbrace{{\mathbf{x}}_i}}+\underset{\begin{array}{c}\text{how far along}\\ 0\to 1\end{array}}{\underbrace{\lambda }}\cdot \underset{\text{the arrow from start to neighbour}}{\underbrace{({\mathbf{x}}_{\text{neighbor}}-{\mathbf{x}}_i)}}$

Reading it in plain words: "Stand on the start dot. Look at the arrow pointing to the neighbour. Walk a fraction $\lambda$ of the way along that arrow. Plant the new dot there."

---

Step 5 — The three cases of $\lambda$ (including the degenerate ends)

WHAT. The dial $\lambda$ can land anywhere in $[0,1]$. We must check every possibility, especially the two ends, so the reader never meets a surprise.

WHY. The two endpoints are degenerate — they produce a duplicate rather than a fresh point. Knowing this tells us why SMOTE draws $\lambda$ from the open-ish interior almost always, and why duplicates (harmless but useless) are rare.

$\lambda$	Result	What it looks like
$\lambda =0$	${\mathbf{x}}_{\text{new}}={\mathbf{x}}_i$	new dot sits exactly on the start — a copy
$\lambda =1$	${\mathbf{x}}_{\text{new}}={\mathbf{x}}_{\text{neighbor}}$	new dot sits exactly on the neighbour — a copy
$0<\lambda <1$	a genuine in-between point	new dot strictly on the open segment

PICTURE. Three stars along one segment: one glued to each end (grey, "duplicate") and one healthy amber star in the middle.

---

Step 6 — Repeat until balanced: counting the new dots

WHAT. One synthetic dot barely helps. We repeat Steps 3–4 many times. How many? Enough to lift the amber count up to the cyan count.

WHY. Balance is the goal, so the number to create is simply "the shortfall":

$N_{\text{synthetic}}=\underset{\text{cyan count}}{\underbrace{N_{\text{maj}}}}-\underset{\text{amber count}}{\underbrace{N_{\text{min}}}}$

PICTURE. The sparse amber scatter fills in: real dots stay, and a swarm of synthetic amber stars appears strictly on the segments between them, thickening the minority cloud until it rivals the cyan.

---

The one-picture summary

Everything above, compressed: pick an amber dot, look only at amber neighbours, draw the segment, slide the dial $\lambda$, plant a fresh amber star, repeat until the amber cloud matches the cyan one.

Recall Feynman retelling — say it like you'd explain to a friend

We had a map with tons of cyan dots and hardly any amber dots, and our model kept ignoring the amber ones. So we made fake amber dots that look real. Here's the trick: take one real amber dot, find the five nearest amber dots to it (never cyan — we don't want to accidentally invent a fake in enemy territory), pick one of those five, and imagine a straight rope between them. Then we roll a random number between 0 and 1, walk that fraction of the way along the rope, and drop a brand-new amber dot right there. Roll a 0 and you land back on the start; roll a 1 and you land on the neighbour — those are just copies, so they're wasted but harmless. Any number in between gives a genuinely new "in-between" case. We do this over and over — exactly (cyan count minus amber count) times — and now both clouds are the same size, we deleted nothing, and the model finally has to learn the amber pattern.

---

Where to go next

• Distances need features on comparable scales → 2.1.3-Feature-scaling-and-normalization (a feature measured in millions dominates the Euclidean ruler otherwise).
• Judge the payoff with the right yardsticks, not accuracy → 3.2.4-Precision-recall-and-F1-score.
• Resample inside each fold, never before → 2.1.8-Cross-validation-and-holdout-methods (SMOTE-ing the whole set before splitting leaks synthetic twins into the test set).
• Prefer to reweight instead of resample? → 3.4.7-Cost-sensitive-learning.
• Fill gaps before you measure distances → 2.1.10-Dealing-with-missing-data.
• Models that consume the rebalanced set → 4.1.2-Decision-trees-and-random-forests.