Stratified and leave-one-out cross-validation: Visual walkthrough

This page rebuilds the two big ideas from the parent note using nothing but pictures and counting. If you have never seen the word "fold" before, start here — we build every symbol from scratch.

Before we start, one word we will use constantly:

We will assume you have met plain k-fold cross-validation and know what a training/test split is. Everything else we derive.

Step 1 — The data as coloured dots

WHAT. Picture every sample as a dot. We colour each dot by its class — the category we are trying to predict (e.g. lavender = "healthy", coral = "diseased"). We have $n$ dots in total. The number of dots of class $i$ is written $n_{i}$ .

WHY. Cross-validation is really a story about how the colours get distributed when we chop the pile. To see the problem, we first have to see the colours. When one colour is rare (few coral dots), that is exactly when trouble starts.

PICTURE. Look at the two rows in the figure. The top strip is our whole dataset: mostly lavender, a sprinkle of coral. Count them: $70$ lavender, $30$ coral, so $n = 100$ .

Figure — Stratified and leave-one-out cross-validation

Step 2 — Why a random chop can go wrong

WHAT. Suppose we blindly shuffle the dots and deal them into $k = 5$ piles. Each pile grabs dots at random, ignoring colour.

WHY. We need to feel the danger before we fix it. Randomness is fine when a colour is common, but a rare colour is like drawing a rare marble — some piles may get none at all.

PICTURE. In the figure, watch the middle pile: pure lavender, zero coral. A model tested on that pile never gets asked about coral, so its score is a lie.

How likely is a "no-coral" pile? Each coral dot lands in our chosen pile with probability $\frac{1}{k}$ , so it misses that pile with probability $1 - \frac{1}{k}$ . All $n_{i}$ coral dots must miss for the pile to be empty of coral:

Step 3 — The fix: sort by colour first

WHAT. Instead of shuffling everything together, we first separate the dots by colour into $C$ groups $D_{1}, D_{2}, \dots, D_{C}$ . Here $C = 2$ : a lavender group of $70$ and a coral group of $30$ .

WHY. If we control each colour separately, no colour can vanish by accident. This is the whole trick of stratification — divide-and-conquer, one colour at a time. It is the same spirit as stratified sampling methods.

PICTURE. Two clean rows now — all lavender on top, all coral below, ready to be sliced.

Step 4 — Slice each colour into $k$ equal sub-piles

WHAT. Cut the lavender row into $k = 5$ equal chunks, and cut the coral row into $k = 5$ equal chunks — independently.

WHY. By cutting within a colour, each chunk holds a guaranteed slice of that colour. Lavender: $70/5 = 14$ each. Coral: $30/5 = 6$ each. No luck involved.

PICTURE. Both rows now show $5$ tidy blocks. The vertical dashed lines mark the cuts.

Step 5 — Reassemble: one slice of each colour per fold

WHAT. Fold $j$ is built by grabbing the $j$ -th chunk from every colour and stacking them: $F_{j} = D_{1}^{(j)} \cup D_{2}^{(j)} \cup \dots \cup D_{C}^{(j)}$

WHY. Stacking one guaranteed slice per colour forces each fold to carry all colours in the right amounts. This is where the proportion gets locked in.

PICTURE. Fold 1 = $14$ lavender + $6$ coral = $20$ dots. Follow the arrows: one lavender chunk and one coral chunk merge into a single mixed fold that mirrors the original strip from Step 1.

Now check the colour proportion inside fold $j$ :

Step 6 — Now push $k$ to its limit: LOOCV

WHAT. Forget colours for a moment. Ask: what if we make the folds as small as possible? Set $k = n$ . Then each fold is a single dot, and the training pile is everything else — $n - 1$ dots.

WHY. Bigger training piles mean each practice-model is nearly the real model we will ship. Pushing $k$ to $n$ squeezes out the most training data possible. This is Leave-One-Out CV.

PICTURE. The ladder shows training fraction climbing as $k$ grows: $50%$ at $k = 2$ , $80%$ at $k = 5$ , $90%$ at $k = 10$ , and $(n - 1) / n \approx 100%$ at $k = n$ .

Step 7 — LOOCV by hand (5 dots)

WHAT. Work a tiny example so the formula becomes muscle memory. Five dots; leave each out once; fit a line to the other four; measure the miss.

WHY. Averages hide the mechanics. Watching one iteration makes the sum concrete.

PICTURE. Iteration 1: the coral dot (sample 1) is hidden; the mint line is fitted to the four lavender dots; the vertical arrow is the error — the gap between true $y = 2$ and predicted $\overset{y}{^} = 3.2$ .

Iteration 1: $\hat{f}^{(- 1)} (1) = 0.7 (1) + 2.5 = 3.2$ , so $e_{1} = (2 - 3.2)^{2} = 1.44$ . Iteration 2: $\hat{f}^{(- 2)} (2) = 0.8 (2) + 1.8 = 3.4$ , so $e_{2} = (4 - 3.4)^{2} = 0.36$ . The remaining errors come out to $e_{3} = 0.64, e_{4} = 0.25, e_{5} = 0.49$ .

$CV_{LOOCV} = \frac{1.44 + 0.36 + 0.64 + 0.25 + 0.49}{5} = \frac{3.18}{5} = 0.636$

Step 8 — The price: bias vs variance

WHAT. LOOCV gives almost zero bias (training on $n - 1$ ≈ training on $n$ ) but high variance, because the $n$ training piles overlap in $n - 2$ dots and are nearly identical.

WHY. You cannot get both cheap and stable. This tension is the theme of bias–variance in CV. Small $k$ = low variance but higher bias; $k = n$ = low bias but wobbly, correlated errors.

PICTURE. As $k$ rises, the bias curve falls toward zero while the variance curve climbs. LOOCV sits at the far-right: bias floor, variance ceiling.

The one-picture summary

The left half is stratification: sort by colour, slice each colour, reassemble so every fold mirrors the original mix. The right half is LOOCV: shrink the test fold to a single dot, maximising training data at the cost of variance and compute.

Recall Feynman retelling — say it back in plain words

Imagine a bag of $100$ marbles, $70$ lavender and $30$ coral. If I blindly deal them into $5$ cups, one cup might have no coral — and a model quizzed only on that cup never learns coral exists. So instead I sort first: lavender in one line, coral in another. I cut the lavender line into $5$ equal bits ( $14$ each) and the coral line into $5$ equal bits ( $6$ each). Then I build each cup from one lavender bit plus one coral bit — every cup is $14 + 6 = 20$ marbles, always $70%/30%$ , just like the bag. That's stratified CV, and the two "divide by $5$ "s cancel so the ratio survives exactly.

LOOCV is the opposite extreme of a different question — how much do I train on? Instead of $5$ big cups, I make $100$ cups of one marble each. Each round I hide one marble, train on the other $99$ , and check if I guessed the hidden one. I do this $100$ times and average the misses. I train on almost everything (great — low bias), but my $100$ mini-datasets barely differ, so my scores jiggle together (high variance) and I paid for $100$ full trainings. Stratify to protect rare classes; go leave-one-out only when data is tiny and precious.

Recall

What single quantity does stratification keep identical between every fold and the full dataset? ::: The class proportion $p_{i} = n_{i} / n$ . In LOOCV, how many samples train the model each iteration? ::: $n - 1$ (all but the one held out). Why does LOOCV have high variance? ::: Its $n$ training sets overlap in $n - 2$ samples, so the error estimates are highly correlated. Probability a random fold misses a class with $n_{i}$ members? ::: $(1 - \frac{1}{k})^{n_{i}}$ .