Stratified and leave-one-out cross-validation: Worked examples

This page is a workbench. In Stratified and leave-one-out cross-validation you met the two ideas; here we drive them through every situation they can meet — clean splits, ugly non-divisible splits, the degenerate one-sample-per-class case, a multi-class remainder mess, the extreme LOOCV limit, a real-world imbalance disaster, and an exam twist. Before each example, you forecast the answer. Guessing first is how the intuition sticks.

Prerequisites we lean on: 2.6.03-K-fold-cross-validation (the base split), 2.6.04-Bias-variance-tradeoff-in-cross-validation (why fold count matters), 2.6.01-Training-validation-and-test-sets (train/test roles), 3.2.05-Handling-imbalanced-classes (why minority classes break naive splits), and 2.3.08-Sampling-methods (how we draw without bias).

The scenario matrix

Every problem this topic can throw is one cell below. The worked examples that follow each declare which cell they cover, so together they fill the whole grid.

Cell	Scenario class	What is special / dangerous
A	Stratified, class count divisible by $k$	Clean case, exact per-fold counts
B	Stratified, class count NOT divisible by $k$	Remainder samples — floor/ceil distribution
B2	Stratified with 3+ classes, several remainders	Each class distributes its own remainder
C	Stratified with severe imbalance (rare class)	Random fold can hit zero minority samples
D	Degenerate: a class has fewer samples than $k$	Cannot put one in every fold — limit case
E	LOOCV numeric error on a small dataset	$k = n$ , average of per-sample losses
F	LOOCV classification (0/1 loss)	Loss is right/wrong, gives an accuracy
G	Limiting behaviour as $k \to n$	Cost and bias trend, why we stop somewhere
H	Real-world word problem	Translate a story into counts and folds
I	Exam twist: probability a fold misses a class	Uses the $(1 - 1/ k)^{n_{i}}$ formula

First, the symbols (earned before use)

The three symbols $⌊ \cdot ⌋$ , $⌈ \cdot ⌉$ and $mod$ appear constantly below, so define them on a picture.

Figure — Stratified and leave-one-out cross-validation

Look at the figure: the strip of length $n_{i}$ is cut into $k$ chunks. When the cut is not even, the plum chunks are one taller (the ceilings) and the teal chunks are the floors. Count the extra tiles: exactly $n_{i} mod k$ chunks (the remainder we just defined) get the extra one.

Cell A — clean divisible split

Worked example A: 100 samples, 70 negative vs 30 positive, 5-fold stratified

Statement: class 0 (negative) has $n_{0} = 70$ , class 1 (positive) has $n_{1} = 30$ , $k = 5$ . How many of each class per fold, and does each fold keep the proportions $p_{0}, p_{1}$ ?

Forecast: guess the two per-fold counts and whether each fold's positive proportion equals $p_{1} = 0.30$ .

Compute the target proportions. $p_{0} = 70/100 = 0.70$ , $p_{1} = 30/100 = 0.30$ . These are the numbers each fold must reproduce. Why this step? Stratification's contract is "keep $p_{i}$ per fold," so we first write down the $p_{i}$ we must hit.
Fold each class (algorithm step 2). $70/5 = 14$ negatives per fold; $30/5 = 6$ positives per fold. Why this step? Folding each class separately makes the counts deterministic, not left to chance.
Combine into folds (algorithm step 3). Fold size $= 14 + 6 = 20$ ; positive proportion $= 6/20 = 0.30 = p_{1}$ . Why this step? This is the whole point — confirm the fold reproduces $p_{1}$ exactly.

Verify: $14 \times 5 = 70$ ✓ and $6 \times 5 = 30$ ✓ (all placed once). Fold positive proportion $6/20 = 0.30 = p_{1}$ ✓; negative proportion $14/20 = 0.70 = p_{0}$ ✓.

Cell B — remainder, not divisible

Worked example B: 73 samples of a class, 5-fold — the leftover 3

Statement: One class has $n_{i} = 73$ samples, $k = 5$ . How are they distributed across folds?

Forecast: $73/5 = 14.6$ . How many folds get 15, how many get 14?

Floor first (algorithm step 2). $⌊ 73/5 ⌋ = 14$ . Give every fold 14. That uses $14 \times 5 = 70$ . Why this step? Guarantee a fair baseline everyone gets; only then hand out the leftovers.
Leftover $= 73 - 70 = 3$ . Equivalently $73 mod 5 = 3$ (the modulo we just defined). Why this step? The remainder is exactly how many "extra" samples still need a home.
Distribute leftovers one each. 3 folds get $⌈ 73/5 ⌉ = 15$ ; the other 2 folds keep 14. Why this step? Spreading remainders one-per-fold keeps folds as equal as possible (max difference = 1).

Verify: $3 \times 15 + 2 \times 14 = 45 + 28 = 73$ ✓. No fold differs from another by more than 1 sample ✓.

Refer back to the plum/teal strip in s01: here exactly 3 chunks are plum (15) and 2 are teal (14).

Cell B2 — three classes, several remainders at once

Worked example B2: multi-class stratified split,

$C = 3$ , k=4 Statement: A dataset has three classes: $n_{1} = 22$ (class A), $n_{2} = 15$ (class B), $n_{3} = 7$ (class C), total $n = 44$ . Use $k = 4$ stratified folds. How does each class distribute its remainder, and does every fold reproduce all three proportions?

Forecast: each class handles its own remainder independently — guess how many folds get the "extra" for A, for B, for C.

Target proportions. $p_{1} = 22/44 = 0.5$ , $p_{2} = 15/44 \approx 0.341$ , $p_{3} = 7/44 \approx 0.159$ . Why this step? These are the three ratios every fold must approximate simultaneously.
Fold class A. $⌊ 22/4 ⌋ = 5$ , remainder $22 mod 4 = 2$ . So 2 folds get 6, 2 folds get 5. Why this step? Each class independently splits by the same $k$ ; A's leftover is 2.
Fold class B. $⌊ 15/4 ⌋ = 3$ , remainder $15 mod 4 = 3$ . So 3 folds get 4, 1 fold gets 3. Why this step? B's leftover is 3 — a different number of "extra" folds than A. That is the multi-class subtlety.
Fold class C. $⌊ 7/4 ⌋ = 1$ , remainder $7 mod 4 = 3$ . So 3 folds get 2, 1 fold gets 1. Why this step? C's leftover is also 3, but from a much smaller pile — its per-fold sizes are just 1 or 2.
Assemble a representative fold. A fold receiving A=6, B=4, C=2 has size $12$ ; positive-of-A proportion $6/12 = 0.5 = p_{1}$ ; B $4/12 \approx 0.333$ ; C $2/12 \approx 0.167$ — all within one sample of the targets. Why this step? Confirms the fold approximates all three proportions at once, the multi-class guarantee.

Verify: A: $2 \times 6 + 2 \times 5 = 22$ ✓; B: $3 \times 4 + 1 \times 3 = 15$ ✓; C: $3 \times 2 + 1 \times 1 = 7$ ✓; grand total $22 + 15 + 7 = 44$ ✓. Each class's "extra" fold count equals its own $n_{i} mod k$ ( $2, 3, 3$ ) — they need not match ✓.

Cell C — severe imbalance (rare class saved)

Worked example C: 1000 patients, 950 healthy (negative) vs 50 diseased (positive), 10-fold

Statement: Rare-disease data ( $p_{1} = 5%$ positive), $k = 10$ . Show stratified counts per fold and per training set, then note the random-fold risk (quantified fully in Cell I).

Forecast: does each fold get any diseased patients under stratification? How many?

Target proportions. $p_{0} = 950/1000 = 0.95$ healthy, $p_{1} = 50/1000 = 0.05$ diseased. Why this step? Each fold must reproduce this 5% positive rate.
Fold each class (step 2). Healthy: $950/10 = 95$ per fold. Diseased: $50/10 = 5$ per fold. Why this step? Folding each class separately forces 5 diseased into every test fold — impossible to hit zero.
Test fold (step 3). $95 + 5 = 100$ , positive rate $5/100 = 0.05 = p_{1}$ ✓. Why this step? Confirms the test fold mirrors the real 5% prevalence, so metrics are honest.
Training fold = everything else $= (950 - 95) + (50 - 5) = 855 + 45 = 900$ , positive rate $45/900 = 0.05 = p_{1}$ . Why this step? The model always trains on 45 diseased examples — it can never collapse to "always predict healthy."

The random-fold danger: without stratification a random fold could contain zero diseased patients, and a model trained on it learns "always predict healthy" — 95% accuracy, useless. How likely is that empty fold? We derive that probability from scratch in Cell I below; for this $k = 10, n_{1} = 50$ case it comes out around $0. 9^{50} \approx 0.005$ per fold — small individually, but dangerous across many folds and many rare classes. See 3.2.05-Handling-imbalanced-classes.

Verify: $95 \times 10 = 950$ ✓, $5 \times 10 = 50$ ✓. Test rate $5/100 = 0.05 = p_{1}$ ✓; train rate $45/900 = 0.05 = p_{1}$ ✓.

Cell D — degenerate: class smaller than $k$

Worked example D: a class with only 3 samples in 5-fold

Statement: $k = 5$ but one class has $n_{i} = 3$ samples. Can every fold contain this class? What happens to its proportion $p_{i}$ ?

Forecast: with 3 samples and 5 folds, is it even possible to give each fold one?

Per-fold target $= ⌊ 3/5 ⌋ = 0$ with remainder $3 mod 5 = 3$ . Why this step? Floor is 0 — the "everyone gets a baseline" step gives zero.
Distribute the 3 leftovers. Only 3 of the 5 folds get a single sample of this class; 2 folds get none. Why this step? You physically cannot place 3 items into 5 non-empty boxes.
Consequence. The class proportion $p_{i}$ cannot be preserved — some folds show $p_{i} = 0$ . This is the hard limit of stratification. Fix: reduce $k$ (use $k \leq n_{i}$ so $k = 3$ here), merge rare classes, or use 2.3.08-Sampling-methods such as resampling. Why this step? Recognizing the degenerate case prevents silently broken folds.

Verify: $3$ folds $\times 1 + 2$ folds $\times 0 = 3$ ✓ (all placed). Number of empty-of-class folds $= k - n_{i} = 5 - 3 = 2$ ✓. The stratification guarantee ( $p_{i}$ in every fold) fails whenever $n_{i} < k$ .

Cell E — LOOCV numeric error (regression)

Worked example E: LOOCV squared-error on 5 samples

Statement: The five per-sample squared errors from the parent's worked run are $e_{1} = 1.44, e_{2} = 0.36, e_{3} = 0.64, e_{4} = 0.25, e_{5} = 0.49$ . Compute the LOOCV error.

Forecast: the average will be near the "middle-ish" error — guess between 0.5 and 0.7.

Recall the formula. $CV_{LOOCV} = \frac{1}{n} \sum_{i = 1}^{n} e_{i}$ with $n = 5$ . Why this step? Every sample is tested exactly once; averaging the $n$ single-sample losses is the definition.
Sum: $1.44 + 0.36 + 0.64 + 0.25 + 0.49 = 3.18$ . Why this step? Total held-out loss across all $n$ leave-one-out fits.
Divide by $n$ : $3.18/5 = 0.636$ . Why this step? Turns total into a per-sample expected error — comparable across datasets of different size.

Verify: $3.18/5 = 0.636$ ✓. Units: squared error (same units as $y^{2}$ ). It lies inside our forecast band ✓.

Cell F — LOOCV classification (0/1 loss)

Worked example F: LOOCV accuracy with right/wrong loss

Statement: 8 samples. In the 8 leave-one-out fits, the held-out sample is predicted correctly in 6 iterations and wrong in 2. Give the LOOCV error and accuracy.

Forecast: 2 wrong out of 8 — error should be a quarter.

Define the per-sample loss $e_{i}$ . Here the loss is the 0/1 loss: $e_{i} = 0$ if sample $i$ is predicted correctly, $e_{i} = 1$ if wrong. So each $e_{i}$ is literally "was this one wrong?" as a 0 or 1. Why this step? Classification has no "distance"; being right or wrong is the natural loss, and it plugs into the same $CV_{LOOCV} = \frac{1}{n} \sum e_{i}$ formula as Cell E — only the definition of $e_{i}$ changed.
LOOCV error $= \frac{1}{n} \sum_{i = 1}^{8} e_{i} = \frac{0 + 0 + 0 + 0 + 0 + 0 + 1 + 1}{8} = \frac{2}{8} = 0.25$ . Why this step? Six $e_{i}$ are 0 (correct) and two $e_{i}$ are 1 (wrong); the average of these 0/1 values is the fraction misclassified.
Accuracy $= 1 - 0.25 = 0.75 = 75%$ . Why this step? Accuracy is the complement of 0/1-loss error.

Verify: $2/8 = 0.25$ ✓, $1 - 0.25 = 0.75$ ✓. With 8 held-out tests the estimate can only take values $0, \frac{1}{8}, \frac{2}{8}, \dots$ — coarse, a known LOOCV feature.

Cell G — limiting behaviour as $k \to n$

Worked example G: training fraction and cost as

$k$ grows Statement: For $n = 200$ , tabulate the training fraction $\frac{k - 1}{k}$ and the number of models trained ( $= k$ ) for $k = 2, 5, 10, n$ . What is the trend?

Forecast: as $k$ climbs, does training data go up or down? Does cost go up or down?

Training fraction $= \frac{k - 1}{k} = 1 - \frac{1}{k}$ . Why this step? Each fold holds out $1/ k$ of the data, so the model trains on the rest.
Evaluate: $k = 2 \to 0.5$ ; $k = 5 \to 0.8$ ; $k = 10 \to 0.9$ ; $k = n = 200 \to 199/200 = 0.995$ . Why this step? Shows LOOCV (= $k = n$ ) trains on almost 100% — hence its near-zero bias (see 2.6.04-Bias-variance-tradeoff-in-cross-validation).
Cost = number of models = $k$ : $2, 5, 10, 200$ . Why this step? LOOCV multiplies compute by $n$ — the price of that low bias.

Verify: $1 - 1/2 = 0.5$ , $1 - 1/5 = 0.8$ , $1 - 1/10 = 0.9$ , $1 - 1/200 = 0.995$ ✓. Trend: training fraction $↑$ toward 1, cost $↑$ linearly — the tradeoff that stops us using LOOCV on huge datasets.

The figure plots both curves: the teal training-fraction climbing and flattening at 1, the burnt-orange cost rising as a straight line. LOOCV sits at the far right of both.

Cell H — real-world word problem

Worked example H: churn dataset from a story

Statement: A telecom keeps records of 4200 customers; 630 of them churned (left — the positive class). You want 6-fold stratified CV. How many churners and non-churners in each test fold and each training set, and does every split keep $p_{1}$ ?

Forecast: churn rate is $630/4200$ — guess the percentage first, then the per-fold churner count.

Global proportions. $p_{1} = 630/4200 = 0.15 = 15%$ churn (positive); $p_{0} = 3570/4200 = 0.85$ (negative). Why this step? Stratification must reproduce this 15% inside every fold.
Fold each class (step 2). Churners: $630/6 = 105$ per fold. Non-churners: $(4200 - 630) /6 = 3570/6 = 595$ per fold. Why this step? Fold each class separately for exact control — both divide evenly, so no remainder here.
Test fold (step 3). $105 + 595 = 700$ ; churn rate $105/700 = 0.15 = p_{1}$ ✓. Why this step? Confirms the test fold mirrors the real 15% churn rate.
Training set. The other 5 folds combined $= 4200 - 700 = 3500$ customers. Churners in training $= 630 - 105 = 525$ ; non-churners $= 3570 - 595 = 2975$ . Training churn rate $= 525/3500 = 0.15 = p_{1}$ ✓. Why this step? Confirms the training set also mirrors 15%, so the model always sees a realistic churn share while learning.

Verify: $105 \times 6 = 630$ ✓, $595 \times 6 = 3570$ ✓, $105/700 = 0.15 = p_{1}$ ✓, $525 + 2975 = 3500$ ✓, $525/3500 = 0.15 = p_{1}$ ✓.

Cell I — exam twist: probability a random fold misses a class

Cells C and D worried about a random fold containing zero of a class. Here we finally derive how likely that is — the formula both earlier cells pointed to.

Worked example I: chance a random fold has zero minority samples

Statement: $k = 5$ , a minority class has $n_{i} = 10$ samples, folds drawn at random (no stratification). What is the approximate probability that one given fold contains none of this class? Then repeat for $n_{i} = 4$ , and for the Cell C case ( $k = 10, n_{i} = 50$ ).

Forecast: more minority samples → less chance of missing them. Guess which of $n_{i} = 10$ vs $n_{i} = 4$ gives the bigger miss probability.

Model one sample's placement. Under the independence approximation, each of the $n_{i}$ minority samples lands in the chosen fold with probability $1/ k$ , so it avoids that fold with probability $1 - 1/ k = 1 - 1/5 = 0.8$ . Why this step? A fold is "missing" the class only if all $n_{i}$ minority samples avoid it, so we first need the avoid-chance of a single sample.
All avoid together. Treating the placements as independent, multiply the single-sample avoid-chances: $P (miss) \approx (1 - \frac{1}{k})^{n_{i}}$ . Why this step? "All of them avoid" = the product of each one avoiding, which is exactly the exponent $n_{i}$ .
Plug in $n_{i} = 10$ : $0. 8^{10} \approx 0.1074$ — about a 10.7% chance this fold is empty of the class. Why this step? This is the number the parent note quoted for $k = 5, n_{i} = 10$ ; we have now derived it.
Plug in $n_{i} = 4$ : $0. 8^{4} = 0.4096$ — about a 41% chance the fold misses the class. Why this step? Shrinking the minority class from 10 to 4 makes the miss far more likely — the danger explodes for rarer classes.
Plug in Cell C ( $k = 10, n_{i} = 50$ ): $(1 - 1/10)^{50} = 0. 9^{50} \approx 0.00515$ . Why this step? This closes the loop with Cell C — that ~0.5% per-fold miss chance we deferred is now derived here.

Verify: $0. 8^{10} \approx 0.1074$ ✓, $0. 8^{4} = 0.4096$ ✓, $0. 9^{50} \approx 0.00515$ ✓. As forecast, the smaller $n_{i}$ gives the larger miss probability ( $0.41 > 0.11$ ). This is exactly why stratification — or 3.2.05-Handling-imbalanced-classes resampling — is mandatory for rare classes.

Recall Quick self-test

Divisible split, 60 of a class, k=6, per fold? ::: $60/6 = 10$ per fold. Non-divisible: 62 of a class, k=6 — how many folds get the extra? ::: $62 mod 6 = 2$ folds get $⌈ 62/6 ⌉ = 11$ , the rest get 10. Three classes 22/15/7 in k=4 — do their "extra" fold counts have to match? ::: No; they are $2, 3, 3$ ( $= n_{i} mod k$ each), independent per class. A class has 4 samples, k=10 — is stratification possible? ::: No; $n_{i} < k$ so 6 folds get zero of it (Cell D degenerate). LOOCV error if per-sample squared errors sum to 3.18 over 5 samples? ::: $3.18/5 = 0.636$ . Probability a random k=5 fold misses a class with 10 samples? ::: $0. 8^{10} \approx 0.107$ (independence approximation).

Related routes onward: for ordered data where random folding is wrong entirely, see 5.4.02-Time-series-cross-validation.