Stratified and leave-one-out cross-validation: Question bank

This page is a question bank for the ideas in the parent note. Each line is a trap. Read the left side, answer out loud, then reveal the right side. The answers are reasoning, not "yes/no" — that reasoning is the whole point.

First, a shared vocabulary so no term is used before it is anchored:

Definition Words and symbols you'll see below

Fold — one of the $k$ equal-ish chunks the data is cut into. In each round, one fold is the test set, the rest are the training set.
$n$ — total number of samples (rows) in the dataset.
Class $i$ — one of the possible labels, numbered $i = 1, 2, \dots, C$ where $C$ is the number of classes.
$n_{i}$ — how many rows carry label class $i$ . They add up to the whole: $\sum_{i} n_{i} = n$ .
$p_{i}$ — the proportion of class $i$ , i.e. $p_{i} = \frac{n _{i}}{n}$ . Example: 30 positives out of 100 rows gives $p_{pos} = \frac{30}{100} = 0.3$ .
Stratified — folds are built so each one copies the whole dataset's class proportions $p_{1}, p_{2}, \dots, p_{C}$ .
LOOCV — the extreme where $k = n$ : every fold is a single row.
Bias of a CV estimate — how far the average estimate sits from the true error you'd measure on infinite fresh data.
Variance of a CV estimate — how much the estimate jumps around if you reran on a slightly different dataset.

Two pictures ground everything below. The first shows why random splitting is dangerous and stratified splitting is safe; the second shows why LOOCV has high variance.

Figure — Stratified and leave-one-out cross-validation

Look at the top row (random split): one fold caught zero red (minority) rows — that fold's metric is worthless. The bottom row (stratified split): every fold carries the same red-to-black ratio as the full bar on the left.

The red overlap band shows that any two LOOCV training sets share $n - 2$ rows. Nearly-identical training sets make nearly-identical mistakes → correlated errors → the averaged estimate stays jittery (high variance).

True or false — justify

Stratified CV changes the model's training algorithm.

False. It only changes which rows land in which fold. The learner, its loss, and its hyperparameters are untouched — stratification is a splitting rule, not a training rule.

Stratified CV can be applied to regression targets exactly as to classification labels.

False as stated. "Class" needs discrete labels. For regression you must first bin the continuous target into buckets (e.g. quantile bins), then stratify on the buckets — otherwise "same class distribution" is undefined.

If every fold has the same class proportion as the whole dataset, then every fold's training set also has that proportion.

True. The training set is the union of the other

k - 1

folds; a union of pieces that each carry proportion

p_{i}

also carries

p_{i}

. Both test and train mirror the original (see the parent's medical example: 95% healthy in each).

LOOCV always gives lower test-error variance than 5-fold CV.

False. LOOCV has low bias but often high variance, because its

n

training sets overlap in

n - 2

samples (figure s02), making the

n

error terms highly correlated. Correlated errors don't average away.

LOOCV is guaranteed to be more accurate than 10-fold CV.

False. "More data per fit" lowers bias, but the high variance can make a single LOOCV number noisier as an estimate. Accuracy of the estimate is bias and variance together.

Stratified 5-fold CV and plain 5-fold CV cost the same in compute.

True. Both train exactly

k = 5

models on

\approx \frac{4}{5} n

rows each. Stratification only reorders which rows go where — no extra fits.

For a perfectly balanced dataset (50/50), stratified and random splitting give each fold the same class proportions.

False in general. In expectation each fold is 50/50, but a single random split can still drift (e.g. 55/45 in one fold). Stratification pins every fold to 50/50 exactly; balance only shrinks how big the random drift typically is. (This is about fold proportions, not about a particular model's metric.)

LOOCV with

n

samples performs

n

training runs.

True. Each of the

n

rows is held out exactly once, so there are

n

folds and

n

fits — this is why LOOCV is expensive for large

n

Increasing

k

toward

n

always reduces the bias of the CV error estimate.

True (as a trend). Larger

k

means each training set is a larger fraction

\frac{k - 1}{k} n

of the data, so each fitted model is closer to the final full-data model, shrinking bias.

Stratified CV removes class imbalance from the dataset.

False. It preserves imbalance faithfully in every fold. Fixing imbalance is a different job — see 3.2.05-Handling-imbalanced-classes. Stratification just stops random splits from worsening it.

Spot the error

"We have 950 healthy and 50 diseased. In random 10-fold CV each fold will contain about 5 diseased patients, so stratification is unnecessary."

Error: "about 5" is only the average. Random draws fluctuate; some folds can get 0 or 1 diseased (top row of figure s01). The whole point (parent's disaster case) is that the minority count is small enough for a fold to miss the class.

"To stratify, I shuffle all rows, then cut into

k

equal blocks."

Error: That is plain k-fold. Stratifying requires splitting within each class first (all

C

classes), then combining one sub-group per class into each fold. Blind shuffling doesn't lock the proportions.

"LOOCV has zero bias, therefore its error estimate equals the true test error."

Error: Bias

\approx 0

means the expected estimate is near the truth. Any single LOOCV run can still be far off due to high variance. Low bias

\neq =

low error on one run.

"Since LOOCV trains on

n - 1

samples, the model it validates is a different model from my final one, so the estimate is biased."

Error:

n - 1

is almost identical to

n

, so the validated model is essentially the final model — this is precisely why LOOCV bias is near zero, not why it's biased.

"I stratified on the target, so my time-series CV is now valid."

Error: Stratification ignores order. For time series you must not let future rows train a model tested on the past — that needs 5.4.02-Time-series-cross-validation, not stratification.

"Each fold in stratified CV has exactly

\frac{n _{i}}{k}

samples of class

i

Error: Only when

n_{i}

is divisible by

k

. Otherwise folds carry

⌊ \frac{n _{i}}{k} ⌋

⌈ \frac{n _{i}}{k} ⌉

— a

\pm 1

rounding wobble the parent notes explicitly. Here

n_{i}

is the count of class

i

(see vocabulary).

Why questions

Why can a random fold end up with zero minority samples?

Because each minority row independently "misses" a fold with probability

1 - \frac{1}{k}

; for a small count

n_{i}

the chance all of them miss,

(1 - \frac{1}{k})^{n_{i}}

, is non-trivial (e.g.

0. 8^{10} \approx 0.107

for

k = 5

Why does stratification make validation metrics more reliable, not just prettier?

Because a fold that reflects the true 5% positive rate produces a metric measured under realistic conditions; a fold with 0% positives measures nothing about the model's ability on positives, so its "accuracy" is meaningless.

Why do LOOCV's

n

error terms end up correlated?

Any two of its training sets share

n - 2

of the same rows (figure s02), so they fit almost the same model and make almost the same mistakes — their errors move together instead of independently.

Why does correlation between fold-errors raise the variance of the averaged estimate?

Averaging kills variance only when terms are independent. Correlated terms don't cancel each other's fluctuations, so the mean stays jittery — high variance survives the averaging.

Why is stratified CV especially tied to 3.2.05-Handling-imbalanced-classes?

Imbalance is exactly when random splits break: the rare class is fragile. Stratification is the first line of defence that guarantees the rare class appears in train and test of every fold.

Why does the parent frame LOOCV as the "limit" of k-fold rather than a separate method?

Because it is k-fold with

k = n

— same averaging formula

\frac{1}{n} \sum_{i} L (y_{i}, \hat{f}^{(- i)} (x_{i}))

, just the extreme setting. Seeing it as a limit explains its low-bias/high-variance profile via the 2.6.04-Bias-variance-tradeoff-in-cross-validation.

Edge cases

What happens to stratified CV if a class has fewer than

k

samples?

You cannot place one member in every fold — some folds get zero of that class no matter what (our class C with

n_{3} = 4

dies at

k = 5

). The stratification guarantee breaks; you must lower

k

or merge/handle that rare class first.

With many classes of varying rarity (say

C = 10

, some with 3 rows, some with 500), how do you assign folds?

Stratify class by class: split each

n_{i}

into

k

near-equal sub-groups and take one per fold. The binding constraint is the rarest class —

k

must satisfy

k \leq min_{i} n_{i}

, or the rare classes can't spread across all folds.

How would you stratify a continuous (regression) target?

There are no classes to copy, so you manufacture them: cut the target into quantile bins (each bin holds an equal count), then stratify on the bin labels. This "quantile-based folding" keeps each fold's target distribution — not just its mean — close to the whole.

Can you stratify a multi-label target (each row carries several labels at once)?

Not with the simple per-class recipe, because label combinations overlap. Specialized iterative stratification algorithms greedily place rows to balance every label's proportion simultaneously; plain single-label stratification would break some labels while fixing others.

What does LOOCV cost when

n = 1, 000, 000

One million model fits, each on

\approx n

rows — usually infeasible. This is why LOOCV is reserved for small

n

(or models with closed-form leave-one-out shortcuts).

For a 50/50 balanced, huge dataset, is stratification worth it?

The benefit is tiny (random drift is small when both

n

and balance are large), but the cost is also ~zero, so it's harmless and still the safe default.

If every sample is identical in features but labels split 70/30, does stratification help the model learn?

No — the model can't separate identical inputs. Stratification still balances the folds, but it fixes split fairness, never the underlying learnability of the data.

With only 2 classes and

k = n

(LOOCV), can each single-row test fold be "stratified"?

No. A one-row fold holds one label, so it can never mirror a 70/30 ratio. Stratification is meaningful only when folds are large enough to hold a mix — it and LOOCV pull in opposite directions.

Does stratified CV protect you if your data was collected with a sampling bias?

No. It preserves whatever distribution the data has, bias included. Fixing collection bias is upstream work — see 2.3.08-Sampling-methods; CV cannot repair a skewed sample.

Recall Quick self-test

One-fold-missing-a-class probability formula ::: $(1 - \frac{1}{k})^{n_{i}}$ — the chance all $n_{i}$ minority rows avoid a given fold. LOOCV bias vs variance in one phrase ::: near-zero bias, often high variance from overlapping training sets. The one thing stratification never changes ::: the actual class imbalance — it copies it, it doesn't cure it. Constraint on $k$ for stratified CV with rarest class count $min_{i} n_{i}$ ::: $k \leq min_{i} n_{i}$ , else some folds miss the rarest class.