Cross-validation pitfalls and nested CV: Worked examples

This page is a drill hall. The parent note built the theory of why standard cross-validation lies and how nesting fixes it. Here we walk through every kind of situation the topic can throw at you, one worked example at a time, so that when you meet one in the wild — or in an exam — you have already seen its twin.

Before starting, three plain-language reminders so we never use a symbol we did not earn:

Definition The words and symbols we will use constantly

Bias (of an estimate), with a fixed sign convention: take the true generalisation error $E_{true}$ (what the model really scores on brand-new data) minus the number you actually write in your report $E_{reported}$ (whatever CV hands you), i.e. $bias = E_{true} - E_{reported}$ . With this convention a positive bias means the report is smaller than the truth — the model looks better than it is. We call that optimistic. So throughout this page, positive = optimistic (report too rosy), negative = pessimistic (report too harsh). In every worked example below the arithmetic " $E_{true} - E_{reported}$ " is spelled out explicitly so you always see which number plays which role. Every leakage we meet gives a positive (optimistic) bias.
Leakage: any moment where information from data you pretend is unseen sneaks into a decision. Choosing features, scaling, or hyperparameters using the held-out fold is leakage. See 2.6.16-Data-leakage.
Dataset symbols: write $D$ for the whole labelled dataset (every row you have). When cross-validation splits it, $D_{train}$ is the rows a step is allowed to learn from and $D_{test}$ is the rows held out to grade that step. Any function that makes a decision — e.g. a feature-picker written $S (\cdot)$ — is "honest" only if it is fed $D_{train}$ and never $D_{test}$ .
Hyperparameter count $m$ : the number of distinct hyperparameter settings (configs) you try. Tuning Ridge over $α \in {0.01, 0.1, 1, 10}$ means $m = 4$ ; fixing one $α$ up front means $m = 1$ .

If 2.6.2-K-fold-cross-validation (splitting data into K equal chunks, training on K−1 and testing on 1, rotating) is fuzzy, read that first — everything below assumes it.

The scenario matrix

Every cross-validation mistake is one of a small number of shapes. This table lists every shape; the examples that follow each fill one or more cells.

Cell	Scenario class	The dangerous move	Covered by
A	Hyperparameter tuning, healthy data	report min-over-λ CV score	Ex 1
B	Feature selection before CV	rank features on all data	Ex 2
C	Preprocessing before CV (scaling)	fit scaler on all data	Ex 3
D	Degenerate: only ONE λ tried ( $m = 1$ )	(no search → no selection bias)	Ex 4
E	Limiting case: tiny data / huge feature count	noise correlates by chance	Ex 5
F	Reporting the wrong number after correct nested CV	quote inner score	Ex 6
G	Real-world word problem (medical, grouped data)	random split breaks groups	Ex 7
H	Exam twist: which number to deploy	confuse estimate vs final model	Ex 8

The "sign axis" of this whole page. Picture a single number line for the bias $E_{true} - E_{reported}$ . Its origin (zero) is an honest report; everything to the right (positive) is optimistic — the report looks better than reality; everything to the left (negative) would be pessimistic. The rule that ties every cell together: leakage can only push you rightward (optimistic); it never pushes left. Cell D is exactly the origin — with $m = 1$ there is no minimum-over-random-variables trick, so no selection bias arises. Keep that one-directional asymmetry in mind; it is the through-line of every example.

Figure — Cross-validation pitfalls and nested CV

Reading the figure (s01) — this is the map for the whole page. Alt-text / caption: "Two error curves versus number of configs searched; a flat teal honest curve, a dipping orange leaky curve, and a plum optimism band between them, touching at m=1." The horizontal axis is $m$ , how many configs you searched (also read as "how much you leaked"). The vertical axis is the reported error. The flat teal curve is the honest nested estimate $E_{true}$ — it does not improve just because you searched more. The dipping orange (red) curve is the leaky standard estimate $E_{reported}$ — it sinks lower the more configs you try, because taking a minimum over more noisy numbers finds a luckier one. The vertical height of the shaded plum band between them is exactly $E_{true} - E_{reported}$ , the optimism (positive bias) from the definition above. At the far left ( $m = 1$ , the black dot) the two curves touch — the band has zero height — and that is Cell D, the honest origin. Read every example below as "how far right on this band am I?"

Example 1 — Cell A: hyperparameter tuning done two ways

Forecast: guess before reading — which number is smaller, and by roughly how much?

Standard CV number = the minimum over the $m = 4$ configs. $min {0.30, 0.26, 0.23, 0.28} = 0.23$ . This is the $E_{reported}$ of standard CV. Why this step? Standard practice picks the best-scoring config and quotes that same score. Taking a minimum over $m = 4$ noisy numbers systematically lands below their average — this is the selection bias the parent note derived.
Nested CV number = the average of the outer-fold errors. This is our truth-proxy $E_{true}$ . $\frac{0.31 + 0.19 + 0.25 + 0.21 + 0.27}{5} = \frac{1.23}{5} = 0.246$ Why this step? Each outer test fold ( $D_{test}$ for that fold) was never seen while its own α was chosen. Averaging (not minimising) over independent estimates removes the optimism.
Optimism $= E_{true} - E_{reported} = 0.246 - 0.23 = 0.016$ . Why this step? With our sign convention, taking nested ( $E_{true} = 0.246$ ) minus standard ( $E_{reported} = 0.23$ ) gives $+ 0.016$ — a positive, hence optimistic, gap. Report the standard 0.23 and you claim ~1.6 error-points better than reality.

Verify: the standard score must be ≤ the mean of the four CV scores ( $= 0.2675$ ). Indeed $0.23 \leq 0.2675$ ✓, and the optimism is positive (nested > standard) ✓ — bias sits to the right of zero (optimistic), matching Cell A's sign rule.

Example 2 — Cell B: feature selection inside vs outside the loop

Forecast: will putting feature selection inside the loop raise or lower the reported accuracy?

Outside-loop selection uses all of $D$ , so features 2 and 4 (correlations 0.60, 0.55) were chosen partly because they matched the held-out rows too. Why this step? Let $S (\cdot)$ be the feature-picker (recall from the definitions: it is honest only if fed $D_{train}$ ). Here it was fed the whole $D$ , so it saw the $D_{test}$ rows it will later be graded on — textbook leakage.
The honest pipeline is "select then evaluate on unseen". Inside the loop, for each fold the selector is fed only that fold's $D_{train}$ (the 90% training slice); the 10% $D_{test}$ rows had no vote. Why this step? We want to estimate performance of the whole pipeline " $S$ then $f$ " — first pick features on $D_{train}$ , then fit the model — on rows it never touched. Formally we compare $Loss (f_{S (D)}, D)$ (leaky, uses all of $D$ ) against the honest $Loss (f_{S (D_{train})}, D_{test})$ .
Report the inside-loop 0.78, discard 0.92. In error terms (1 − accuracy) this is $E_{true} = 0.22$ vs $E_{reported} = 0.08$ , so bias $= 0.22 - 0.08 = + 0.14$ ; equivalently the accuracy drop $0.92 - 0.78 = 0.14$ . Why this step? The drop is the leakage you removed. Bigger drop ⇒ selection was fitting noise.

Verify: the two retained features are exactly those with correlation $> 0.5$ ({0.60, 0.55}) ✓; honest ≤ biased (0.78 ≤ 0.92) ✓; optimism direction = higher accuracy for the leaky method ✓ (accuracy leaks up, error leaks down — both land on the positive/optimistic side).

Example 3 — Cell C: scaling fit on the whole dataset

Forecast: is the leak here large or subtle compared to Example 2?

Leaky mean $= \frac{2 + 4 + 4 + 4 + 5 + 5}{6} = \frac{24}{6} = 4.0$ . Why this step? This mean was computed with the future $D_{test}$ values included — so the scaler is contaminated.
Which std? Population vs sample — call it out. Deviations from the mean are ${- 2, 0, 0, 0, 1, 1}$ , squares ${4, 0, 0, 0, 1, 1}$ , summing to $6$ .
- Population std (divide by $n = 6$ , the convention scikit-learn's StandardScaler uses): variance $= 6/6 = 1.0$ , so $σ_{pop} = 1.0 = 1.0$ .
- Sample std (divide by $n - 1 = 5$ , the default of numpy.std(ddof=1), pandas.std, and most stats courses): variance $= 6/5 = 1.2$ , so $σ_{samp} = 1.2 \approx 1.095$ . Why this step? The number you get for the scale depends on this $n$ -vs- $(n - 1)$ choice, and libraries disagree — this is a classic silent mismatch. For leakage the point is identical either way; we quote both so you are never surprised by which one a tool returns.
Correct fix: inside each outer fold, fit the scaler on training rows ( $D_{train}$ ) only, then transform the $D_{test}$ rows. The scaler must never fit on data it is about to be judged on. Why this step? Preprocessing is part of the pipeline; it must live inside the loop exactly like the model.

Verify: mean $= 4.0$ ✓; population std $= 1.0$ ✓ and sample std $= 1.2 \approx 1.095$ ✓. This is a subtle leak (unlike Ex 5): scaling rarely inflates scores much unless the feature distribution is wild — but it is still Cell C leakage.

Example 4 — Cell D: the ZERO case (only one hyperparameter, $m = 1$ )

Forecast: does the parent note's "minimum over $m$ configs" argument even apply here?

Count the configurations searched: $m = 1$ (recall $m$ = number of hyperparameter settings tried). Why this step? The optimism came from taking a $min$ over several noisy CV scores. With one config there is nothing to minimise over.
CV score = plain average $= \frac{0.24 + 0.22 + 0.26 + 0.23 + 0.25}{5} = \frac{1.20}{5} = 0.24$ . Why this step? This is already an (approximately) unbiased estimate of generalisation — no selection happened, so $E_{reported} = 0.24 \approx E_{true}$ and the bias $\approx 0$ .
Conclusion: nested CV is unnecessary for the estimate when $m = 1$ and no feature selection / preprocessing decisions use the folds. Why this step? This is the degenerate cell: bias $\approx 0$ , sitting exactly at the origin of our sign axis. It is the honest baseline the other cells deviate from.

Verify: plain average $= 0.24$ ✓; with $m = 1$ the "min over $m$ " equals the single value, so the selection-bias term vanishes ✓.

Example 5 — Cell E: limiting case, tiny n, huge p (Cawley–Talbot flavour)

Forecast: with 10× more features than samples, guess how much accuracy is illusory.

Honest skill above chance $= 73% - 50% = 23%$ . Why this step? Anything above 50% for balanced binary is genuine signal — the nested number is trustworthy, so it is our $E_{true}$ side (in accuracy terms).
Illusory (leaked) accuracy $= 91% - 73% = 18%$ . Why this step? With $p = 1000 ≫ n = 100$ , many features correlate with $y$ by chance. Selecting on all of $D$ validates on the very rows those features were fit to. In error terms $E_{true} - E_{reported} = (1 - 0.73) - (1 - 0.91) = 0.27 - 0.09 = + 0.18$ , a large positive/optimistic bias.
Interpretation: almost as much of the biased headline is noise-fitting (18 points) as is real (23 points). This is the far-right end of the plum band in Figure s01. Why this step? This is why the topic exists — small-n/large-p is where naive CV fails catastrophically. Links to 2.7.3-Bias-variance-tradeoff: high-variance selection on tiny data.

Verify: honest lift $= 23$ ✓; leaked lift $= 18$ ✓; both non-negative and sum to the total lift above chance ( $23 + 18 = 41 = 91 - 50$ ) ✓.

Example 6 — Cell F: correct nested run, wrong number reported

Forecast: is the inner 0.16 optimistic or pessimistic relative to the truth?

The performance estimate is the OUTER average $= 0.246$ (from Ex 1). This is $E_{true}$ . Why this step? Inner scores exist only to choose λ; they are contaminated by selection within that fold, so they cannot serve as the honest performance number.
The inner 0.16 is doubly optimistic, so setting $E_{reported} = 0.16$ is unjustified: it is (a) the winning config's own tuning score — already a minimum over the inner search — and (b) then hand-picked as the smallest across the five outer folds. Why this step? Two nested "pick the best" operations each drag the number below its honest average (the same min-over-noise mechanism as Cell A, applied twice), so 0.16 is far to the optimistic right of $E_{true}$ and must be rejected as a performance claim.
Overstatement $= E_{true} - E_{reported} = 0.246 - 0.16 = 0.086$ . Why this step? Reporting 0.16 claims your model is ~8.6 error-points better than the honest nested estimate — a large positive/optimistic gap.

Verify: outer average $= 0.246$ ✓; the cherry-picked inner value $0.16$ is less than every plausible honest number ( $0.16 < 0.246$ ) ✓; overstatement $= 0.086$ ✓.

Example 7 — Cell G: real-world word problem, grouped data

Forecast: does random row shuffling help or hurt the honesty here?

Random shuffling puts the same patient in both $D_{train}$ and $D_{test}$ . A patient's visit 3 can train while visit 4 tests. Why this step? The model memorises patient identity, not disease pattern — a classic leakage (see 2.6.16-Data-leakage).
Group-wise split keeps all 5 visits of a patient together, so test patients are genuinely new. Why this step? The regulator's question is about new patients, so the evaluation unit must be the patient, not the row; the grouped 0.71 is the honest $E_{true}$ and the random 0.88 is the leaky $E_{reported}$ .
Answer the regulator with 0.71, the grouped number. Leakage inflated AUC by $0.88 - 0.71 = 0.17$ (positive/optimistic, since higher AUC = rosier report). Why this step? The question defines the correct split; matching the split to the deployment scenario is the whole game.

Verify: grouped ≤ random (0.71 ≤ 0.88) ✓; inflation $= 0.17$ ✓; direction optimistic (AUC leaks up) ✓ — same one-directional sign law as every other cell.

Example 8 — Cell H: exam twist, "which number to deploy?"

Forecast: are these two answers the same object or different objects? (They are different — that is the trap.)

(a) Expected error = outer average = 0.246. This is the honest $E_{true}$ you quote. Why this step? Nested CV estimates the procedure's performance, not any one model. This is the number you report.
(b) The shipped model: aggregate the outer folds' chosen α, then retrain on ALL 100 samples. The two standard aggregation rules are (i) majority vote — take the α that appears in the most outer folds — and (ii) the "refit" rule — ignore the per-fold winners and simply run one ordinary inner-CV hyperparameter search on the entire dataset. Both are accepted practice (scikit-learn's typical workflow uses (ii): the outer loop is only for scoring, and the deployed model comes from a final GridSearchCV.fit on all data). Here the per-fold winners are ${0.1, 1, 1, 1, 0.1}$ ; the majority-vote rule gives counts $α = 1$ : 3 times, $α = 0.1$ : 2 times ⇒ choose $α = 1$ , then retrain on all 100 rows. Why this step? You want the strongest final artefact, so use every row — but its performance is still estimated by 0.246, not re-measured on the training data. Majority vote is the simplest defensible aggregator when the per-fold winners are stable (as here, 3 vs 2); if they disagreed wildly you would prefer rule (ii).
Do NOT report a new CV score for the retrained-on-all model. That model has no held-out data left, so any number it produces would drop $E_{reported}$ artificially below $E_{true}$ — re-introducing optimism. Why this step? The estimate (a) and the artefact (b) are separate deliverables. Confusing them re-introduces the very optimism nested CV removed. (Compare choosing when to stop in 3.4.5-Early-stopping: the stopping decision is tuning, its score is not the final estimate.)

Verify: majority-vote winner of ${0.1, 1, 1, 1, 0.1}$ is $1$ (count 3 > 2) ✓; deployment estimate stays 0.246, unchanged by retraining ✓.

Recall Quick self-test

Standard CV reports the ??? over the $m$ configs, nested reports the ??? over outer folds. ::: minimum ; average With our sign convention $E_{true} - E_{reported}$ , does leakage give a positive or negative bias? ::: positive (optimistic) If you search only ONE hyperparameter config ( $m = 1$ ) and touch no folds for preprocessing, is your CV biased? ::: No — selection bias needs $m \geq 2$ (Cell D, the origin) After nested CV you deploy the model by retraining on ??? data, using an ??? rule for α, and quote the ??? as its error. ::: all ; aggregation (majority vote or a final full-data search) ; outer average