Question bank — Cross-validation pitfalls and nested CV

True — for one model chosen in advance, the K held-out folds are genuinely independent of training, so the average $\mathrm{CV}$ is honest. The bias only appears once you select among many models using those same scores.

False — you never trained on them. The bias comes from choosing ${\lambda }^{\ast }$ because it won on those specific folds; selection, not training, is the leak.

True — the minimum of the $m$ noisy scores $\mathrm{CV}(\lambda )$ sits below their mean, so ${\min }_{\lambda }\mathrm{CV}(\lambda )$ is negatively biased as an error estimate; the more configs $m$, the larger the dip.

True — each outer test set is set aside before the inner loop tunes, so no decision ever touched it, making the outer average an honest estimate of the whole pipeline.

False — the inner score is optimistic by construction (it's a min over $m$ configs). It is only for picking $\lambda$; performance is the outer average.

False — a tie does not undo selection bias. You still have to pick one config using the same folds that would then grade it, and the reported score is still ${\min }_{\lambda }\mathrm{CV}(\lambda )$ over all $m$ configs — the winning value is low partly because it got lucky on those folds. A single tie changes nothing about the search-wide optimism that nested CV exists to price in.

False — different outer folds may pick different ${\lambda }^{\ast (i)}$. The outer average estimates pipeline performance; to ship, you retune on all data (or use the most-frequent ${\lambda }^{\ast }$), separately from the estimate.

False — larger data means lower-variance $\mathrm{CV}$ scores, so the "lucky minimum" wins by less; the optimism shrinks. Small data is where standard CV lies most.

The feature-selection step saw the validation folds, so the chosen features are tuned to noise in those very folds — this is leakage; feature selection must live inside each outer training split.

The scaler learned statistics from the held-out folds, leaking their distribution into training. Fit the scaler on the training portion only, inside the loop, and apply it to the held-out fold.

You cherry-picked an optimistic, selection-biased score from one fold's tuning. Report the outer fold average instead — that's the number no decision contaminated.

The inner folds would then include outer-test samples for some outer folds, re-leaking the test data into tuning. Inner splits must be drawn within each outer training set only.

With $m=200$ the min-over-configs bias is large; that score is a lower bound on error, not an estimate. You measured "best of 200 lucky draws," not generalization.

Early stopping is a decision made from validation performance, so those folds are no longer independent — the reported error is optimistic just like hyperparameter tuning.

Outer folds share overlapping training data, so their errors are correlated; the naive variance underestimates true uncertainty — treat the spread as a rough guide, not a clean standard error.

The single-model score never involved a choice; the tuned score is the winner of a competition judged on the same folds, so it reflects both skill and luck on those folds.

Each of the $m$ configurations is another independent chance to fluke a low $\mathrm{CV}$ on these particular folds, and the minimum of more noisy numbers drifts further below the mean.

Because each outer fold reruns the entire recipe — feature selection, tuning, fitting — from scratch on its own training data, so the outer error reflects everything the recipe would do on genuinely new data.

With 1000 features and 100 samples, many features correlate with $y$ by pure chance; standard CV selected those on the same data it validated on, fitting noise, while nested CV exposes that the noise doesn't transfer.

Not training on data is only half of independence — every decision (features, $\lambda$, stopping) informed by that data spends its independence, even without a single gradient step on it.

Each of the $K_{\text{outer}}$ outer folds runs a full inner CV (which itself trains $K_{\text{inner}}$ models per config), so the total number of model fits multiplies across both loops.

A truthful 73% lets you set correct expectations and make sound deployment decisions; a fake 91% invites failure in production and destroys trust when reality bites.

No — with $m=1$ there is no selection and no minimum to exploit, so plain CV is an honest estimate; the pitfall requires an actual choice among configs.

No — a stable choice is reassuring but the outer folds still provide the unbiased performance estimate; consistency of ${\lambda }^{\ast }$ tells you tuning is robust, not that you can skip the outer loop.

As a rule of thumb you want each outer test fold to hold at least ~10–20 samples and each inner fold likewise, so with 5×5 nesting you already need on the order of 100+ samples; below ~50 the inner folds shrink to a handful of points, tuning picks $\lambda$ essentially at random, and the outer estimate — while still unbiased — becomes too noisy to trust. Under a few dozen samples, prefer repeated CV or a Bayesian/hold-out alternative.

It removes selection bias (each single point was never used to tune its own prediction), but LOO's outer errors are highly correlated and high-variance, so the estimate is unbiased yet unstable.

No — a fixed, data-independent transform learns nothing from any fold, so there is nothing to leak; only steps that fit on data (scaling, imputation, feature selection) must live inside.

No — random folds let future rows train models tested on the past, a different leak; you need forward-chaining (time-respecting) splits in both loops, with the same never-touch-the-test rule.

Yes, if you stratify both loops so each fold keeps the class ratio; without stratification a fold may miss a rare class, wrecking both tuning and the estimate regardless of the nesting.