2.6.4 · D5Model Evaluation & Selection
Question bank — K-fold cross-validation
Before we start, a one-line refresher of the words used everywhere below:
Recall Vocabulary you must have before answering
Fold ::: One of the K equal-sized chunks the dataset is chopped into; it plays "validation set" exactly once. ==CV estimate () ::: The average of the K scores, one score per fold used as validation. Bias (of the estimate) ::: A systematic gap between the CV score and the score the final all-data model would truly get. Variance (of the estimate)== ::: How much would jump around if you re-ran CV on a fresh dataset or reshuffled. Data leakage ::: Information from a validation fold sneaking into training (or into preprocessing), making scores look better than reality.
True or false — justify
Each item: decide true/false and give the reason. A right answer with a wrong reason is a wrong answer.
Every data point is used for validation exactly once across all K folds.
True. Each fold is the validation set in exactly one iteration , and the folds are disjoint and cover , so no point is validated twice or skipped.
Every data point is used for training exactly once.
False. Each point sits in training sets — it is trained on times and validated on once; only validation is a "once" event.
Larger K always gives a better (more trustworthy) estimate.
False. Larger K lowers bias but raises variance and cost; K=10 is a balance, not a maximum-is-best knob. LOOCV (K=N) is nearly unbiased yet has the highest variance.
is an estimate of how the final model (trained on all N points) will generalize.
True, but approximately. Each fold model trains on only points, so slightly underestimates the all-data model — this gap is the bias.
If all K fold scores are nearly identical, the model is guaranteed to be good.
False. Low spread means the estimate is stable/consistent, not that the level is high. A model can be consistently bad (all folds ≈ 0.55).
K-fold CV eliminates overfitting.
False. It detects and estimates generalization performance; it does not change how the model is trained. You can still overfit inside each training fold.
For a fixed dataset, running 5-fold CV twice with different random shuffles gives identical results.
False. Different shuffles produce different fold memberships, hence different scores — this run-to-run wobble is exactly the variance we worry about.
The standard error across folds measures the noise in a single test example.
False. It measures how much the per-fold scores vary around — it is about fold-level stability, not individual predictions.
Doubling K roughly doubles the computational cost.
True. You train one model per fold, so cost scales linearly with K (K trainings). K=10 is ~10 model fits.
Spot the error
Read the (flawed) reasoning; name the mistake.
"I normalized my whole dataset first, then split into folds — cleaner code, same result."
Data leakage. The mean/std were computed using validation points too, so each validation fold peeked at the training statistics; fit the scaler on the training fold only.
"I have 90 class-A and 10 class-B samples, so plain random 5-fold is fine."
Risk of empty-class folds. Random splitting can dump all class-B into one or two folds, leaving some validation folds with zero class-B — use stratified K-fold to preserve the 9:1 ratio in every fold.
"I ran 10-fold CV to pick hyperparameters and reported that same CV score as my final performance."
Optimistic (selection) bias. The folds were used to choose the model, so reporting their score double-counts. Use nested CV or a separate held-out test set for the final estimate.
"I tuned a scaler and model in a pipeline, but fit the scaler once outside the CV loop for speed."
Leakage again. Any learned preprocessing (scaling, PCA, imputation, feature selection) must be refit inside each fold on the training portion only.
"My time-series model got 95% with 5-fold CV, so it'll work in production."
Shuffling breaks temporal order. Random folds let the model train on the future to predict the past — use forward-chaining / time-series CV instead.
"With K=N (leave-one-out) I average N nearly-perfect training runs, so variance is tiny."
Backwards. LOOCV training sets are almost identical, so the N models are highly correlated → high variance of the estimate, not low.
"I oversampled the minority class to balance data, then did 5-fold CV on the oversampled set."
Leakage through duplicates. Copies of the same point land in both train and validation folds, inflating the score; oversample inside each training fold only.
Why questions
Explain the reason, not just the fact.
Why do we average the K scores instead of picking the best fold's score?
Because averaging cancels the lucky/unlucky noise of individual splits (variance reduction); the max would just report our luckiest fold, a biased-optimistic number.
Why must the folds be (roughly) equal in size?
So each validation set represents the same fraction of the data, making the K scores comparable and their average a fair, evenly-weighted estimate.
Why does smaller K produce a pessimistic (upward-biased-error) estimate?
Small K means each model trains on a smaller fraction (e.g., K=3 → only 2N/3), and models trained on less data perform worse, so the CV score undershoots the all-data model's true ability.
Why does the standard error tell us something the mean alone cannot?
The mean gives the performance level; the SE reveals reliability — a large SE flags that the model is sensitive to which data it trains on (possible instability or a non-representative fold).
Why is K=5 or K=10 the usual recommendation rather than K=2 or K=N?
They sit in the sweet spot: training sets are 80–90% of the data (low bias) while only 5–10 models are averaged (moderate variance and cost) — a practical bias-variance-compute compromise.
Why must preprocessing be refit inside the loop even though it "seems deterministic"?
Any statistic learned from data (mean, std, class frequencies, principal components) carries information; computing it once over all data lets validation folds influence their own training, silently leaking the answer.
Why does stratified K-fold specifically help imbalanced classification?
It forces every fold to keep the global class ratio, so no validation fold ends up missing a class — which would otherwise make that fold's metric undefined or meaningless.
Edge cases
Boundary and degenerate scenarios — the reader must never be surprised in the wild.
What happens if K equals N?
You get Leave-One-Out CV: N models, each validated on a single point — nearly unbiased but maximum variance and maximum compute.
What if K = 1?
It's undefined/degenerate: with one fold there is no held-out validation data at all, so you'd train and "test" on everything — not cross-validation.
What if N is not divisible by K (e.g., N=100, K=7)?
Folds can't all be exactly equal; implementations make some folds one sample larger (sizes 15 or 14 here), which is fine — "equal-sized" means as-even-as-possible.
What if a class has fewer than K members in stratified K-fold?
You cannot place one of that class in every fold, so at least one fold lacks it — reduce K, merge rare classes, or gather more data.
What does it mean if one fold's score is wildly different from the others?
That fold likely contains an unrepresentative slice (outliers, a rare subgroup, or a leakage/labeling issue); investigate rather than blindly average it away.
If two models have overlapping CV score ± SE ranges, can you declare one better?
Not confidently — the difference may be within the noise of the split; you'd need a paired statistical test across matched folds, not just a mean comparison.
What if the dataset is tiny (say 20 samples)?
Prefer larger K or LOOCV to keep training sets as large as possible (low bias); accept the higher variance because wasting any data on a fixed holdout would hurt more.