5.6.5 · D5Machine Learning (Aerospace Applications)
Question bank — Cross-validation — k-fold
Before you start, make sure three words mean something concrete to you:
True or false — justify
True or false: 5-fold CV trains one model and tests it five times.
False. It trains five separate models, each on a different four-fold training set; a single trained model tested five times would be tested partly on its own training data.
True or false: In k-fold CV every data point is used for testing exactly once.
True. Each row lives in exactly one fold, and each fold is the test set exactly once — so every row is tested once and used for training times.
True or false: Increasing always makes the CV estimate more accurate.
False. Larger lowers bias (bigger training sets) but raises variance (tiny, noisy test folds whose overlapping training sets make scores correlated), so it is a tradeoff, not a free win. See 5.6.10-Bias-variance-tradeoff-in-model-selection.
True or false: The final CV number is a property of one specific trained model.
False. It estimates the performance of the learning procedure on data of this size — none of the models is your deliverable. You retrain on all rows afterward.
True or false: A CV mean of tells you your deployed model will score .
Mostly true but only approximately: the deployed model is retrained on all rows (slightly more data), so its true performance is usually a hair better than the CV estimate, which is why CV is a mild underestimate.
True or false: Stratified k-fold is only about balancing classes and never changes the folds for regression.
True for the standard tool — stratification keeps class proportions equal, which only applies to classification. For a continuous target you'd need a binned or custom scheme. See 5.6.04-Stratified-sampling.
True or false: Reporting only the CV mean is enough to trust a model.
False. The mean hides consistency; a model is trustworthy while signals the score depends heavily on which rows landed in the test fold. Always report the standard deviation too.
True or false: k-fold CV eliminates overfitting.
False. It detects and estimates overfitting by showing the gap between training and test-fold scores; it does not prevent the model from memorizing the training folds.
True or false: With shuffle=False, k-fold cuts the data into contiguous blocks.
True. It slices in row order, so if rows are sorted by year, fold 1 is all early data and fold 5 all late data — usually a bad idea unless order carries meaning.
True or false: For time-ordered flight logs, ordinary shuffled k-fold is the correct default.
False. Shuffling lets the model train on the future to predict the past — leakage of temporal order. Use forward-chaining instead: 8.5.02-Time-series-cross-validation.
Spot the error
Spot the error: "I scaled all features on the full dataset, then ran 5-fold CV."
The scaler learned the mean/std using the test folds too — that is leakage. Fit any preprocessing inside each fold on the training portion only, then apply it to the test fold.
Spot the error: "I tuned hyperparameters on my CV score, then reported that same CV score as the model's performance."
The CV set was used to choose the model, so it is no longer unseen — reporting it is optimistic. You need an outer loop that never touched tuning: 5.6.07-Nested-cross-validation.
Spot the error: "My classes are 95/5 imbalanced, so I used plain KFold and got wildly varying fold scores."
Random folds can end up with 2% vs 10% failures, so each fold measures a different problem. Use
StratifiedKFold to hold the 5% rare-class rate steady in every fold.Spot the error: "I oversampled the minority class to balance the data, then split into folds."
Synthetic/duplicated copies of the same sample can land in both train and test folds — leakage. Resample after splitting, using only the training fold.
Spot the error: "I have duplicate flight records from the same aircraft, and I let KFold split them randomly."
Near-identical rows from one aircraft can straddle the train/test boundary, leaking the answer. Use group-aware splitting so all rows of one aircraft stay in a single fold.
Spot the error: "I used random_state=42 in KFold but a different seed inside RandomForest, then said the folds caused the score change."
Two different randomness sources are moving at once; you can't attribute variance to folds. Fix both seeds so a change is traceable to one cause.
Spot the error: "My dataset is only 40 rows so I picked (LOOCV) to be safe."
LOOCV here gives high-variance, correlated single-row scores and is 40 model fits; small samples usually favour a modest . See 5.6.06-Leave-one-out-cross-validation-(LOOCV).
Spot the error: "I did feature selection (kept the top 20 sensors by correlation with the label) on the whole dataset, then CV'd."
Selection peeked at the labels of every fold, so the chosen features already 'know' the test rows — leakage. Redo selection inside each training fold.
Why questions
Why do we average the fold scores instead of just taking the best one?
The best fold is the luckiest split, not the honest one; the mean estimates typical performance and, by averaging, damps the noise of any single split.
Why divide by (not ) in the fold standard deviation?
The scores are compared to their own sample mean, which uses up one degree of freedom; dividing by gives an unbiased estimate of the spread rather than a systematically-too-small one.
Why does small raise bias but large raise variance?
Small means each model trains on less data, so it looks worse than the full-data model (pessimistic bias). Large means test folds are tiny and noisy, and the heavily-overlapping training sets make the scores correlated, inflating variance.
Why is the F1 score used instead of accuracy for rare engine failures?
With 5% failures, always predicting "no failure" scores 95% accuracy while catching zero failures. F1 combines precision and recall, so it collapses when the rare class is missed.
Why must preprocessing live inside the fold loop?
Because the test fold must simulate genuinely unseen future data; letting a scaler or imputer see it means your model has knowledge at test time it won't have in deployment.
Why does k-fold CV still slightly underestimate the final model's performance?
Each fold model trains on only of the data, whereas the deployed model trains on all rows, so it has a touch more to learn from than any fold model did.
Why prefer 5-fold CV over a single 80/20 holdout for scarce aerospace data?
The holdout uses 20% of precious labelled flights purely for testing and gives one number, while 5-fold reuses every row for both roles and yields five numbers, exposing variance. Contrast with 5.6.02-Holdout-method and 5.6.01-Train-test-split-validation-set.
Edge cases
Edge case: 1003 rows and — what happens to the "equal" folds?
They can't be exactly equal; the library gives some folds 201 rows and others 200. It's fine — folds are approximately equal by design, not exactly.
Edge case: a fold ends up with zero positive (failure) examples.
F1 (and recall) become undefined or zero for that fold, poisoning the mean. Stratified k-fold prevents this by forcing the rare-class proportion into every fold.
Edge case: you set (one row per fold).
This is leave-one-out CV — nearly full-size training sets (low bias) but each test score is a single, noisy prediction and you fit models. See 5.6.06-Leave-one-out-cross-validation-(LOOCV).
Edge case: you set .
Meaningless — you'd have one fold used for both training and testing, so there is no unseen data at all. The smallest sensible value is .
Edge case: you set .
Legal, but each model trains on only half the data (high pessimistic bias) and there are just two scores, so the std is barely informative. It behaves like a symmetric double holdout.
Edge case: all your rows are nearly identical.
Every fold scores about the same, giving a tiny std that looks like great stability but really reflects no diversity in the data — CV can't warn you about a scenario absent from the data.
Edge case: CV mean is high but every fold's training score is far higher still.
Large train-minus-test gap across all folds is the fingerprint of overfitting; the model memorizes training folds and CV is correctly reporting weaker generalization.
Edge case: you run CV during grid search over 100 hyperparameter combos and pick the best CV score.
With 100 tries, the winning score is partly luck (the multiple-comparisons trap); confirm it on an untouched outer fold via 5.6.07-Nested-cross-validation before trusting it. See also 7.2.03-Hyperparameter-tuning-grid-search.
Recall Quick self-test
What is the single most common cause of an over-optimistic CV score? ::: Data leakage — preprocessing, feature selection, resampling, or tuning that peeked at the test fold before it was tested. Does k-fold CV give you a model to deploy? ::: No; it estimates the procedure's performance, after which you retrain on all rows.