Exercises — Cross-validation — k-fold

• Models trained: one per fold, so $==5==$ models.
• Test appearances per point: the whole point of k-fold is that every sample lands in the test fold exactly once. So each point is tested $==1==$ time (and used for training in the other $k-1=4$ rounds).

Look at figure s01: follow any single row across the four columns — it is coloured "test" (pink) in exactly one column and "train" (blue) in the other three.

Each fold holds $n/k=200/4===50==$ samples.

• Test set = one fold = $50$ samples.
• Training set = the other $k-1=3$ folds = $3\times 50===150==$ samples.

${\text{CV}}_3=\frac{0.80+0.86+0.83}{3}=\frac{2.49}{3}===0.83==.$ We take the mean because averaging independent estimates reduces variance — no single fold's luck dominates.

Fold size $=1000/5=200$.

• Test set $=200$, training set $=1000-200===800==$.
• Fraction used for training $=\frac{k-1}{k}=\frac{4}{5}===0.8===80\%$.

This is why k-fold estimates are close to full-data performance: every model still sees $80\%$ of everything, unlike LOOCV's tiny hold-out (see 5.6.06-Leave-one-out-cross-validation-(LOOCV)).

Mean: ${\text{CV}}_5=\frac{0.82+0.79+0.85+0.81+0.83}{5}=\frac{4.10}{5}===0.820==.$ Deviations from mean (each ${\text{Score}}_i-0.82$): $[0.00,-0.03,0.03,-0.01,0.01].$ Squared: $[0,0.0009,0.0009,0.0001,0.0001]$, sum $=0.0020$. Sample variance: divide by $k-1=4$: $\frac{0.0020}{4}=\mathrm{0.0005.}$ Std: $\sqrt{0.0005}\approx ==0.0224==$. So report $\text{F1}=0.820\pm 0.022$.

We divide by $k-1$, not $k$, because we already "spent" one degree of freedom estimating the mean; using $k$ would systematically underestimate the spread.

With shuffle=False, fold 1 = oldest flights, fold 5 = newest. Each round tests on a time-contiguous block that may differ systematically from the training block (older engines, seasonal weather, different routes). The model is asked to generalise across a distribution shift it never trained on, so scores drop — but this reflects the split, not the model. Fix: shuffle (shuffle=True), which makes each fold a random sample of all dates. ⚠️ Exception: if the task is genuinely a forecast, you must not shuffle — use 8.5.02-Time-series-cross-validation instead.

• For B: its mean F1 ($0.85$) beats A's ($0.82$).
• Against B: its spread ($\pm 0.09$) is over four times A's ($\pm 0.02$). A worst-plausible fold for B sits near $0.85-0.09=0.76$, below A's worst fold near $0.82-0.02=0.80$. B is inconsistent — its score depends heavily on which slice of data it sees, a red flag for a safety task.
• What to do: the means are only $0.03$ apart while B's std is $0.09$; the difference is not clearly significant. Run a paired comparison across the same folds, or increase $k$/repeats. In aerospace, prefer the model whose lower bound is safest — often A. See 5.6.10-Bias-variance-tradeoff-in-model-selection.

• Bias ↓ (lower): each training set now has $n-1$ samples, almost the full dataset, so each model's performance is nearly what the full-data model would achieve. Small-$k$ models train on only $\frac{k-1}{k}$ of the data and slightly under-perform, biasing the estimate pessimistically.
• Variance ↑ (higher): two mechanisms. (1) Each test set is a single sample, so each ${\text{Score}}_i$ is $0$ or $1$-ish — extremely noisy. (2) The $n$ training sets overlap in $n-2$ of their samples, so the models are almost identical and their errors are correlated; averaging correlated numbers barely reduces variance.
• Net: LOOCV trades bias for variance, and costs $n$ model fits. That is why $k=5$ or $10$ is the usual sweet spot.

Each fold has $200$ samples; the expected positives per fold are $50/5=10$. A fold with only $2$ positives means:

• Recall is measured over just $2$ true positives — miss one and recall drops from $1.0$ to $0.5$; a single mistake swings the score wildly.
• The training folds now hold $48$ positives concentrated elsewhere, so class balance differs round to round, injecting extra variance.

Fix: Stratified k-fold (5.6.04-Stratified-sampling) forces each fold to hold $\approx 5\%$ positives, i.e. $\approx 10$ per fold, stabilising every ${\text{Score}}_i$.

• Flaw: the CV score was used to choose max_depth. Reporting that same score as the final estimate is optimistically biased — you peeked at the validation folds to make a decision, so they are no longer unseen. The best-of-$4$ CV scores is inflated by selection.
• Correct design — nested CV (5.6.07-Nested-cross-validation):
  • Outer loop ($5$ folds): each outer training set is passed to an inner CV that tunes max_depth (this is grid search, 7.2.03-Hyperparameter-tuning-grid-search).
  • The winning depth is refit on the whole outer-training set and scored once on the untouched outer-test fold.
  • Average the $5$ outer-test scores → the honest estimate.
• The key principle: the data used to select a model must be disjoint from the data used to estimate its performance.

Nested-CV fit count (ignoring the final refits, or counting them separately): $\text{fits}=k_o\times (\text{settings})\times k_i=k_o\cdot 4\cdot k_i.$ Try $k_o=3,k_i=5$: $3\times 4\times 5===60==.$ That exactly hits the budget. If you also count one refit per outer fold ($k_o=3$ extra fits) you'd exceed it, so drop to $k_i=4$: $3\times 4\times 4=48$ fits, leaving room. Either $(k_o,k_i)=(3,5)$ at the ceiling or $(3,4)$ with headroom is defensible; state your assumption.

• Leakage risk: two flights from the same aircraft share serial-number-specific sensor quirks. If some flights of aircraft #3 are in training and others in test, the model can "recognise" that aircraft rather than learn general failure signatures — optimistic and unsafe. So splits must be at the aircraft (group) level: all $30$ flights of an aircraft go entirely to train or entirely to test. This is group k-fold.
• Class balance: with $6\%$ failures we also want each fold near $6\%$ — that argues for stratification. Combining both gives stratified group k-fold: partition by aircraft while balancing the failure rate across groups.
• Natural $k$: $8$ aircraft → up to $k=8$ folds, each fold = the $30$ flights of one held-out aircraft (a group-LOOCV over aircraft). If $8$ is too costly or too variable, group them $2$ aircraft per fold for $k=4$.
• Why this is the mastery answer: it simultaneously respects the two independence assumptions k-fold silently relies on — that (a) folds are disjoint in the units that matter (aircraft, not rows), and (b) each fold represents the true label distribution.

Mean: ${\text{CV}}_k=\frac{1}{k}{\sum }_{i=1}^{k}c=\frac{1}{k}(kc)===c==.$ Std: each deviation ${\text{Score}}_i-{\text{CV}}_k=c-c=0$, so the sum of squares is $0$ and ${\sigma }_{\text{CV}}=\sqrt{0}===0==.$ Interpretation: in practice fold scores are never exactly equal; a reported ${\sigma }_{\text{CV}}=0$ almost always signals a bug — e.g. the same data leaked into every fold, or the metric is being computed on the training set. Perfect consistency is a warning, not a trophy.

k-fold trains ==$k$ models, and each sample is tested exactly once==. The training fraction per model is ::: $\frac{k-1}{k}$ Sample std of fold scores divides the sum of squares by ::: $k-1$ Fix for rare-class folds ::: stratified k-fold Fix for same-aircraft leakage ::: group (aircraft-level) k-fold Honest estimate while tuning hyperparameters ::: nested cross-validation