Worked examples — Cross-validation — k-fold

Before anything, three plain-word reminders so no symbol here is unearned:

Recall What do

$n$ and $k$ mean again? $n$ ::: the total number of samples (rows) in your dataset. $k$ ::: the number of folds — how many equal piles we cut the data into. One fold ::: one of those piles, used once as the test set while the other $k-1$ piles train the model.

Recall What is a "score" and why do we average?

Score of one fold ::: a single number (accuracy, F1, RMSE…) telling how well the model did on that held-out pile. Why average the $k$ scores ::: one split can get lucky or unlucky; the mean of $k$ splits is a steadier estimate of true performance.

The two metrics we will actually score with

Every fold's "score" on this page is one of two numbers. Define them once, here, so nothing below is borrowed on faith.

---

The scenario matrix

Every k-fold problem is a combination of the axes below. Each cell is a distinct thing that can go wrong or behave differently — so each gets its own worked example.

Cell	Case class	What makes it special	Covered by
A	Clean, balanced, $n$ divisible by $k$	The textbook case, every fold identical size	Example 1
B	$n$ not divisible by $k$ (remainder)	Folds have unequal sizes — how are leftovers spread?	Example 2
C	Degenerate small: $k=n$ (LOOCV limit)	Test fold has one sample; score is 0 or 1	Example 3
D	Degenerate small: $k=2$	Only two folds — closest to a single split	Example 4
E	Imbalanced classes (rare event)	Random folds distort class ratio → stratify	Example 5
F	Regression metric (RMSE) not accuracy	Averaging error, not "correctness"; units matter	Example 6
G	Time-ordered data (leakage trap)	Shuffling steals the future → forward-chaining	Example 7
H	Exam twist: variance vs mean judgement	Two models, same mean, different std — which wins?	Example 8

---

Example 1 — Cell A: the clean divisible case

Forecast: guess the mean before reading on — is it above or below $0.85$?

The figure below shows the rotation: each row is one of the five folds, blue cells are the 8 training samples, and the single orange block of 2 cells is that fold's test set. Trace down the rows and you can see the orange block slide left-to-right — every sample lands in the test set exactly once, and each row's per-fold accuracy is printed on the right.

Figure 1. Five stacked rows, one per fold; a two-cell orange test block slides rightward down the rows while the other eight blue cells train, and the fold's accuracy ($0.90,0.80,1.00,0.80,0.90$) is printed at each row's right edge.

1. Find fold size. $n/k=10/5=2$. Each fold holds exactly 2 samples (one orange block in the figure), each model trains on the other 8 (the blue cells). Why this step? We must confirm folds are equal — Cell A only holds when $n$ divides evenly by $k$.
2. Average the scores (the CV score). ${\text{CV}}_5=\frac{0.90+0.80+1.00+0.80+0.90}{5}=\frac{4.40}{5}=0.88$ Why this step? The parent formula ${\text{CV}}_k=\frac{1}{k}\sum {\text{Score}}_i$ — averaging is what turns 5 lucky/unlucky numbers into one estimate.
3. Spread across folds (sample standard deviation). ${\sigma }_{\text{CV}}=\sqrt{\frac{1}{k-1}{\sum }_i({\text{Score}}_i-0.88{)}^2}$ The squared deviations are $0.0004,0.0064,0.0144,0.0064,0.0004$, summing to $0.028$. Divide by $k-1=4$: $0.007$. Square root: $0.0837$. Why this step? We divide by $k-1$, not $k$, because the mean itself was estimated from the same data — one degree of freedom is "used up". This is the sample standard deviation.

Forecast check: you were asked to guess if the mean beats $0.85$. It does — $0.88>0.85$. The eyeball intuition works because three of the five scores ($0.90,1.00,0.90$) sit above $0.85$ and only pull the average up; that "count how many are above the line" trick is the mental shortcut the forecast was training.

Verify: the five scores lie in $[0.80,1.00]$; a mean of $0.88$ sits inside that range ✓. Total $=4.40$ split into 5 equal shares of $0.88$ ✓.

---

Example 2 — Cell B: $n$ not divisible by $k$

Forecast: $23/5=4.6$ — you cannot have $4.6$ samples in a fold. So what happens to the leftover?

The figure below lays the 23 samples out as five coloured bars. Notice the first three bars are one cell longer than the last two: that extra cell is the remainder being handed out, one per fold, to the earliest folds. The caption confirms the sizes add back to $n=23$.

Figure 2. Five horizontal bars labelled fold 1–5 with lengths $5,5,5,4,4$; the first three are one cell longer (the remainder handed out one-per-fold), and a caption confirms $5+5+5+4+4=23=n$.

1. Split the remainder. $23=5\times 4+3$. The base size is $4$; the remainder $3$ is spread one-per-fold to the first three folds (the three longer bars). Fold sizes: $\boxed{5,5,5,4,4}$. Why this step? scikit-learn's KFold gives the first n % k folds one extra sample so every sample is used exactly once and no fold is empty. Sizes stay "approximately equal", matching $|D_i|\approx n/k$.
2. Check total. $5+5+5+4+4=23$ ✓ — every sample is tested once, none twice. Why this step? The defining guarantee of k-fold is every point tests exactly once; unequal folds must still sum to $n$.
3. Average the RMSEs. ${\text{CV}}_5=\frac{12+15+11+14+13}{5}=\frac{65}{5}=13\text{ kg}$ Why this step? RMSE is a per-fold number (defined above as $\sqrt{\frac{1}{m}\sum (y-\hat{y}{)}^2}$); the CV estimate is still their mean.

Forecast check: your instinct that $4.6$ can't be a fold size was right — the fix is not rounding but distributing the $3$ leftover samples, giving the mixed sizes $5,5,5,4,4$. The number to remember is n % k: it counts exactly how many folds get the bonus sample.

Verify: sizes sum to $n=23$ ✓. Mean $13$ lies between min $11$ and max $15$ ✓. Units stay kg (we averaged kg values) ✓.

---

Example 3 — Cell C: the LOOCV limit $k=n$

Forecast: with only one test point per fold, what values can a single fold's accuracy take?

1. Each test fold has one sample. $n/k=4/4=1$. Train on $3$, test on $1$. Why this step? This is the extreme $k\to n$ named in the parent as LOOCV.
2. Each fold score is 0 or 1. One sample is either right ($1$) or wrong ($0$) — no in-between. Scores: $[1,1,0,1]$. Why this step? accuracy $=\frac{\text{correct}}{\text{tested}}=\frac{\{0\text{ or }1\}}{1}$. This is why LOOCV fold scores are high variance / noisy — each is a coin-flip-like $0$ or $1$.
3. Average. ${\text{CV}}_4=\frac{1+1+0+1}{4}=0.75$ Why this step? The mean of the $0/1$ scores equals the overall fraction correct — exactly $3$ of $4$.

Forecast check: the answer to the prompt is "only $0$ or $1$" — a single test point can only be fully right or fully wrong. That is exactly why the average ($0.75$) is the informative number in LOOCV while any single fold's score is almost useless on its own.

Verify: $3$ correct out of $4$ total $=0.75$ ✓. Every value averaged was $0$ or $1$, confirming the single-sample test folds ✓.

---

Example 4 — Cell D: the two-fold extreme $k=2$

Forecast: with only 2 folds, how much of the data trains each model?

1. Fold size and training fraction. $n/k=8/2=4$. Each model trains on $k-1=1$ fold $=4$ samples $=50\%$ of data. Why this step? Small $k$ means each model sees less training data — here only half — so each score is a pessimistic (higher-bias) estimate of the full-data model.
2. Average. ${\text{CV}}_2=\frac{0.70+0.86}{2}=0.78$ Why this step? Same averaging rule; two folds still each test once ($4+4=8$). Each F1 here is the balanced precision–recall blend defined at the top.
3. Interpret the tie to a single split. With $k=2$, fold 1 trains on fold 2 and vice-versa: it is two mirror-image train/test splits. Averaging them is barely more robust than one split — that is why we prefer $k=5$ or $10$.

Forecast check: the answer to "how much trains each model?" is exactly half ($50\%$). That low training fraction is the whole reason $k=2$ is biased-pessimistic — the model is judged while starved of data it would have in production.

Verify: $4+4=8=n$ ✓. Mean $0.78\in [0.70,0.86]$ ✓.

---

Example 5 — Cell E: rare event → stratified k-fold

Forecast: with $12$ failures split into $4$ folds, how many failures should each fold hold on average?

The figure contrasts the two strategies as bar charts of "failures per fold". On the left (random k-fold) the bars are jagged — one fold is red at height $0$, meaning it caught no failures at all. On the right (stratified) all four bars sit flat at the target height of $3$, matching the $6\%$ rate. The dashed grey line marks that target of $3$.

Figure 3. Two side-by-side bar charts of failures-per-fold for $k=4$. Left ("Random k-fold") is jagged with one red bar at height $0$; right ("Stratified k-fold") shows four equal green bars at height $3$; a dashed grey line marks the target of $3$ per fold.

1. True class rate. $12/200=0.06=6\%$ failures. Why this step? This baseline is what each fold should mirror. The full dataset ratio is the target for stratified sampling.
2. Expected failures per fold. $12/4=3$ failures per fold on average (the dashed line in the figure). Why this step? If randomness deals a fold $0$ failures (the red bar), that fold has no positives to detect.
3. Why the zero-failure fold's F1 collapses. F1 needs true positives (the $TP$ in its formula above). With zero actual positives in the test fold, recall's denominator $TP+FN$ is $0$ → F1 is undefined (scikit-learn returns $0$). That fold's score is meaningless and drags/spikes the average. Why this step? Rare-event scoring is only valid if positives exist in every test fold.
4. Stratified fix. Stratified k-fold forces each fold to hold $\approx 6\%$ failures — i.e. $200/4\times 0.06=3$ failures per fold (the flat right-hand chart), guaranteeing every fold can compute a real F1.

Forecast check: the "should be" number is 3 per fold, and that is precisely the value stratification enforces on every fold (the flat green bars). The random scheme's failure was that it deviated from this 3 — one fold got 0. Your forecast of 3 is the yardstick that exposes the random scheme as broken.

Verify: $6\%\times 50$ samples-per-fold $=3$ failures/fold, and $3\times 4=12$ = total failures ✓. $12/200=0.06$ ✓.

---

Example 6 — Cell F: regression, RMSE units

Forecast: is the CV RMSE just the mean of these five, or do we have to square-and-root something?

1. Average the fold RMSEs directly. ${\text{CV}}_5=\frac{210+190+205+195+200}{5}=\frac{1000}{5}=200\text{ kg}$ Why this step? Each fold already is an RMSE (the $\sqrt{\frac{1}{m}\sum (y-\hat{y}{)}^2}$ from the definition above). The classic recipe averages the per-fold scores — we do not re-pool residuals. (Pooling residuals is a different, valid variant; the parent's formula uses the mean of scores.)
2. Spread across folds. Deviations from $200$: $[10,-10,5,-5,0]$. Squares: $[100,100,25,25,0]$, sum $=250$. Divide by $k-1=4$: $62.5$. Root: $\approx 7.906$ kg. Why this step? Same $\frac{1}{k-1}$ sample-std reasoning as Example 1 — one degree of freedom used by the mean.
3. State units. Both mean and std are in kg (same units as the target), because RMSE inherits the target's units through that final square root. A "% accuracy" interpretation would be wrong here. Why this step? A metric's units decide how you may interpret and combine its values; treating a kg-error like a fractional accuracy would silently corrupt every downstream comparison.

Forecast check: the answer to "mean, or square-and-root?" is just the mean — because each fold value is already an RMSE (the squaring-and-rooting happened inside each fold). The trap the forecast guarded against is double-processing numbers that are already error summaries.

Verify: sum $=1000$, mean $200$ ✓; std $=\sqrt{250/4}=\sqrt{62.5}\approx 7.906$ ✓; units kg ✓.

---

Example 7 — Cell G: time-ordered data, the leakage trap

Forecast: what is wrong with letting the model train on $t_5$ while testing on $t_2$?

The figure stacks the four forward-chaining folds as rows. In every row the blue cells (training past) sit strictly to the left of the single orange cell (the next-day test), and the grey cells are future days not yet available. The bottom arrow marks time flowing left-to-right — no orange test cell ever has a blue training cell to its right, which is exactly the "no future leaks backward" guarantee.

Figure 4. Four stacked rows of the six days $t_1$–$t_6$; in each row blue cells (past = train) lie strictly left of one orange cell (next day = test), grey cells are unavailable future, and a bottom arrow marks time flowing left-to-right.

1. Spot the leakage. Shuffled folds let a model train on future days ($t_5,t_6$) to predict a past day ($t_2$). In deployment you never have the future, so shuffled CV over-estimates performance. Why this step? k-fold's fairness assumes each test fold is "unseen future"; time order breaks that assumption. This is the domain of time-series CV.
2. Why $k=3$ shuffled must be abandoned, and what replaces it. We cannot keep the colleague's $3$ shuffled folds at all — shuffling is the very act that leaks the future. In time-series CV, $k$ is not chosen freely; it is derived from how many forward steps you take. We fix an initial training window (here the first $2$ days, needed to fit any model) and then advance one day at a time. With $6$ days and a window of $2$, that yields $6-2=4$ forward folds — so effectively $k=4$, and $k$ is dictated by the data length, not picked for convenience. Why this step? It replaces the illegal shuffled splits with ordered ones and explains the change from the stated $k=3$ to $4$ folds: the fold count follows from n − (initial window), not from a free choice.
3. Build forward-chaining folds. Always train on the past (blue cells), test on the next block (orange cell):
   • train $\{t_1,t_2\}$ → test $t_3$
   • train $\{t_1,t_2,t_3\}$ → test $t_4$
   • train $\{t_1,t_2,t_3,t_4\}$ → test $t_5$
   • train $\{t_1,\dots ,t_5\}$ → test $t_6$ Why this step? Every training set precedes its test point in time — no future leaks backward.
4. Count test evaluations. Points tested: $t_3,t_4,t_5,t_6$ → $4$ test points. Unlike plain k-fold, forward chaining does not test every point: the first two days ($t_1,t_2$) only ever seed the earliest training window and are never themselves held out. So the "every sample tested exactly once" guarantee of ordinary k-fold is deliberately given up here in exchange for never peeking at the future. Why this step? Recognising that early samples are train-only is the whole point — it is the price time-series CV pays for realism.

Forecast check: the answer to "what's wrong with training on $t_5$ to predict $t_2$?" is that $t_5$ is the future relative to $t_2$ — in real deployment you would not yet have it, so the score is dishonestly inflated. The forward-chaining scheme is exactly the arrangement that makes such time-travel impossible.

Verify: with $n=6$ and initial train window $2$, tested points $=n-2=4$ ✓. Each training set is a strict prefix of indices, so max(train index) $<$ test index in every fold ✓.

---

Example 8 — Cell H: exam twist, mean vs variance

Forecast: identical means — so what breaks the tie?

1. Confirm the means match. $\bar{P}=\frac{0.83+0.81+0.82+0.82+0.82}{5}=\frac{4.10}{5}=0.82,\bar{Q}=\frac{0.90+0.70+0.95+0.72+0.83}{5}=\frac{4.10}{5}=0.82$ Why this step? The trap is assuming equal means ⇒ equal quality. They tie on average, so the mean alone cannot decide.
2. Compute each spread (the fold-to-fold std). For P, deviations $[0.01,-0.01,0,0,0]$, squares sum $=0.0002$, $/4=0.00005$, root $\approx 0.00707$. For Q, deviations $[0.08,-0.12,0.13,-0.10,0.01]$, squares $[0.0064,0.0144,0.0169,0.0100,0.0001]$ sum $=0.0478$, $/4=0.01195$, root $\approx 0.1093$. Why this step? The variance across folds measures consistency — a safety-critical concern that the mean hides.
3. Decide. ${\sigma }_P\approx 0.007\ll {\sigma }_Q\approx 0.109$. Deploy Model P: same expected F1 but far steadier across data subsets. Model Q's swing ($0.70$ to $0.95$) means on some real flight populations it could fail badly. Why this step? In aerospace, predictable mediocrity beats erratic brilliance — a model that is occasionally excellent but sometimes poor is a safety liability.

Forecast check: the tie-breaker you were asked to name is the standard deviation — the second of the "always two numbers" pair. Equal means forced you to look past the average to consistency, which is the entire lesson of this exam-style cell.

Verify: both means $=4.10/5=0.82$ ✓; ${\sigma }_P=\sqrt{0.0002/4}\approx 0.00707$, ${\sigma }_Q=\sqrt{0.0478/4}\approx 0.10932$, and ${\sigma }_P<{\sigma }_Q$ ✓.

---

Recall Which example covers which trap?

Unequal fold sizes ::: Example 2 (remainder spread to first n % k folds). Single-sample test folds ::: Example 3 (LOOCV, scores are 0/1). Rare event / stratification ::: Example 5. Time leakage / forward chaining ::: Example 7 (only $n-\text{init}$ points tested). Same mean, pick by std ::: Example 8.

Parent: Cross-validation — k-fold · Hinglish: 5.6.05 Cross-validation — k-fold (Hinglish)