Cross-validation — k-fold: Visual walkthrough

Before line one, three plain words we will lean on:

Data point: one row of your table — one flight, with its sensor readings and its known answer (failed / did not fail).
Model: a machine that, after looking at some data points, tries to guess the answer for a data point it has never seen.
Score: a single number saying "how often was the guess right on the unseen points" (higher = better).

That is the whole vocabulary. Let's draw.

Step 1 — Lay all your data in a row

WHAT. Picture every data point as one coloured tile. We have $n$ of them — that is, $n = 1000$ flight records. Line them up left to right.

WHY. Before we can split anything, we must see the whole thing as one object. The tiles are our raw material; everything else is just choosing which tiles to hide.

PICTURE. Look at the single strip below. Each tile is one flight. Nothing is hidden yet — the model would train on all of it and we'd have nothing left to test on honestly. That is exactly the problem we are about to fix.

Step 2 — Cut the strip into $k$ equal blocks

WHAT. Slice the strip into $k$ side-by-side blocks of the same length. Each block is called a fold. We choose $k = 5$ , so five blocks, each holding $n / k = 1000/5 = 200$ tiles.

WHY. We need several separate pools of unseen data, not one. If we only ever hide the same 200 tiles, a lucky (or unlucky) block could fool us. Cutting into $k$ blocks gives us $k$ different "unseen" pools to try later — one at a time.

PICTURE. The strip now has 5 coloured blocks labelled $D_{1}$ through $D_{5}$ . They do not overlap (no tile is in two blocks) and together they cover everything.

Step 3 — Round 1: hide the first block, train on the rest

WHAT. Take block $D_{1}$ and set it aside as the test set — the model is forbidden to look at it. The other four blocks ( $D_{2}, D_{3}, D_{4}, D_{5}$ ) become the training set. Train one model, call it $M_{1}$ , on those four blocks only.

WHY. The model must be scored on tiles it has never seen, otherwise the score is a memory test, not a generalization test. Hiding $D_{1}$ guarantees $M_{1}$ has no memory of it.

PICTURE. Below, block 1 glows (the hidden test block) and blocks 2–5 are dimmed into a single "training" region with an arrow feeding the model box $M_{1}$ .

Step 4 — Grade round 1, then rotate

WHAT. Ask $M_{1}$ to guess the answers for the hidden block $D_{1}$ , compare to the true answers, and record one number $Score_{1}$ . Then rotate: now hide $D_{2}$ instead, train a fresh $M_{2}$ on the other four blocks, grade it → $Score_{2}$ . Keep rotating until every block has taken its turn as the hidden one.

WHY. One rotation gives one opinion. Rotating means every tile eventually gets to be an unseen test tile exactly once, so no region of the data is left ungraded and none is graded twice. That even coverage is the whole point of k-fold.

PICTURE. Five rows, one per round. In each row a different block is lit (the test block) while the rest are the training region. Read it top to bottom: the lit block marches left→right, so the "test" spotlight sweeps the entire dataset.

We end this step holding five numbers, e.g. $[0.82, 0.79, 0.85, 0.81, 0.83]$ .

Step 5 — Collapse the five numbers into one: the mean

WHAT. Add the five scores and divide by five. That average is the k-fold cross-validation estimate.

WHY. Each round's score wobbles because its particular hidden block was a bit easy or a bit hard. Averaging lets those wobbles cancel: high rounds pull up, low rounds pull down, and the middle is a steadier guess of true performance than any single round. This is the law of large numbers doing our work.

PICTURE. Five dots at heights $0.79$ – $0.85$ , and a bold horizontal line at their mean $0.82$ . The dashed band around the line is the spread we measure next.

Step 6 — Measure the wobble: standard deviation

WHAT. Measure how far, on average, the five dots sit from their mean line. That distance is the standard deviation $σ_{CV}$ .

WHY. A mean of $0.82$ hides whether the rounds agreed. $0.82 \pm 0.02$ (dots hug the line) means the model is consistent. $0.82 \pm 0.15$ (dots scattered) warns you the model is fragile — its score swings wildly depending on which tiles it happened to see.

PICTURE. Same five dots as before, now with a vertical arrow from each dot to the mean line marking its gap $(Score_{i} - CV_{k})$ . The spread band's half-width is $σ_{CV}$ .

Final report: $0.820 \pm 0.022$ .

Step 7 — The edge cases: what $k$ at its extremes looks like

WHAT. Two degenerate choices of $k$ that break the balance.

WHY / PICTURE. See the two panels below.

$k = 2$ (left panel), the smallest legal value. Only two blocks, so each model trains on just half the data. Half a dataset is a weak apprentice, so scores come out pessimistically low — high bias. But each test block is huge (500 tiles), so individual scores are stable — low variance.
$k = n$ (right panel), one tile per block — this is LOOCV. Each model trains on $n - 1$ tiles (almost the whole set → low bias), but each test block is a single tile, so each score is either "right" or "wrong" — extremely noisy (high variance), and all $n$ training sets overlap almost completely, so the scores are correlated. You also train $n$ whole models — brutally slow.

The one-picture summary

The whole procedure on a single canvas: the strip is cut into 5 folds; five rows rotate the lit test block across the dataset; each row yields one score; the scores collapse into a mean $\pm$ spread.

Recall Feynman retelling — say it like a story

I lay all my flights in one long row. I chop the row into 5 equal blocks. Round one: I cover block 1 with my hand, let the model study the other four, then uncover block 1 and see how many flights it guessed right — that's one score. I slide my hand to block 2 and do it all again with a fresh model. Five rounds, my hand sweeping across the whole row, so every flight gets covered exactly once. Now I have five scores. I average them — that steady number is my real estimate of how good the model is. I also check how spread out the five were: tight together means trustworthy, scattered means the model is jumpy. If I use too few blocks the model learns from too little and looks worse than it is; if I use as many blocks as flights I train forever and the scores get noisy. Five or ten blocks is the comfortable middle.

Recall Quick self-test

Why must the folds be disjoint? ::: So no tile is both trained on and tested on — that would leak the answer and inflate the score. What does averaging the $k$ scores buy us? ::: It cancels the round-to-round wobble, giving a steadier estimate than any single split. What is the legal range of $k$ ? ::: $2 \leq k \leq n$ — at least two folds (else the training set is empty), at most one tile per fold (LOOCV). What happens when $n$ does not divide evenly by $k$ ? ::: Spread the leftover tiles one per fold, so some folds hold one extra tile (ceil vs floor). When would you not shuffle before folding? ::: When the data is time-ordered and you must forecast the future — use time-series CV.

Related build-ups: 5.6.01-Train-test-split-validation-set, 5.6.02-Holdout-method, 5.6.07-Nested-cross-validation, 7.2.03-Hyperparameter-tuning-grid-search.