Visual walkthrough — K-fold cross-validation

Before line one, let us agree on the words we will use. Nothing below is used before it is defined.

---

Step 1 — Start with the honest question

WHAT. We have $N$ dots. We want a number that predicts performance on future, unseen dots.

WHY. If we test the model on the very same dots it learned from, it can just memorise them and score 100%. That number is a lie — it tells us nothing about the future. So we must hide some dots from the model during learning, and only reveal them at test time.

PICTURE. Look at the figure: the blue dots on the left are shown to the model (it learns). The yellow dots on the right are hidden until testing. The model never sees yellow while learning — that is the only way the test is honest.

---

Step 2 — Cut the data into K equal slices

WHAT. Chop the dataset $D$ (our whole pile of dots, defined above) into $K$ equal, non-overlapping pieces called folds. Write it as:

$D=\underset{\text{slice 1}}{\underbrace{F_1}}\cup \underset{\text{slice 2}}{\underbrace{F_2}}\cup \dots \cup \underset{\text{slice K}}{\underbrace{F_K}}$

Here $\cup$ means "glued together" — stack all the slices and you get the whole pile $D$ back. Each slice $F_i$ holds about $\frac{N}{K}$ dots (total dots divided by number of slices).

WHY. Equal slices means every future "test" uses the same amount of data, so every turn is judged on a level playing field. No slice gets an easier or harder exam.

PICTURE. The strip below is the dataset. The dashed white lines are the $K-1$ cuts that produce $K$ coloured slices of the same width.

---

Step 3 — Let one slice be the test, the rest teach

WHAT. For each round $i$ (going $i=1,2,\dots ,K$), pick slice $F_i$ to be the validation set (the hidden test), and glue all the other slices together to be the training set:

$V_i=\underset{\text{this round’s test}}{\underbrace{F_i}}{\text{Tr}}_i=\underset{\text{everything except }{F}_i}{\underbrace{D\setminus F_i}}={\bigcup }_{j\ne i}{F}_j$

The symbol $\setminus$ means "remove": $D\setminus F_i$ is the whole pile $D$ minus the one test slice. The big ${\bigcup }_{j\ne i}$ just says "glue every slice $F_j$ whose number $j$ is not $i$." We write the training set as ${\text{Tr}}_i$ (short for Training) so it never gets confused with the training procedure coming in Step 4.

WHY. This is the honest test from Step 1 — but now we do not waste the hidden slice. Every dot spends one turn being tested and $K-1$ turns teaching. No dot is wasted.

PICTURE. Row $i$ of the grid shows round $i$: the yellow block is the tested slice $F_i$, the blue blocks are the training slices ${\text{Tr}}_i$. Notice the yellow block slides down one row at a time.

---

Step 4 — Train a model each round and score it

WHAT. In round $i$, train a fresh model on the training set ${\text{Tr}}_i$, producing settings ${\theta }_i$. Then measure how well it does on the hidden slice $V_i$, giving a score:

${\theta }_i=\text{Fit}({\text{Tr}}_i){\text{score}}_i=M(\underset{\text{model from round }i}{\underbrace{{\theta }_i}},\underset{\text{its hidden test}}{\underbrace{F_i}})$

• $\text{Fit}(\cdot )$ = the training procedure (turns a training set into a trained model). We name it "Fit" so it is clearly a process, not the training set ${\text{Tr}}_i$.
• ${\theta }_i$ = the trained model's learned settings for round $i$.
• $M(\cdot ,\cdot )$ = the metric — the ruler that measures quality (e.g. accuracy: fraction guessed right).

WHY. Each ${\text{score}}_i$ is one honest exam result. Because each model was tested on a slice it never saw, no cheating is possible. We get $K$ independent report cards.

PICTURE. Each row now has a score on the right — five little report cards, one per round. Note they are close but not identical: that wobble is the noise a single split would have suffered from.

---

Step 5 — Average the scores: the CV estimate

WHAT. Add up all $K$ scores and divide by $K$:

${\text{CV}}_K=\frac{1}{K}{\sum }_{i=1}^K{\text{score}}_i\text{where}{\text{score}}_i=M({\theta }_i,F_i)$

• ${\sum }_{i=1}^K$ = "add these up for $i$ going 1 to $K$."
• $\frac{1}{K}$ = divide by how many, i.e. take the mean.
• ${\text{score}}_i=M({\theta }_i,F_i)$ = the same round-$i$ score defined in Step 4.

WHY the mean and not just one score? Each individual score is a noisy guess at the true skill — some rounds land on an easy slice, some on a hard one. Averaging cancels the ups and downs (Law of Large Numbers: many noisy readings average to something steadier than any single one). One number, far more trustworthy than any single split could give.

PICTURE. The five report cards drop onto a number line; the yellow tick is their average ${\text{CV}}_K$. See how the average sits calmly in the middle of the scatter.

---

Step 6 — Measure the wobble: the standard error

WHAT. First measure how spread out the $K$ scores are around their average — that spread is the standard deviation $s$:

$s=\sqrt{\frac{1}{K-1}{\sum }_{i=1}^K(\underset{\text{gap from the average}}{\underbrace{{\text{score}}_i-{\text{CV}}_K}}{)}^2}$

Then the standard error of the mean — how uncertain the average ${\text{CV}}_K$ itself is — divides that spread by $\sqrt{K}$:

$\text{SE}=\frac{s}{\sqrt{K}}$

• ${\text{score}}_i-{\text{CV}}_K$ = how far one round is from the mean (a gap, plus or minus).
• squaring makes every gap positive (so + and − gaps don't cancel).
• dividing by $K-1$ (not $K$) is the standard unbiased way to estimate spread from a sample.
• the  ​ brings us back to the original units (accuracy, not accuracy-squared).
• dividing $s$ by $\sqrt{K}$ turns "spread of one score" into "uncertainty of the average" — more folds pin the average down tighter.

WHY. A small SE means the folds all agree → a reliable, stable estimate of the mean. A big SE means the average could easily have landed elsewhere → a warning sign (fragile model or messy data). SE turns "trust me" into a measured $\pm$.

PICTURE. The same number line, now with an error bar of width SE around the mean. Tight bar = confident; wide bar = shaky.

---

Step 7 — The choose-your-K trade-off

WHAT. $K$ controls slice thinness, and that controls bias (is the estimate systematically too low?) and variance (how jittery is it?).

WHY. Each round trains on only $\frac{K-1}{K}$ of the data, but the final model will use all of it. Training on less data than the final model makes the CV number a touch pessimistic — that gap is the bias.

$\text{training fraction}=\frac{K-1}{K}$

• Small $K$ (say 3): training fraction $=\frac{2}{3}\approx 67\%$ → far from 100% → more bias, but only 3 rounds → less variance, and fast.
• Large $K$ (say 10): training fraction $=\frac{9}{10}=90\%$ → close to 100% → less bias, but 10 noisy rounds → more variance, and slow.

PICTURE. Two curves versus $K$: bias falls as $K$ grows (blue, dropping), variance rises (red, climbing). They cross near the sweet spot — which is why $K=5$ and $K=10$ are the classic picks.

---

Step 8 — The degenerate extreme: K = N (Leave-One-Out)

WHAT. Push $K$ all the way to $N$. Now each slice is a single dot. Every round trains on $N-1$ dots and tests on the one left out.

WHY show this edge case. It is the limit of Step 7. Training fraction $=\frac{N-1}{N}$ is almost 1 → almost zero bias. But we run $N$ trainings (expensive) and the $N$ test-scores are wildly noisy (each is judged on ONE dot, so each score is 0 or 1) → maximum variance. This is why LOOCV is reserved for tiny datasets where every dot is precious.

PICTURE. The strip is now sliced into $N$ hair-thin lines; one lone yellow dot is the test, everything else teaches. Compare its width to the fat slices of Step 2.

---

The one-picture summary

Everything above, compressed: cut into $K$, slide the test slice down, score each round, average the scores, put an error bar on it.

Recall Feynman retelling — say it to a 12-year-old

Imagine you baked a big tray of cookies and want to know how good your recipe really is, not just how good this one batch tasted to you. So you cut the tray into 5 equal squares. Five times over, you set ONE square aside as the "judge's square," bake — I mean, teach — using the other four, then let the judge taste only the set-aside square and give a score. Because the judge never helped bake its own square, its score is honest. Do this so every square gets one turn as judge. Now you have five honest scores. Average them — that's your recipe's real rating. Look at how much the five scores disagree — a little wobble means a rock-solid recipe; a big wobble means it's fragile and got lucky. Cut into more, thinner squares and each teaching batch is bigger (rating gets more accurate) but the tasting gets jumpier and takes longer. Cut into just a few fat squares and it's fast but a bit pessimistic. Cut so every single cookie gets its own turn as judge — that's the extreme, super accurate on average but painfully slow and jumpy. That's it: slice, take turns, average, and read the wobble.

Recall Quick self-check

What does $D\setminus F_i$ mean in words? ::: The whole dataset $D$ with slice $F_i$ removed — i.e. this round's training set ${\text{Tr}}_i$. Why average the K scores instead of trusting one? ::: Each score is noisy; averaging cancels the ups and downs into a steadier, more reliable estimate. Why is small K more biased? ::: It trains on a smaller fraction $\frac{K-1}{K}$ than the final model uses, so it underestimates true performance. What is the CV estimate and standard error for scores 0.85, 0.82, 0.88, 0.84, 0.86? ::: CV = 0.850, standard deviation ≈ 0.0224, standard error of the mean = 0.010.