K-fold cross-validation: Exercises — AI-ML deep dive

Before we start, let us make sure every symbol on this page is earned.

Definition The symbols we will use

$D$ = the whole dataset (all your data points together — think of it as one big bag).
$N$ = total number of data points (rows in $D$ ).
$K$ = number of folds — the number of equal piles we chop the data into. We always need $K \leq N$ (you cannot make more piles than you have points).
$F_{i}$ = the $i$ -th fold (one pile). Picture $K$ boxes; $F_{i}$ is box number $i$ .
$T_{i}$ = the training set for iteration $i$ = "everything in $D$ except box $i$ ", written $T_{i} = D ∖ F_{i}$ (the backslash $∖$ means "remove").
$score_{i}$ = the performance number (accuracy, error, ...) you get when you test on box $i$ after training on all the other boxes.
$CV_{K}$ = the final answer = the average of all $K$ scores.
$SD_{folds}$ = a number that says how spread out the $K$ scores are. Small value = scores agree = trustworthy.

The two formulas, restated so you never have to scroll up: $CV_{K} = \frac{1}{K} \sum_{i = 1}^{K} score_{i} SD_{folds} = \frac{1}{K} \sum_{i = 1}^{K} (score_{i} - CV_{K})^{2}$ Read $\sum_{i = 1}^{K}$ as "add up, letting $i$ run $1, 2, \dots, K$ "; $CV_{K}$ is just the mean of the fold scores.

Look at the figure: the horizontal strip is your whole dataset $D$ . One coloured box is the test fold ( $F_{i}$ ); the rest are the training folds ( $T_{i}$ ). As $i$ moves from $1$ to $K$ , the coloured box slides along — that is the entire algorithm in one picture.

L1 — Recognition

Recall Solution 1.1

WHAT: Each fold is an equal pile, so it holds $N / K$ samples. $\frac{N}{K} = \frac{200}{4} = 50 samples per fold.$ WHY that many models: In K-fold we train one model per fold (each fold takes a turn as the test set). So the number of models trained equals $K = 4$ . Answer: $50$ samples per fold, $4$ models trained.

Recall Solution 1.2

False. In any single iteration $i$ , the folds are disjoint: the test fold $F_{i}$ is completely removed from training ( $T_{i} = D ∖ F_{i}$ , i.e. "all of $D$ except box $i$ "). A point is either being tested or being trained on in that iteration — never both at once. (Across different iterations a point is used for both, just not simultaneously.)

Recall Solution 1.3

$K = N$ means each fold has exactly one sample ( $N / K = N / N = 1$ ). Testing on one point at a time, using all $N - 1$ others to train, is called Leave-One-Out Cross-Validation (LOOCV). Note this is the largest legal $K$ , since $K \leq N$ .

L2 — Application

Recall Solution 2.1

WHAT: average the four scores. $CV_{4} = \frac{0.80 + 0.90 + 0.85 + 0.85}{4} = \frac{3.40}{4} = 0.85.$ Answer: $CV_{4} = 0.85$ (i.e. $85%$ ).

Recall Solution 2.2

WHY subtract the mean first: $SD_{folds}$ measures spread around the average, so every score is compared to $0.85$ .

(0.80 - 0.85)^{2} (0.90 - 0.85)^{2} (0.85 - 0.85)^{2} (0.85 - 0.85)^{2} = (- 0.05)^{2} = 0.0025 = (0.05)^{2} = 0.0025 = 0 = 0

Sum $= 0.0050$ . Divide by $K = 4$ : $0.0050/4 = 0.00125$ . Take the square root: $SD_{folds} = 0.00125 \approx 0.0354.$ Answer: $SD_{folds} \approx 0.035$ , so the model is $85.0% \pm 3.5%$ .

Recall Solution 2.3

Each fold = $N / K = 1000/10 = 100$ test samples. Training uses the other $K - 1 = 9$ folds: $train size = N - \frac{N}{K} = 1000 - 100 = 900.$ Fraction $= \frac{K - 1}{K} = \frac{9}{10} = 0.90 = 90%$ . Answer: $900$ training samples, $90%$ of the data.

L3 — Analysis

Recall Solution 3.1

Compute the spread. Model A's scores hug $0.85$ tightly; Model B swings from $0.60$ to $0.99$ . Model A $SD_{folds}$ : deviations $- 0.01, 0, 0.01, 0, 0$ → sum of squares $= 0.0002$ → $0.0002/5 = 0.00004$ → $SD_{A} = 0.00004 \approx 0.0063$ . Model B $SD_{folds}$ : deviations $- 0.15, 0.14, - 0.25, 0.13, 0.13$ → squares $0.0225, 0.0196, 0.0625, 0.0169, 0.0169$ → sum $= 0.1384$ → $/5 = 0.02768$ → $SD_{B} = 0.02768 \approx 0.166$ . Interpretation: same average, but $SD_{B} \approx 26 \times SD_{A}$ . Model B is wildly sensitive to which data it trains on — a red flag (unstable model or non-representative folds). Trust Model A: consistent scores mean the $0.85$ estimate is reliable.

Recall Solution 3.2

The core idea: more training data usually → better model.

$K = 3$ : each CV model trains on $\frac{K - 1}{K} N = \frac{2}{3} \cdot 1000 \approx 667$ samples. That is far short of the final $1000$ , so those models are weaker than the deployed one → the reported score underestimates true performance. That downward gap is the bias.
$K = 10$ : each CV model trains on $900$ samples — much closer to $1000$ → its performance closely mirrors the final model → less bias. Trade-off summary: larger $K$ shrinks bias but averages over noisier, more-overlapping training sets (higher variance) and costs $K$ trainings.

Recall Solution 3.3

The failure: random splitting could dump many/all of the $10$ class-B points into just one or two folds. A validation fold with zero class-B points can only measure class-A accuracy — the minority class is invisible, so the score is misleading (a model that ignores class B could still look "good"). The fix — stratified K-fold: keep each fold at the same $90 : 10$ ratio as the whole set. For $K = 5$ : each fold gets $90/5 = 18$ class-A and $10/5 = 2$ class-B samples. Now every validation fold contains both classes and truly represents the data distribution.

L4 — Synthesis

Recall Solution 4.1

Correct procedure:

Shuffle $D$ once, then split it into 5 folds and fix that split.
For each setting (say $H_{1}$ and $H_{2}$ ): a. For each fold $i$ : train the model with that setting on the other 4 folds. b. Do all preprocessing (scaling, imputing) fitted on the training folds only, then apply to $F_{i}$ . c. Record $score_{i}$ . d. Compute $CV_{5}$ for that setting.
Pick the setting with the higher $CV_{5}$ .
Retrain that chosen setting on all data for deployment. The classic leak: fitting the scaler / choosing the hyperparameter using information from the validation fold (e.g. normalizing the whole dataset before splitting). Preprocessing statistics must be learned inside each training fold only.

Recall Solution 4.2

Split each class into 5 equal parts:

Class A: $100/5 = 20$ per fold.
Class B: $20/5 = 4$ per fold. One fold $F_{i}$ : $20$ A $+$ $4$ B $= 24$ samples, preserving the $100 : 20 = 5 : 1$ ratio. Training size per model: $N - ∣ F_{i} ∣ = 120 - 24 = 96$ samples (which is $20 \times 4$ A $+ 4 \times 4$ B $= 80$ A $+ 16$ B — still $5 : 1$ ). Answer: each fold $= 24$ samples ( $20$ A/ $4$ B); each model trains on $96$ samples.

L5 — Mastery

Recall Solution 5.1

LOOCV means $K = N = 5$ folds, one point each. The per-fold "score" here is the squared error. $CV_{5} = \frac{4 + 1 + 9 + 0 + 1}{5} = \frac{15}{5} = 3.0.$ Fold spread: deviations from $3.0$ : $1, - 2, 6, - 3, - 2$ . Squares: $1, 4, 36, 9, 4$ , sum $= 54$ . Divide by $K = 5$ : $10.8$ . Square root: $SD_{folds} = 10.8 \approx 3.286.$ Answer: LOOCV MSE $= 3.0$ , $SD_{folds} \approx 3.29$ . The large spread relative to the mean warns that with only $5$ points the estimate is very shaky — exactly the "maximum variance" LOOCV is known for.

Recall Solution 5.2

When the scaler's mean/standard-deviation are computed on the full dataset, each validation fold's statistics have already influenced the numbers the model trains on — the model gets a subtle hint about the test data. This inflates performance, so $91%$ is optimistically biased. The corrected pipeline (fit scaler on training folds only, apply to the held-out fold) removes that hint and gives an honest $86%$ . Publish $86%$ — it estimates true generalization. Reporting $91%$ would over-promise and the model will underperform in production.

Recall Solution 5.3

Each strategy trains $K$ models, so total time $= K \times 2$ minutes.

(a) $K = 5$ : $5 \times 2 = 10$ minutes.
(b) $K = 10$ : $10 \times 2 = 20$ minutes.
(c) LOOCV, $K = N = 500$ : $500 \times 2 = 1000$ minutes $\approx 16.7$ hours. Rule: cost grows linearly in $K$ . LOOCV's tiny bias rarely justifies a $50 \times$ – $100 \times$ cost jump, so $K = 5$ or $K = 10$ is the default; reserve LOOCV for very small datasets where every sample is precious.

Common mistake L5 Trap: chasing LOOCV for its "unbiasedness" on big data

Why it feels right: "unbiased" sounds strictly better, so why would you ever accept the small bias of $K = 10$ when you could have (nearly) none? Beginners treat "less bias" as an unconditional win. Why it's wrong: LOOCV pays for that tiny bias reduction with maximum variance (Exercise 5.1: with each fold being a single point, the fold scores swing wildly) and maximum cost (Exercise 5.3: $16.7$ hours vs $10$ minutes — a $\sim 100 \times$ jump). On large datasets the $N$ training sets differ by only one point, so the models are nearly identical and the near-zero bias buys you almost nothing real. Fix: match $K$ to your data size and compute budget — use $K = 10$ for moderate data, $K = 5$ when compute is tight, and reserve LOOCV for the genuinely tiny- $N$ case where every single sample is precious and you can afford $N$ trainings.

Recall Quick self-check (cloze)

The two aggregation formulas are $CV_{K} =$ ::: $\frac{1}{K} \sum_{i = 1}^{K} score_{i}$ (the mean of fold scores). Each K-fold model trains on a fraction ::: $\frac{K - 1}{K}$ of the data. LOOCV corresponds to $K =$ ::: $N$ (one sample per fold; the largest legal $K$ ). Before splitting into folds you should always ::: shuffle the data once (so sorted data does not bias the folds). A large fold spread with a good mean means ::: the model is unstable across folds — do not trust the single number. Fitting a scaler on the whole dataset before splitting causes ::: data leakage (optimistically biased scores).