Training, validation, and test error: Worked examples

This page is a workout. The parent note told you what the three errors are and why they exist. Here we take that machinery and run it through every kind of situation it can meet — big data, tiny data, the overfitting trap, the underfitting trap, the "gap goes negative" oddity, a real-world word problem, and the sneaky exam twist where someone tunes on the test set and gets a lie for an answer.

Before any example, let us make one promise: no symbol appears before it is earned. So a 30-second refresher on the four quantities we will keep computing.

Here "average mistake" means: take every example, measure how wrong the prediction is (that per-example wrongness is called the loss $L$ ), and average. Suppose the set we are scoring holds $m$ examples — here $m$ is simply the count of examples in whatever set we are averaging over (it will be the training-set size when we score training data, the validation-set size when we score validation data, and so on). Then:

$E = \frac{1}{m} \sum_{i = 1}^{m} L (y_{i}, \hat{f} (x_{i}))$

where $y_{i}$ is the true answer for example number $i$ , $\hat{f} (x_{i})$ is the model's guess for it, and $\sum$ ("sum over $i$ ") just means "add these up for $i = 1, 2, \dots, m$ ". For classification we will use the misclassification loss: $L = 1$ if the guess is wrong, $L = 0$ if right — so $E$ becomes simply the fraction of examples gotten wrong. For regression we use squared error $L = (y - \hat{f} (x))^{2}$ , because it punishes big misses much harder than small ones (and because its derivative is clean for gradient descent — that is why squares and not absolute value here).

Two named quantities we will re-use, defined here once so no symbol is used unearned:

Definition

$n_{val}$ and the standard error

$n_{val}$ ::: the number of examples in the validation set — i.e. the value of $m$ in the formula above when the set we are averaging over is the validation set. (Likewise $n_{train}$ , $n_{test}$ for the other two sets.)
standard error ::: when we estimate an error from a finite held-out set, our number wobbles depending on which examples we happened to draw. The standard error is the typical size of that wobble — "how far off, on average, this estimate is from the truth it is trying to hit."

For an error measured as a fraction-of-wrong on $n_{val}$ independent validation examples, the standard error has the exact form $SE = \frac{p ( 1 - p )}{n _{val}}$ where $p$ is the true error rate — the fraction the model would get wrong averaged over all possible data, the thing $E_{val}$ is estimating. Where does this come from? Each validation example scores a 0-or-1 loss whose spread (variance) is $p (1 - p)$ ; averaging $n_{val}$ independent such scores shrinks that spread by a factor $n_{val}$ , and the square root turns variance back into a typical distance. We use this formula in Example 1 and watch its consequences (the $1/ n$ shrinking) play out fully in Example 7.

The scenario matrix

Every case this topic can throw is one of these cells. The examples below are labelled with the cell they cover, and together they cover every row.

#	Case class	The defining feature	Covered by
C1	Large data split	$N$ huge → tiny % validation is still thousands of points	Ex 1
C2	Medium data split	80/10/10, straightforward	Ex 2
C3	Small data → k-fold	one split too noisy; rotate folds	Ex 3
C4	Overfitting ( $Δ ≫ 0$ )	train error low, val error high	Ex 4
C5	Underfitting ( $Δ \approx 0$ , both high)	model too weak; both errors bad	Ex 4
C6	Negative gap ( $Δ < 0$ )	val easier than train (noise / unlucky split)	Ex 5
C7	Degenerate: $E_{train} = 0$	interpolation, capacity = data	Ex 6
C8	Limiting behaviour ( $n_{val} \to \infty$ )	val error converges to true error	Ex 7
C9	Word problem (real world)	translate a business story into a split	Ex 8
C10	Exam twist: leakage	tuning on the test set fakes a good score	Ex 9

Figure — Training, validation, and test error

The figure above is the map of this whole page. As model complexity grows left→right, training error keeps falling but validation error dips then rises. Read it as a tour of the examples:

Ex 4's underfit model P sits at the far left (both curves high).
Ex 4's good model Q sits at the violet sweet-spot dot (validation minimum).
Ex 4's overfit model R and Ex 6's degree-9 interpolator live at the far right, where the training curve has crashed toward zero but the validation curve has climbed back up.
Ex 7 is about how tightly the validation curve itself can be pinned down as its sample count grows.

Keep glancing back at this map; every example is a point on it.

The worked examples

Example 1 — Large dataset split (cell C1)

Forecast: guess — is 1% of two million a laughably small validation set, or actually plenty?

Compute each slice. $98% \times 2, 000, 000 = 1, 960, 000$ train; $1% \times 2, 000, 000 = 20, 000$ val; $20, 000$ test. Why this step? Percentages of the whole tell us the raw counts, and raw counts — not percentages — are what determine estimate stability.
Judge stability by count, not fraction. Use the standard error formula defined just above the scenario matrix, $SE = p (1 - p) / n_{val}$ , where $p$ is the true error rate and $n_{val}$ is the validation-set size. Even at the worst-case $p = 0.5$ (which maximises $p (1 - p) = 0.25$ ), with $n_{val} = 20, 000$ we get $SE = 0.25/20000 = 1.25 \times 1 0^{- 5} \approx 0.0035$ . Why this step? We want to know how much $E_{val}$ jitters just from which 20k samples we drew. Using the worst-case $p$ gives a safe upper bound on that jitter. If even the bound is tiny compared to the differences between models, 1% is enough.

Verify: $1, 960, 000 + 20, 000 + 20, 000 = 2, 000, 000$ ✓ (all samples accounted for, none double-counted). Worst-case standard error $\approx 0.0035 < 0.01$ , so estimates are stable. → See Model Selection for how these compare candidates.

Example 2 — Medium dataset split (cell C2)

Forecast: will the "combine train+val for the final fit" step give you back the full 40,000, or only part?

Split. $0.8 \times 40000 = 32, 000$ train; $0.1 \times 40000 = 4, 000$ val; $0.1 \times 40000 = 4, 000$ test. Why this step? 10% of 40k is 4,000 — comfortably above the ~1,000-sample floor where fraction estimates get noisy.
Combine after tuning. Once hyperparameters are chosen, val is no longer needed as a referee. Merge it back: $32, 000 + 4, 000 = 36, 000$ final training samples. Why this step? Every held-out point you stop holding out is extra fuel for the final model — you only kept val separate to make an honest choice, and that choice is now made.
Test stays sacred. The 4,000 test points are never merged in. They score the finished model once. Why this step? The moment test data influences any decision, its score stops being unbiased — that is Data Leakage.

Verify: $32000 + 4000 + 4000 = 40000$ ✓. Final model sees $36, 000$ , i.e. $90%$ of all data ✓, test untouched at $4, 000$ ✓.

Example 3 — Small data, k-fold cross-validation (cell C3)

Forecast: guess the average error and whether each round trains on more or fewer points than a plain 80/20 split.

Fold size. $N / k = 1000/5 = 200$ points per fold. Why this step? We chop the data into $k$ equal blocks so each block gets one turn as the validation fold.
Per-round training count. Each round holds out one 200-point fold and trains on the other four: $4 \times 200 = 800$ . Why this step? Rotating which fold sits out means every point is used for validation exactly once and for training four times — no point is wasted.
Average the fold errors. $\frac{0.30 + 0.34 + 0.28 + 0.32 + 0.36}{5} = \frac{1.60}{5} = 0.32$ . Why this step? Averaging five independent-ish estimates cancels the luck of any single split — that is the whole point of cross-validation (see Cross-Validation).

Verify: fold size $200$ ✓ ( $5 \times 200 = 1000$ ); per-round train $800$ ✓; mean error $= 1.6/5 = 0.32$ ✓.

Example 4 — Overfitting vs underfitting from a table (cells C4 & C5)

Worked example Statement

Three models on the same data. Classify each as underfit, good, or overfit using the gap $Δ = E_{val} - E_{train}$ , then pick the one to deploy.

Model	$E_{train}$	$E_{val}$
P (linear)	0.40	0.42
Q (medium)	0.12	0.15
R (giant net)	0.01	0.38

Forecast: which model is memorising, which is too weak, and which do you ship?

Compute gaps. $Δ_{P} = 0.42 - 0.40 = 0.02$ ; $Δ_{Q} = 0.15 - 0.12 = 0.03$ ; $Δ_{R} = 0.38 - 0.01 = 0.37$ . Why this step? The gap isolates overfitting specifically — how much worse new data is than studied data.
Read the gaps together with the level. P: tiny gap but both errors high → underfitting (cell C5): the model is too simple to even fit the training data (far left of the map figure). R: huge gap, near-zero train error → overfitting (cell C4): memorised noise (far right of the map). Q: small gap and low errors → good fit (the violet sweet-spot dot). Why this step? A small gap alone is not enough — an underfit model also has a small gap. You must check whether the level of error is acceptable, not just the gap.
Deploy Q. Lowest $E_{val}$ (0.15) among all three. Why this step? Validation error is our proxy for real-world error; we pick the smallest. Regularizing R (see Overfitting and Regularization) could also rescue it, but Q already wins.

Verify: $Δ_{P} = 0.02, Δ_{Q} = 0.03, Δ_{R} = 0.37$ ✓; smallest $E_{val}$ is Q's $0.15$ ✓; P has both errors $\geq 0.40$ (underfit) ✓.

Example 5 — Negative gap (cell C6)

Forecast: is a negative gap always a bug, or can it happen honestly?

Compute the gap. $Δ = 0.20 - 0.25 = - 0.05$ . Why this step? We first confirm the sign is genuinely negative, not a typo.
List honest causes. (i) The validation fold, by luck of the draw, contains easier examples than training. (ii) Regularization/dropout is active during training (raising $E_{train}$ ) but switched off at validation. (iii) The training set carries more label noise. Why this step? $Δ < 0$ is rare but legal. Before blaming a bug, we check whether a benign explanation fits.
The real red flag to rule out — leakage in the wrong direction. If validation somehow leaked into training or the split isn't i.i.d., that's a genuine bug. Here, with dropout on, cause (ii) fully explains it → not broken. Why this step? We separate "surprising but fine" from "actually wrong" so we don't waste hours debugging a healthy run.

Verify: $Δ = 0.20 - 0.25 = - 0.05 < 0$ ✓; sign is negative and a valid non-bug explanation (dropout on during train) exists ✓.

Example 6 — Degenerate case: perfect interpolation (cell C7)

Forecast: what training error does a curve with as many free knobs as data points achieve?

Count knobs vs points. A degree-9 polynomial has $9 + 1 = 10$ coefficients — exactly the number of data points. Why this step? When free parameters $p$ equal samples $n$ , the curve can thread through every point exactly. This is the boundary case $p = n$ .
Training error is exactly zero. Each prediction equals its target, so every loss term $(y_{i} - \hat{f} (x_{i}))^{2} = 0$ , and $E_{train} = \frac{1}{10} \sum 0 = 0$ . Why this step? This is the algebraic meaning of "interpolation": zero residual at every fitted point.
Why validation still suffers. Between the fitted points, the wiggly curve swings wildly up and down (look at the violet overshoot in the figure). New validation $x$ -values land between the training points, right on those wild swings, so the curve's prediction there is far from the true trend — hence each validation loss $(y_{j} - \hat{f} (x_{j}))^{2}$ is large and $E_{val}$ is large. The model spent all its capacity memorising the 10 training points, none of it on capturing the smooth underlying pattern that would generalise. Why this step? Zero training error is the most misleading possible number — this is the clearest warning that low $E_{train}$ alone means nothing (see Generalization Error).

Verify: coefficients $= 9 + 1 = 10 = n$ ✓; interpolating fit → $E_{train} = 0$ ✓ (checked numerically by fitting and evaluating).

Example 7 — Limiting behaviour of the validation estimate (cell C8)

Forecast: does the estimate keep getting better forever, and how fast?

Recall what one sample contributes. Each validation example scores a loss that is either $1$ (wrong) or $0$ (right). Its long-run average is exactly $p$ , the true error rate. Such a single 0/1 outcome is a Bernoulli draw, and a fact about Bernoulli draws is that its variance (the average squared distance from its mean) equals $p (1 - p)$ . Why this step? We must know the spread of one sample before we can know the spread of an average of many — this is the fact the standard-error formula above the matrix was built on.
Standard error of the average. $E_{val}$ is the average of $n_{val}$ independent such 0/1 losses. For an average of $n$ independent things, variance shrinks by a factor $n$ , so the standard error (its square root) is $SE = p (1 - p) / n_{val}$ — the very formula stated earlier. This is why the Law of Large Numbers works: the wobble carries a $1/ n$ factor that drives it to zero. Why this step? This turns the vague promise "more data → better estimate" into an exact, computable rate.
Plug in. With $p (1 - p) = 0.30 \times 0.70 = 0.21$ :
- $n = 100 : 0.21/100 = 0.0021 \approx 0.0458$
- $n = 10, 000 : 0.21/10000 \approx 0.00458$
- $n = 1, 000, 000 : 0.21/1000000 \approx 0.000458$ Why this step? Concrete numbers show the $1/ n$ law: multiply $n$ by 100, the error shrinks by 10.
Take the limit. As $n_{val} \to \infty$ , $SE \to 0$ , so $E_{val} \to E_{true} = 0.30$ . Why this step? This is the precise sense in which validation error "approximates generalization" — it converges to the truth as the held-out set grows.

Verify: $SE (100) \approx 0.0458$ , $SE (10, 000) \approx 0.00458$ , $SE (1, 000, 000) \approx 0.000458$ ✓; each $100 \times$ in $n$ gives a $10 \times$ smaller SE ✓; the limit as $n \to \infty$ is $0$ , so $E_{val} \to 0.30$ ✓.

Example 8 — Real-world word problem (cell C9)

Forecast: will random shuffling over- or *under-*estimate the model's real 2024 performance?

Assign records by year. Train 2019–2022 $= 4 \times 2000 = 8, 000$ ; validate early-2023 $= 1000$ (half of 2023); test late-2023 $= 1000$ . Why this step? Deployment is in the future, so the honest test is "train on the past, predict the future" — exactly what a time split simulates.
Why random shuffling lies. Random mixing lets the model train on late-2023 records and be tested on early-2023 records — it peeks at the future. That is a subtle form of Data Leakage, and it makes $E_{test}$ optimistically low. Why this step? If evaluation conditions don't match deployment conditions, the reported score is a fantasy.
Conclusion. The senior's time split gives a realistic (higher, honest) error estimate. Why this step? An honest-but-worse number beats an optimistic-but-fake one, because reality won't be fooled.

Verify: $8000 + 1000 + 1000 = 10, 000$ ✓; train fraction $8000/10000 = 0.80$ ✓; each of val and test $= 1000$ ✓.

Example 9 — Exam twist: tuning on the test set (cell C10)

Forecast: is reporting the best-of-50 test scores an honest estimate — and if not, which way does it lie?

Spot the mechanism. With 50 models each scored on the same test set, some model gets a lucky draw of easy test points by pure chance. Picking the minimum of 50 noisy estimates is systematically below the average. Why this step? Choosing the best-looking of many random results biases you toward flukes — the "winner's curse."
Quantify the underestimate. Reported $0.18$ vs true $0.25$ : underestimate $= 0.25 - 0.18 = 0.07$ , i.e. the score is understated by $0.07/0.25 = 28%$ . Why this step? Naming the gap makes the danger concrete: a claimed 18% error is really 25%.
The fix. Use a validation set for the 50-way comparison, then touch the test set once on the single chosen model. The test set stays an unbiased referee. Why this step? This is the entire reason the three-way split exists — the moment the test set influences model choice, it becomes a second validation set and its number stops being trustworthy.

Verify: underestimate $= 0.25 - 0.18 = 0.07$ ✓; relative $= 0.07/0.25 = 0.28 = 28%$ ✓.

Recall

Recall Which single number tells you a model is overfitting — and its trap?

The gap $Δ = E_{val} - E_{train}$ . Trap: a small gap can also mean underfitting if both errors are high — always check the error level too. ::: gap plus level

Recall Why is 1% validation fine for 2 million samples but not for 1 thousand?

Stability depends on the absolute count ( $1%$ of 2M $=$ 20,000 points, worst-case SE $\approx 0.0035$ ), not the percentage. For $N = 1000$ , 1% is 10 points — useless. ::: absolute size rules

Recall What is the standard error of a fraction-of-wrong estimate, and why?

$SE = p (1 - p) / n_{val}$ , because $E_{val}$ averages $n_{val}$ independent 0/1 (Bernoulli) losses, each with variance $p (1 - p)$ ; averaging shrinks variance by $n_{val}$ . ::: Bernoulli over n

Recall What happens to

$E_{val}$ as the validation set grows without bound? Its standard error $p (1 - p) / n_{val} \to 0$ , so $E_{val} \to E_{true}$ (Law of Large Numbers). ::: converges to truth

Recall When may

$E_{train}$ be exactly 0 yet the model be terrible? When parameters $\geq$ samples (interpolation): the curve threads every point but wiggles wildly between them, so $E_{val}$ is large. ::: interpolation degeneracy

Recall Reporting the best-of-many

test errors — which way does it lie? Optimistically (too low) — the winner's curse. Use validation to choose, test once at the end. ::: understates true error