Training, validation, and test error: Question bank

This page is a question bank of traps — the sneaky, easy-to-get-wrong ideas hiding inside the training/validation/test split. Each item is a one-line reveal: read the prompt, say your answer out loud, then reveal. The answer side always gives you the reasoning, not a bare yes/no.

Before we start, let us earn every symbol and word this page will use.

Definition Symbols and words used on this page

A sample = one data point, a pair $(x, y)$ : an input $x$ (e.g. an image) and its correct answer $y$ (e.g. the label "cat").
$N$ = the total number of samples in the whole dataset, before any splitting.
$n_{train}, n_{val}, n_{test}$ = the number of samples that land in the train, validation, and test subsets. So $N = n_{train} + n_{val} + n_{test}$ .
$p$ = the number of parameters (tunable numbers, the "knobs") inside the model. A bigger $p$ means a more flexible, more complex model.
$\hat{f}$ = the trained model itself — a function that takes an input $x$ and outputs a prediction $\hat{f} (x)$ .
$L (y, \hat{f} (x))$ = the loss: a single number saying how wrong the prediction $\hat{f} (x)$ is compared to the true answer $y$ (bigger = more wrong).
$P_{data}$ = the true data distribution: the imaginary infinite pool of all possible real-world samples, from which our dataset is just a finite handful.
i.i.d. = independent and identically distributed: each sample is drawn fresh and separately ("independent") from the same pool $P_{data}$ ("identically distributed") — like drawing marbles from one bag, replacing each time.

The figure below fixes the whole picture in one glance — refer back to it as the traps arrive.

Figure — Training, validation, and test error

And here is what the two big failure words — underfitting and overfitting — look like as model complexity $p$ grows:

Now the traps.

True or false — justify

TF1. A model with training error

0

is a great model.

False. Zero training error often means the model memorised the training set including its noise — the tell-tale sign of overfitting, not skill. You only learn its worth from

E_{val}

E_{test}

TF2. If

E_{val} < E_{train}

, something must be broken.

False. A negative gap

Δ < 0

is unusual but legitimate — it can happen when the validation split is "easier" by luck, when regularisation (dropout) is on during training but off during evaluation, or when the training set carries more label noise than the validation set.

TF3. The test set can be used to decide when to stop training (early stopping).

False. Early stopping is a decision, and any decision made using the test set turns it into a second validation set — that is data leakage, and

E_{test}

stops being unbiased.

TF4. A larger validation set always gives a lower validation error.

False. More validation data lowers the variance of the estimate (it gets more reliable), not its expected value.

E_{val}

converges toward the true generalization error by the Law of Large Numbers — it does not slide downward.

TF5. Because the test set is untouched during training,

E_{test}

is guaranteed to equal the true generalization error.

False. It is an unbiased estimate, but a finite sample still has sampling noise, and if the test distribution differs from real-world data (distribution shift), even an unbiased sample of the wrong distribution misleads you.

TF6. If two configs have the same

E_{val}

, they are equally good and it doesn't matter which you ship.

Roughly true but risky. Equal validation error on a small validation set can be a coin-flip; prefer the simpler model (lower variance, better generalization margin) or confirm with cross-validation.

TF7. Doing model selection on the validation set makes

E_{val}

a biased (optimistic) estimate of generalization.

True. By picking the model that scored lowest on that specific validation set, you exploit its lucky quirks, so

E_{val}

of the winner is biased downward — this is exactly why a separate untouched test set exists.

TF8. Increasing model complexity always increases the training-validation gap

Δ

Generally true, but not monotonically from step one. As the parameter count

p

grows toward the number of training samples

n_{train}

, training error falls toward

0

while validation error eventually rises, so

Δ

widens — but at very low complexity both errors are high and

Δ

can be tiny (underfitting, not overfitting).

Spot the error

SE1. "I have 200 samples, so I'll do a single 80/10/10 split to be safe."

With only

N = 200

points, a

20

-sample validation set is so tiny its error estimate swings wildly. Use k-fold cross-validation instead, keeping one held-out test set.

SE2. "I normalised the whole dataset (computed the mean and std over all rows), then split into train/val/test."

The scaling statistics leaked information from val and test into training — that's data leakage. Compute mean/std on the training set only, then apply those same numbers to val and test.

SE3. "My validation error looked bad, so I tried 30 different architectures until one gave low

E_{val}

, and I reported that as my expected accuracy."

Searching over many configs fits the validation set the way training fits the training set. The reported

E_{val}

is optimistically biased; the true score must come from the untouched test set.

SE4. "I picked the best model on validation, then reported

E_{test}

. It was worse than

E_{val}

, so I tweaked regularisation and re-ran the test."

The moment you tweaked because of the test result, the test set became a validation set. Its next number is no longer unbiased — you'd need fresh held-out data.

SE5. "Train and test both come from the same big file, so I took the first 80% as train and last 20% as test."

If the file is ordered (by time, class, or source), the split is not i.i.d. — train and test see different distributions. Shuffle first (or split by time deliberately for time-series, never accidentally).

SE6. "Config C has the lowest training error (

0.02

), so it's my best model."

Training error measures memorisation, not generalization. Config C's high validation error (

0.55

) exposes overfitting; low

E_{val}

, not low

E_{train}

, is the selection criterion.

SE7. "After choosing hyperparameters with validation, I threw the validation data away and shipped the model."

Wasteful. Once tuning is done, retrain the chosen config on train + validation combined to use maximum data, then report the earlier test score.

SE8. "I balanced my classes by oversampling the minority class, then split into train/test."

Duplicated minority samples can land in both train and test, so the model is tested on rows it effectively memorised — leakage. Split first, oversample the training set only.

Why questions

WHY1. Why do we need three sets instead of just train and test?

The validation set lets us make selection decisions without touching test. If we tuned on test, test error would be optimistically biased and no longer an honest final score.

WHY2. Why does averaging validation losses approximate the true generalization error?

Each i.i.d. validation point is an unbiased sample of the loss

L (y, \hat{f} (x))

; by the Law of Large Numbers the average of many independent unbiased samples converges to the expectation over

P_{data}

, i.e.

E_{true}

WHY3. Why is the generalization gap

Δ

mostly driven by variance, not bias?

Bias (wrong assumptions) hurts training and validation equally, so it barely changes their difference. Variance is sensitivity to the particular training set, which lowers

E_{train}

while leaving

E_{val}

high — that's the gap.

WHY4. Why can adding regularization raise training error yet lower validation error?

Regularization restricts the model's freedom to fit noise, so it fits the training data slightly worse (higher

E_{train}

) but the reduced variance transfers to unseen data (lower

E_{val}

WHY5. Why does the recommended validation percentage shrink for huge datasets (1% instead of 10%)?

Estimate reliability depends on the count of validation samples

n_{val}

, not the fraction. At total size

N = 100, 000

, even

1%

n_{val} = 1000

samples — plenty — so the rest is better spent on training.

WHY6. Why is a learning curve (error vs. training-set size) more informative than a single train/val number?

A curve reveals why the gap exists: if both errors are high and flat you have high bias (need a bigger model); if

E_{train}

is low but

E_{val}

stays high, you have high variance (need more data or regularization).

WHY7. Why must the validation set be independent of the training process to give a valid estimate?

If gradient updates ever saw the validation data, the model partly memorised it, so

E_{val}

is biased downward and no longer reflects performance on genuinely unseen data.

WHY8. Why does k-fold cross-validation give a more stable estimate than one split on small data?

Every point gets to be in the validation fold exactly once, so the estimate averages out the luck of any single split — it reduces the variance of the error estimate.

Edge cases

EC1. What happens to the training-validation gap as the parameter count

p

approaches the number of training samples

n_{train}

The model can interpolate every training point (

E_{train} \to 0

) by fitting noise, so the gap

Δ

typically blows up — the classic overfitting regime.

EC2. Your validation set contains zero examples of a rare class. What does

E_{val}

tell you about that class?

Nothing — you cannot estimate performance on a class you never evaluate. Use stratified splitting so each subset carries every class in proportion.

EC3. Both

E_{train}

and

E_{val}

are high and nearly equal. Overfitting or underfitting?

Underfitting (high bias). A small gap with high errors means the model isn't expressive enough or is under-trained — the cure is more capacity or longer training, not regularization.

EC4.

E_{train}

is tiny and

E_{val}

is huge. Which way is the model failing, and what fixes it?

Overfitting (high variance). Fixes: more data, regularization (L2, dropout, augmentation), or a simpler model.

EC5. In the limit

n_{val} \to \infty

, what does

E_{val}

converge to?

It converges to the true generalization error

E_{true} = E_{P_{data}} [L (y, \hat{f} (x))]

— the average loss over the whole data pool

P_{data}

— provided the validation data is i.i.d. from the same distribution as deployment data.

EC6. You retrain the selected config on train + val combined. Is the earlier

E_{val}

still meaningful for the new model?

Not exactly — the retrained model is a different model trained on more data, so it should generalize at least as well; the honest score for it is still the untouched

E_{test}

EC7. What breaks if training and test data are not drawn from the same distribution (distribution shift)?

E_{test}

no longer estimates real-world performance — you're honestly measuring performance on the wrong distribution. No amount of clean splitting fixes a mismatch between test and deployment data.

EC8. A degenerate split gives the test set exactly one sample (

n_{test} = 1

). What's wrong with the resulting

E_{test}

A single-point average has enormous variance — the estimate is essentially a coin flip and tells you almost nothing about true generalization.

Recall Quick self-check

The three biggest traps on this page, in one line each ::: (1) Zero training error ≠ good model; (2) any decision made with the test set leaks and biases it; (3) tuning on the validation set makes its own number optimistic — that's why test is separate.