Train - validation - test splitting: Question bank

Before we start, three words appear again and again, so let's pin them down in one breath:

Here is the whole page in one picture: the data is one deck of cards, split three ways, and only the sealed pile is opened at the very end.

Figure — Train - validation - test splitting

A word we'll lean on: generalization means "how well the model does on data it has never seen". Everything on this page is ultimately about not fooling ourselves about that one number.

Two more ideas will keep appearing, so let's pin them to pictures now.

Leakage, drawn. When you fit a scaler (or anything) on all the data before splitting, the arrow of information runs backwards — from test into train. The figure shows the forbidden arrow in red.

Why the test estimate has a $\pm$ band. Accuracy measured on a finite test set is itself a coin-flip experiment: each of the $n$ test rows is either right or wrong, so the measured accuracy jitters around the true value. Let $p$ be the estimated accuracy (the fraction correct on the test set, a number between 0 and 1) and let $n$ be the test-set size (how many rows we scored). Then the 95% band half-width is

Formula Where the

$\pm$ band comes from $half-width \approx 1.96 \frac{p ( 1 - p )}{n}$ Reading each symbol: $p$ = fraction of test rows the model got right; $n$ = number of test rows; $1.96$ = the "how many standard deviations cover 95%" number for a bell curve. Why this shape? Getting each test row right/wrong is like flipping a biased coin with success chance $p$ . Counting successes over $n$ flips is a binomial count; its standard deviation as a fraction is $p (1 - p) / n$ . For large $n$ that binomial is well-approximated by a bell curve (this is the normal approximation), and $\pm 1.96$ standard deviations of a bell curve holds 95% of the probability. When it breaks: the normal approximation needs $n$ reasonably large and $p$ not too close to $0$ or $1$ (rule of thumb: both $n p$ and $n (1 - p)$ above ~10). Near $p = 0$ or $p = 1$ the band is lopsided, not symmetric, and this formula overstates it.

The figure below shows that band shrinking as $n$ grows — the $1/ n$ law in one glance.

True or false — justify

The claim is stated as if true. Decide, then give the reason — that's where the learning is.

A bigger test set always gives a better model.

False. The test set is never used to change the model, so growing it cannot improve the model — it only makes the estimate of performance more precise while stealing samples from training.

If train and test accuracy are both high, the split was done correctly.

False. They could both be high because test data leaked into training (e.g. scaling fit on all data), which makes the test score dishonestly high while looking healthy.

Stratification is only about the training set.

False. Every split — train, validation and test — is stratified so all three carry the same class proportions; a skewed test set gives meaningless metrics.

A random shuffle-and-slice already guarantees each split has the right class balance.

False. Random shuffling balances splits on average only; for small or imbalanced data a single draw can dump too few (or zero) minority samples into a split. True stratification splits each class separately.

Stratification only makes sense for classification.

False. For regression you can stratify on binned target values (e.g. deciles of the price), and for multi-label you use iterative stratification that balances each label roughly — the goal (matching target distribution across splits) is the same everywhere.

Once you report the test score, you can go tune the model to raise it and report again.

False. The moment you tune toward the test score, that score becomes a validation score in disguise and stops estimating generalization — the test set is spent.

The validation set measures real-world performance.

False. Because we pick the model that scored best on validation, the validation score is optimistically biased; only the untouched test set estimates real-world performance.

Two-way (train/test) splitting is always insufficient.

False. It is fine when you do no model selection. You only need the third set once you compare many models — otherwise there's no selection bias to protect against.

Cross-validation and a three-way split solve the same problem.

Partly. Both estimate generalization, but cross-validation reuses data across folds for a low-variance estimate on small datasets, while a fixed three-way split keeps a permanently sealed test set — they're complementary, not identical.

Nested cross-validation is just cross-validation done twice for fun.

False. It has an inner loop that selects hyperparameters and an outer loop that scores the selected model on data the inner loop never saw — so it gives an unbiased estimate while tuning, which a single CV loop cannot when tuning is deep.

With 10 million rows, an 80/10/10 split is the right default.

False. With huge

N

, 5% is already tens of thousands of test samples — plenty. Pushing to 90/5/5 or even 95/2.5/2.5 hands more data to training with no real loss in estimate precision.

Setting a fixed random seed makes the split "correct".

False. A seed only makes the split reproducible. A reproducible but unstratified or leaky split is reliably wrong.

Spot the error

Each item describes a workflow. Name the flaw and say why it corrupts the estimate.

scaler.fit(X_all) then split into train/test. What broke?

The scaler's mean and standard deviation were computed using test rows, so test information leaks into training — this is classic data leakage via preprocessing. Fit the scaler on train only, then transform val and test.

We trained 200 models and reported the best test accuracy as our result. What broke?

Picking the best of 200 test scores is selecting on the test set — the winner is partly luck. That best number is optimistically biased; selection should happen on validation, and the test set is read once for the chosen model.

We used early stopping and stopped when test loss rose. What broke?

Early stopping is a decision about the model, so it must watch the validation set. Watching test loss lets the test set steer training — leakage again.

We ran a heavy hyperparameter search with one fixed validation set, then reported that validation score. What broke?

Deep tuning against a single validation set overfits the validation set; its score is now optimistic. Either keep a sealed test set for the final number, or use nested cross-validation so an outer loop scores on untouched data.

For a time-ordered dataset (daily sales) we shuffled rows, then split randomly. What broke?

Shuffling destroys temporal ordering, so future rows land in training and predict past rows in test — the model "sees the future". Time-series needs a chronological split, not a random one.

We fit the scaler on train, but re-fit a fresh scaler on validation before evaluating. What broke?

Validation and test must use the training statistics via transform, never their own. Re-fitting gives each split a different scale, so the model receives inputs unlike what it trained on.

We split first, then removed duplicate rows across the whole dataset. What broke?

If a duplicate sits in both train and test, the model memorizes it and "predicts" it perfectly at test time — a hidden leak. De-duplicate before splitting, or ensure duplicates stay within one split.

We stratified by the target but the same patient appears in both train and test. What broke?

Rows from one patient are correlated; splitting rows instead of groups leaks patient-specific signal. Use group-aware splitting so all of a patient's rows land in one split.

For a regression target we stratified by rounding to the nearest integer, but 95% of values were 0. What broke?

Binning collapsed almost everything into one bin, so stratification did nothing useful. Bin by quantiles (equal-count bins) so each bin actually splits, and check the tails aren't starved.

Why questions

The "why" is the whole point — a memorized rule you can't justify will fail on a new variant.

Why do we hide the test set until the very end instead of checking it now and then?

Every peek is a chance to (consciously or not) tweak choices toward it, which converts it into a validation set and destroys its role as an unbiased generalization estimate.

Why fit preprocessing on the training set only?

At deployment the model meets data whose statistics it couldn't have known in advance; using only train statistics simulates that honestly and prevents leakage. See feature scaling for the fit/transform pattern.

Why does a validation set specifically fight overfitting during model selection?

Overfitting is fitting noise in the training data; a separate validation score rewards models that generalize, so we select the one that does well on data it never trained on rather than the one that memorized hardest.

Why does the 95% confidence interval on test accuracy shrink as the test set grows?

The half-width is

1.96 p (1 - p) / n

with

p

the estimated accuracy and

n

the test size, so it falls with

1/ n

— quadrupling the test size roughly halves the band.

Why use nested cross-validation instead of a single CV loop when tuning many hyperparameters?

A single loop reuses the same folds to both pick hyperparameters and score them, biasing the score up. The nested outer loop scores on folds the tuning never touched, giving an honest estimate while still tuning.

Why prefer cross-validation over a fixed three-way split when

N

is small?

A tiny fixed validation/test set gives a noisy, luck-dependent estimate. Cross-validation rotates every point through the held-out role, averaging out that luck for a more stable estimate.

Why stratify by the target class rather than by some feature?

The metric we care about (accuracy, recall) depends on class balance; matching class proportions across splits keeps those metrics comparable and stops one split from being artificially easy or hard.

Why is the validation score usually a little too optimistic even when done correctly?

Because we choose the model that scored best on validation. Choosing the maximum of several noisy scores biases it upward — hence a final, unseen test set is still needed.

Edge cases

Boundaries and degenerate inputs — where confident rules quietly break.

A class has only 3 samples total. What happens to a 70/15/15 stratified split?

0.15 \times 3 = 0.45

, and

⌊ 0.45 ⌋ = 0

, so validation and test get zero of that class. You can't reliably evaluate a class no split contains — merge classes, gather more data, or use cross-validation.

Your dataset has 500 rows. Is a fixed three-way split a good idea?

Usually no. Splitting 500 rows leaves a validation/test set so small its score is dominated by noise; below ~1000 rows, prefer cross-validation.

The full dataset is 100% one class (no positives at all). Can any split help?

No split fixes this — with a single class there is nothing to classify and every metric is degenerate. This is a data-collection problem, not a splitting one.

You have a regression target (house price) — how do you "stratify" with no classes?

Bin the target into quantiles (e.g. 10 equal-count price bands), stratify on the bin label, then split — this keeps each split's price distribution matched to the whole.

You have a multi-label problem (an image can carry several tags at once). Why is plain stratification hard?

Each row belongs to many classes at once, so you can't cleanly assign it to one class's split; use iterative/greedy multi-label stratification that balances every label's count across splits as closely as possible.

You must split but the classes are 99.9% / 0.1%. What extra care is needed?

Plain stratification may still starve the rare class in the smallest split; consider larger validation/test sizes, stratified cross-validation, or grouping so the few positives are shared sensibly.

The confidence-interval formula gives a tiny band when the model is 100% accurate on test (

p = 1

). Trust it?

No — at

p = 1

the term

p (1 - p) = 0

so the band collapses to zero, which is nonsense. The normal approximation fails near

p = 0

p = 1

; use an exact (e.g. Clopper–Pearson) interval there.

Two split ratios both give an integer test size — say 20% of 5000 vs 20% of 5004. Does the leftover row matter?

Rounding leaves a remainder (e.g. after

⌊ \cdot ⌋

per class); it's assigned to one split by convention, shifting sizes by one or two rows — negligible for estimates but worth checking your splits sum to

N

with no dropped or duplicated rows.

You have 3500 train / 500 val / 1000 test. Why give test more rows than val here?

Validation only needs to reliably rank a handful of model variants, while the test set must give a trustworthy final number; the wider

\pm

tolerance on a mere ranking lets validation be smaller.

Recall One-line self-test

Cover the answers and give the reason for each: (1) Why three sets not two? (2) Why fit the scaler on train only? (3) Why does a bigger test set never improve the model? (4) What does nested CV add over vanilla CV? ::: (1) A separate validation set absorbs the optimistic bias of model selection, keeping test as an untouched generalization estimate. (2) To simulate unseen data and prevent leakage. (3) Test data never updates weights — it only sharpens the estimate, at the cost of training samples. (4) An outer loop that scores the tuned model on data the tuning never saw, so the estimate stays unbiased under deep tuning.