Training, validation, and test error

Overview

The three-way split of data into training, validation, and test sets forms the backbone of honest model evaluation. Each subset serves a distinct purpose in the machine learning pipeline, and confusing them leads to overly optimistic (or pessimistic) performance estimates.

The fundamental tension: you want to know how your model performs on truly unseen data, but you also need feedback during development to improve it. The solution is compartmentalization—each dataset answers a different question.

The Three Errors: Definitions & Roles

$E_{\text{train}} = \frac{1}{n_{\text{train}}} \sum_{i=1}^{n_{\text{train}}} L(y_i, \hat{f}(x_i; \theta^*))$

where $\theta^*$ are the parameters that minimize this very sum. This error measures fit, not generalization.

WHY it exists: You need a metric to guide optimization. Gradient descent needs a function to descend.

WHAT it tells you: How well the model memorizes the training data. Low training error ≠ good model (overfitting).

HOW to use it: Monitor during training to ensure learning is happening. If training error doesn't decrease, check learning rate, initialization, or data preprocessing.

$E_{\text{val}} = \frac{1}{n_{\text{val}}} \sum_{j=1}^{n_{\text{val}}} L(y_j, \hat{f}(x_j; \theta^*))$

WHY it exists: To estimate generalization during development without "peeking" at the test set. You tune hyperparameters (learning rate, regularization, architecture) by comparing validation errors.

WHAT it tells you: Which configuration generalizes best among the options you tried.

HOW to use it: Train multiple models with different hyperparameters, pick the one with lowest $E_{\text{val}}$. Stop training (early stopping) when $E_{\text{val}}$ stops decreasing.

$E_{\text{test}} = \frac{1}{n_{\text{test}}} \sum_{k=1}^{n_{\text{test}}} L(y_k, \hat{f}(x_k; \theta^*))$

WHY it exists: To provide an unbiased estimate of generalization to real-world data. This is your "competition performance."

WHAT it tells you: The true expected error on new data (up to sampling noise and distribution shift).

HOW to use it: Compute once at the very end, after all decisions are finalized. If you iterate based on test error, it becomes a second validation set (data leakage).

Derivation: Why Validation Error Approximates Generalization

Start with the true generalization error (expected loss over the data distribution):

$E_{\text{true}} = \mathbb{E}_{(x,y) \sim P_{\text{data}}}[L(y, \hat{f}(x))]$

We cannot compute this—we don't have the full distribution $P_{\text{data}}$. But if the validation set ${(x_j, y_j)}_{j=1}^{n_{\text{val}}}$ is drawn i.i.d. from $P_{\text{data}}$, then by the Law of Large Numbers:

$E_{\text{val}} = \frac{1}{n_{\text{val}}} \sum_{j=1}^{n_{\text{val}}} L(y_j, \hat{f}(x_j)) \xrightarrow{n_{\text{val}} \to \infty} \mathbb{E}_{P_{\text{data}}}[L(y, \hat{f}(x))] = E_{\text{true}}$

Why this step? Each validation sample contributes an unbiased estimate of the true loss at that point. Averaging independent unbiased estimates converges to the expectation.

Crucially, the validation set must be independent of the training process. If the model saw validation data during gradient updates, the estimate is biased downward (the model "memorized" validation patterns).

Similarly, $E_{\text{test}}$ approximates $E_{\text{true}}$, but with one key difference: test error is computed on a model whose hyperparameters were chosen using validation error. If you don't have a separate validation set and tune on the test set, then $E_{\text{test}}$ underestimates $E_{\text{true}}$ (because you picked the model that happened to do well on that test set by chance).

Typical Dataset Split Ratios

• Large datasets ($N > 100{,}000$):

  • Train: 98%, Val: 1%, Test: 1%
  • Why? With 100k samples, 1% = 1000 samples—enough for stable error estimates.

• Medium datasets ($10{,}000 < N < 100{,}000$):

  • Train: 80%, Val: 10%, Test: 10%
  • Why? Balances having enough training data with reliable validation/test estimates.

• Small datasets ($N < 10{,}000$):

  • Use k-fold cross-validation instead: split into $k$ folds, rotate which fold is validation, average results. Then evaluate once a held-out test set.
  • Why? With small data, a single validation split has high variance. Cross-validation reduces this.

Derivation of fold size: Suppose $N = 1000$, $k=5$. Each fold has $N/k = 200$ samples. You train on $4 \times 200 = 800$ samples and validate on $200. Repeat 5 times, each time using a different fold as validation. Average the5 validation errors to get a stable estimate.

The Training-Validation Gap

$\Delta = E_{\text{val}} - E_{\text{train}}$

Interpretation:

• $\Delta \approx 0$: Model generalizes well (good fit, no overfitting).
• $\Delta \gg 0$: Overfitting—model memorized training data but fails on new data.
• $\Delta < 0$: Rare, but possible if training set is unrepresentative or has label noise.

Why does $\Delta$ grow with model complexity?

Consider a model with $p$ parameters trained on $n$ samples. As $p \to n$, the model can fit every training point exactly (interpolation). Training error $\to 0$. But the model has "spent" all its capacity memorizing noise, so validation error remains high.

Bias-variance decomposition (informal): $E_{\text{val}} \approx \text{(irreducible noise)} + \text{(bias)}^2 + \text{(variance)}$

• Bias: error from wrong model assumptions (underfitting).
• Variance: error from sensitivity to training set fluctuations (overfitting).

As model complexity increases:

• Bias $\downarrow$ (more expressive model)
• Variance $\uparrow$ (more ways to overfit)
• Training error $\downarrow$ (better fit)
• Validation error: $\downarrow$ then $\uparrow$ (sweet spot in middle)

The gap $\Delta$ is primarily driven by variance. Regularization (L2, dropout, data augmentation) reduces variance, shrinking the gap.

Worked Examples

Step 1: Split training into 45,000 train, 5,000 validation.

Why this step? You need a validation set to compare hyperparameters. The test set must remain untouched.

Step 2: Train 10 models with different $(lr, \lambda)$ pairs on the45k images.

| Config | $lr$ | $\lambda$ | $E_{\text{train}}$ | $E_{\text{val}}$ | |-----|------|-------------------|----------------| | A | 0.01 | 0.001 | 0.15 | 0.42| | B | 0.001| 0.01 | 0.08 | 0.38 | | C | 0.1 | 0 | 0.02 | 0.55 |

Why compute both? Training error alone doesn't tell you which model generalizes. Config has lowest training error but highest validation error (overfitting).

Step 3: Pick Config B (lowest $E_{\text{val}}$).

Step 4: Retrain Config B on all 50k training images (combining train+val) to use maximum data. Evaluate once on the10k test set.

Why retrain? You held out 5k samples just for tuning. Now that tuning is done, use them to improve the final model.

Result: $E_{\text{test}} = 0.40$. This is your unbiased estimate of generalization.

Split: 70 train, 15 validation, 15 test.

Training Results (MSE):

| Degree $d$ | $E_{\text{train}}$ | $E_{\text{val}}$ | $E_{\text{test}}$ | |----------|---------------------|----------------| | 1 | 2.8 | 3.1 | (not computed yet)| | 2 | 0.26 | 0.29 | (not computed yet)| | 5 | 0.18 | 0.52 | (not computed yet)| | 10 | 0.09 | 1.24 | (not computed yet)|

Why this step? Degree 1 (linear) underfits—high bias, both errors large. Degree 2 matches true function—low bias, low variance. Degree 5 starts overfitting—training error drops, validation error rises. Degree 10 severely overfits—memorizes noise.

Decision: Pick $d = 2$ (lowest $E_{\text{val}}$).

Test evaluation: $E_{\text{test}} = 0.31$ for $d=2$ model. Close to $E_{\text{val}}$, confirming good generalization.

If we had picked based on training error: We'd choose $d=10$ (lowest $E_{\text{train}} = 0.09$), then discover $E_{\text{test}} \approx 1.2$ — disaster!

Common Mistakes

Why it feels right: The test set is "unseen," so doesn't performance on it reflect generalization?

The fix: Once you look at test performance and make decisions based on it (e.g., "model A beat model B, let's tweak A further"), the test set is no longer independent. You're implicitly optimizing for that specific test set. This is data leakage.

Steel-man: The confusion arises because both validation and test sets are "held out." The key distinction is when you use them. Validation is for development-time decisions. Test is for final reporting.

Correct approach: Never look at test error until the very end. Use validation for all tuning. If you want to iterate further after seeing test error, collect a new test set.

Why it feels right: High accuracy sounds good, and it's the metric you optimized.

The fix: Training accuracy measures memorization, not generalization. A model can achieve100% training accuracy by overfitting (.g., a lookup table). Always report validation or test performance.

Example: A 10-degree polynomial can fit 10 noisy points perfectly ($E_{\text{train}} = 0$) but wildly oscillate between points ($E_{\text{val}} \gg 0$).

Steel-man: Beginers focus on training error because it's what gradient descent directly minimizes. The leap to "this must mean the model is good" is natural but wrong.

Why it feels right: You already trained the model—why train again?

The fix: The model you tuned was trained on only the training subset (excluding validation). To maximize performance, retrain with the chosen hyperparameters on train+val combined before final testing. You held out validation data only to make decisions, not because it's contaminated.

When to skip this: If training is extremely expensive (days of GPU time), you might skip retraining and accept slightly suboptimal performance. But for most cases, retrain.

Active Recall Questions

#flashcards/ai-ml

What are the three types of dataset splits in ML and their purposes? :: Training set (optimize model parameters), Validation set (tune hyperparameters and model selection), Test set (unbiased final evaluation).

1. Practice problems (training): You solve these over and over, learning the techniques. You check your answers and improve. These are easy for you now because you've memorized them.

2. Rehearsal problems (validation): New problems you haven't practiced, but you use them to test which technique works best. "Should I draw a diagram or make a table?" You try both on rehearsal problems and see which gets more right. You pick the best technique.

3. Competition problems (test): Brand new problems at the actual competition. You never saw these before. Your score here shows how good you really are, not just at the problems you practiced.

If you peeked at the competition problems while practicing, you'd think you're better than you are—that's cheating! In machine learning, we keep the test set secret until the very end for the same reason.

Or think: TV Test — you rehearse for a TV appearance (training), have a dress rehearsal (Validation), then the live broadcast (Test—one shot, no redos).

Connections

• Bias-Variance Tradeoff: The training-validation gap is driven by variance (overfitting). High-bias models have small gaps but high errors overall.
• Overfitting and Regularization: Regularization techniques (L2, dropout, early stopping) are chosen by monitoring validation error.
• Cross-Validation: Alternative to single train-val split for small datasets; provides more robust error estimates.
• Model Selection: Validation error is the criterion for choosing among model architectures, hyperparameters, and feature sets.
• Learning Curves: Plots of training and validation error vs. training set size or epochs; diagnose underfitting vs. overfitting.
• Generalization Error: Test error is our best estimate of the true generalization error on the data distribution.
• Data Leakage: Using test data for any decision (even indirectly) biases estimates upward—a critical mistake in practice.

Concept Map

Hinglish (regional understanding)