Visual walkthrough — Training, validation, and test error

This page rebuilds ONE idea from the parent note from absolute zero: why a number computed on a small held-out pile of data (the validation error) is a trustworthy stand-in for how the model does on the whole infinite world of future data (the true error). We will earn every symbol before we use it, and every step gets a picture.

---

Step 1 — What is a "mistake"? Defining loss on one example

WHAT. Before averaging anything, we need to score a single prediction. A model is a function $\hat{f}$: you feed it an input $x$ (say, an image) and it returns a guess $\hat{f}(x)$. The real answer is $y$. The loss $L(y,\hat{f}(x))$ is just a number saying "how wrong was this one guess". Zero means perfect; bigger means worse.

WHY loss and not "right/wrong"? A yes/no correctness flag throws away how wrong you were. Loss keeps the size of the error, so we can do smooth maths (averaging, calculus) with it.

PICTURE. Look at one dot: the true value $y$ sits on the black line, the prediction $\hat{f}(x)$ sits on the orange curve, and the red gap between them is the loss.

---

Step 2 — The number we wish we could compute: true error

WHAT. The world produces examples according to some hidden recipe we call $P_{\text{data}}$ — a probability rule that says how likely each $(x,y)$ pair is. The true error is the average loss over the entire world, weighting each possible example by how often it appears:

$E_{\text{true}}=\underset{\text{average over the whole world}}{\underbrace{{\mathbb{E}}_{(x,y)\sim P_{\text{data}}}}}\left[\underset{\text{loss on one example}}{\underbrace{L(y,\hat{f}(x))}}\right]$

WHY this is the real target. This is exactly the "competition performance" from the parent's music analogy — the mistake we make on genuinely new data. Everything else on this page is an attempt to estimate this one number.

WHY we can't just compute it. The symbol $\mathbb{E}$ ("expected value") here means "add up the loss on every possible example, each weighted by its probability". That is an infinite sum over a recipe $P_{\text{data}}$ we never get to see. So $E_{\text{true}}$ is a real, fixed number — but it is hidden from us.

PICTURE. The whole cloud of grey dots is the world $P_{\text{data}}$. $E_{\text{true}}$ is the average red gap over all of them — but we only ever get to see a handful.

---

Step 3 — The trick: replace the infinite average with a finite one

WHAT. We grab a validation set: $n_{\text{val}}$ examples $(x_j,y_j)$ that the model did not train on. We compute the plain average loss over just these:

$E_{\text{val}}=\underset{\text{divide by count}}{\underbrace{\frac{1}{n_{\text{val}}}}}\underset{\text{add over held-out examples}}{\underbrace{\sum _{j=1}^{n_{\text{val}}}}}\underset{\text{loss on example }j}{\underbrace{L(y_j,\hat{f}(x_j))}}$

WHY an average of a sample? Because we can compute this — it is a finite sum over data we actually hold. The bet we are making is: this measurable average is close to the unmeasurable $E_{\text{true}}$. Step 4 justifies the bet.

WHY "not trained on" matters (previews Step 6). If the model had seen these points during training, it could have memorised them, making $E_{\text{val}}$ artificially tiny — a lie about the true error.

PICTURE. We highlight a small blue sub-cloud pulled out of the grey world. $E_{\text{val}}$ is the average red gap over only the blue dots.

---

Step 4 — Why the sample average converges: the Law of Large Numbers

WHAT. Each validation example gives one loss value $L_j=L(y_j,\hat{f}(x_j))$. Because the points are drawn i.i.d. (independent, and each from the same recipe $P_{\text{data}}$), each $L_j$ is a fresh random draw whose expected value is exactly $E_{\text{true}}$:

$\mathbb{E}[L_j]=E_{\text{true}}\text{for every }j.$

The Law of Large Numbers says: averaging many independent draws pulls the average toward the common expected value.

$E_{\text{val}}=\frac{1}{n_{\text{val}}}{\sum }_{j=1}^{n_{\text{val}}}{L}_j\underset{n_{\text{val}}\to \infty }{\to }{E}_{\text{true}}$

Term by term: the left side is what we measure; the arrow with $n_{\text{val}}\to \infty$ means "as the held-out pile grows"; the right side is the hidden truth we were chasing in Step 2.

WHY this exact tool and not another? We specifically need a theorem that connects a finite average to an expected value. The Law of Large Numbers is precisely that bridge — it is the mathematical reason a poll of 1000 people predicts millions of voters. Averaging is the only operation that makes independent random errors cancel out.

HOW fast? The wobble of $E_{\text{val}}$ around $E_{\text{true}}$ shrinks like $1/\sqrt{n_{\text{val}}}$ — quadruple the validation set to halve the noise. This is why the parent note says 1% of 100{,}000 (=1000 samples) is "enough": $1/\sqrt{1000}\approx 0.032$, a small wobble.

PICTURE. As $n_{\text{val}}$ grows left→right, the noisy $E_{\text{val}}$ curve settles onto the flat dashed line $E_{\text{true}}$, inside a shrinking $1/\sqrt{n}$ funnel.

---

Step 5 — The two silent assumptions (and where they hide)

WHAT. The convergence in Step 4 rests on exactly two words we underlined: i.i.d. and independent of training. Break either and the arrow no longer points at $E_{\text{true}}$.

WHY call them out? Because in real projects they fail quietly — no error message, just a wrong number. This is the Data Leakage trap.

• Assumption A — same distribution (i.i.d.): validation data must come from the same world the model will face. If future data drifts (new camera, new users), $E_{\text{val}}$ estimates the old world, not the new one.
• Assumption B — independence from training: if a validation row also sat in the training set (a duplicate, or a leaked feature), the model may have memorised it.

PICTURE. Left: honest split — training (orange) and validation (blue) clouds are separate, so the blue average is unbiased. Right: leakage — a blue point overlaps an orange one; the model "already knows the answer," and $E_{\text{val}}$ drops below $E_{\text{true}}$ (green dashed).

---

Step 6 — Why the test set exists: selection re-biases validation

WHAT. Suppose we train many models and pick the one with the lowest $E_{\text{val}}$ (this is Model Selection). Even though each individual $E_{\text{val}}$ was unbiased in Step 4, the minimum of several noisy numbers is systematically below the truth — we selected the model that got lucky on this particular validation set.

$\underset{\text{winner we chose}}{\underbrace{\underset{m}{\min }{E}_{\text{val}}^{(m)}}}\le E_{\text{true}}^{(\text{winner})}\text{(on average, biased low)}$

WHY a separate test set fixes it. The test set is touched once, after all choices are locked. We never optimise against it, so no "luckiest of many" bias sneaks in — its average is a clean estimate for the already-chosen model.

PICTURE. Several validation errors scatter around $E_{\text{true}}$ (grey band); we circle the lowest (green) — it sits below the band. The single test measurement (red) lands back on the true line.

---

Step 7 — Degenerate cases: what happens at the extremes

WHAT. Every scenario the reader could hit:

• $n_{\text{val}}=0$ (no held-out data): the arrow in Step 4 has nothing to average. You have no estimate of generalisation — only training error, which overfitting makes meaningless.
• $n_{\text{val}}$ tiny (e.g. 5): the $1/\sqrt{n}$ wobble is huge; $E_{\text{val}}$ is noise. Fix with Cross-Validation — rotate every fold through the validation role and average, so all data contributes to the estimate.
• $n_{\text{val}}$ huge (e.g. 40% of data): the estimate is stable but you starved training, raising both bias and variance of the model itself (see Learning Curves).
• Validation not i.i.d. (time series, grouped data): Step 4 fails silently; split by group / by time so the held-out set truly mimics the future.

WHY show these. The clean convergence of Step 4 is a best case; real datasets live near these edges, and knowing the failure shape tells you which knob to turn.

PICTURE. The bias–variance trade-off of the estimate itself vs held-out fraction $q$: too small → high variance (red), too large → the model degrades (orange). The green valley is the sweet spot the parent's split ratios target.

---

The one-picture summary

The whole page in one frame: the infinite grey world ($E_{\text{true}}$, the hidden target) → sampled into a finite blue validation pile ($E_{\text{val}}$, measurable) → the Law of Large Numbers closes the gap as $n_{\text{val}}$ grows → but selecting the best model biases validation low, so a once-only red test measurement gives the honest final number.

Recall Feynman retelling — say it back in plain words

We want to know how badly our model will mess up on all future data, but "all future data" is infinite, so that number is hidden. Trick: grab a small honest handful of examples the model has never studied, measure the average mistake on them, and trust that this average is close to the hidden number. The reason we can trust it is the Law of Large Numbers — averaging many independent measurements cancels the randomness, and the leftover wobble shrinks like one over the square root of how many examples we grabbed. Two catches: the handful must come from the same world (i.i.d.) and the model must not have peeked at it during training, or the average lies low. And once we pick the best-scoring model out of many, that best score is luckily low, so we keep one final untouched pile — the test set — measured exactly once, to report the honest score. That's the whole reason for three sets. See Generalization Error for the formal version and Overfitting and Regularization for what makes the training–validation gap grow.

Recall Quick self-check

Why can't we compute $E_{\text{true}}$ directly? ::: It averages loss over the entire infinite data distribution $P_{\text{data}}$, which we never fully observe. What theorem lets $E_{\text{val}}$ approximate $E_{\text{true}}$? ::: The Law of Large Numbers — the average of i.i.d. losses converges to their expected value. How does the estimate's noise scale with validation size? ::: Like $1/\sqrt{n_{\text{val}}}$ — quadruple the set to halve the wobble. Why is a separate test set needed after model selection? ::: Choosing the minimum validation error biases it low; the once-only test set gives an unbiased score for the chosen model. What makes $E_{\text{val}}$ dip below $E_{\text{true}}$? ::: Data leakage — the model trained on (or memorised) validation examples, violating independence.