Foundations — Training, validation, and test error

Before you can read a single formula in the parent note, you need to earn every symbol it uses. This page introduces them one at a time, from absolute zero, so that by the end you will understand every piece of a line like "average loss over the validation cards" — including the funny subscripts and Greek letters.

0. The picture everything hangs on

Imagine a big box of index cards. Each card has a question on the front and the true answer on the back. That box is our dataset. Everything in this topic is about how we deal these cards into three smaller boxes.

Look at the three coloured boxes. The orange box (train) is huge — that is where the model studies. The teal box (validation) is smaller — that is where we compare study methods. The plum box (test) is sealed shut until the very end — that is our final exam.

1. A data point: $(x,y)$

• $x$ = the question. It could be an image, a row of numbers, a sentence. Picture the front of one card.
• $y$ = the true label. Picture the back of the same card.

We write them glued together as a pair $(x,y)$ — "this input goes with this target."

Why the topic needs it: every error we ever compute is built by going card-by-card and comparing the model's guess to the real $y$. No cards, nothing to measure.

2. The model as a guessing machine: $\hat{f}$ and $\hat{f}(x)$

The little hat ($\hat{}$) is a universal maths symbol meaning "an estimate — our best guess, not the truth." So $\hat{f}$ is our guessing function, and $\hat{f}(x)$ is the guess it makes for card $x$.

In the figure: a card front $x$ goes into the orange machine, and out comes $\hat{f}(x)$. Next to it sits the real answer $y$. The gap between the two arrows is what "error" will measure.

Why the topic needs it: the model is the whole point. Training, validation, and test errors are all just "how far is $\hat{f}(x)$ from $y$", counted over different boxes of cards.

3. The knobs: parameters $\theta$ and the best setting ${\theta }^{\ast }$

• $\theta$ = any setting of the dials.
• ${\theta }^{\ast }$ (theta-star) = the one special setting that makes the guesses fit the training cards as closely as possible. The star always means "the best / chosen one."

That is why the parent writes $\hat{f}(x;{\theta }^{\ast })$ — read "the guess for $x$ using the chosen dial settings". The semicolon just separates "the input we feed" from "the dials we've fixed."

Why the topic needs it: training is the act of searching for ${\theta }^{\ast }$. Everything after training uses that frozen ${\theta }^{\ast }$.

4. Being wrong, measured: the loss $L(y,\hat{f}(x))$

Read $L(y,\hat{f}(x))$ as "the loss between the real answer and our guess."

A simple example of a loss is the squared gap (used when $y$ is a number, like a price or a temperature): $L(y,\hat{f}(x))=(y-\hat{f}(x){)}^2$ Picture a ruler measuring the vertical distance between the dot (truth) and the line (guess), then squaring it so negatives don't cancel positives.

In the figure each teal dot is a true $y$; the orange line is $\hat{f}$. The vertical dashed segments are the gaps. Loss turns each segment into a positive number.

Why the topic needs it: you cannot average "wrongness" until you have turned each guess into a single number. $L$ does exactly that, one card at a time — whether the answer is a number (squared loss) or a category (0-1 loss).

5. Counting and adding: $n$, $\sum$, and the average

• $i$ is a counter — it walks through the cards one by one.
• $n$ is how many cards are in the box (so $n_{\text{train}}$ = number of training cards, $n_{\text{val}}$ = number of validation cards, and so on).

The subscript $i$ on $x_i$ and $y_i$ just means "the $i$-th card's front and back."

Why the topic needs it: all three errors you meet next are this exact formula. The only thing that changes is which box of cards you loop over.

6. The three errors, named at last: $E_{\text{train}}$, $E_{\text{val}}$, $E_{\text{test}}$

Now that you own $L$, $\sum$, $n$, and the average, the three errors are simply the same average run over three different boxes. The little word underneath each $E$ (the subscript) tells you which box.

7. Where the cards come from: the distribution $P_{\text{data}}$

Picture a giant lottery machine tumbling infinitely many cards; our dataset is one scoop.

Why the topic needs it: the whole promise of "validation error tells you about real-world performance" relies on the validation cards being a fair, random slice from the same factory that will produce future cards.

8. The expected value $\mathbb{E}$ and the true error

So the true generalization error is $E_{\text{true}}={\mathbb{E}}_{(x,y)\sim P_{\text{data}}}[L(y,\hat{f}(x))]$ read: "the average loss over all cards the factory could ever make." The subscript $(x,y)\sim P_{\text{data}}$ says "cards drawn from the factory."

Why the topic needs it: $E_{\text{true}}$ is the target we secretly care about; $E_{\text{val}}$ and $E_{\text{test}}$ are the measurable shadows of it.

9. Comparing two errors: the gap $\Delta$

• $\Delta \approx 0$: studies and exam scores match → healthy.
• $\Delta \gg 0$ (much bigger than zero): aced the studying, flunked the exam → overfitting (see Overfitting and Regularization).

Why the topic needs it: the gap is the single most important diagnostic number the topic teaches — it is the signal of overfitting.

How the foundations feed the topic

Each box you build on this page becomes a symbol you use in the parent note. Follow the arrows: cards + model + knobs → loss → averaged → the three errors → their difference.

Where these lead next

• The full three-box strategy: the parent note
• What to do when $\Delta$ is large: Overfitting and Regularization
• Squeezing reliable estimates from tiny datasets: Cross-Validation
• Choosing between models honestly: Model Selection
• Watching errors change as data grows: Learning Curves
• The quantity all of this estimates: Generalization Error
• The classic way to accidentally cheat: Data Leakage

Equipment checklist

Cover the right side and answer out loud — if you can, you are ready for the parent note.