Exercises — Training, validation, and test error

1. Training — only training data is allowed to move the parameters.
2. Validation — comparing hyperparameter settings is a model-selection decision, exactly what the validation set is for.
3. Test — the final, report-once number is the test error.

Usually $E_{\text{train}}\le E_{\text{val}}\approx E_{\text{test}}$.

• Training error is smallest because the model optimised on that exact data.
• Validation and test are on unseen data, so they are higher and roughly equal (both estimate the same true generalization error).
• If $E_{\text{val}}\ll E_{\text{test}}$, you tuned too hard on validation (leakage of a mild kind, see Data Leakage).

False. Looking at test error to decide anything (pick a model, stop training, change a feature) turns it into a validation set. It stops being unbiased. Rule: compute test error once, at the end.

Recall $E=\frac{1}{n}\sum L_i$ (plain average of the losses). $E_{\text{train}}=\frac{0.10+0.20+0.30}{3}=0.20$ $E_{\text{val}}=\frac{0.40+0.60}{2}=0.50$ $E_{\text{test}}=\frac{0.50+0.70+0.30+0.50}{4}=0.50$ $\Delta =E_{\text{val}}-E_{\text{train}}=0.50-0.20=0.30$ A positive gap of $0.30$ hints at some overfitting.

Large dataset → a 98 / 1 / 1 split works.

• Val $=1\%\times 200,000=2,000$ samples.
• Test $=1\%\times 200,000=2,000$ samples.
• Train $=98\%\times 200,000=196,000$. Why acceptable: $2,000$ is large enough that the error estimate is stable, and you keep the vast majority for training. On a small dataset you'd instead use k-fold cross-validation.

Deploy B: it has the lowest validation error ($0.38$), and validation error is the decision metric.

• C has the lowest training error ($0.02$) but the highest validation error ($0.55$) → classic overfitting: it memorised the training pile.
• Its gap ${\Delta }_C=0.55-0.02=0.53$ is huge; B's gap ${\Delta }_B=0.38-0.08=0.30$ is milder.

• P: gap $\Delta =0.45-0.05=0.40$ (large). → Overfitting. Fix: add regularization, more data, or early stopping (see Learning Curves).
• Q: gap $\Delta =0.42-0.40=0.02$ (tiny) but both errors high. → Underfitting (high bias). Fix: bigger / more expressive model, train longer, better features. Key insight: a small gap alone is not success — Q generalises "well" only in the sense that it's equally bad everywhere.

By trying 50 models and keeping the minimum test error, you are selecting the model that got lucky on that particular test pile's sampling noise. The minimum of many noisy estimates is biased below the true error.

• This is exactly leakage through the selection process: the test set silently became a validation set.
• Fix: reserve a separate validation set for the 50 comparisons; touch test once on the single chosen model. Its number will typically be worse — and honest.

• Bias ↓ (a more flexible model can represent the truth).
• Variance ↑ (more ways to chase noise).
• $E_{\text{train}}$ ↓ monotonically (always fits training data better).
• $E_{\text{val}}$ = a U-shape: falls (bias shrinking dominates) then rises (variance blowing up dominates).
• Best model = bottom of the U, where the marginal bias drop equals the marginal variance rise. See the figure below.

Step 1 — Carve off test first. Hold out $15\%$ → $150$ test points, locked away. Remaining $850$ for development. Step 2 — 5-fold CV on the 850. Each fold $=850/5=170$ points.

• For each degree $d$: train on $4\times 170=680$ points, validate on the held-out $170$, rotate 5 times.
• Average the 5 validation errors → one stable ${\bar{E}}_{\text{val}}(d)$ per degree (see Cross-Validation). Step 3 — Select. Pick the $d$ with the smallest ${\bar{E}}_{\text{val}}(d)$. Step 4 — Refit. Retrain that degree on all 850 development points (max data now that tuning is done). Step 5 — Report. Evaluate once on the $150$ test points → $E_{\text{test}}$, the unbiased final grade. Why CV, not one split? With only 850 points a single 170-point validation set is noisy; averaging 5 folds cuts that variance.

• During tuning: train on $45,000$, validate on $5,000$.
• After choosing the config, retrain on $45,000+5,000=50,000$ images.
• Why allowed: the validation set's only job was choosing the hyperparameters; that decision is now frozen. Feeding those $5,000$ back in adds data and never let them influence which config we picked, so no leakage into model selection.
• Test still stays at $10,000$, untouched → $E_{\text{test}}$ remains honest.

(a) The average of $n$ independent numbers each with variance $s^2$ has variance $s^2/n$, so $\text{SE}(E)=\frac{s}{\sqrt{n}}.$ This is the Law of Large Numbers made quantitative: more held-out data → smaller spread → more trustworthy $E_{\text{val}}$ or $E_{\text{test}}$. (b) Require $\frac{s}{\sqrt{n}}\le 0.01$ with $s=0.5$: $\sqrt{n}\ge \frac{0.5}{0.01}=50\Rightarrow n\ge 2500.$ So you need at least $2,500$ held-out points — which is exactly why a $1\%$ slice of a $250$k dataset is plenty, but a $15$-point validation set is dangerously noisy.

Each $E_{\text{val}}$ is the true error plus noise ${\epsilon }_i$ (mean $0$, spread $\text{SE}$). Selecting the minimum over $m$ noisy values, ${\min }_i(\text{true}+{\epsilon }_i),$ systematically picks a large-negative ${\epsilon }_i$. The expected minimum of $m$ zero-mean noises drifts more negative as $m$ grows (with $\approx$ Gaussian noise it grows like $\text{SE}\sqrt{2\ln m}$). Consequence: the validation score of your winner is optimistic, and more so the more configs you tried. Practical rules: (1) keep a truly untouched test set so this optimism is caught; (2) don't run thousands of configs against a tiny validation set; (3) prefer cross-validation to shrink $\text{SE}$.

Which pile may move the parameters? ::: Only the training pile. Which pile chooses hyperparameters? ::: The validation pile. How many times do you touch the test pile? ::: Once, at the very end. Standard error of an error estimate over $n$ points with per-point spread $s$? ::: $s/\sqrt{n}$. Sign of a healthy gap $\Delta =E_{\text{val}}-E_{\text{train}}$? ::: Small and positive (near $0$).