2.6.1 · D5Model Evaluation & Selection
Question bank — Bias-variance tradeoff
This page is a conceptual gym. No heavy arithmetic here — every item tests whether you truly understand the parent tradeoff, not whether you can crunch a polynomial. Cover each reveal line before flipping it.
Before we start, one shared vocabulary reminder so no symbol is used unearned:
- Bias² = how far your average prediction (averaged over many possible training sets) sits from the truth. A systematic miss.
- Variance = how much your prediction jumps around when you swap one training set for another. Instability.
- Irreducible error () = pure noise baked into the data; no model can remove it.
- Total expected error .
True or false — justify
TF1. A model with zero training error must have low bias.
False — zero training error means it fit this sample perfectly, which usually signals high variance (memorising noise), not low bias measured across many datasets.
TF2. Variance is a property of a single trained model.
False — variance is defined across the ensemble of models you'd get from different training sets; a single model has no variance until you imagine resampling the data.
TF3. The irreducible error can be driven to zero with a good enough model.
False — is the noise in itself; it is a floor on error that no model, however clever, can cross.
TF4. A high-bias model and a high-variance model can have the same total error.
True — total error is a sum, so a stiff underfit ( large, Variance tiny) can tie a wobbly overfit (Variance large, tiny). See Overfitting and Underfitting.
TF5. Averaging many high-variance models (bagging) reduces bias.
False — averaging shrinks variance while leaving bias roughly unchanged; that is precisely why Ensemble Methods like bagging target the variance term.
TF6. If two models have identical average predictions over all datasets, they have identical bias.
True — bias depends only on ; if their average predictions match, their bias matches even if their variance differs wildly.
TF7. More training data lowers bias.
False — more data lowers variance (each point matters less); bias comes from the model class being too rigid and survives any sample size. See Mistake 1 in the parent.
TF8. Regularization can only help, never hurt.
False — regularization trades variance down for bias up; over-regularise and the added bias overwhelms the variance saving, raising total error. See Regularization Techniques.
TF9. The bias-variance decomposition requires the noise to be independent of and mean-zero.
True — the cross terms in the derivation vanish only because and is independent of the estimator; drop these and the clean three-way split breaks.
TF10. A model at the bottom of the U-shaped error curve has zero bias and zero variance.
False — the optimum balances the two; both are typically nonzero there, and total error still sits above .
Spot the error
SE1. "My deep net memorised the training set, so it has low variance."
The error: memorising the sample is the signature of high variance — swap datasets and the fit changes drastically. Low training error ≠ low variance.
SE2. "L2 regularization makes the model simpler, so it must reduce bias."
The error: 'simpler' here means less flexible, which increases bias by pulling parameters toward zero; the payoff is reduced variance, not reduced bias.
SE3. "With infinite data the tradeoff disappears entirely."
The error: infinite data sends variance → 0, but bias remains unless the model class contains the true . The tradeoff shifts (optimum gets more complex) but a wrong functional form is still biased.
SE4. "k=1 nearest neighbour underfits because it only looks at one point."
The error: k=1 overfits — it chases every point including outliers, giving low bias and huge variance. It's k=n (majority class everywhere) that underfits.
SE5. "The gap between training and validation error measures bias."
The error: a large train/val gap signals variance (overfitting). Bias shows up as both errors being high and close together. See Learning Curves.
SE6. "Increasing polynomial degree always lowers total error."
The error: raising degree lowers bias but raises variance; total error follows a U, so past the optimum, error rises again.
SE7. "Cross-validation removes variance from my model."
The error: Cross-Validation estimates generalization error and helps you pick complexity — it doesn't change the trained model's inherent variance.
SE8. "Since bias and variance both cause error, minimising one minimises total error."
The error: they move in opposite directions with complexity; pushing one down usually pushes the other up, so you must minimise their sum, not either alone.
Why questions
WHY1. Why does a more flexible model (higher Model Capacity) tend to have higher variance?
More parameters give more ways to bend the fit toward the specific noise in this dataset, so the fit changes a lot when the dataset changes.
WHY2. Why does the cross term in the decomposition vanish?
Because it factors into an expectation of a mean-zero quantity — and — so the product of a centred term with a constant averages to zero.
WHY3. Why is the total error curve U-shaped rather than monotone?
Bias falls and variance rises as complexity grows; a decreasing plus an increasing curve sum to a valley with a unique minimum.
WHY4. Why does bagging reduce variance but boosting can reduce bias?
Bagging averages independent-ish models (averaging cuts variance); boosting sequentially fits residuals, letting a weak, high-bias base learner build up flexibility. See Ensemble Methods.
WHY5. Why does a linear model on curved data keep high bias no matter how much data you collect?
Bias is about the best possible line still not being able to match a curve — the wrong functional form is a limitation of the model class, not the sample size.
WHY6. Why do we square the bias but not the variance in the decomposition?
Bias is a difference , and error is squared error, so its contribution is that difference squared; variance is already defined as an expected squared deviation.
WHY7. Why does PAC Learning Theory care about model capacity when studying generalization?
Higher capacity means more ways to fit noise (more variance / worse generalization gap), so capacity bounds how much training performance can be trusted to transfer.
WHY8. Why can adding a good feature reduce bias without necessarily blowing up variance?
If the feature captures real structure in , the model needs less contortion to fit — it lowers systematic miss without adding wild flexibility for noise to exploit.
Edge cases
EC1. What happens to variance as in k-NN?
It collapses to zero — every dataset predicts the single global majority class, identical everywhere, so predictions don't change across datasets (but bias is maximal).
EC2. What happens to bias for a degree-9 polynomial fit to 10 points?
Bias can be near zero (the average of many wild fits tracks the true curve), yet each individual fit is terrible — a pure high-variance regime.
EC3. What is the total error when your model class exactly contains the true and you have infinite data?
It equals , the irreducible error — bias → 0 (right class) and variance → 0 (infinite data), leaving only noise.
EC4. What does the error look like for a constant (degree-0) model as the dataset changes?
Its prediction barely moves (nearly zero variance) but sits far from a wiggly truth (high bias) — the extreme stiff-model corner.
EC5. If (noiseless data), can a sufficiently flexible model reach zero total error?
Yes in principle — with no noise, a model rich enough to contain and enough data drives both bias and variance to zero, and there's no irreducible floor.
EC6. What happens to the optimal complexity as you add more training data?
It shifts rightward — extra data suppresses variance, so you can 'afford' a more complex, lower-bias model at the new optimum. See Mistake 1 in the parent.
EC7. At the exact bottom of the U-curve, which term dominates the reducible error?
Neither dominates by design — the optimum is where the marginal decrease in bias² equals the marginal increase in variance, so they're balanced at the margin.
EC8. What is the variance of a model that ignores the input entirely (predicts a fixed constant chosen before seeing data)?
Zero — its prediction never depends on the training set, so it cannot vary across datasets (bias, however, is whatever that fixed constant's miss happens to be).
Recall Fast self-check
Which term does more data attack? ::: Variance (not bias). Which term does the wrong model class create? ::: Bias. Which term is a hard floor no model beats? ::: Irreducible error . A big train/val gap signals which term? ::: Variance (overfitting).