Bias-variance trade-off: Question bank — Coding deep dive

This is a self-test page. Each line hides a reveal after the :::. Read the prompt, commit to an answer out loud with a reason, then reveal. Bare "true"/"false" gains you nothing — the reasoning is the point. Every trap here targets a specific misconception that this topic reliably produces.

If any word feels unfamiliar, first re-read the parent: Bias-Variance Trade-off (or the Hinglish version). Related ideas live in 5.6.03-Overfitting-and-Regularization, 5.6.05-Cross-Validation-Techniques, 5.6.08-Hyperparameter-Tuning, and 3.4.07-Polynomial-Regression.

Notation recap — read this first

Before any trap, let us pin down every symbol so we never argue about what a letter means.

Recall The noise model behind

$σ^{2}$ (needed for the derivation traps) We assume $y = f (x) + ϵ$ , where $ϵ$ is random noise with $E [ϵ] = 0$ and $E [ϵ^{2}] = σ^{2}$ , and crucially $ϵ$ is independent of the training procedure — the noise on a new test point is not something our model $\hat{f}$ saw or could learn. That independence is what makes $E [ϵ \cdot (f (x) - \hat{f} (x))] = E [ϵ] E [f (x) - \hat{f} (x)] = 0$ , killing the cross-term. If noise were not independent (e.g. it leaked into features), that factorization would fail and the clean three-term split would break.

True or false — justify

True or false: A model with zero training error must have low bias.

False. Zero training error only means it fits the points it saw; that is a symptom of high variance (memorizing noise), not low bias. A degree-10 polynomial through 15 points has ~0 training error yet generalizes terribly.

True or false: If you collect more training data, high bias will eventually vanish.

False. Step through it: bias is

E [\hat{f} (x)] - f (x)

. As the sample size

N \to \infty

\hat{f}

converges to the best fit inside its own family — call it

\hat{f}_{\infty}

. If the family is "all straight lines" but

f (x)

is quadratic, then

\hat{f}_{\infty}

is still the best straight line, and

E [\hat{f}_{\infty}] - f (x) \neq = 0

. More data collapses variance to ~0 but leaves this assumption-driven bias frozen.

True or false: More training data reduces variance.

True. Variance is

E [(\hat{f} (x) - E [\hat{f} (x)])^{2}]

, the scatter of the estimate. Averaging over more samples makes any single noisy point matter less, so re-training on a fresh dataset gives more consistent predictions; for many estimators variance scales like

\sim σ^{2} / N

, vanishing as

N

grows.

True or false: The irreducible error

σ^{2}

can be driven to zero by a good enough model.

False.

σ^{2} = E [ϵ^{2}]

is noise in the data itself (sensor jitter, unmeasured variables). It appears in the total error regardless of $\hat{f}$ , so no model — however clever — can predict genuine randomness; it sets a hard floor.

True or false: A high-variance model always has low bias.

Usually but not guaranteed. Extra flexibility tends to lower bias, but a badly-designed flexible model (wrong features, bad optimization) can be both biased and unstable at once.

True or false: Regularization increases bias.

True, and that's the point. Regularization shrinks

\hat{f}

's parameters toward zero, pulling

E [\hat{f} (x)]

away from

f (x)

(more bias) but tightening the dart spread (less variance) — a good trade when overfitting.

True or false: The bias-variance decomposition

E [(y - \hat{f} (x))^{2}] = Bias^{2} + Variance + σ^{2}

holds for classification accuracy the same way it holds for squared error.

False. That clean three-term additive split is a special property of squared-error loss. Under 0/1 or cross-entropy loss the "decomposition" still separates a systematic part from a variability part, but the terms combine multiplicatively / non-additively (bias can even help variance flip predictions), so it is not a simple sum.

True or false: Two models with identical test error must have identical bias and variance.

False. Test error is

Bias^{2} + Variance + σ^{2}

; a high-bias/low-variance model and a low-bias/high-variance model can land on the same total from opposite directions.

True or false: On the U-shaped total-error curve, the minimum sits where bias equals variance.

False. The minimum sits where the slopes balance:

\frac{d ( Bias ^{2} )}{d c} + \frac{d ( Variance )}{d c} = 0

(complexity

c

), i.e. where falling bias and rising variance change at equal-and-opposite rates. That is generally not the point where the two values are equal — see the s02 figure.

Spot the error

"My model has 99% training accuracy and 99% test accuracy, so it's overfitting."

Error: overfitting shows as a gap — high training, low test. Matching high scores means it's generalizing well; nothing is wrong.

"I'll fix underfitting by adding L2 regularization."

Error: regularization increases bias to fight variance. For underfitting (already high bias) you want the opposite — a more flexible model or fewer constraints.

"Variance is the error, bias is just a bonus term we can ignore."

Error: both are genuine error contributions.

Bias^{2}

and Variance both add to expected test error; ignoring bias is why linear models on nonlinear data fail no matter what. Concretely: a flat line fit to

y = x^{2}

has near-zero variance yet huge error — all of it bias.

"For k-NN, larger k means more overfitting because k is a bigger number."

Error: it's the reverse. Look at the s03 figure: with k=1 the boundary is jagged and wraps around every noisy point (overfit, high variance); as k grows it averages more neighbors, smoothing toward a nearly flat, high-bias boundary. Big k = underfit.

"A degree-2 polynomial on truly quadratic data has zero bias, so its test error is zero."

Error: even with zero bias, Variance and the irreducible noise

σ^{2}

remain. Test error bottoms out near

σ^{2}

, not at zero.

"Cross-validation removes the bias-variance trade-off."

Error: cross-validation estimates generalization error more reliably (see 5.6.05-Cross-Validation-Techniques); it helps you locate the sweet spot but does not abolish the trade-off itself.

"The noise term

ϵ

has mean zero, so it never affects the total error."

Error:

E [ϵ] = 0

kills the cross term in the derivation, but

E [ϵ^{2}] = σ^{2}

survives and lands directly in total error.

"Deeper neural network = lower bias always, so deeper is always better for flight-control prediction."

Error: added depth lowers bias but raises variance and can memorize sensor noise. Beyond the sweet spot, test error climbs — the U-curve applies to depth too.

"Squared error is the only loss where bias and variance matter."

Error: they matter under any loss. Squared error is just the one where they split additively. For cross-entropy / logistic loss you still have a systematic term (your average predicted probability is off) and a variability term (predictions swing with the training set); the algebra is different but the shaky-vs-mis-aimed intuition is identical.

Why questions

Why does the middle cross-term

2 E [ϵ (f (x) - \hat{f} (x))]

vanish in the decomposition?

Because the test-point noise

ϵ

is independent of the training procedure that produced

\hat{f} (x)

, the expectation of the product factors:

E [ϵ (f - \hat{f})] = E [ϵ] E [f - \hat{f}]

. Then

E [ϵ] = 0

zeroes it, leaving only

Bias^{2}

, Variance, and

σ^{2}

. Without that independence the product would not factor and the term could survive.

Why can't we minimize bias and variance simultaneously by tuning complexity?

Because complexity moves them in opposite directions along one dial: more flexibility lowers bias but raises variance. The best you can do is minimize their sum, not each alone.

Why does a 50-degree polynomial on 20 points give wildly different curves each retrain?

With more parameters

p

than data points

N

(

p > N

), the model has slack to "chase" each dataset's specific noise; different noise → drastically different fits → high variance. This is the

N < p

regime discussed in the edge cases below.

Why is k=1 nearest-neighbor guaranteed 0% training error yet often terrible on new data?

Every training point is its own nearest neighbor, so it reproduces its own label perfectly — but that means it also faithfully reproduces every noisy label, giving jagged, unstable boundaries (see s03) that fail on fresh pixels.

Why do we care so much about this trade-off specifically in aerospace?

Aerospace data (flight tests, satellite passes) is expensive and the operating conditions are unseen and safety-critical; a model that overfits one dataset can fail catastrophically on the next real mission.

Why does adding a good regularizer often lower total test error despite raising bias?

The bias it adds is tiny, but the variance it removes is large; since total error sums

Bias^{2} + Variance

, a small increase plus a big decrease nets a lower total.

Why does the true function

f (x)

never appear in the Variance term?

Variance measures how much

\hat{f} (x)

scatters around its own average

E [\hat{f} (x)]

; the distance from that average to the truth is captured separately by bias, keeping the two concepts cleanly split.

Edge cases

What is the bias of a model that always predicts the exact true mean of

f

but has huge scatter around it?

Zero bias — its average prediction hits the truth, so

E [\hat{f} (x)] - f (x) = 0

— but large variance from the scatter, so its error is dominated entirely by the variance term.

What happens to bias and variance in the limit of infinite model complexity with fixed data?

Bias → near zero (it can represent anything), but variance blows up (it fits every noise wiggle); total error diverges upward — the right end of the U-curve in the s02 figure.

What happens at the opposite limit — a "model" that ignores

x

and predicts a fixed constant?

Variance is essentially zero (same output regardless of training set), but bias is maximal unless the truth happens to be that constant; this is the extreme-underfitting left end of the U-curve.

If the data has zero noise (

σ^{2} = 0

), can total test error still be nonzero?

Yes. Even with no irreducible noise, a mismatched model contributes

Bias^{2}

and finite-sample Variance, so error is only zero if the model also perfectly matches the truth and is fully determined.

When the number of data points

N

roughly equals the number of parameters

p

(

N \approx p

), what tends to happen to variance?

It typically spikes. With just enough equations to pin down every parameter, the fit is fully determined by the noise-corrupted points and has no averaging left to damp it — this is the notorious "interpolation threshold" where test error often peaks before (in some over-parameterized models) descending again.

What happens in the strict

N < p

regime (fewer data points than parameters)?

The training system is under-determined: infinitely many parameter settings fit the data exactly (training error 0), and which one you land on depends entirely on the optimizer and the noise — enormous variance, textbook overfitting.

How does variance scale with sample size

N

for a fixed model, roughly?

For a fixed-complexity estimator variance shrinks like

\sim σ^{2} / N

: quadruple the data, roughly halve the standard deviation of the prediction. Bias, being assumption-driven, does not follow this and stays put.

If you had access to infinitely many independent training sets and averaged all their models, which error term would you eliminate?

You'd wipe out Variance (averaging cancels the dataset-to-dataset scatter), leaving only

Bias^{2} + σ^{2}

— this is exactly why ensembling/bagging helps high-variance models.

For a perfectly correct model on noiseless data, what is the theoretical total error?

Exactly zero:

Bias^{2} = 0

, Variance

= 0

, and

σ^{2} = 0

. This ideal essentially never occurs in real aerospace data, which is why the trade-off always bites.

Recall One-line summary to lock in

Bias $= E [\hat{f} (x)] - f (x)$ = wrong on average (underfit); Variance $= E [(\hat{f} (x) - E [\hat{f} (x)])^{2}]$ = inconsistent across datasets (overfit); $σ^{2}$ = unbeatable noise floor. Total error is their sum, and complexity trades the first for the second.