Bias-variance trade-off: Exercises — Coding deep dive

These exercises climb from recognising the words to engineering real aerospace models. Do each on paper first, then open the solution. Everything you need was built in the parent note Bias-Variance Trade-off; if a symbol feels unfamiliar, re-read the definition there before peeking.

Level 1 — Recognition

Can you name the pieces and read the picture?

Recall Solution

What the numbers say. Large training error means the model cannot even fit the data it was shown — its assumptions are too rigid. Test error being equally large (not much bigger) means the model is not especially sensitive to which data it saw.

Big error on training data → cannot capture the pattern → high bias.
Train ≈ test (both bad) → predictions barely change across datasets → low variance.

Answer: high bias → underfitting. On the U-curve of the figure above, you are on the left side (model too simple).

Recall Solution

$Bias^{2} \geq 0$ and $Variance \geq 0$ (both are squares / averages of squares — they can never be negative). So the smallest the total can ever be is $Total_{m i n} = 0 + 0 + σ^{2} = 0.01.$ The claimed $0.004 < 0.01$ would require the noise term to shrink, but $σ^{2}$ is irreducible — it lives in the data, not the model. No algorithm can beat it. The colleague made a measurement or bookkeeping error.

Level 2 — Application

Plug into the formula and get a number.

Recall Solution

The formula wants Bias squared, not bias. That squaring is the whole reason a signed systematic error becomes a positive contribution: $Bias^{2} = (- 0.03)^{2} = 0.0009.$ Then add the three pieces: $Total = 0.0009 + 0.0004 + 0.0001 = 0.0014.$ Answer: $0.0014$ . Notice bias² ( $0.0009$ ) dominates — this model's biggest problem is being too simple, not too jumpy.

Worked example Exercise 2.2 — Which degree wins?

Using the parent note's simulated table for the UAV drag model:

Degree	Bias²	Variance
1	0.04	0.001
2	0.001	0.005
10	0.0001	0.08

The noise is $σ^{2} = 0.0001$ . Compute total error for each degree and pick the winner.

Recall Solution

Add all three columns row by row:

Degree 1: $0.04 + 0.001 + 0.0001 = 0.0411$ .
Degree 2: $0.001 + 0.005 + 0.0001 = 0.0061$ .
Degree 10: $0.0001 + 0.08 + 0.0001 = 0.0802$ .

Winner: degree 2, total $= 0.0061$ . This is the bottom of the U: degree 1 is bias-dominated (left side), degree 10 is variance-dominated (right side). See 3.4.07-Polynomial-Regression for why higher degree buys flexibility at the cost of stability.

Level 3 — Analysis

Explain the mechanism, don't just compute.

Recall Solution

The knob $k$ = how many neighbours vote.

$k = 1$ (few voters): one training pixel decides its whole neighbourhood → boundary is jagged, wraps every point. Bias low (can trace any shape), variance high (one noisy pixel flips a whole region; a different image gives a wildly different map).
$k = 100$ (many voters): you average over a huge crowd → boundary is smooth, nearly straight. Bias high (cannot follow a sharp river/forest edge), variance low (averaging cancels noise, so different images give similar maps).

Rule: larger $k$ = simpler model = slide left→right becomes right→left on the U. Big $k$ ↑bias ↓variance; small $k$ ↓bias ↑variance.

Recall Solution

(a) More data. Variance measures how much predictions wobble across different training sets. With far more data, any single noisy point carries less weight, so re-sampling changes the fit less → variance falls. But if the model's form was already flexible enough to represent the truth, its bias is unchanged — more data doesn't change your assumptions, only how firmly they're pinned. Data attacks variance, not bias.

(b) Shrink the network. Fewer parameters = less flexibility = variance falls but bias rises (you may now be too simple). This is the trade-off in one move.

Best when data is available: (a), because it lowers variance with no bias penalty — a rare free lunch. When data is expensive (satellites!), you fall back to (b) or regularization (5.6.03-Overfitting-and-Regularization).

Level 4 — Synthesis

Combine ideas; design a procedure.

Recall Solution

(i) Both curves converge to a high plateau and sit close together. Close curves = low variance; high plateau = high error the model cannot escape = high bias / underfitting.

(ii) Useful:

Increase model capacity (deeper network, higher polynomial degree — 3.4.07-Polynomial-Regression).
Add richer features (e.g. interaction terms like airspeed×pitch-rate).

Useless here: collecting more data — the curves have already flattened and converged, so extra samples change nothing. That's the tell-tale sign of a bias-limited model.

Recall Solution

Recipe:

Split data → train / validation / test (5.6.02-Training-Validation-Test-Sets); lock the test set in a vault.
For each candidate degree $d = 1, 2, \dots, 10$ : use k-fold cross-validation (5.6.05-Cross-Validation-Techniques) on the train+validation data to estimate error. Cross-validation averages over multiple splits, which directly estimates the variance term by seeing how the fit changes across folds.
Plot the averaged CV error vs. degree — it should be U-shaped.
Pick the degree at the bottom of the U (here, degree 2). This is 5.6.08-Hyperparameter-Tuning in miniature: degree is a hyperparameter.
Only now touch the locked test set once, to report an honest final error.

Protecting tool: cross-validation — it lets you estimate generalisation error many times without burning the test set.

Level 5 — Mastery

Prove or derive; defend against edge cases.

Recall Solution

Start from the decomposition proved in the parent note: $E [(y - \hat{f} (x))^{2}] = \geq 0 Bias^{2} + \geq 0 Variance + σ^{2} .$

$Bias^{2} = (f (x) - E [\hat{f} (x)])^{2} \geq 0$ — it's a real number squared.
$Variance = E [(\hat{f} (x) - E [\hat{f} (x)])^{2}] \geq 0$ — an average of squares.

Adding two non-negative quantities to $σ^{2}$ can only increase it: $E [(y - \hat{f} (x))^{2}] \geq σ^{2} .$ Equality holds iff $Bias^{2} = 0$ and $Variance = 0$ simultaneously — i.e. the model equals the true $f (x)$ on average AND is perfectly stable across all training sets. This "perfect estimator" is essentially unreachable in practice, which is why $σ^{2}$ is called the irreducible error. ∎

Recall Solution

Assumption used: the noise $ϵ$ has mean zero and is independent of the training-based estimate $\hat{f} (x)$ (the noise on the new test point $y$ is not the same noise the model trained on).

By independence, expectation factorises: $E [ϵ (f (x) - \hat{f} (x))] = E [ϵ] \cdot E [f (x) - \hat{f} (x)] .$ Since $E [ϵ] = 0$ , the entire cross term is $2 \cdot 0 \cdot (\dots) = 0$ . ✔

If independence failed — e.g. if the test point's noise leaked into training (data leakage) — then $E [ϵ \hat{f} (x)] \neq = E [ϵ] E [\hat{f} (x)]$ , the cross term would survive, and the clean "Bias² + Variance + noise" split would collapse. The reported error would look artificially low. This is the mathematical shadow of the L4 test-set-leak trap.

Recall Solution

We minimise a sum of a falling term ( $σ^{2} / k$ , the variance) and a rising term ( $k^{2} / n^{2}$ , the bias²) — exactly the U-shape. Why calculus here? Because "bottom of the U" means the slope is zero; the derivative is the tool that locates a zero slope.

Differentiate with respect to $k$ and set to zero:

\;\Longrightarrow\; \frac{2k}{n^2}=\frac{\sigma^2}{k^2} \;\Longrightarrow\; k^3=\frac{\sigma^2 n^2}{2}.$$ $$\boxed{\,k^\star=\left(\frac{\sigma^2 n^2}{2}\right)^{1/3}\,}$$ **Sanity of the second derivative:** $T''=2\sigma^2/k^3+2/n^2>0$, so it's genuinely a minimum (the U opens upward). ✔ **Limits:** - $k\to 1$: variance term $\sigma^2/1$ blows up → **overfitting corner**. - $k\to n$: bias² term $(n/n)^2=1$ dominates, model averages *everything* → flat prediction → **underfitting corner**. The optimum sits between, and grows with more data ($k^\star \propto n^{2/3}$) — with more flights you can afford a larger, more stable neighbourhood.

Recall Self-test cloze

The total expected error splits into bias squared, variance, and irreducible error.

High bias with train ≈ test ::: underfitting; make the model more complex or add features. Low train error but high test error ::: overfitting (high variance); simplify, regularize, or add data. Tool that estimates generalisation error without touching the test set ::: cross-validation. The only error term no model can reduce ::: the irreducible noise $σ^{2}$ .