Bias-variance trade-off: Worked examples — Coding deep dive

This is a hands-on companion to the parent topic. There we derived why total error splits into three pieces. Here we drill every possible situation the trade-off can throw at you — one small model at a time — so that no exam question or real flight-data problem catches you off guard.

Before anything: three words we will use on every line.

The whole page rests on one equation, the bias-variance decomposition. It says the expected squared error at a point splits into exactly three non-negative pieces:

Look at the four dartboards above. Each board is one row of the equation "turned on". Read them left to right:

Board 1 — low bias, low variance: the cyan darts cluster tightly on the amber bullseye. Both $Bias^{2}$ and $Variance$ are small — the goal we chase.
Board 2 — low bias, high variance: the darts average to the centre (small bias) but scatter wildly (large variance). This is overfitting.
Board 3 — high bias, low variance: the darts cluster tightly (small variance) but the whole cluster sits off-centre (large bias). This is underfitting.
Board 4 — high bias, high variance: off-centre and scattered — the worst of both.

Keep this map in mind: every example below lands the darts in one of these four patterns and names which term of the decomposition is to blame. Board 4 (both errors large) is the trap in Example 8.

The scenario matrix

Every bias-variance problem is one of these cells. Our job in this page is to work an example for each row so you have seen it before it appears on a test.

Cell	What is being varied	Bias	Variance	Real-world flavour
A. Too simple	model far below true complexity	high	low	linear fit to a curve
B. Just right	model capacity ≈ true complexity	low	low–moderate	quadratic fit to quadratic drag
C. Too flexible	model far above true complexity	low	high	degree-10 fit to 15 points
D. Degenerate: 1 point	not enough data to learn	—	undefined/huge	fitting a line to a single measurement
E. Limiting: infinite data	$N \to \infty$	unchanged	$\to 0$	variance vanishes, bias remains
F. Zero-noise data	$σ^{2} = 0$	can hit 0 error	still can overfit	perfect simulation output
G. Discrete knob (k-NN)	neighbours $k$ small→large	low→high	high→low	satellite pixel classifier
H. Word problem	choose a model under a data budget	pick from curve	—	fuel-burn predictor for a new route
I. Exam twist	"add data or add regularization?"	reasoning	reasoning	which lever moves which error
J. Worst corner	wrong shape AND too flexible	high	high	wrong features + tiny dataset

We will cover A, B, C in Example 1, D in Example 2, E in Example 3, F in Example 4, G in Example 5, H in Example 6, I in Example 7, and J (dartboard 4) in Example 8. Every numeric answer is machine-checked in #Verification at the bottom.

Worked examples

Forecast (guess first!): Which degree do you think minimises total error? Write it down before reading on.

Step 1 — Read the picture. The amber curve is the true quadratic. The blue straight line (degree 1) cannot bend, so it sits below the truth in the middle and above at the ends — a systematic miss. Why this step? Bias is exactly "systematic miss", so seeing the line never touch the curve's belly tells us bias is high before any arithmetic.

Step 2 — Score degree 1 (cell A). A line has 2 knobs ( $a, b$ ) but the truth needs 3 (it has an $α^{2}$ term). No data can supply the missing curvature. So bias is high; but two different random samples both give nearly the same best-fit line, so variance is low. Simulated numbers: $Bias^{2} \approx 0.04$ , $Variance \approx 0.001$ . Why this step? We separate "can the model shape match?" (bias) from "does the model wobble between samples?" (variance).

Step 3 — Score degree 2 (cell B). Now the model has exactly the 3 knobs the truth needs, and we have 15 points feeding 3 knobs — a healthy 5-to-1 ratio. Bias collapses; variance stays modest. Simulated: $Bias^{2} \approx 0.001$ , $Variance \approx 0.005$ . Why this step? Matching model capacity to the true form is what drives bias to near-zero without paying much variance — the definition of the U-curve's floor.

Step 4 — Score degree 10 (cell C). The degree-10 polynomial contains the true quadratic as a special case (set the coefficients of $α^{3}, \dots, α^{10}$ to zero), so its true bias is exactly 0 — it can represent the truth. The tiny $Bias^{2} \approx 0.0001$ we quote is not a modelling limitation; it is leftover estimation residue: with only 15 noisy points the averaged fit $E [\hat{f}]$ doesn't perfectly recover those zero high-order coefficients, so a whisker of apparent bias survives the simulation. What dominates is variance: the wiggles differ violently sample-to-sample. Simulated: $Bias^{2} \approx 0.0001$ , $Variance \approx 0.08$ . Why this step? We distinguish representational bias (can the shape match? — here, yes, so ~0) from tiny finite-sample estimation residue, and confirm the real cost of over-flexibility shows up entirely in the variance term.

Step 5 — Add them up (ignore the common $σ^{2}$ , it's added to all three equally):

$Total_{1} = 0.041, Total_{2} = 0.006, Total_{10} = 0.0801$ Why this step? The decomposition lets us just sum the two model-dependent terms and compare, since $σ^{2}$ is identical across models.

Verify: Degree 2 gives the smallest total ( $0.006 < 0.041 < 0.0801$ ) — the U-curve bottoms out at "just right", matching the parent note's claim. Units: all quantities are in (drag-coefficient) $^{2}$ , dimensionless, so the comparison is valid. ✓

Forecast: Can you even find a unique line through one point?

Step 1 — Count knobs vs. equations. A line has 2 unknowns $a, b$ . One point gives 1 equation. Two unknowns, one equation ⇒ infinitely many solutions. Why this step? Whether a fit is well-posed is just "equations ≥ unknowns". Here $1 < 2$ , so the problem is underdetermined.

Step 2 — What does variance become? Variance asks "how much does the fitted line change across resampled datasets?" With one point, any tiny noise wiggle sends the chosen line spinning arbitrarily (the solver picks some line among infinitely many). Variance is effectively unbounded / undefined unless we pin down a tie-break rule. Why this step? An underdetermined solver has a whole space of equally-valid answers, so an infinitesimal change in data can jump it to a totally different line — that is exactly what unbounded variance means.

Step 3 — What does bias become? With the raw least-squares problem there is no stable "average model" $E [\hat{f}]$ to compare against the truth, so bias is undefined too. But the standard conventions do pin it down: (i) the minimum-norm solution (what numpy.linalg.lstsq and ridge-at-zero return) picks the single line closest to the origin in coefficient space; (ii) any regularization term $λ ∥(a, b) ∥^{2}$ with $λ > 0$ makes the solution unique. Under either convention $E [\hat{f}]$ is well-defined — and it will be heavily biased, because pulling coefficients toward zero drags the line away from the true quadratic. Why this step? "Undefined" is only true for the naive solver; naming the minimum-norm / ridge tie-break shows how real libraries recover a definite (and biased) answer, so the reader isn't left with a false impossibility.

Step 4 — The fix. Reduce model capacity below the data. With one point, only a degree-0 model (a constant $\hat{f} = 0.9$ ) is well-posed without any tie-break: 1 knob, 1 equation. It has huge bias (a flat line for a curve) but finite variance.

Verify: knobs of degree-0 model $= 1 =$ number of data points $= 1$ , so the system is exactly determined and solvable. ✓ The lesson (cell D): never let parameter count exceed data count — the parent's "11 parameters, 15 points" was already flirting with this edge.

Forecast: Does more data cure high bias, high variance, both, or neither?

Step 0 — State the assumptions (they earn the law). We assume the noise $ϵ$ is iid (independent, identically distributed) with mean 0 and constant variance $σ^{2}$ , and the inputs $α$ come from a fixed design distribution (same range each time). Under these standard ordinary-least-squares (OLS) assumptions the estimator's variance at a point is $σ^{2} x^{⊤} (X^{⊤} X)^{- 1} x$ , and $X^{⊤} X$ grows proportionally to $N$ , so its inverse shrinks like $1/ N$ . Why this step? A " $1/ N$ law" is not free — it holds exactly only under iid noise and a fixed design; naming those assumptions is what licenses the clean scaling below.

Step 1 — Variance shrinks like $1/ N$ . From Step 0, $Variance \propto \frac{1}{N}$ . Doubling data roughly halves variance. Why this step? $X^{⊤} X \propto N$ means each extra point adds equally to the "information", and inverting linearly growing information gives the $1/ N$ decay — the law of large numbers made precise for OLS.

Step 2 — Numeric trace. Take the degree-10 variance $0.08$ at $N = 15$ and scale it: at $N = 150$ it is $0.08 \times \frac{15}{150} = 0.008$ ; at $N = 1500$ , $0.0008$ . Why this step? Plugging concrete $N$ values into the $1/ N$ law turns an abstract rate into numbers you can check, showing variance falling 10× per 10× data.

Step 3 — Bias does not move. Bias is a property of the model shape, not the sample size. A degree-10 model can represent a quadratic exactly, so its bias was already ~0 and stays ~0. But a linear model's high bias would stay high forever, no matter how much data.

Step 4 — Conclusion. More data is a variance cure, not a bias cure. As $N \to \infty$ , the degree-10 model's variance $\to 0$ and it becomes as good as the "just right" model. This is why big-data teams can afford flexible models.

Verify: variance at $N = 1500$ is $0.08 \times 15/1500 = 0.0008$ , a 100× drop for a 100× data increase — consistent with the $1/ N$ law. ✓

Forecast: With perfectly clean data, is overfitting impossible?

Step 1 — Recall the decomposition. $E [(y - \hat{f})^{2}] = Bias^{2} + Variance + σ^{2} .$ Set $σ^{2} = 0$ . The floor of achievable error is now zero — a perfect model could nail every point. Why this step? We are checking which of the three terms actually vanishes. Only the third one does.

Step 2 — Variance can still be nonzero. If your 15 simulator points come from different regions each run (say different $α$ grids), the degree-10 fit still wiggles between runs. So variance need not be 0 even when $σ^{2} = 0$ . Why this step? Variance measures sensitivity to which points you sampled, not to measurement noise — so changing the sampling grid alone can keep variance alive even with perfectly clean readings.

Step 3 — Overfitting is a variance issue, not a bias issue. Overfitting means the model varies wildly between samples — that is high variance. With clean data there is no measurement noise to "chase", so the usual noise-driven variance largely disappears; what remains is only the milder sampling-grid variance of Step 2. If instead the grid is fixed and clean, the degree-10 model interpolates exactly and generalises well between points. Why this step? This nails the vocabulary: overfitting lives in the variance term. Removing noise ( $σ^{2} = 0$ ) starves the variance of its main fuel, which is why clean data is so forgiving of flexible models.

Step 4 — The takeaway. So "high degree = bad" is really "high degree + noise = bad". Zero-noise simulator data is the one place aerospace engineers can safely use very flexible models.

Verify: total error floor $= Bias^{2} + Variance + 0$ . For a degree-10 model on clean fixed-grid data, both remaining terms $\to 0$ , so total error $\to 0$ — achievable only because $σ^{2} = 0$ . ✓

Forecast: Which $k$ is high-variance — small or large?

Step 1 — Read the boundaries. At $k = 1$ (left) the decision boundary is jagged — each training pixel owns a little cell, so one stray green pixel makes a green island. At $k = 100$ (right) the boundary is nearly straight — 100 votes drown out any single pixel. Why this step? In k-NN, $k$ is the complexity knob: small $k$ = flexible = complex; large $k$ = smooth = simple.

Step 2 — Assign bias/variance.

$k = 1$ : bias low (can trace any shape), variance high (jagged, resample-sensitive). Cell C behaviour.
$k = 100$ : bias high (over-smoothed, misses sharp river/forest edges), variance low. Cell A behaviour.
$k = 10$ : middle — smooth enough to ignore lone noisy pixels, sharp enough to follow curved coastlines. Cell B. Why this step? Reading each board straight off the decomposition tells us which error term dominates at each $k$ before we commit to a choice.

Step 3 — The direction rule. As $k$ increases, variance decreases and bias increases — the opposite direction from polynomial degree. Small $k$ ↔ high degree; large $k$ ↔ low degree. Why this step? Every algorithm has a complexity knob, but it points in a different direction; knowing which end is "flexible" stops you from turning the dial the wrong way and worsening the very error you meant to cut.

Step 4 — Pick. For moderate noise, $k = 10$ balances both. It sits at the bottom of the U-curve: large enough to average away lone noisy pixels (taming variance) yet small enough to hug curved coastlines (keeping bias low). Why this step? The winner is always the knob setting that minimises the sum of the two terms, and $k = 10$ is where neither term is large — the U-curve floor for this problem.

Verify: monotonic check — variance ordering $V (k = 1) > V (k = 10) > V (k = 100)$ and bias ordering $B (k = 1) < B (k = 10) < B (k = 100)$ . The two orderings run opposite ways, confirming the trade-off. ✓

Forecast: More layers = more accuracy... right?

Step 1 — Identify the real risk. The test data (route 4) is distributionally new. The danger is a model that memorised the 3 training routes — that is high variance / overfitting. Why this step? Choose the lever that fights the dominant error. Here generalisation to unseen conditions matters more than squeezing the last drop of training fit.

Step 2 — Rule out (c). A 12-layer net has thousands of knobs; 200 samples cannot pin them down (echo of cell D: knobs ≫ data). Variance would be enormous on an unseen route. Why this step? When knobs far exceed data the model is underdetermined, exactly the high-variance regime we must avoid.

Step 3 — Rule out (a) if physics is nonlinear. Fuel burn depends nonlinearly on weight, altitude, and headwind, so a pure line has high bias — it will systematically miss. Why this step? A model whose shape cannot match the physics carries irreducible bias no data can fix — the opposite failure from (c).

Step 4 — Choose (b). A modest degree-3 model captures the main curvature (low-ish bias) and regularization (see regularization) shrinks the coefficients to tame variance. It sits nearest the U-curve's minimum given the data budget. Why this step? (b) is the only option with bias and variance both under control for a 200-sample budget — the middle of the U-curve.

Step 5 — Validate the choice properly. Do not trust training error. Use cross-validation holding out an entire route each fold, so the score mimics "predict a new route". Why this step? Random-point cross-validation would leak similar flights across folds; holding out a whole route makes the validation match the real deployment task.

Verify: knobs-to-data ratio for (b) ≈ 4 knobs / 200 points = 0.02 (very safe, ≪ 1), while (c) with even 2000 knobs gives 2000/200 = 10 (≫ 1, unsafe). The safe model is (b). ✓

Forecast: High training error — is that a bias problem or a variance problem?

Step 1 — Diagnose from the two error numbers.

Training error high AND test error high ⇒ the model can't even fit what it has seen ⇒ high bias (underfitting).
Training error low but test error high ⇒ it memorised the training set ⇒ high variance (overfitting). Here both are high ⇒ this is a bias problem. Why this step? The gap between train and test error is the tell-tale: small gap + high error = bias; large gap = variance.

Step 2 — Answer question 1: collect more data? No. Example 3 proved data drives variance toward 0 but leaves bias — a model-shape property — untouched. A too-simple model stays wrong on both train and test no matter how many rows you add. Why this step? We match the lever to the term: data only moves the variance term, so it cannot help a pure bias problem.

Step 3 — Answer question 2: add capacity / features? Yes. Increasing model complexity (higher degree, more features, more layers) directly lowers bias, which is the term that is large here. This is the correct lever. Why this step? Bias is high exactly because the model shape is too poor to match the truth; adding capacity enriches that shape.

Step 4 — Answer question 3: stronger regularization? No — it makes things worse. Regularization raises bias to cut variance; here variance is already fine and bias is the culprit, so we would be pushing the wrong term up. Why this step? Regularization trades variance for bias; applying it to a bias-limited model moves us the wrong way on the U-curve.

Step 5 — State the general map.

Symptom	Diagnosis	Fix	Do NOT
high train + high test	high bias	add capacity, add features, reduce regularization	add data, regularize
low train + high test	high variance	more data, regularize, simplify	add more capacity

Verify: consistency with Example 3 (data ↓ variance only) and Example 5 (increasing $k$ ↑ bias): the "more data" fix appears only under high variance, and "add capacity" only under high bias. No contradictions. ✓

Forecast: Can a model be both systematically wrong and wildly unstable at once?

Step 1 — Spot the bias source. The chosen feature (colour) is nearly unrelated to thrust, so no setting of the coefficients can trace the true thrust curve. The averaged prediction $E [\hat{f}]$ sits far from the truth ⇒ high bias. Why this step? Bias comes from the model being unable to represent the truth; here the failure is upstream in the features, not the degree — a cause students often miss.

Step 2 — Spot the variance source. Degree 9 gives 10 knobs for 11 points — a knobs-to-data ratio of $10/11 \approx 0.91$ , right up against the cell-D edge. The fit chases noise, so it swings violently between samples ⇒ high variance. Why this step? Variance comes from too much flexibility relative to data; the near-1 ratio guarantees the model wiggles wildly, independent of the feature problem.

Step 3 — Combine. Both terms are large simultaneously and for independent reasons (bad features → bias; too many knobs → variance). This is dartboard 4: darts scattered and off-centre. Why this step? It shows the four dartboards are not mutually exclusive labels — a single model can trigger both error sources at once, which is the corner our matrix (cell J) must not leave uncovered.

Step 4 — The fix needs two moves. Cutting only the degree (variance) still leaves the bad-feature bias; adding good features (bias) but keeping degree 9 still leaves the variance. You must fix the features AND lower the degree to escape. Why this step? Because the two errors have independent causes, one lever cannot fix both — the practical lesson of the worst corner.

Verify: knobs-to-data ratio $= 10/11 \approx 0.909 > 0.5$ (deep in the high-variance zone) and feature–target relation ≈ 0 (high bias), so both error terms are large together. ✓

Recall

Recall Which cell has small

$k$ in k-NN? Small $k$ ::: high variance / low bias (flexible, cell C) — same family as a high-degree polynomial.

Recall Does collecting more data reduce bias?

No ::: more data drives variance toward 0 (like $1/ N$ ) but leaves bias, a model-shape property, unchanged.

Recall Train error high and test error high — what is wrong, and what fixes it?

High bias / underfitting ::: fix by adding capacity or features; do NOT add data or regularize.

Recall Is overfitting a bias problem or a variance problem?

Variance ::: overfitting means the model jiggles wildly between samples, which is exactly the variance term of the decomposition.

Recall With zero data noise (

$σ^{2} = 0$ ), is overfitting still possible? Only via sampling variance ::: on a fixed clean grid a flexible model can interpolate perfectly, so noise-driven overfitting largely disappears.

Recall How can a model land in dartboard 4 (high bias AND high variance)?

Two independent faults ::: wrong/uninformative features cause the bias, while too many knobs for too little data cause the variance — you must fix both.

Verification

Every numeric claim on this page — the degree-1/2/10 totals (Ex 1), the one-point knob count (Ex 2), the $1/ N$ variance trace (Ex 3), the zero-noise floor (Ex 4), the k-NN orderings (Ex 5), the knobs-to-data ratios (Ex 6, 8), and the lever-to-error map (Ex 7) — is machine-checked in the ===VERIFY=== block that accompanies this note. Each check declares its symbols, recomputes the quantity with sympy, and asserts the stated result.

See also: Polynomial Regression, Training/Validation/Test splits, and Hyperparameter Tuning for choosing the knob automatically.