Bias-variance trade-off: Visual walkthrough — Coding deep dive

This page rebuilds the parent result — the error decomposition — from absolute zero, as a sequence of pictures. By the end you will see why every prediction error splits into exactly three pieces, and why you cannot squeeze all three to zero at once.

Before any symbol appears, three plain-word promises:

A target is the true number we wish we knew.
A guess is what our trained model outputs.
An average ( $E [\cdot]$ , read "expected value") means: repeat the whole experiment many times and take the mean of the thing in the brackets. That's all $E$ ever means on this page.

Step 1 — The target is a true curve plus a shiver of noise

WHAT. We say the real measured value is $y = true value at x f (x) + random shiver ϵ$

$x$ — the input we feed in (say, angle of attack).
$f (x)$ — the true, unknowable answer nature would give with no measurement error.
$ϵ$ (epsilon) — a tiny random wobble added every time we measure: sensor jitter, gusts, rounding.

WHY. No real instrument is perfect. If we pretend $y = f (x)$ exactly, we've already lied about reality. Splitting $y$ into a fixed part and a random part is the honest starting point.

PICTURE. The smooth curve is $f (x)$ . The scattered dots are actual measurements $y$ — each dot sits a little above or below the curve by $ϵ$ .

Step 2 — Our model is one guess drawn from a whole family of guesses

WHAT. We call our trained model $\hat{f} (x)$ (read "f-hat"). The hat means estimate — our attempt at $f (x)$ .

WHY. Here is the crucial idea most people miss: $\hat{f}$ depends on which training set we happened to get. Train on 15 wind-tunnel runs today, 15 different runs tomorrow, and you get a different curve each time. So $\hat{f} (x)$ is itself random — not because of measurement noise at prediction time, but because of which data raised it.

PICTURE. Many pale training-set draws each give a slightly different fitted curve (thin lines). Their average curve is the bold one — that average is written $E [\hat{f} (x)]$ .

$\hat{f} (x)$ — one thin curve (one training set).
$E [\hat{f} (x)]$ — the bold curve, the average model over all possible training sets. It is a fixed, non-random reference we'll lean on hard.

Step 3 — Write down the quantity we actually care about

WHAT. The expected squared error at a point $x$ : $E [(y - \hat{f} (x))^{2}]$

$(y - \hat{f} (x))$ — the gap between a real measurement and our guess.
squaring it — makes every gap positive and punishes big misses more than small ones.
$E [\cdot]$ around it — average that squared gap over both sources of randomness: the noise $ϵ$ and the training-set-driven wobble in $\hat{f}$ .

WHY squared, not absolute value? Squaring is the tool that lets the algebra split cleanly — cross-terms will vanish because of the zero-mean rules from Steps 1–2. Absolute value would not factor like this. That clean split is the whole payoff.

PICTURE. The vertical red bar is one raw gap $y - \hat{f} (x)$ ; we square it, then average over the whole dot-cloud.

Step 4 — Split off the noise using the "add-zero" trick, part one

WHAT. Substitute $y = f (x) + ϵ$ and expand:

= E\big[(f(x)-\hat{f}(x))^2\big] + 2\,E[\epsilon]\,E[f(x)-\hat{f}(x)] + E[\epsilon^2]$$ Term by term on the right: - $E\big[(f(x)-\hat{f}(x))^2\big]$ — how far our model sits from the *true curve* (noise removed). - $2\,E[\epsilon]\,E[f(x)-\hat{f}(x)]$ — a cross-term. Since $E[\epsilon]=0$ (Step 1), this **whole term dies**. - $E[\epsilon^2] = \sigma^2$ — the raw noise variance, survives untouched. **WHY.** We're peeling reality's own randomness ($\epsilon$) off the model's error. The noise is independent of the model, so its cross-term averages away, leaving a lonely $\sigma^2$. **PICTURE.** The total gap is a chain: model→true-curve (blue) plus true-curve→measured-dot (yellow, that's $\epsilon$). The blue and yellow pieces don't correlate, so their cross-term vanishes. ![[deepdives/dd-coding-5.6.04-d2-s04.png]] So far: $$E\big[(y-\hat{f}(x))^2\big] = \underbrace{E\big[(f(x)-\hat{f}(x))^2\big]}_{\text{still to split}} + \underbrace{\sigma^2}_{\text{irreducible}}$$ > [!definition] Irreducible error > ==$\sigma^2$== is the **irreducible error** — noise baked into the data. No model, however clever, can remove it. It sets a floor on how small the error can ever get. --- ## Step 5 — Add zero again, this time around the *average model* **WHAT.** Take the leftover term $E\big[(f(x)-\hat{f}(x))^2\big]$ and insert $+E[\hat{f}(x)]-E[\hat{f}(x)]$ (a fancy zero): $$f(x)-\hat{f}(x) \;=\; \underbrace{\big(f(x)-E[\hat{f}(x)]\big)}_{\text{fixed offset}} + \underbrace{\big(E[\hat{f}(x)]-\hat{f}(x)\big)}_{\text{random wobble}}$$ - $f(x)-E[\hat{f}(x)]$ — distance from the *true curve* to the *average model*. This is a **constant** — no randomness left in it. - $E[\hat{f}(x)]-\hat{f}(x)$ — how far one particular model strays from the average model. This is the **random wobble**, and it has mean zero *by construction*. **WHY.** We're separating two totally different failures: (a) the average model being systematically off, and (b) individual models scattering around that average. These are the two remaining villains. **PICTURE.** From the true point $f(x)$: a fixed blue arrow to the bold average model, then a pink zig-zag from the average to each individual thin model. ![[deepdives/dd-coding-5.6.04-d2-s05.png]] --- ## Step 6 — Square it and watch the middle term die **WHAT.** Squaring the sum from Step 5 and taking $E[\cdot]$: $$\big(f(x)-E[\hat{f}(x)]\big)^2 + 2\big(f(x)-E[\hat{f}(x)]\big)\,\underbrace{E\big[E[\hat{f}(x)]-\hat{f}(x)\big]}_{=\,0} + E\big[(E[\hat{f}(x)]-\hat{f}(x))^2\big]$$ - **First term** — constant squared, passes straight through the average unchanged. - **Middle term** — the wobble's average is $E[\hat f]-E[\hat f]=0$, so this cross-term **vanishes** (the second clean kill, same trick as Step 4). - **Third term** — the average squared wobble, survives. **WHY.** Same zero-mean magic. The constant offset and the zero-mean wobble refuse to correlate, so no cross-term contaminates the sum. **PICTURE.** The three pieces laid side by side: a fixed blue square, a pink cloud, and the vanishing cross-term crossed out. ![[deepdives/dd-coding-5.6.04-d2-s06.png]] --- ## Step 7 — Name the two survivors **WHAT.** The two surviving pieces already *are* the definitions from the parent note: $$\text{Bias}^2 = \big(f(x)-E[\hat{f}(x)]\big)^2 \qquad\qquad \text{Variance} = E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big]$$ - ==Bias== — how far the **average** model misses the truth. A *systematic* aim error. Underfitting. - ==Variance== — how much a **single** model jiggles around its own average. A *stability* error. Overfitting. **WHY.** Bias asks "is my dartboard aim centered?" Variance asks "are my darts tightly grouped?" They are genuinely different questions, which is why they earned separate names. **PICTURE.** The dartboard picture: high-bias/low-variance = tight group off-center; low-bias/high-variance = centered but scattered. ![[deepdives/dd-coding-5.6.04-d2-s07.png]] > [!formula] The assembled result > $$\boxed{\,E\big[(y-\hat{f}(x))^2\big] = \text{Bias}^2 + \text{Variance} + \sigma^2\,}$$ > Three villains, no leftovers. This matches the parent's decomposition exactly. ∎ --- ## Step 8 — The degenerate cases (never leave a gap) The formula must survive its extreme corners. Check all of them: > [!example] Corner cases > - **Perfect model, $\hat f = f$ always.** Then $E[\hat f]=f$, so $\text{Bias}=0$ and $\text{Variance}=0$. Error $=\sigma^2$ — the noise floor. You **cannot** beat this. This is why $\sigma^2$ is called irreducible. > - **Constant model, $\hat f(x)=c$** (predicts the same number no matter what). Variance $=0$ (it never wobbles — same output every training set), but Bias is huge wherever $f(x)\neq c$. Pure underfitting. > - **Interpolating model** (degree-10 polynomial through 15 points, or $k{=}1$ nearest neighbour). Bias $\approx 0$ (flexible enough to hit any shape), but Variance explodes — retrain on new dots and the curve thrashes. Pure overfitting. > - **Zero noise, $\sigma^2=0$.** The irreducible term disappears; error $=\text{Bias}^2+\text{Variance}$ only. Even here you still can't zero both at once with finite data. **WHY these matter.** Every real model lives somewhere between the constant and the interpolator. See [[5.6.03-Overfitting-and-Regularization]] for how regularization slides you back from the interpolator corner, and [[5.6.05-Cross-Validation-Techniques]] for how we *measure* the variance corner without ever seeing test data. **PICTURE.** The U-curve: complexity on the x-axis, error on the y-axis. Bias falls, variance rises, their sum dips in the middle — the sweet spot. The three corner models are pinned onto it. ![[deepdives/dd-coding-5.6.04-d2-s08.png]] > [!mistake] The trap > "More complex = always better." No. Past the U-curve's bottom, every gain in bias is *more than paid for* by a loss in variance. The parent's degree-10 drag example lands here: $\text{Bias}^2\approx0.0001$ but $\text{Variance}\approx0.08$ — the variance dwarfs everything. Compare [[3.4.07-Polynomial-Regression]] and pick degree by validation, not by ambition — [[5.6.08-Hyperparameter-Tuning]]. --- ## The one-picture summary Everything on one board: the true curve, the noisy dots, the family of models scattered around their average, and the three arrows — **Bias** (average-model to truth), **Variance** (one model to average-model), **Noise** (truth to dot) — snapping together into the boxed sum. ![[deepdives/dd-coding-5.6.04-d2-s09.png]] > [!recall]- Feynman retelling — say it to a 12-year-old > Imagine throwing darts. The bullseye is the *true answer*. But the board itself trembles a little in the wind — that trembling is **noise** ($\sigma^2$), and no thrower can fix it. Now, every time you retrain your model you throw one dart. Do it many times: where's the *center* of your dart cluster? If that center is off the bullseye, that's **bias** — your aim is systematically wrong (too-simple model). And how *spread out* is the cluster? That spread is **variance** — an over-eager model that lands somewhere new every time it's retrained. Your total wrongness is exactly aim-error-squared **plus** spread **plus** the board's tremble. The whole derivation is just algebra proving those three add up with no cross-talk, because the noise and the wobble both average to zero and refuse to mix. The punchline: tighten your grip (simpler model) and the cluster shrinks but drifts off-center; loosen it (complex model) and you re-center but scatter. The sweet spot is the bottom of the U. > [!recall]- Check yourself > Why does the cross-term in Step 4 vanish? ::: Because $E[\epsilon]=0$ — the noise is zero-mean, so $2E[\epsilon]E[f-\hat f]=0$. > Why does the cross-term in Step 6 vanish? ::: Because $E[E[\hat f]-\hat f]=E[\hat f]-E[\hat f]=0$ — the wobble around the average model is zero-mean by construction. > What is $\sigma^2$ and can a better model reduce it? ::: It is the irreducible noise variance in the data itself; no model can reduce it — it is the error floor. > A model that outputs the same constant for every input has which error high and which zero? ::: High bias, zero variance. > Bias measures the miss of the *average* model or a *single* model? ::: The average model, $E[\hat f(x)]$.