Visual walkthrough — Bias-variance tradeoff

This page rebuilds the Bias-variance tradeoff central result — the splitting of prediction error into three pieces — from absolutely nothing. No prior formula is assumed. Every symbol is drawn before it is used. If a word like "expectation" or "square of an error" is new to you, you are in exactly the right place: we start from a dartboard and end at the boxed equation.

Step 1 — What "prediction error" even means

WHAT. We have some true relationship in the world. Feed it an input $x$ (say, "hours studied") and it returns a true output $f(x)$ (say, "true skill level"). But the data we actually observe is dirty: it comes with random wobble $ϵ$ (a lucky or unlucky exam). So the number we see is

$y=f(x)+ϵ$

• $y$ — the observed value (what a data point actually shows).
• $f(x)$ — the true, clean value (the signal, invisible to us).
• $ϵ$ — the noise, a random nudge up or down each time.

WHY. Before we can talk about a model's error we must be honest that even the data is not the truth. The model $\hat{f}$ (read "f-hat", the hat means estimate) is our guess of $f$. Its error at $x$ is how far its guess lands from the observed $y$, squared so up-misses and down-misses both count as positive:

$\text{error}=(y-\hat{f}(x){)}^2$

We square (rather than take absolute value) because squaring is smooth and — as we will see in Step 5 — makes cross-terms politely cancel. That is the specific reason this tool is chosen here.

PICTURE. Below: the smooth true curve $f$, a scattered observed point $y$ sitting off it by $ϵ$, and a model's guess $\hat{f}(x)$. The three vertical gaps are the three characters of our whole story.

Step 2 — Why we average over many training sets

WHAT. Your model $\hat{f}$ was trained on one dataset $\mathcal{D}$. But that dataset was random — a different random sample would have produced a different $\hat{f}$. So $\hat{f}(x)$ is itself a random quantity. To talk about it fairly we imagine training the same recipe on many independent datasets and averaging the results.

We write this averaging as $\mathbb{E}[\cdot ]$ — the expected value, meaning "the average over all the randomness."

WHY. A single fit tells you nothing about stability. Only by looking at the whole cloud of fits can we separate "wrong on average" from "jumpy." So we now define one crucial anchor point:

$\bar{f}(x)\equiv {\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$

• $\bar{f}(x)$ — the average prediction at $x$, over all possible training sets. This is the center of the prediction cloud.

PICTURE. Five wiggly fits from five different random datasets, drawn faintly; their pointwise average $\bar{f}$ drawn boldly through the middle; and the true $f$ for comparison.

Step 3 — The dartboard: naming the two model errors

WHAT. With $\bar{f}(x)$ (cloud center) and $f(x)$ (truth) in hand, two natural distances appear:

$\underset{\text{how far the center is from truth}}{\underbrace{\bar{f}(x)-f(x)}}\text{and}\underset{\text{how far one fit is from the center}}{\underbrace{\hat{f}(x)-\bar{f}(x)}}$

• The first is bias: a systematic miss that stays even after averaging.
• The second is spread: how a single fit deviates from its own average.

WHY. These are genuinely different failures. You can have a cloud perfectly centered on the bullseye yet spread all over (low bias, high variance), or a tight cluster parked in the wrong corner (high bias, low variance). No single number captures both, so we keep them apart.

PICTURE. A dartboard. The bullseye is $f(x)$. The cloud's centroid is $\bar{f}(x)$. The arrow from bullseye to centroid is bias; the fuzzy scatter around the centroid is variance.

Step 4 — The add-and-subtract trick, seen as vectors

WHAT. Inside the error $(y-\hat{f}(x){)}^2$ we now perform the key algebraic move: substitute $y=f+ϵ$ and insert $\bar{f}-\bar{f}$ (adding zero). This rewrites the single gap $y-\hat{f}$ as a sum of three named gaps:

$y-\hat{f}(x)=\underset{-\text{bias }B}{\underbrace{(f(x)-\bar{f}(x))}}+\underset{-\text{deviation }V}{\underbrace{(\bar{f}(x)-\hat{f}(x))}}+\underset{\text{noise}}{\underbrace{ϵ}}$

• $B=\bar{f}(x)-f(x)$ — the constant bias gap (fixed once we pick a model recipe).
• $V=\hat{f}(x)-\bar{f}(x)$ — the random deviation of this fit from the cloud center.
• $ϵ$ — the data noise, independent of everything about the model.

WHY. We chose these three particular pieces because each depends on a different source of randomness: $B$ depends on nothing random (it is a fixed constant), $V$ depends only on which training set we drew, and $ϵ$ depends only on the observation noise. Splitting by source is what will let the cross-terms die in Step 5.

PICTURE. Three arrows laid head-to-tail — a fixed black bias arrow, a blue random deviation arrow, a pink noise arrow — summing to the total error arrow.

Step 5 — Squaring, and watching the cross-terms vanish

WHAT. Square the three-part sum. Any square $(B+V+ϵ{)}^2$ opens into three "self" terms plus three "cross" terms:

$(B+V+ϵ{)}^2=B^2+V^2+{ϵ}^2+2BV+2Bϵ+2Vϵ$

Now take the expectation $\mathbb{E}$ of the whole thing. Each cross-term dies:

• $\mathbb{E}[2BV]=2B\mathbb{E}[V]=2B\cdot 0=0$, because $\mathbb{E}[V]=\mathbb{E}[\hat{f}-\bar{f}]=\bar{f}-\bar{f}=0$: the deviation is centred at zero by definition of $\bar{f}$.
• $\mathbb{E}[2Bϵ]=2B\mathbb{E}[ϵ]=0$, because noise averages to zero ($\mathbb{E}[ϵ]=0$).
• $\mathbb{E}[2Vϵ]=2\mathbb{E}[V]\mathbb{E}[ϵ]=0$, because model deviation and observation noise are independent, and each averages to zero.

WHY. This is the whole payoff of the add-and-subtract trick. By building three pieces that each centre on zero (or are constant), every mixed product averages away, leaving a clean sum of squares. That is why we squared instead of taking absolute values — squares expand into cross-terms that expectation can annihilate.

PICTURE. A 3×3 grid of the nine expanded terms; the three diagonal boxes (the squares) glow gold and survive; the six off-diagonal cross-terms are crossed out in pink with "$\mathbb{E}=0$."

Step 6 — The three survivors: the decomposition

WHAT. Only the diagonal survives:

$\mathbb{E}[(y-\hat{f}(x){)}^2]=\underset{{\text{Bias}}^2}{\underbrace{\mathbb{E}[B^2]}}+\underset{\text{Variance}}{\underbrace{\mathbb{E}[V^2]}}+\underset{{\sigma }^2}{\underbrace{\mathbb{E}[{ϵ}^2]}}$

Since $B=\bar{f}-f$ is constant, $\mathbb{E}[B^2]=B^2=(\bar{f}(x)-f(x){)}^2$. And $\mathbb{E}[{ϵ}^2]={\sigma }^2$ (noise variance). So:

• Bias² $=(\bar{f}-f{)}^2$ — squared distance from cloud-center to bullseye.
• Variance $=\mathbb{E}[(\hat{f}-\bar{f}{)}^2]$ — average squared scatter around the cloud center.
• ${\sigma }^2$ — the noise floor, present no matter how good the model is.

WHY. This is the object we set out to build. It says total expected error is literally a sum — you cannot reduce it below ${\sigma }^2$, and the two model pieces are the only knobs you control.

PICTURE. Three stacked bars whose total height is the fixed prediction error; as complexity grows, Bias² shrinks while Variance grows, keeping the sum in a U-shape.

Step 7 — The edge cases: reading the extremes off the picture

WHAT. Check the boxed formula at the two degenerate ends of complexity. These correspond to the models in the Overfitting and Underfitting discussion.

Case A — the constant model (maximum stiffness, e.g. $k=n$ in k-Nearest Neighbours or degree-0 polynomial). Same prediction on every dataset $\Rightarrow \hat{f}=\bar{f}$ always $\Rightarrow$ Variance $=0$. But $\bar{f}$ ignores all structure, so Bias² is large. Total error $\approx {\text{Bias}}^2+{\sigma }^2$.

Case B — the interpolating model (maximum flexibility, e.g. degree-9 on 10 points). It threads every training point, so its cloud-center $\bar{f}$ can sit near truth $\Rightarrow$ Bias² small. But one noisy point yanks the whole curve $\Rightarrow$ Variance huge. Total error $\approx \text{Variance}+{\sigma }^2$.

Case C — infinite data. As sample size grows, each point's influence shrinks $\Rightarrow$ Variance $\to 0$. But if the model class cannot represent $f$, Bias² stays put — more data never fixes a wrong shape. (This is exactly Mistake 1 from the parent note, and connects to Learning Curves.)

WHY. Reading the two survivors at the extremes shows why the tradeoff is unavoidable: killing variance forces bias up (Case A), killing bias forces variance up (Case B), and the noise floor ${\sigma }^2$ never moves.

PICTURE. Three mini-panels: (A) flat line + tight cloud; (B) wild interpolation + fat cloud; (C) shrinking cloud but a stubborn offset from truth.

The one-picture summary

Everything above compressed: complexity increases left→right; Bias² falls, Variance rises, ${\sigma }^2$ is a flat floor, and their sum is the U-curve whose valley is the sweet spot you tune with Cross-Validation and Regularization Techniques.