Foundations — Bias-variance trade-off

This page assumes you have seen none of the notation in Bias-Variance Trade-off. We build every symbol from the ground up, in the order the ideas depend on each other. Read top to bottom; nothing appears before it is earned.

1. The dataset: dots on a page

Before any formula, picture the raw material of machine learning: measurements. In aerospace we might measure, for one wing, its angle of attack (how tilted it is into the wind) and the drag it produces. One measurement is one dot.

Why the topic needs this: bias and variance are both measured as gaps between where a dot actually is and where our model thinks it should be. No dots, no gaps, no story.

2. The true function $f(x)$ — the law of nature

Nature does not scatter dots randomly. There is a hidden rule: "at this angle, the drag tends to be this much." That hidden rule is a smooth curve running through the middle of the cloud.

Why we need it: it is the target we are trying to hit. Bias is literally "how far our aim drifts from $f(x)$."

3. Noise $ϵ$ and irreducible error ${\sigma }^2$

Real measurements are never exactly on the true curve. A gust of wind, a sensor wobble — each nudges a dot off the line. That random nudge has a name.

Two facts about $ϵ$ we will lean on constantly:

• Its wobbles average out to nothing: on the picture, dots land above and below the curve equally often.
• Its typical size is captured by one number, ${\sigma }^2$.

4. The model $\hat{f}(x)$ — our best guess at the law

We don't know $f(x)$. We only see dots. So we build a guess: a curve of our own choosing (a straight line, a parabola, a wiggly polynomial). That guess gets a hat.

5. Expectation $E[\cdot ]$ — the "average over many tries"

Here is the subtlest symbol on the page, so we go slow. Imagine you don't train your model once — you train it a thousand times, each on a fresh batch of dots drawn from the same fuzzy band. Each time you get a slightly different curve $\hat{f}(x)$. Expectation asks: what happens on average across all those attempts?

Why the topic needs it: you cannot talk about "how much would predictions change" without imagining many datasets. Variance is that change; expectation is the tool that makes "many datasets" a precise idea.

6. Bias — the aim being off

Now assemble the payoff. Compare the average model curve $E[\hat{f}(x)]$ to the truth $f(x)$.

Bias is about being consistently wrong in the same direction — even the average of a million tries still misses. A too-simple model (a straight line for a curved truth) has high bias because it can't bend to match, no matter how many dots you feed it.

7. Variance — the aim being twitchy

Variance is about being inconsistent: retrain on different dots and you get a wildly different answer. A too-flexible model (a 10-wiggle polynomial through 15 dots) has high variance because it chases the noise, and the noise is different every time.

8. Squared error and MSE — the score we actually minimise

To rank models we need a single number saying how bad they are.

The headline result of the parent note is that this one number splits into three understandable pieces: $\underset{\text{total error}}{\underbrace{E[(y-\hat{f}(x){)}^2]}}=\underset{\text{aim off}}{\underbrace{{\text{Bias}}^2}}+\underset{\text{aim twitchy}}{\underbrace{\text{Variance}}}+\underset{\text{noise floor}}{\underbrace{{\sigma }^2}}$

Every symbol in that equation is now something you can see: an average of square-areas equals a steady-gap-squared plus a swarm-fatness plus a fuzzy-band-thickness. That is the entire topic in one line, and you built every letter of it.

9. Model complexity — the dial we turn

One last idea ties them together: the knob we control.

Prerequisite map

This foundation feeds directly into 5.6.03-Overfitting-and-Regularization (high variance is overfitting), 5.6.02-Training-Validation-Test-Sets (how we measure the swarm's error on unseen dots), and 5.6.05-Cross-Validation-Techniques (how we estimate variance in practice).

Equipment checklist

Test yourself: cover the right side and answer before revealing.