Foundations — Bias-variance tradeoff

Before you can read a single line of the parent note, you need to own every symbol it throws at you. This page builds each one from nothing — plain words, a picture, and the reason the topic can't live without it. Read top to bottom; nothing here uses a symbol before it is earned.

1. A function $f(x)$ — the hidden truth

Picture a conveyor belt. Every input $x$ that rides in comes out transformed into $f(x)$. In this topic, $f$ is the true, hidden rule that generated your data — the perfect answer nature is using but never shows you.

Look at the yellow curve above: that is $f(x)=\sin (2\pi x)$, the true rule used in the parent's Example 1. You never get to see this curve directly. All you ever receive are dots scattered near it — and even those dots are nudged off the curve by noise (next section).

2. Noise $ϵ$ and the data equation $y=f(x)+ϵ$

Why do we need it? Because real measurements are never clean. A sensor flickers, a survey answer is fuzzy, a coin lands unpredictably. So the value $y$ you actually record is the true value plus a random bump:

$y=f(x)+ϵ$

• $y$ (read "why") = the number you actually see in your dataset.
• $f(x)$ = where the truth says it should be.
• $ϵ$ = the random shove up or down.

The red dots are the $y$ values. Each red dot sits near the yellow truth curve but is knocked off by its own $ϵ$ — some up, some down.

3. Expectation $\mathbb{E}[\cdot ]$ — the "average over all possible worlds"

This is the most important and most confusing symbol in the whole topic. Slow down here.

Picture rolling a fair die. Any single roll is 1–6, unpredictable. But $\mathbb{E}[\text{roll}]=3.5$: the average you converge to over millions of rolls. You never see 3.5 on the die — it is the average of a random thing.

The little subscript tells you which random thing you are averaging over:

4. Training dataset $\mathcal{D}$ — one draw of dots

Picture reaching into a jar of infinitely many possible datasets and pulling out one handful of dots. That handful is $\mathcal{D}$. If you reach in again, the noise $ϵ$ on each point is different, so you pull out a different handful — a different $\mathcal{D}$.

Above are three different draws of $\mathcal{D}$ from the same truth. Same underlying yellow curve, different red dots each time. This "many possible datasets" idea is the engine behind ${\mathbb{E}}_{\mathcal{D}}$ and the whole notion of variance: your model would be trained on just one of these, but its behaviour is judged across all of them.

Why does the topic need $\mathcal{D}$? Because bias and variance are questions about what happens if you had gotten a different dataset — and $\mathcal{D}$ is the symbol for "the dataset you happened to get." Link forward: how you slice $\mathcal{D}$ into pieces to estimate error is the job of Train-Validation-Test Split and Cross-Validation.

5. The estimator $\hat{f}(x)$ — your model's guess

So:

• $f(x)$ = truth (unknown, no hat).
• $\hat{f}(x)$ = your model's prediction at $x$ (built from $\mathcal{D}$, wears a hat).

Because $\hat{f}$ is trained on a random $\mathcal{D}$, it is itself random: train on a different handful of dots, get a different curve. That is why we can talk about ${\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$ (its average curve) and how much it scatters.

How wiggly $\hat{f}$ is allowed to be — a straight line vs. a degree-9 polynomial — is exactly Model Capacity. More capacity can lower bias but raises variance, which is the Overfitting and Underfitting story.

6. Squaring and $({)}^2$ — why errors get squared

The parent measures error with $(y-\hat{f}(x){)}^2$. Why square it?

The parabola shows it: whether you miss left or right, squaring lifts the penalty above zero, and the penalty grows steeply for large misses.

7. Variance $\text{Var}(\cdot )$ — average squared scatter

Read it inside-out:

1. $\mathbb{E}[Z]$ — the average value (the center).
2. $Z-\mathbb{E}[Z]$ — how far this particular draw sits from the center.
3. square it (section 6) so ups and downs don't cancel.
4. $\mathbb{E}[\cdots ]$ — average that squared distance over all draws.

Apply it to a model: $\text{Var}(\hat{f}(x))$ asks "across all the different datasets $\mathcal{D}$ I might have trained on, how much does my prediction at $x$ jump around?" That is precisely the "variance" pillar of the decomposition.

The constant ${\sigma }^2$ (read "sigma squared") in the parent is just the variance of the noise: $\text{Var}(ϵ)={\sigma }^2$. It is the irreducible floor — you cannot out-model random noise. See it shrink with cleaner data or grow with messy data; Learning Curves make this floor visible.

8. Bias — the steady, systematic miss

Notice both symbols are now earned: ${\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$ is the average model (section 3–5), $f(x)$ is truth (section 1). Their difference is a steady error that stays even after averaging away all the scatter — a rule that is simply the wrong shape (a straight line trying to be a sine wave). Fighting bias with penalties is the surprising twist in Regularization Techniques and Ensemble Methods.

Prerequisite map

Every arrow says "you must understand the tail before the head." The three streams — truth-and-noise, expectation-and-scatter, dataset-and-model — all converge on the tradeoff itself.

Equipment checklist

Cover the right side and answer out loud. If any fails, reread that section.