Worked examples — Bias-variance tradeoff

This is the worked-examples deep dive for the Bias-Variance Tradeoff. The parent note derived the decomposition

$\mathbb{E}[(y-\hat{f}(x){)}^2]=\underset{{\text{Bias}}^2}{\underbrace{(\mathbb{E}[\hat{f}(x)]-f(x){)}^2}}+\underset{\text{Variance}}{\underbrace{\mathbb{E}[(\hat{f}(x)-\mathbb{E}[\hat{f}(x)]{)}^2]}}+\underset{\text{Irreducible}}{\underbrace{{\sigma }^2}}$

Here we make it concrete with numbers. Before we start, let me re-anchor every symbol in plain words so you never meet an unexplained mark.

Every example below plugs real numbers into these three buckets.

---

The scenario matrix

Bias-variance is not one formula; it is a spectrum of regimes. Every model you ever build lands in exactly one of these cells. This page works one example per cell so you never meet a case blind.

Cell	Regime	Bias	Variance	Where you meet it
A	Model too simple (underfit)	High	Low	constant / linear on curved data
B	Model too complex (overfit)	~0	Very high	degree-9 poly on 10 points, 1-NN
C	Just-right complexity	Low	Low-ish	the U-curve minimum
D	Degenerate low end ($k=n$, constant)	Max	Zero	predict-the-mean baseline
E	Degenerate high end (interpolation)	0	Max	model passes through every point
F	Limiting input: $n\to \infty$ (infinite data)	unchanged	$\to 0$	asymptotic behaviour
G	Effect of noise ${\sigma }^2$ (irreducible floor)	unchanged	unchanged	error can't go below ${\sigma }^2$
H	Regularization dial (trade bias up for variance down)	↑	↓	ridge / L2
I	Real-world word problem	—	—	forecasting sales
J	Exam twist (compute the three terms exactly)	—	—	closed-form check

We now sweep the whole table. Figure 1 is your map: the U-shaped total-error curve with each cell marked.

---

Example 1 — Cells A & D: the constant model (max bias, zero variance)

Forecast: Guess now — will variance be big or tiny for a model that ignores its data almost entirely?

1. Find the constant the model settles on. Using oracle knowledge, its value is the mean of the true curve: $c={\int }_0^1\sin (2\pi x)dx=[-\frac{\cos (2\pi x)}{2\pi }{]}_0^1=0.$ Why this step? We deliberately fix $c$ from the noise-free $f$ (not from data) so that no dataset can shift it — that is what makes the variance exactly zero for this teaching case. Averaging a full sine cycle gives $0$.

2. Bias at $x=0.25$. The true value there is $f(0.25)=\sin (\pi /2)=1$. The average prediction is $\mathbb{E}[\hat{f}]=c=0$. $\text{Bias}=0-1=-1,{\text{Bias}}^2=1.$ Why this step? Bias is average prediction minus truth — here a full unit off. That is cell A: maximal systematic miss.

3. Variance. Because $c$ was pinned by oracle knowledge (not by any noisy sample), it never changes across datasets: $\text{Var}(\hat{f}(0.25))=0.$ Why this step? Zero variance is the signature of cell D: with $c$ frozen independent of the data, the model is perfectly stable but blind.

Verify: Bias${}^2$ + Var $=1+0=1$. Sanity: a model that always says $0$ at a point where truth is $1$ should be off by exactly $1$, and never wobble. ✔

---

Example 2 — Cells B & E: degree-9 polynomial interpolating 10 points

Forecast: If the fit hits every point exactly, what is its bias at those points? And what happens when the noise changes?

1. At a training point, the prediction equals the observed $y_i$. So $\hat{f}(x_i)=f(x_i)+{ϵ}_i$. Why this step? Interpolation means "copy the data". This is cell E: forced to pass through everything.

2. Bias at $x_i$. Average over datasets: $\mathbb{E}[\hat{f}(x_i)]=f(x_i)+\mathbb{E}[{ϵ}_i]=f(x_i)+0=f(x_i)$. $\text{Bias}=f(x_i)-f(x_i)=0.$ Why this step? On average the noise cancels, so the interpolator is unbiased at data points — classic cell B: near-zero bias.

3. Variance at $x_i$. $\hat{f}(x_i)-\mathbb{E}[\hat{f}(x_i)]={ϵ}_i$, so $\text{Var}(\hat{f}(x_i))=\mathbb{E}[{ϵ}_i^2]={\sigma }^2.$ Why this step? Since the prediction copies the noise directly, its scatter across datasets equals the noise's own scatter, ${\sigma }^2$. And between data points a degree-9 curve swings enormously (Runge oscillation), so its variance there is many times ${\sigma }^2$. Figure 2 shows five fits on five noise draws diverging.

Verify: At $x_i$: Bias${}^2$+Var $=0+{\sigma }^2={\sigma }^2$. With $\sigma =0.1$ (so ${\sigma }^2=0.01$), that's $0.01$. The overfit model's point error equals pure noise — it has learned nothing but the noise. ✔

See Overfitting and Underfitting and Model Capacity for why 10 parameters on 10 points is the danger line.

---

Example 3 — Cell C: the just-right degree-3 fit (the U-minimum)

Forecast: Complexity climbs left to right. Bet on which one wins before summing.

1. Add the three buckets for each model (total $=$ Bias${}^2+$Var$+{\sigma }^2$):

   • Degree 1: $0.21+0.002+0.01=0.222$
   • Degree 3: $0.02+0.010+0.01=0.040$
   • Degree 9: $0.001+0.15+0.01=0.161$ Why this step? The decomposition says total error is just the sum — no cross terms survive.

2. Pick the minimum. Degree 3 gives $0.040$, the smallest. That is cell C, the bottom of the U. Why this step? Neither extreme wins: degree 1 pays in bias, degree 9 pays in variance.

Verify: $0.040<0.161<0.222$. The bottom sits at moderate complexity — exactly the U-shape the parent note predicted. ✔ Use Cross-Validation or Learning Curves to find this minimum in practice.

---

Example 4 — Cell F: infinite data limit ($n\to \infty$)

Forecast: Common belief: "with enough data, error → 0." True or false here?

1. Variance as $n$ grows. With $p=2$ parameters, ${\sigma }^2=0.01$: $n=10:\text{Var}\approx \frac{2(0.01)}{10}=0.002,n=1000:\frac{2(0.01)}{1000}=2\times 10^{-5},n\to \infty :\text{Var}\to 0.$ Why this step? Each extra point dilutes any single point's influence, so scatter shrinks. This is cell F.

2. Bias stays put. A straight line cannot bend into a sine no matter how many points. The best line still misses the curvature, so Bias${}^2$ stays at its floor (say $0.21$). Why this step? Bias is a property of the model class, not the sample size.

Verify: As $n\to \infty$, total error $\to 0.21+0+0.01=0.22$, not zero. More data helps variance only; the tradeoff persists. ✔ (This is Mistake 1 from the parent note made numeric.)

---

Example 5 — Cell G: the irreducible floor

Forecast: Perfect model — surely zero error?

1. Plug into the decomposition with Bias $=0$, Var $=0$: $\mathbb{E}[(y-\hat{f}{)}^2]=0+0+{\sigma }^2=\mathrm{0.04.}$ Why this step? The ${\sigma }^2$ term is measurement noise living in $y$ itself, not in the model. No model can predict a coin flip's specific outcome. This is cell G.

2. Interpret. Since ${\sigma }^2=0.04$, the standard deviation is $\sigma =\sqrt{0.04}=0.2$, the RMS floor: predictions cannot beat $\pm 0.2$ typical error on average.

Verify: Total $=0.04$; a perfect predictor still eats the noise. If someone reports error below ${\sigma }^2$, they measured on the training set (data leak) — see Train-Validation-Test Split. ✔

---

Example 6 — Cell H: regularization dials bias up, variance down

Forecast: $\lambda =0$ is no regularization. Will the best $\lambda$ be $0$, or something positive?

1. Write the total and differentiate. $T(\lambda )=0.01{\lambda }^2+\frac{0.30}{(1+\lambda {)}^2},T^{\prime }(\lambda )=0.02\lambda -\frac{0.60}{(1+\lambda {)}^3}.$ Why this step? The minimum of a smooth U is where its slope $T^{\prime }$ hits zero — that's why we use the derivative, the tool that reads "instantaneous steepness".

2. Set $T^{\prime }(\lambda )=0$ and solve numerically. $0.02\lambda (1+\lambda {)}^3=0.60\Rightarrow \lambda (1+\lambda {)}^3=30.$ Trying $\lambda =1.5$: $1.5\cdot (2.5{)}^3=1.5\cdot 15.625=23.4$ (too small). $\lambda =1.7$: $1.7\cdot (2.7{)}^3=1.7\cdot 19.68=33.5$ (too big). $\lambda \approx 1.62$: $1.62\cdot (2.62{)}^3=1.62\cdot 17.98=29.1$. $\lambda \approx 1.63$: $1.63\cdot (2.63{)}^3=1.63\cdot 18.19=29.6$. So ${\lambda }^{\star }\approx 1.635$. Why this step? Positive ${\lambda }^{\star }$ proves some regularization helps: cell H.

3. Confirm the trade. As $\lambda$ rises from 0: bias $0.01{\lambda }^2$ grows, variance $0.30/(1+\lambda {)}^2$ shrinks. The optimum accepts a little bias to buy a big variance cut. Why this step? Reading the two curves against each other is the whole point of the exercise — it shows the trade is a genuine exchange, not a free lunch.

Read Figure 3 now. The cyan Bias${}^2$ curve climbs from zero as $\lambda$ grows; the amber Variance curve falls steeply from $0.30$ at $\lambda =0$. Their sum, the white Total curve, dips to a clear minimum marked by the amber dot at ${\lambda }^{\star }\approx 1.64$. Notice how far below the $\lambda =0$ starting height that minimum sits — that vertical drop is the benefit of regularizing.

Verify: At ${\lambda }^{\star }\approx 1.635$: $T=0.01(1.635{)}^2+0.30/(2.635{)}^2=0.0267+0.0432=0.0699$. Compare $\lambda =0$: $T=0+0.30=0.30$. Regularizing cut error from $0.30$ to $0.07$. ✔ More in Regularization Techniques.

---

Example 7 — Cell I: real-world word problem (ensemble averaging)

Forecast: Averaging 10 models — which term drops, bias or variance?

1. Bias of the average. Each tree averages to the same biased value, so their mean has the same bias: Bias${}^2=100$ still. Why this step? Averaging identical systematic misses doesn't fix the miss.

2. Variance of the average. Independence gives ${\text{Var}}_{\text{ens}}=\frac{400}{10}=40.$ Why this step? Uncorrelated wobbles partly cancel — this is exactly what Ensemble Methods (bagging) exploits. Cell I.

3. Total. Bias${}^2+$Var $=100+40=140$ vs. single tree $100+400=500$. Why this step? Summing the buckets lets us compare the ensemble against the single tree on one honest number.

Verify: Ensemble error $140<500$ single tree — a $72\%$ cut, all from variance reduction. Units: sales${}^2$, so RMS drops from $\sqrt{500}\approx 22.4$ to $\sqrt{140}\approx 11.8$ units. ✔

---

Example 8 — Cell J: exam twist (compute all three terms exactly)

Forecast: The sample mean is famously "unbiased" — so is one of the three terms exactly zero?

1. Bias. $\mathbb{E}[\hat{f}]=\frac{1}{n}\sum \mathbb{E}[y_i]=\frac{1}{n}\sum 5=5$. $\text{Bias}=5-5=0.$ Why this step? Averaging unbiased observations stays unbiased — cell A/D's opposite: no systematic miss.

2. Variance. For i.i.d. terms, $\text{Var}(\hat{f})=\frac{{\sigma }^2}{n}=\frac{9}{9}=1$. Why this step? This is the ${\sigma }^2/n$ rule from Example 4, now exact.

3. Total expected error (comparing prediction to a fresh noisy $y=5+ϵ$): ${\text{Bias}}^2+\text{Var}+{\sigma }^2=0+1+9=10.$ Why this step? The final ${\sigma }^2=9$ is the noise in the new observation we're judged against — the irreducible floor of cell G.

Verify: Total $=10$; note the estimator's own MSE is Bias${}^2+$Var$=1$, and the extra $9$ is the noise in the target. Doubling $n$ to $18$ would halve the variance to $0.5$ but leave bias $0$ and floor $9$. ✔ This connects to sample-complexity ideas in PAC Learning Theory.

---

Recall Quick self-test

Constant model on a sine has which extreme profile? ::: Maximum bias, zero variance (cells A & D). A degree-9 poly through 10 points has what bias at the training points? ::: Zero (it copies $y_i$, and noise averages out). Does infinite data remove bias? ::: No — bias is a property of the model class; only variance $\to 0$ (cell F). Regularization $\lambda$ trades what for what? ::: Raises bias, lowers variance; optimal $\lambda >0$ (cell H). Averaging 10 independent models divides which term by 10? ::: Variance, not bias (cell I). Can total error drop below ${\sigma }^2$? ::: No, that's the irreducible floor (cell G).