Exercises — Bias-variance tradeoff

(a) Bias. The word "average prediction... far from truth" is literally $\mathbb{E}[\hat{f}(x)]-f(x)$. (b) Variance. "Changes across training sets" is exactly ${\mathbb{E}}_{\mathcal{D}}[(\hat{f}-\mathbb{E}[\hat{f}]{)}^2]$. (c) Irreducible error ${\sigma }^2$. It survives even when $\hat{f}=f$ exactly.

(B) high bias, low variance. A flat line cannot bend to a sine → big systematic miss → high bias. But no matter which noisy sample you draw, the best constant is roughly the data mean → the line barely moves → low variance.

All three terms are already in the same "squared" currency, so add directly: $\text{Total}=0.09+0.04+0.04=\mathrm{0.17.}$ Watch: the term in the formula is the noise variance ${\sigma }^2$. If a problem ever gives you the standard deviation $\sigma$ instead, you must square it before adding.

Rearrange the one formula: ${\text{Bias}}^2=\text{Total}-\text{Variance}-{\sigma }^2=0.30-0.08-0.05=\mathrm{0.17.}$

As $k$ ↑: bias ↑, variance ↓.

• $k=1$: boundary hugs every point → low bias, high variance (jagged, chases outliers).
• $k=n$: predicts the global majority everywhere → high bias, zero variance (same output on every dataset).

With 10 free coefficients and 10 points, the polynomial interpolates exactly — it passes through every training point.

• Variance is huge: if noise nudges one point upward, the curve must bend violently to reach it, and it overshoots elsewhere (Runge-style wiggles). A fresh dataset with different noise → a totally different curve → large $\mathbb{E}[(\hat{f}-\mathbb{E}[\hat{f}]{)}^2]$.
• Bias is low: average the wildly-different curves over many datasets and the over-/under-shoots cancel, so $\mathbb{E}[\hat{f}(x)]\approx \sin (2\pi x)$.

Key idea: variance is about individual fits; bias is about their average. See Overfitting and Underfitting for the same story as a symptom checklist.

• Variance ↓ toward 0: with far more points than coefficients, each point has little individual leverage, so fits stop swinging.
• Bias unchanged (for this model class): degree-9 polynomials can represent $\sin$ well, so its bias was already low — but the tradeoff itself never vanishes; the optimal complexity just moves rightward. With infinite data variance→0 but bias stays whatever the model class allows.
• The friend is partly right (you can afford more complexity) but wrong that the tradeoff disappears — you still balance at the new optimum. See Learning Curves.

Total $E(c)=\frac{1}{c}+0.01c+0.05$. Differentiate and set to zero: $E^{\prime }(c)=-\frac{1}{c^2}+0.01=0\Rightarrow c^2=100\Rightarrow c^{\star }=10.$ Why the derivative? The minimum of a smooth U-curve is where its slope flattens to zero. Is it really a minimum? Check the second derivative: $E^{\prime \prime }(c)=\frac{2}{c^3},E^{\prime \prime }(10)=\frac{2}{1000}>0,$ and $E^{\prime \prime }>0$ for all $c>0$, so $E$ is convex and $c^{\star }=10$ is a genuine (global) minimum, not a maximum or saddle. Plug back: $E(10)=\frac{1}{10}+0.01(10)+0.05=0.1+0.1+0.05=\mathrm{0.25.}$ Notice ${\text{Bias}}^2$ and variance are equal ($0.1$ each) at the optimum — a recurring signature. The figure below plots all four curves so you can see this.

How to read the figure below: the magenta curve is ${\text{Bias}}^2=1/c$ — it starts sky-high on the left (a too-simple model) and drops as complexity grows. The violet curve is $\text{Variance}=0.01c$ — a straight line climbing as the model gets more flexible. The orange dashed flat line is the noise floor ${\sigma }^2=0.05$ you can never beat. The navy curve is their sum, the Total. Follow the navy curve: it falls, bottoms out, then rises — the classic U. The navy dot marks $c^{\star }=10$ where the U bottoms at $E=0.25$; drop your eye straight down and notice the magenta and violet curves cross there — that crossing (bias² = variance = 0.1) is the visual signature of the optimum. To the left of the dot bias dominates (underfit); to the right variance dominates (overfit).

Fix A — increase $\lambda$ (L2): shrinks coefficients toward 0. → variance ↓ (parameter space constrained, fits stabilise), bias ↑ (systematic pull away from true optimum). Net win if variance was the dominant term. See Regularization Techniques. Fix B — collect more data: → variance ↓ (less leverage per point), bias ≈ unchanged. Strictly a variance-killer; safe but expensive. Both fixes attack the same weak spot: overfitting is fundamentally variance too large relative to the amount of data. Either fix improves that ratio — more data grows the "amount of data" side; stronger regularization shrinks the effective flexibility (and hence the variance) side. Combining them lets you keep the high degree yet control variance. Diagnose which is cheaper with a validation curve — see Cross-Validation and Train-Validation-Test Split.

Let ${\hat{f}}_1,\dots ,{\hat{f}}_M$ each have $\mathbb{E}[{\hat{f}}_m]=\mu$ (so signed bias $\mu -f=B$) and variance $V$. Ensemble $\bar{f}=\frac{1}{M}{\sum }_m{\hat{f}}_m$.

• Bias: $\mathbb{E}[\bar{f}]=\frac{1}{M}\sum \mu =\mu$, so the signed bias stays $B$ (and $B^2$ stays $B^2$) — averaging does not fix a shared systematic error.
• Variance: in general $\text{Var}\left({\sum }_m{\hat{f}}_m\right)={\sum }_m\text{Var}({\hat{f}}_m)+{\sum }_{m\ne \ell }\text{Cov}({\hat{f}}_m,{\hat{f}}_{\ell })$. The pairwise-uncorrelated assumption kills every covariance term ($\text{Cov}=0$), so the double sum collapses to ${\sum }_m\text{Var}=MV$. Hence $\text{Var}(\bar{f})=\frac{1}{M^2}\cdot MV=\frac{V}{M}.$

So averaging $M$ uncorrelated models cuts variance by a factor $M$ while leaving bias untouched — the core reason Ensemble Methods (bagging) work. Real models are positively correlated, so the surviving covariance terms make the real gain smaller — which is exactly why random forests decorrelate their trees.

One tree: $0.02+0.18+0.05=0.25$. Ensemble of 9: $B^2$ unchanged $=0.02$; variance $=0.18/9=0.02$; noise $=0.05$. $E_{\text{ens}}=0.02+0.02+0.05=\mathrm{0.09.}$ Error drops from $0.25$ to $0.09$ — nearly all of it by killing variance. (Real trees are correlated, so gains are smaller.)

Let $R$ denote the number of repeated training sets we draw — i.e. we independently sample $R$ separate training datasets (say $R=200$) so we can average over datasets, which is exactly what the expectations ${\mathbb{E}}_{\mathcal{D}}$ in the decomposition mean.

Simulation ($f$ known):

1. Fix a grid of test points $x$. For each degree $d$: draw $R$ independent training sets, fit ${\hat{f}}_r$ on each ($r=1,\dots ,R$).
2. Estimate $\mathbb{E}[\hat{f}(x)]\approx \bar{f}(x)=\frac{1}{R}{\sum }_{r=1}^R{\hat{f}}_r(x)$.
3. $\widehat{{\text{Bias}}^2}(x)=(\bar{f}(x)-f(x){)}^2$; $\widehat{\text{Var}}(x)=\frac{1}{R}{\sum }_{r=1}^R({\hat{f}}_r(x)-\bar{f}(x){)}^2$.
4. Average over the $x$-grid; add ${\sigma }^2$; plot vs $d$. Expect a U with minimum near $d^{\star }\approx 3\text{–}5$.

Real setting ($f$ unknown) — still separable via resampling: you cannot use step 3's $(\bar{f}-f{)}^2$ because $f$ is unknown, but you can still split bias from variance:

1. Draw $R$ bootstrap resamples of your data; fit ${\hat{f}}_r$ on each. This is the empirical stand-in for $R$ fresh training sets.
2. Variance is estimated directly (it needs no $f$): $\widehat{\text{Var}}(x)=\frac{1}{R}{\sum }_r({\hat{f}}_r(x)-\bar{f}(x){)}^2$ — the spread of the resampled fits.
3. Total error is estimated on a held-out set (cross-validation), giving $\widehat{\text{Total}}$.
4. Bias² is then obtained by subtraction, using the one formula rearranged: $\widehat{{\text{Bias}}^2}=\widehat{\text{Total}}-\widehat{\text{Var}}-\widehat{{\sigma }^2}$, where $\widehat{{\sigma }^2}$ is estimated from the residual noise floor (e.g. the irreducible error of the most flexible model). So both terms are recoverable separately even without $f$ — variance from resampling spread, bias² by subtraction. This resampling machinery is Cross-Validation applied to the decomposition.

Larger Model Capacity → lower bias but a wider generalization gap (gap between train and test error) that scales with capacity ÷ sample size — the variance side, expressed as a bound. PAC Learning Theory formalises this: to learn a hypothesis class of complexity $H$ to error $\epsilon$ with confidence $1-\delta$, you need roughly $m=O\left(\frac{1}{\epsilon }\left(\ln |H|+\ln \frac{1}{\delta }\right)\right)$ samples. More capacity ($\ln |H|$) demands more data to keep the gap (variance) small — the tradeoff, quantified.

Model 1 (train 0.02 / val 0.35): huge gap, tiny train error → high variance / overfitting. Actions in order:

1. Collect more training data.
2. Increase regularization strength $\lambda$.
3. Reduce capacity / polynomial degree.
4. Use a bagging ensemble to average out variance.

See Overfitting and Underfitting.

Model 2 (train 0.30 / val 0.33): both errors high, gap tiny → high bias / underfitting. The model is too rigid to fit even the training data. Actions in order:

1. Increase capacity / polynomial degree, or add richer features.
2. Decrease regularization $\lambda$ (currently over-constraining the model).
3. Train longer / reduce optimisation error (if using an iterative learner).

Do NOT collect more data for Model 2 — its train and validation curves have already converged to a high floor, so more data leaves both essentially unchanged. The remedy is a richer model, not more examples.

The gap itself is the diagnostic: large gap ⇒ variance problem; both-high-small-gap ⇒ bias problem.

Bias measures the ::: gap between average prediction and truth, $\mathbb{E}[\hat{f}]-f$ Variance measures the ::: spread of predictions across training sets Averaging M uncorrelated models divides variance by ::: M (bias unchanged) A large train-validation gap signals ::: high variance (overfitting) Both errors high with a small gap signals ::: high bias (underfitting) The optimal complexity sits at the ::: bottom of the U-shaped total-error curve In Problem 3.3, ${\text{Bias}}^2$ and variance at the optimum are each equal to ::: 0.1