Bias-variance tradeoff

The bias-variance tradeoff is the fundamental tension in machine learning: simple models miss important patterns (high bias), while complex models memorize noise (high variance). Every model sits somewhere on this spectrum, and our job is to find the sweet spot where total error is minimized.

Core Intuition

In ML: a linear model on nonlinear data has high bias (wrong functional form) but low variance (stable predictions across datasets). A deep neural net has low bias (can fit any pattern) but high variance (sensitive to training data quirks). The tradeoff arises from balancing model complexity against finite, noisy data: with only a limited, noise-corrupted sample, a more flexible model chases the noise (variance) while a stiffer model ignores real structure (bias).

Mathematical Derivation from First Principles

• Variance of $\hat{f}$ at $x$: how much predictions change across different training sets $\text{Var}(\hat{f}(x)) = \mathbb{E}_{\mathcal{D}}\left[(\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)])^2\right]$

• Irreducible error: noise $\epsilon$ in data, $\mathbb{E}[\epsilon^2] = \sigma^2$

Deriving the Decomposition

Start with the expected prediction error (expected over both training sets $\mathcal{D}$ and noise $\epsilon$) at single point $x$:

Setup: True model is $y = f(x) + \epsilon$ where $\mathbb{E}[\epsilon] = 0$, $\text{Var}(\epsilon) = \sigma^2$.

Goal: Decompose $\mathbb{E}_{\mathcal{D}, \epsilon}\left[(y - \hat{f}(x))^2\right]$

Step 1: Expand the squared error $\mathbb{E}_{\mathcal{D}, \epsilon}\left[(y - \hat{f}(x))^2\right]$

Why this step? We want to understand what contributes to prediction error.

Step 2: Substitute $y = f(x) + \epsilon$ $= \mathbb{E}_{\mathcal{D}, \epsilon}\left[(f(x) + \epsilon - \hat{f}(x))^2\right]$

Why? Separate true signal $f(x)$ from noise $\epsilon$.

Step 3: Add and subtract $\mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ (the average prediction) $= \mathbb{E}_{\mathcal{D}, \epsilon}\left[\left(f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)] + \mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x) + \epsilon\right)^2\right]$

Why? This algebraic trick lets us separate systematic error (bias) from random fluctuation (variance). We're creating a "center point" to measure deviations from.

Step 4: Expand the square ($(A + B + C)^2$) $= \mathbb{E}_{\mathcal{D}, \epsilon}\left[\left(f(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]\right)^2 + \left(\mathbb{E}_{\mathcal{D}}[\hat{f}(x)] - \hat{f}(x)\right)^2 + \epsilon^2 + \text{cross terms}\right]$

Why? We need to isolate three distinct sources of error.

Step 5: Show cross terms vanish

• $\mathbb{E}_{\mathcal{D}}[\hat{f}(x) - \mathbb{E}_{\mathcal{D}}[\hat{f}(x)]] = 0$ by definition of expectation
• $\mathbb{E}_{\epsilon}[\epsilon] = 0$ by assumption
• Cross terms like $2(f(x) - \mathbb{E}[\hat{f}]) \cdot \mathbb{E}[\epsilon] = 0$

Why? Centered random variables have zero expected cross products with constants or independent variables.

Step 6: Take expectations (noting $f(x)$ and $\mathbb{E}_{\mathcal{D}}[\hat{f}(x)]$ don't depend on $\mathcal{D}$ once expectation is taken)

Physical Meaning:

• Bias²: How far is your average prediction from truth? (systematic miss)
• Variance: How much do individual predictions scatter around your average? (instability)
• Irreducible error: Noise you cannot eliminate (fundamental limit)

Why the Tradeoff?

Increasing model complexity (more parameters, deeper networks, higher polynomial degree):

• Decreases bias: More flexible function can fit true $f(x)$ better
• Increases variance: More parameters = more ways to overfit noise in specific dataset
• Total error = Bias² + Variance goes through a U-shaped curve

The optimal complexity sits at the bottom of this U.

Worked Examples

Models to compare:

1. Degree 0 (constant): $\hat{f}(x) = c$
2. Degree 1 (linear): $\hat{f}(x) = a + bx$
3. Degree 9 (high polynomial): $\hat{f}(x) = \sum_{j=0}^{9} a_j x^j$

Analysis:

| Model | Bias | Variance | Why? | |----|----------|---| | Degree 0 | High | Low | Constant cannot capture sine wave (systematic error), but prediction is stable (same $c$ for all datasets) | | Degree 1 | High | Low | Linear mises curvature, but still stable | | Degree 9 | Low | Very High | Can fit sine wave perfectly, but 10 parameters on 10 points → memorizes noise, wildly different fits on different samples |

Why degree 9 has high variance? With 10 datapoints and 10 parameters, the polynomial interpolates every training point exactly. If noise bumps one point up, the entire curve bends to pass through it. On a new dataset with different noise, you get a completely different curve. Average over many datasets (bias) might be close to true sine, but individual fits (what you actually get) swing wildly.

Optimal choice: Degree 3-5 polynomial or Fourier basis with ~3 terms balances the two.

k=1 (nearest neighbor):

• Bias: Low (can form any boundary by piecewise constant regions)
• Variance: Very high (one noisy point flips entire local region, boundaries change drastically between datasets)
• Symptom: Decision boundary is jagged, follows every training point including outliers

k=n (all neighbors → majority class):

• Bias: Very high (ignores local structure, predicts global majority everywhere)
• Variance: Zero (same prediction for all datasets)
• Symptom: Predicts same class everywhere, mises circular boundary

k=5-20 (moderate):

• Bias: Moderate (smooths over noise but can still approximate circle)
• Variance: Moderate (local averaging reduces sensitivity to individual points)
• Optimal for this problem.

Why this step (choosing k)? We're controlling model flexibility. Small k = high flexibility = low bias, high variance. Large k = low flexibility = high bias, low variance. The tradeoff is explicit in the hyperparameter.

Common Mistakes & Misconceptions

The error: More data reduces variance (each datapoint has less individual influence) but does not reduce bias (a linear model on nonlinear data remains biased no matter the sample size). The tradeoff persists—you still need to choose appropriate complexity.

The fix: More data shifts the optimal complexity rightward (allows more complex models) but does not eliminate the need to balance bias and variance. With infinite data, variance →0, but bias remains unless model class includes true function.

Steel-man: The intuition is partially correct—with more data, the optimal model is indeed more complex (you can "afford" more parameters). But you still face a tradeoff at that new optimal point.

The error: Regularization increases bias (pulls parameters toward zero, adding systematic error) to reduce variance (shrinks parameter space, stabilizing predictions). It does not reduce bias—it trades for variance.

The fix: Think of regularization as a "complexity knob" that tunes the bias-variance tradeoff. Stronger regularization = simpler effective model = higher bias, lower variance. It is a tool to find the optimal tradeoff point, not to escape the tradeoff.

The error: The base learners (individual trees) face the tradeoff. Bagging averages high-variance models to reduce variance while preserving their low bias. But you paid the computational cost of multiple models. The tradeoff for a single model remains; ensembles navigate it by combining models.

The fix: Ensembles do not violate the tradeoff—they exploit it cleverly. Bagging works because averaging reduces variance (by $1/\sqrt{n}$ for $n$ independent models) without affecting bias. Boosting sequentially reduces bias by fitting residuals. Both operate within the bias-variance framework.

Practical Strategy: The 80/20

20% of effort for 80% of results:

1. Start simple (linear/logistic regression, shallow tree)
2. Plot learning curves (training vs. validation error vs. dataset size)
   • Gap between train/val → high variance (overfit) → regularize or get more data
   • Both errors high → high bias (underfit) → increase complexity
3. Use cross-validation to estimate bias-variance empirically (see Cross-Validation)
4. Regularize complex models (dropout, L2, early stopping) before abandoning them

Key metric: Validation error is the observable proxy for bias + variance. You cannot measure bias and variance separately on a single dataset, but their sum appears as generalization error.

Alternative: "The dartboard duality"—Bias = aim, Variance = scatter. You're either consistently off-target or all over the place.

Active Recall Practice

Now imagine you try to improvise every shot, adjusting your arms and legs randomly. Sometimes you get lucky and hit, but most shots go all over the place—left, right, short, long. You are inconsistent. That is variance: you are unreliably wrong.

The best shooters find a technique that is both aimed correctly (low bias) and repeatable (low variance). But here's the catch: if you try to be super flexible (adjusting to every tiny factor—wind, lighting, your mood), you will scatter more. If you lock into a rigid technique to be consistent, you might lock in a flaw. Machine learning models face the same dilemma: simple models are consistent but wrong; complex models can be right on average but are all over the place on individual tries.

#flashcards/ai-ml

What is the bias-variance tradeoff? :: The fundamental tension where simple models have high bias (systematic error, underfitting) and low variance (stable predictions), while complex models have low bias but high variance (overfitting, sensitivity to training data). Total error = Bias² + Variance + Irreducible error.

Connections

• Cross-Validation: Empirical method to estimate generalization error (bias + variance) and select model complexity
• Regularization Techniques: L1/L2 penalties, dropout, early stopping—all trade bias for variance
• Learning Curves: Diagnostic tool to visualize bias-variance issues
• Ensemble Methods: Bagging reduces variance, boosting reduces bias—both navigate the tradeoff
• Overfitting and Underfitting: Overfitting = high variance, underfitting = high bias
• Train-Validation-Test Split: Validation set estimates generalization error to detect bias-variance problems
• Model Capacity: The fundamental quantity that determines position on the bias-variance spectrum
• PAC Learning Theory: Formal framework for bias-variance tradeoff via approximation-estimation error decomposition

Concept Map

Hinglish (regional understanding)