Bias-variance trade-off

Overview

The bias-variance trade-off is the fundamental tension in machine learning between a model's ability to fit training data (low bias) and its ability to generalize to new data (low variance). This trade-off governs model selection in aerospace applications from flight control prediction to satellite image classification.

Core Concepts

Why this matters: A linear model predicting rocket thrust (which depends nonlinearly on fuel flow, pressure, temperature) has high bias — its assumptions are wrong.

Why this matters: A 50-degree polynomial fit to 20 flight data points will give wildly different predictions if we retrain with 20 different flights — it's too sensitive to the specific training samples.

where $\sigma^2$ is the irreducible error (noise in the data itself).

Derivation from scratch:

Let $y = f(x) + \epsilon$ where $\epsilon$ is random noise with mean 0 and variance $\sigma^2$. We want to find the expected squared error when predicting $y$ with our model $\hat{f}(x)$:

E[(y - \hat{f}(x))^2] &= E[(f(x) + \epsilon - \hat{f}(x))^2] \\ &= E[(f(x) - \hat{f}(x))^2 + 2\epsilon(f(x) - \hat{f}(x)) + \epsilon^2] \end{align}$$ **Why this step?** We're expanding the square and using linearity of expectation. $$\begin{align} &= E[(f(x) - \hat{f}(x))^2] + 2E[\epsilon]E[f(x) - \hat{f}(x)] + E[\epsilon^2] \\ &= E[(f(x) - \hat{f}(x))^2] + 0 + \sigma^2 \end{align}$$ **Why this step?** $E[\epsilon] = 0$ by definition, and $E[\epsilon^2] = \sigma^2$ is the noise variance. Now focus on the first term. Add and subtract $E[\hat{f}(x)]$: $$\begin{align} E[(f(x) - \hat{f}(x))^2] &= E[(f(x) - E[\hat{f}(x)] + E[\hat{f}(x)] - \hat{f}(x))^2] \\ &= E[(f(x) - E[\hat{f}(x)])^2 + 2(f(x) - E[\hat{f}(x)])(E[\hat{f}(x)] - \hat{f}(x)) \\ &\quad + (E[\hat{f}(x)] - \hat{f}(x))^2] \end{align}$$ **Why this step?** Classic "add zero" trick to separate systematic error from variability. $$\begin{align} &= (f(x) - E[\hat{f}(x)])^2 + 2(f(x) - E[\hat{f}(x)])E[\hat{f}(x)] - \hat{f}(x)] \\ &\quad + E[(E[\hat{f}(x)] - \hat{f}(x))^2] \end{align}$$ **Why this step?** The first term has no randomness (it's constants). The middle term's expectation is zero because $E[\hat{f}(x)] - \hat{f}(x)$ has mean zero by definition. $$= (f(x) - E[\hat{f}(x)])^2 + E[(\hat{f}(x) - E[\hat{f}(x)])^2]$$ The first term is $\text{Bias}^2[\hat{f}(x)]$ and the second is $\text{Variance}[\hat{f}(x)]$. Done! ∎ > [!formula] Total Error > $$\boxed{\text{Total Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}}$$ **Key insight**: We can't reduce all three simultaneously. Reducing bias (making the model more complex) increases variance. Reducing variance (simplifying the model) increases bias. ## The Trade-off Curve > [!intuition] The U-Shape > As model complexity increases: > - **Bias decreases**: more flexible model can fit the true pattern better > - **Variance increases**: more flexible model is more sensitive to training data noise > - **Total error follows a U-curve**: there's a sweet spot In aerospace: - **Underfitting** (high bias): Linear model for turbulent airflow — mises critical nonlinear dynamics - **Just right**: Neural network with regularization, validated on hold-out flight data - **Overfitting** (high variance): Deep network memorizing sensor noise instead of actual patterns ## Worked Examples > [!example] Example 1: Polynomial Regression for Drag Coefficient > **Problem**: Predict drag coefficient $C_D$ vs. angle of attack $\alpha$ for a UAV wing. We have 15 wind tunnel measurements. **Setup**: True relationship is roughly $C_D = 0.02 + 0.3\alpha + 0.5\alpha^2$ (simplified). Measurements have noise $\sigma = 0.01$. **Approach 1 — Degree 1polynomial (linear)**: ```python # Model: C_D = a + b*alpha # This is high bias — we're forcing a linear fit on quadratic data ``` **Why underfitting happens**: Our model assumption (linear) contradicts reality (quadratic). No amount of training data will fix this — the model literally cannot represent the truth. **Expected behavior**: - Training error: moderate (can't fit the curve) - Test error: moderate (same systematic error) - Bias: **high** — we systematically underpredict at middle angles - Variance: **low** — two different datasets give very similar lines **Approach 2 — Degree 2 polynomial**: ```python # Model: C_D = a + b*alpha + c*alpha^2 # This matches the true form ``` **Why this works**: Model capacity matches problem complexity. We have3 parameters for a 3-parameter truth, with15 data points (5x the parameters). **Expected behavior**: - Training error: low - Test error: low - Bias: **low** — can represent the true function - Variance: **moderate** — some sensitivity to which15 points we sampled - **This is the sweet spot** **Approach 3 — Degree 10 polynomial**: ```python # Model: C_D = a + b*alpha + c*alpha^2 + ... + k*alpha^10 # This has 11 parameters for 15 data points ``` **Why overfitting happens**: With 11 parameters and only 15 data points, the model has enough flexibility to "chase" the random noise in each measurement. It fits the noise, not the signal. **Expected behavior**: - Training error: **very low** (nearly interpolates all points) - Test error: **high** (wild oscillations between data points) - Bias: **low** — model is flexible enough - Variance: **very high** — drastically different curves from different training sets **Quantitative check** (simulated): ```python # Degree 1: Bias²≈ 0.04, Variance ≈ 0.001 # Degree 2: Bias² ≈ 0.001, Variance ≈ 0.005 # Degree 10: Bias² ≈ 0.0001, Variance ≈ 0.08 ``` Total error is minimized at degree 2. > [!example] Example 2: k-NN for Satellite Image Classification > **Problem**: Classify terrain types from satellite pixels (forests, urban, water). Each pixel has 3 RGB values. **k=1 (one nearest neighbor)**: - **Decision boundary**: Voronoi cells around each training pixel - **Training error**: 0% — every training pixel is its own nearest neighbor - **Why high variance**: If we retrain with a slightly different image sample, boundaries change dramatically. One noisy "forest" pixel in an urban area creates an isolated green island. - **Bias**: very low (can represent arbitrarily complex boundaries) - **Variance**: very high - **Result**: Overfitting — predicts "forest" for a single green car in parking lot **k=100 (hundred nearest neighbors)**: - **Decision boundary**: Very smooth, almost linear in RGB space - **Why high bias**: Averaging 100 neighbors over-smooths. Can't capture the sharp boundary between river and forest. - **Bias**: high (forces nearly linear boundaries) - **Variance**: low (stable across training sets) - **Result**: Underfitting — classifies entire image as "mixed terrain" **k=10 (ten nearest neighbors)** — middle ground: - Smooth enough to ignore noise pixels - Flexible enough to follow curved forest/urban boundaries - **Typically the best for satellite imagery with moderate noise** **Why this step matters**: In aerospace, satellite data is expensive. We can't afford to overfit to one image set and fail on the next orbital pass. > [!example] Example 3: Neural Network Depth for Flight Control > **Problem**: Predict elevator deflection from airspeed, altitude, pitch rate. **1 hidden layer, 5 neurons** (simple): - Can learn basic linear combinations - **High bias** if control law is nonlinear (which it is — dynamic pressure goes as $v^2$) - Low variance **3 hidden layers, 50 neurons each** (complex): - Can approximate any continuous function (universal approximation theorem) - **High variance** with small flight test dataset - Will memorize specific test flights instead of learning general dynamics **2 hidden layers, 20 neurons each + L2 regularization**: - Enough capacity for nonlinear aerodynamics - Regularization penalizes overfitting - **Sweet spot validated by cross-validation on multiple flight regimes** ## Common Mistakes > [!mistake] Mistake 1: "More data always fixes overfitting" > **Why it feels right**: More data means more examples, so the model should generalize better, right? **Steel-man the intuition**: This is partially true! More data *does* reduce variance (the model sees more scenarios). With infinite data, variance goes to zero. **The fix**: More data only helps **if your model has the right capacity**. If you're using a degree-50 polynomial with 50 parameters, going from 100 to 1000 data points helps, but you're still overfitting relative to a degree-2 model. And more data does **nothing for bias** — a linear model will underfit quadratic data no matter how many samples you collect. **The right principle**: Match model complexity to: (data size) × (signal-to-noise ratio) × (true complexity). > [!mistake] Mistake 2: "Low training error means good model" > **Why it feels right**: If the model fits the training data well, it learned the pattern! **Steel-man**: In a perfect world with infinite data and noise, zero training error would mean perfect learning. **The fix**: Training error measures bias, not variance. A degree-100 polynomial gets zero training error by memorizing noise. The real test is **validation error** on held-out data. In aerospace, this means testing on flights/images/trajectories the model never saw during training. **Best practice**: Always split data: 60% train, 20% validation (tune hyperparameters), 20% test (final assessment). Report test error. > [!mistake] Mistake 3: "Regularization always helps" > **Why it feels right**: Regularization (like L2 penalty) is specifically designed to reduce overfitting by penalizing model complexity. **Steel-man**: Regularization *does* reduce variance by constraining the model. **The fix**: Regularization **increases bias**. If your model already underfit (high bias), adding regularization makes it worse. Example: You're using linear regression for quadratic drag, and you add strong L2 penalty — now the model is even more constrained and fits even worse. **When to regularize**: When validation error is higher than training error (sign of overfitting). Don't regularize if both errors are high (sign of underfitting — you need more capacity, not less). ## Practical Techniques ### Cross-Validation Use ==k-fold cross-validation== to estimate both bias and variance: ```python from sklearn.model_selection import cross_val_score # Train model on different subsets, test on held-out fold scores = cross_val_score(model, X, y, cv=5) # Low mean score → high bias (underfitting) # High std of scores → high variance (overfitting) # Both low and stable → just right ``` **Why this works**: By training on different subsets, we directly measure how much predictions vary (variance). The mean score tells us systematic error (bias). ### Learning Curves Plot training and validation error vs. dataset size: **High bias signature**: - Training error: high, plateaus quickly - Validation error: high, converges to training error - **Fix**: Increase model complexity **High variance signature**: - Training error: low - Validation error: high, large gap persists - **Fix**: More data, regularization, or reduce complexity ### Regularization Parameter Tuning For aerospace applications: ```python # Ridge regression (L2): penalizes sum of squared weights # Lasso regression (L1): penalizes sum of absolute weights # Elastic net: combines both from sklearn.linear_model import RidgeCV model = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0]) # Try different penalties model.fit(X_train, y_train) print(f"Best alpha: {model.alpha_}") # Chosen by cross-validation ``` **Why this matters in aerospace**: Flight data is expensive (wind tunnel time, test flights). We typically have small datasets → high risk of overfitting → regularization is critical. > [!mnemonic] BVIR Memory Aid > **B**ias: **B**lind to complexity (model too simple) > **V**ariance: **V**olatile across datasets (model too sensitive) > **I**reducible: **I**nherent noise (can't fix) > **R**elationship: They **R**un opposite directions — increasing complexity reduces bias but increases variance **The Goldilocks Rule**: Not too simple (bias), not too complex (variance), just right (minimize total error). ## Connections - [[5.6.01-Supervised-vsUnsupervised-Learning]] — Bias-variance applies to both paradigms - [[5.6.02-Training-Validation-Test-Sets]] — How we measure the trade-off in practice - [[5.6.03-Overfitting-and-Regularization]] — Regularization as a tool to control variance - [[5.6.05-Cross-Validation-Techniques]] — Empirical method to find the sweet spot - [[5.6.08-Hyperparameter-Tuning]] — Model complexity is a hyperparameter to tune - [[3.4.07-Polynomial-Regression]] — Classic example of bias-variance with varying degree > [!recall]- Explain to a 12-year-old > Imagine you're learning to predict where a basketball will land based on how hard you throw it. If you use a super simple rule like "it always goes10 feet," you're being **biased** — you're not even trying to look at how hard you threw! That's like underfitting. If you memorize exactly where the ball landed in your5 practice shots, then when you throw with slightly different force, your prediction is way off because you memorized the exact throws instead of learning the general pattern. That's **variance** — being too sensitive to the specific practice shots. That's like overfitting. The smart approach is to notice "harder throw → farther distance" but NOT memorize every tiny woble. You learn the pattern without the noise. That's the sweet spot — low bias (you're learning the real rule) and low variance (you're stable across different practice sessions). In machine learning for rockets and planes, we do the same thing: find the model that's "just right" — not too simple, not too memorizing, but captures the real physics. --- #flashcards/coding What is bias in machine learning? :: The error from incorrect assumptions in the learning algorithm; high bias means the model is too simple to capture the true pattern (underfitting). Mathematically: expected difference between model prediction and truth. What is variance in machine learning? ::: The sensitivity of the model's predictions to the specific training dataset; high variance means the model changes drastically with different training data (overfitting). Mathematically: expected squared deviation of predictions from their mean. State the bias-variance decomposition formula ::: Expected error = Bias² + Variance + Irreducible error (noise). These three components are additive and represent systematic error, sensitivity to data, and inherent noise respectively. Why can't we minimize bias and variance simultaneously? ::: They have an inverse relationship. Increasing model complexity reduces bias (model can fit better) but increases variance (model becomes more sensitive to training data). There's a sweet spot where total error is minimized. What is the signature of high bias in learning curves? ::: Training error and validation error are both high and converge quickly. The gap between them is small. This indicates underfitting — the model is too simple. What is the signature of high variance in learning curves? ::: Training error is low but validation error is high with a large persistent gap. This indicates overfitting — the model memorizes training data but doesn't generalize. How does k-fold cross-validation help identify bias vs variance? ::: Low mean cross-validation score indicates high bias (systematic error). High standard deviation of scores across folds indicates high variance (predictions unstable across different training subsets). Why doesn't more data always fix overfitting? :: More data reduces variance but does nothing for bias. If your model is too simple (high bias), infinite data won't help. If your model is too complex, you're just overfitting to a larger dataset. You need the right model complexity for the data size. What does regularization do to the bias-variance trade-off? ::: Regularization increases bias (constrains the model) but decreases variance (makes less sensitive to training data). Use it when overfitting (high variance), but it makes underfitting (high bias) worse. In polynomial regression, why does degree-1 have high bias? ::: Because the true relationship is often nonlinear (e.g., quadratic drag). A linear model literally cannot represent the truth regardless of how much data you have — its assumptions are fundamentally wrong. Why does k=1 in k-NN have high variance? ::: Every training point is its own nearest neighbor (0% training error), but predictions change dramatically with different training sets. One noisy point creates isolated decision regions. The model is too flexible and memorizes noise. What is irreducible error and why can't we eliminate it? ::: The inherent noise in the data (σ²), caused by measurement error, unmodeled variables, or true randomness. No model can predict noise — it's theoretical lower bound on prediction error. ## 🖼️ Concept Map ```mermaid flowchart TD BVT[Bias-Variance Trade-off] Bias[Bias] Variance[Variance] Under[Underfitting] Over[Overfitting] Decomp[MSE Decomposition] Noise[Irreducible Error sigma^2] Model[Model Selection] Aero[Aerospace Predictions] BVT -->|one side| Bias BVT -->|other side| Variance Bias -->|high causes| Under Variance -->|high causes| Over Bias -->|wrong assumptions| Model Variance -->|sensitivity to data| Model Decomp -->|sums| Bias Decomp -->|sums| Variance Decomp -->|adds| Noise Model -->|governs| Aero Under -->|too simple for| Aero Over -->|learns noise in| Aero ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Bias-variance trade-off machine learning ka sabse fundamental concept hai, aur aerospace applications mein yeh bohot critical hai. Socho agar tum ek model bana rahe ho jo aircraft ka fuel consumption predict kare. Agar tumhara model bohot simple hai (jaise sirf linear equation), toh woh true pattern ko capture nahi kar payega — yeh **bias** hai, matlab model ki assumptions hi galat hain. Iska matlab underfitting — model complex reality ko samajh hi nahi pa raha. > > Dosri taraf, agar tumne bohot complex model liya (jaise 50-degree polynomial) aur tumhare pas sirf 20 data points hain, toh model training data ke noise ko bhi memorize kar lega instead of learning the real pattern. Yeh **variance** hai — model itna sensitive hai ki har naye dataset pe predictions bilkul alag dega. Iska matlab overfitting — model ne training flights ko yad kar liya, lekin naye route pe predict karne mein fail ho jayega. > > Real insight yeh hai ki tum dono ko ek sath minimize nahi kar sakte. Jab tum model ko complex banate ho (bias kamarne ke liye), variance badh jata hai. Jab simplify karte ho (variance kam karne ke liye), bias badh jata hai. Total error = Bias² + Variance + Noise, aur tumhe woh sweet spot dhoondhna padta hai jahan total minimum ho. Aerospace mein yeh bohot important hai kyunki flight test data expensive hota hai — tum overfit nahi kar sakte ek test flight set pe, kyunki real missions mein model fail ho jayega. Cross-validation aur learning curves use karo to diagnose karo ki tumhara model underfit kar raha hai ya overfit, aur phir complexity adjust karo accordingly. ![[audio/5.6.04-Bias-variance-trade-off.mp3]]