Underfitting vs overfitting diagnosis

Overview

The central challenge in machine learning: your model must generalize to unseen data. Underfitting and overfitting are the two failure modes, and diagnosing which one you have determines your next move. This note teaches you how to read the signs and take action.

Core Concepts

Diagnostic Framework: The Three Lenses

1. Learning Curves Analysis

Formula Learning Curves: Error vs Training Set Size

Why it works: As you add more training examples, patterns emerge more clearly (reducing overfitting) but your model's limitations become apparent (exposing underfitting). What to plot:

\begin{align} J_{train}(m) &= \frac{1}{m}\sum_{i=1}^{m} L(h_\theta(x^{(i)}), y^{(i)}) \\ J_{cv}(m) &= \frac{1}{m_{cv}}\sum_{i=1}^{m_{cv}} L(h_\theta(x^{(i)}_{cv}), y^{(i)}_{cv}) \end{align}

Where $m$ = number of training examples used, $m_{cv}$ = cross-validation set size.

How to interpret: | Scenario | $J_{train}$ behavior | $J_{cv}$ behavior | Gap | Diagnosis | |----------|------------------|----------------| | Underfitting | High plateau | High plateau | Small | High bias | | Overfitting | Low throughout | High, slow convergence | Large | High variance | | Good fit | Low, gradual rise | Decreases, converges near $J_{train}$ | Small | Just right |

Worked example Learning Curves: Polynomial Regression

Setup: Fitting $y = 3x^2 + 2x + 1 + \epsilon$ (true quadratic relationship)

Case 1 - Underfitting (Linear model $h_\theta(x) = \theta_0 + \theta_1 x$ ):

m=10:   J_train=8.2, J_cv=8.5  
m=100:  J_train=8.1, J_cv=8.3  
m=1000: J_train=8.0, J_cv=8.1

Why this pattern? The linear model cannot represent the quadratic curve. Adding more data doesn't help because the model's capacity is fundamentally insufficient. Both errors plateau at a high value.

Case 2 - Overfitting (Degree-15 polynomial):

m=10:   J_train=0.01, J_cv=156.3  → Memorized 10 points perfectly
m=100:  J_train=0.05, J_cv=45.2   → Still wigling through noise
m=1000: J_train=0.8,  J_cv=12.1   → Constraints force generalization

Why this pattern? With few examples, the 15-degree polynomial twists wildly to hit every training point, capturing noise. As $m$ increases, the model is constrained by more data points and must learn the true quadratic trend. The gap shrinks but slowly.

Case 3 - Just Right (Quadratic model):

m=10:   J_train=0.9, J_cv=1.2  
m=100:  J_train=1.0, J_cv=1.1  
m=1000: J_train=1., J_cv=1.0

Why this pattern? The model capacity matches the problem. Both errors converge to the irreducible error $\epsilon$ (noise in the data). The gap is small throughout.

2. Training vs Validation Error Gap

Formula The Generalization Gap

Derivation from first principles: Start with the expected test error:

\mathbb{E}[\text{Test Error}] = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}

Why this decomposition? Break down by error source:

Bias²: Error from wrong model assumptions → affects training error
Variance: Error from sensitivity to training set → creates train/test gap
Irreducible: Noise in data → minimum possible error

The training error reflects mainly bias (and partial variance):

\mathbb{E}[\text{Train Error}] \approx \text{Bias}^2 + \alpha \cdot \text{Variance} + \text{Irreducible Error}

(where $\alpha < 1$ because the model is optimized on this set)

Therefore, the gap:

\begin{align} \text{Gap} &= \mathbb{E}[\text{Test Error}] - \mathbb{E}[\text{Train Error}] \\ &\approx (1-\alpha) \cdot \text{Variance} \end{align}

The diagnostic:

Large gap → High variance (overfitting)
Small gap + high errors → High bias (underfitting)

Worked example Gap Analysis: Image Classification

Scenario: Building a cat vs dog classifier with a neural network

Experiment1 - Too shallow (2 layers, 10 neurons each):

Training accuracy:   68%  →  Error = 32%
Validation accuracy: 65%  →  Error = 35%
Gap: 3%

Diagnosis: UNDERFITTING (high bias) Why? Both errors are high (far from human-level ~5%), but the gap is small. The model can't even fit the training data well. Action: Increase capacity (more layers/neurons) Experiment 2 - Too deep (20 layers, no regularization):

Training accuracy:   99%  →  Error = 1%
Validation accuracy: 78%  →  Error = 22%
Gap: 21%

Diagnosis: OVERFITTING (high variance) Why? Training error is excellent but validation error is much worse. The model memorized training peculiarities. Action: Add regularization (dropout, L2), get more data, or reduce capacity

Experiment 3 - Just right (8 layers, dropout 0.3, L2 reg):

Training accuracy:   94%  →  Error = 6%
Validation accuracy: 92%  →  Error = 8%
Gap: 2%

Diagnosis: Good fit Why? Both errors are near human-level, gap is minimal. Model has learned generalizable features.

3. Error Components Analysis

Formula Decomposing Your Error Budget

Framework: Compare your errors to baselines

Let $\epsilon^*$ = optimal error (human-level or Bayes error)

\begin{align} \text{Avoidable Bias} &= J_{train} - \epsilon^* \\ \text{Variance} &= J_{cv} - J_{train} \\ \text{Total Avoidable Error} &= J_{cv} - \epsilon^* \end{align}

Why these metrics?

Avoidable bias: Gap between your training performance and the best possible → how much your model limitations hurt you
Variance: The train-validation gap → how much your model's instability hurts you
Total: Where to focus your effort

The 80/20 rule: Focus on whichever component is larger

Worked example Error Budget: Speech Recognition

Context: Building a speech-to-text system

Human-level error: $\epsilon^* = 2\%$ (professional transcribers) Iteration 1:

Training error:    15%
Validation error:  18%
Avoidable bias = 15% - 2% = 13%  ← PRIMARY PROBLEM
Variance       = 18% - 15% = 3%

Diagnosis: Severe underfitting Why the focus on bias? You're leaving 13% on the table just from model limitations vs only 3% from overfitting. Action taken: Switched from shallow RNN to deep Transformer (increased capacity)

Iteration 2 (after capacity increase):

Training error:    4%
Validation error:  12%

Avoidable bias = 4% - 2% = 2%
Variance       = 12% - 4% = 8%   ← PRIMARY PROBLEM

Diagnosis: Now overfitting is the bottleneck Why the shift? The high-capacity model now fits training data well (close to human-level) but doesn't generalize. Action taken: Added data augmentation (speed perturbation, noise injection), applied dropout

Iteration 3 (after regularization):

Training error:    3%
Validation error:  4%

Avoidable bias = 3% - 2% = 1%
Variance       = 4% - 3% = 1%

Diagnosis: Balanced, near-optimal Why this is good? Both error sources are roughly equal and small. Further gains would require better features or more data.

Diagnosis Decision Tree

Recall The Diagnosis Algorithm (Explain to a 12-year-old)

Imagine you're learning to throw darts: Step 1: Look at your training board (practice throws)

Are you missing the bullseye by a lot? → You haven't learned the basics (underfitting)
Are you hitting the bullseye perfectly? → Move to step 2 Step 2: Look at the test board (new game)
Are you still hitting well? → You're skilled! (good fit)
Are you missing badly? → You memorized that specific practice board, not real dart skills (overfitting) The fix:
If step 1 failed: Practice more basics, improve your stance (increase model capacity)
If step 2 failed: Practice on many different boards, not just one (get more data, use regularization)

Formal decision tree:

START: Train your model
    ↓
Is J_train high (>> ε*)?
    ├─ YES → UNDERFITTING
    │   └─ Action: Increase capacity, remove regularization,
    │              engineer features, train longer
    │
    └─ NO → Is (J_cv - J_train) large?
        ├─ YES → OVERFITTING
        │   └─ Action: More data, regularization (L1/L2/dropout),
        │              reduce capacity, data augmentation
        │
        └─ NO → Is J_cv acceptable (≈ ε*)?
            ├─ YES → GOOD FIT (done!)
            └─ NO → HIGH IRREDUCIBLE ERROR
                └─ Action: Better features, cleaner data,
                           redefine problem

Common Mistakes

Common mistake Mistake 2: "Small train-test gap means good model"

The wrong intuition: "My training and validation errors are both40%, and they're close to each other. That's consistent, so it's good!"

Why it feels right: Consistency = reliability, and the model isn't overfitting (small gap).

The truth: ==Small gap with high errors = severe underfitting==. Your model is "consistently bad."

The fix: Check errors against baseline:

If J_train ≈ J_cv but both >> ε*  → Underfitting (increase capacity)
If J_train≈ J_cv and both ≈ ε*   → Good fit (done!)

Example:

Human error on task: 2%
Your model: J_train=40%, J_cv=42%
Gap is only 2%, but you're leaving 38% on the table! Underfitting.

Common mistake Mistake 3: "Just use validation accuracy to diagnose"

The wrong intuition: "If validation accuracy is 90%, my model is great!"

Why it feels right: High accuracy sounds good in isolation.

The truth: You need BOTH training and validation metrics to diagnose. High validation accuracy could mean:

Good fit (if training accuracy is also ~90%)
Underfitting a potentially better solution (if training is 95%, you have capacity for better)
The task is easy (if human-level is 99%, your 90% is underfitting)

The complete picture:

# Always check this trio:
baseline_error = 2%      # Human-level or theoretical minimum
train_error = 3%         # Your training error
val_error = 5%           # Your validation error
# Now diagnose:
if train_error >> baseline_error:
   print("Underfitting - model can't even fit training data")
elif (val_error - train_error) > threshold:
   print("Overfitting - big gap between train and val")
else:
   print("Good fit - near baseline with small gap")

Practical Diagnostic Checklist

Quick Diagnosis (30 seconds)

Compute: $J_{train}$ , $J_{cv}$ , baseline $\epsilon^*$
Check:
- Is $J_{train} \gg \epsilon^*$ ? → Underfitting
- Is $(J_{cv} - J_{train})$ large (>10% of $\epsilon^*$ )? → Overfitting
Act: See decision tree above

Deep Diagnosis (when quick check is unclear)

Plot learning curves: Error vs $m$ (training set size)
- Underfitting: Both curves plateau high
- Overfitting: Large persistent gap
Plot validation curves: Error vs model complexity
- Underfitting: Error decreases as complexity increases
- Overfitting: Training error decreases, validation error increases (U-shape)

Compute error budget:

Avoidable bias = J_train - ε*
Variance = J_cv - J_train
Focus on the larger component (80/20 rule)

Sanity checks:
- Can your model fit a single batch perfectly? (No → architecture issue)
- Does validation error decrease at all during training? (No → data/feature issue)
- Is training loss still decreasing? (Yes → might benefit from more training)

Action Playbook

To Fix Underfitting (High Bias)

| Action | Why It Works | When To Use | |--------|-------------| | Increase model capacity | Allows model to represent complex patterns | $J_{train}$ high, gap small | | Add polynomial features | Captures non-linear relationships | Linear model on curved data | | Reduce regularization | Removes constraints on model flexibility | $\lambda$ too high | | Train longer | Allows optimization to converge | Loss still decreasing | | Remove dropout/L2 | Same as reduce regularization | Regularization too aggressive |

To Fix Overfitting (High Variance)

| Action | Why It Works | When To Use | |--------|-------------| | Get more training data | Provides more constraints | $(J_{cv} - J_{train})$ large | | Data augmentation | Artificially increases data diversity | Can't get more real data | | Add regularization (L1/L2) | Penalizes model complexity | Gap large, enough data | | Add dropout | Forces redundant representations | Deep neural networks | | Reduce model capacity | Limits memorization ability | Severe overfitting | | Early stopping | Stops before memorization dominates | Validation error U-shaped | | Ensemble methods | Averages out individual model variance | Production system |

Connections

2.6.01-Bias-variance-tradeoff - Theoretical foundation for why underfitting/overfitting occur
2.6.03-Learning-curves-interpretation - Detailed analysis of curve patterns
2.6.04-Cross-validation-techniques - Robust error estimation for diagnosis
2.6.05-Regularization-methods - Tools to combat overfitting
3.2.01-Neural-network-capacity - How architecture affects fit
4.1.02-Data-augmentation-strategies - Addressing high variance with synthetic data

Flashcards

#flashcards/ai-ml

What are the two failure modes of model generalization? :: Underfitting (high bias) and overfitting (high variance)

What is the mathematical signature of underfitting?

High training error

J_{train} \gg \epsilon^*

AND small gap

(J_{cv} - J_{train})

is small

What is the mathematical signature of overfitting?

Low training error

J_{train} \approx 0

AND large gap

(J_{cv} - J_{train}) \gg 0

In learning curves, what pattern indicates underfitting?

Both training and validation error curves plateau at a HIGH value

In learning curves, what pattern indicates overfitting?

Large persistent gap between training (low) and validation (high) error curves, gap decreases slowly with more data

What is the formula for avoidable bias?

Avoidable bias =

J_{train} - \epsilon^*

where

\epsilon^*

is human-level or Bayes error

What is the formula for variance in error budget analysis?

Variance =

J_{cv} - J_{train}

(the generalization gap)

If both training and validation errors are 40% and human-level is 5%, what is the diagnosis?

Underfitting (high bias) - small gap but both errors are far from baseline

If training error is 2% and validation error is 25 with human-level at 3%, what is the diagnosis?

Overfitting (high variance) - large gap, training near baseline but validation far above

What are three ways to fix underfitting?

1) Increase model capacity, 2) Add polynomial/interaction features, 3) Reduce regularization

What are three ways to fix overfitting?

1) Get more training data, 2) Add regularization (L1/L2/dropout), 3) Reduce model capacity or use early stopping

Why doesn't training longer help overfitting?

Training longer on the same data gives the model more time to memorize noise, making overfitting worse; you need more DATA not more epochs

What does a small train-test gap with high errors indicate?

Underfitting - the model is "consistently bad" at fitting even the training data

What should you compare your training error against?

A baseline error

\epsilon^*

(human-level performance or theoretical minimum) to measure avoidable bias

According to the 80/20 rule for error budget, where should you focus effort?

Focus on whichever is larger: avoidable bias (

J_{train} - \epsilon^*

) or variance (

J_{cv} - J_{train}

)

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, machine learning mein sabse bada problem ye hai ki tumhara model nayi data pe kaisa perform karega. Training data pe toh bahut acha kar raha hai, lekin real world mein test karo toh fail ho jata hai - yeh overfitting kehte hain. Ya phir ulta, training data pe bhi thek se kuch seekh nahi paya - yeh underfitting hai.

Diagnosis kaise karein? Teen chezein dekho: pehla, learning curves banao (error vs training examples ka graph). Agar dono curves (training aur validation) upar hi ruk gaye high error pe, matlab model itna simple hai ki pattern hi samajh nahi aa raha - underfitting. Agar training error bahut kam hai lekin validation error bahut zyada, aur unke bech bada gap hai, toh model ne training data ko ratt liya hai instead of learning general pattern - overfitting. Dosra tareka hai error budget analysis: apni training error ko baseline (jaise human-level performance) se compare karo. Agar J_train >> baseline, toh capacity badhao (underfitting fix). Agar

Overview

Core Concepts

Diagnostic Framework: The Three Lenses

1. Learning Curves Analysis

2. Training vs Validation Error Gap

3. Error Components Analysis

Diagnosis Decision Tree

Common Mistakes

Practical Diagnostic Checklist

Quick Diagnosis (30 seconds)

Deep Diagnosis (when quick check is unclear)

Action Playbook

To Fix Underfitting (High Bias)

To Fix Overfitting (High Variance)

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Model Evaluation & Selection

Connections