Learning curves analysis

Core Concept

Think of it like learning to play guitar: if you can't play simple songs even after practicing the same three chords for months (high bias), you need a better learning method. If you can play those three songs perfectly but freeze on anything new (high variance), you need to practice more diverse material.

Mathematical Foundation

Why Errors Behave This Way

Training Error Behavior: As training set size $m$ increases, training error typically increases because:

$J_{train}(m) \text{ increases with } m$

Why? With few examples, even a simple model can memorize them perfectly (zero training error). As you add more diverse examples, the model can't fit all of them perfectly, so training error rises.

Validation Error Behavior: As training set size $m$ increases, validation error typically decreases because:

$J_{val}(m) \text{ decreases with } m$

Why? With few training examples, the model learns a poor representation of the underlying pattern—it's under-informed. More data helps it generalize better, reducing validation error.

$J_{train}(m) = \frac{1}{m}\sum_{i=1}^{m} L(h_\theta(x^{(i)}), y^{(i)})$

$J_{val}(m) = \frac{1}{m_{val}}\sum_{i=1}^{m_{val}} L(h_\theta(x^{(i)}_{val}), y^{(i)}_{val})$

As $m \to \infty$:

• High bias: $J_{train}(m) \to \epsilon_{high}$ and $J_{val}(m) \to \epsilon_{high}$, where both converge to a high error
• High variance: $J_{train}(m) \to \epsilon_{low}$ but $J_{val}(m) \gg J_{train}(m)$ with large gap

Diagnostic Patterns

Pattern 1: High Bias (Underfitting)

What you see:

• Training error rises quickly and plateaus at a high value
• Validation error decreases but plateaus at a similar high value
• Small gap between curves
• Both errors remain high even with more data

Why this happens: The model is too simple (e.g., linear model for non-linear data). It can't capture the underlying pattern, so:

1. Even on training data, it makes systematic errors (high training error)
2. The validation error is similarly high because the same systematic errors occur
3. More data doesn't help—the model lacks capacity

Derivation of plateau behavior:

For a model with bias error $\epsilon_{bias}$, as $m$ increases:

$J_{train}(m) \approx \epsilon_{bias} + \frac{\sigma^2}{m}$

where $\sigma^2$ is noise variance. As $m \to \infty$:

$\lim_{m \to \infty} J_{train}(m) = \epsilon_{bias}$

Similarly for validation:

$\lim_{m \to \infty} J_{val}(m) = \epsilon_{bias} + \epsilon_{variance}$

If $\epsilon_{bias}$ is large (simple model, complex data), both plateau high.

Fix: Increase model complexity, add features, reduce regularization

Pattern 2: High Variance (Overfitting)

What you see:

• Training error remains very low (often near zero)
• Validation error is much higher with a large gap
• Gap persists or slowly decreases as $m$ increases
• Validation error still decreasing at the end (hasn't plateaued)

Why this happens: The model is too complex (e.g., high-degree polynomial, deep network with little data). It:

1. Memorizes training data perfectly (low training error)
2. Fails to generalize because it learned noise and specifics (high validation error)
3. More data helps the model learn the true pattern instead of noise

Derivation of the gap:

For a complex model, the training error approaches the irreducible error $\epsilon_{irreducible}$ (noise):

$J_{train}(m) \approx \epsilon_{irreducible}$

But validation error has an additional variance penalty that decreases with $m$:

$J_{val}(m) \approx \epsilon_{irreducible} + \epsilon_{variance} \cdot \frac{k}{m}$

where $k$ relates to model complexity. The gap:

$\text{Gap}(m) = J_{val}(m) - J_{train}(m) \approx \epsilon_{variance} \cdot \frac{k}{m}$

This gap shrinks as $m$ increases, but slowly if $k$ (complexity) is high.

Fix: Get more training data, reduce model complexity, increase regularization, use dropout/augmentation

Pattern 3: Good Fit

What you see:

• Training and validation errors converge to a low value
• Small gap between curves
• Both errors have plateaued

Why this happens: The model complexity matches the data complexity. It learns the true underlying pattern without memorizing noise.

Degree 1 (Linear):

Training set size: [10, 30, 50, 70, 100]
Training error:    [45, 48, 50, 51, 52]  (in $1000s squared)
Validation error:  [50, 52, 53, 53, 54]

Analysis:

• Both errors high (~$50K-54K squared)
• Small gap (2-3K)
• Plateaued early
• Diagnosis: High bias—linear model too simple for housing data

Why this step? We compare final values and gap size. High final values + small gap = underfitting.

Degree 10:

Training set size: [10, 30, 50, 70, 100]
Training error:    [0.1, 2 5, 8, 10]
Validation error:  [80, 60, 45, 35, 28]

Analysis:

• Training error very low (10K squared)
• Validation error much higher (28K squared)
• Large gap (18K) even at m=100
• Validation error still decreasing
• Diagnosis: High variance—polynomial overfits

Why this step? Large persistent gap + training error near zero + validation still decreasing = more data would help.

Optimal Degree 3:

Training set size: [10, 30, 50, 70, 100]
Training error:    [5, 12, 15, 16, 17]
Validation error:  [15, 18, 19, 18, 18]

Analysis:

• Both converge to ~17-18K squared
• Small gap (~K)
• Both plateaued
• Diagnosis: Good fit

With 500 images:

• Training accuracy: 99%
• Validation accuracy: 65%
• Diagnosis: High variance (34% gap). Action: Collect more data or add regularization.

After collecting 5000 images:

• Training accuracy: 95%
• Validation accuracy: 88%
• Diagnosis: Better! Gap reduced to 7%. If validation accuracy acceptable, stop. If need higher accuracy AND gap still large, collect more data.

Why this step? We monitor whether additional data closes the gap and improves validation performance. The7% gap suggests mild overfitting remains but is much improved.

Depth = 1: Learning curves show both errors at40% (accuracy = 60% on both sets).

• Diagnosis: High bias—tree too shallow.
• Action: Increase max_depth.

Depth = 20 (unlimited): Learning curves show training error at 2% but validation error at 35%.

• Diagnosis: High variance—tree memorizes training data.
• Action: Limit depth, prune tree, or use ensemble methods.

Why this step? Tree depth directly controls model complexity. Learning curves guide us to the right complexity level.

Common Mistakes

Why it feels right: More data often does help with overfitting, and it's a common recommendation.

The reality: If you have high bias (both errors plateaued high), more data won't help! Your model lacks capacity to learn the pattern.

Example: You use linear regression for clearly non-linear data. Curves show:

• Training error: 50% (plateaued)
• Validation error: 52% (plateaued)

Collecting 100x more data will keep both errors around 50-52%. The model simply can't represent the relationship.

Fix: First diagnose with learning curves. If high bias, increase model complexity instead of collecting data.

Steel-man: The confusion arises because "more data is better" is true for high variance. The key is diagnosing which problem you have first.

Why it feels right: We expect "more data = better performance," so rising error seems backwards.

The reality: Training error should increase with more data—it's healthy! With few examples, your model can memorize perfectly (zero error). With more data, it can't fit everything perfectly and training error rises to reflect the true difficulty.

What matters: Where validation error goes. If validation error decreases while training error increases, you're learning better generalizations.

Fix: Always plot BOTH curves. Focus on validation error and the gap between curves, not training error alone.

Why it feels right: After collecting 1000 examples, you expect results.

The reality: With high variance, the validation error curve often hasn't plateaued yet. It's still decreasing significantly, indicating more data would continue to help.

How to check: Look at the slope of the validation curve. If it's still steeply decreasing at your largest $m$, extrapolate: you likely need much more data.

Fix:

1. Check if validation error is still decreasing steeply
2. Extrapolate to estimate how much data you'd need for satisfactory error
3. If infeasible, reduce model complexity instead

Decision Framework

Plot learning curves → Observe pattern at largest m:

Pattern A: Both errors HIGH, CONVERGED, small gap
├→ Diagnosis: High bias
├→ More data? NO
└→ Action: Increase complexity, add features, reduce regularization

Pattern B: Training error LOW, validation HIGH LARGE gap, validation still decreasing
├→ Diagnosis: High variance  
├→ More data? YES (if feasible)
└→ Alternative: Reduce complexity, increase regularization, augmentation

Pattern C: Both errors LOW, CONVERGED, small gap
├→ Diagnosis: Good fit
└→ Action: Done, or optimize hyperparameters

Pattern D: Both errors HIGH, but validation BELOW training near start
├→ Diagnosis: Unusual—check data leakage or implementation bugs

Practical Implementation

How to Generate Learning Curves

Algorithm:

For each m in [10, 20, 50, 100, 200, 500, 1000, ...]:
    1. Sample m examples from training set
    2. Train model on these examples
    3. Evaluate on ALL validation examples
    4. Compute J_train(m) on the m training examples
    5. Compute J_val(m) on validation set
    6. Store (m, J_train(m), J_val(m))
Plot both curves against m

Why sample the same set size multiple times? With small $m$, performance varies depending on which examples you sample. Average over multiple random samples for each $m$ value to get stable curves.

Computational Cost

For $k$ different training set sizes and a model that takes $O(m^2)$ to train:

$\text{Total cost} = \sum_{i=1}^{k} O(m_i^2) = O(k \cdot m_{max}^2)$

Why this step? Smaller training sets train faster, so generating curves is often faster than training on the full dataset many times.

Memory aid: "BIAS Both Are Stuck" (both errors stuck high), "VARIANCE Validation Awful, Still Reducing" (gap large, validation improving)

Scenario 1 (High Bias): You only use stick figures. Whether you practice on 10 pictures or 1000, your drawings look equally bad on both practice and new pictures. You're stuck! You need to learn a better drawing technique (more complex model), not just practice more.

Scenario 2 (High Variance): You trace the practice pictures perfectly, but when given new pictures, you can't draw them well. You memorized the practice pictures instead of learning how to draw faces in general. The solution? Practice on MORE different pictures so you learn the general skill, not just specific pictures.

Scenario 3 (Good Fit): Your practice drawings and new drawings both look pretty good and similar quality. You've learned the skill!

Learning curves help you figure out which scenario you're in, so you know whether to practice more (get more data) or learn better techniques (change your model).

Connections

• Bias-Variance Tradeoff: Learning curves visualize bias-variance directly
• Train-Test Split: Requires proper validation set to plot curves
• Cross-Validation: Can use CV scores instead of single validation set for more stable curves
• Regularization: High variance diagnosis leads to increased regularization
• Feature Engineering: High bias diagnosis leads to more/better features
• Model Complexity: Learning curves guide complexity selection
• Overfitting Detection: Primary tool for detecting overfitting
• Sample Size Determination: Predicts how much data you need

#flashcards/ai-ml

What does a learning curve plot? :: A learning curve plots training error and validation error against the size of the training set, showing how model performance changes as you add more training examples.

In a high variance scenario, what happens to training and validation errors? :: Training error stays very LOW (near zero), validation error is much HIGHER, creating a large gap. More data would help close this gap.

Concept Map

Hinglish (regional understanding)