Underfitting vs overfitting diagnosis — Hinglish notes

2.6.2 · AI-ML › Model Evaluation & Selection

Overview

Machine learning ka central challenge yeh hai: your model must generalize to unseen data. Underfitting aur overfitting — yeh do failure modes hain, aur diagnose karna ki aapke paas kaunsa hai, yeh determine karta hai aapka agla step. Yeh note aapko sikhata hai ki signs kaise padhein aur action kaise lein.

Core Concepts

Diagnostic Framework: The Three Lenses

1. Learning Curves Analysis

Formula Learning Curves: Error vs Training Set Size

Yeh kaam kyon karta hai: Jaise aap zyada training examples add karte ho, patterns zyada clearly emerge hote hain (overfitting reduce hota hai) lekin aapke model ki limitations bhi apparent ho jaati hain (underfitting expose hoti hai). Kya plot karein:

J_{t r ain} (m) J_{c v} (m) = \frac{1}{m} i = 1 \sum m L (h_{θ} (x^{(i)}), y^{(i)}) = \frac{1}{m _{c v}} i = 1 \sum m_{c v} L (h_{θ} (x_{c v}^{(i)}), y_{c v}^{(i)})

Jahan $m$ = use kiye gaye training examples ki sankhya, $m_{c v}$ = cross-validation set size.

Kaise interpret karein:

Scenario	$J_{t r ain}$ behavior	$J_{c v}$ behavior	Gap	Diagnosis
Underfitting	High plateau	High plateau	Small	High bias
Overfitting	Low throughout	High, slow convergence	Large	High variance
Good fit	Low, gradual rise	Decreases, converges near $J_{t r ain}$	Small	Just right

Worked example Learning Curves: Polynomial Regression

Setup: $y = 3 x^{2} + 2 x + 1 + ϵ$ fit karna (true quadratic relationship)

Case 1 - Underfitting (Linear model $h_{θ} (x) = θ_{0} + θ_{1} x$ ):

m=10:   J_train=8.2, J_cv=8.5  
m=100:  J_train=8.1, J_cv=8.3  
m=1000: J_train=8.0, J_cv=8.1

Yeh pattern kyon? Linear model quadratic curve ko represent nahi kar sakta. Zyada data add karna help nahi karta kyunki model ki capacity fundamentally insufficient hai. Dono errors ek high value par plateau kar jaati hain.

Case 2 - Overfitting (Degree-15 polynomial):

m=10:   J_train=0.01, J_cv=156.3  → 10 points perfectly memorize kar liye
m=100:  J_train=0.05, J_cv=45.2   → Abhi bhi noise mein wiggling
m=1000: J_train=0.8,  J_cv=12.1   → Constraints generalization force karte hain

Yeh pattern kyon? Kam examples ke saath, 15-degree polynomial wildly twist karta hai har training point ko hit karne ke liye, noise capture karta hai. Jaise $m$ badhta hai, model zyada data points se constrained ho jaata hai aur true quadratic trend seekhna padta hai. Gap shrink hoti hai lekin slowly.

Case 3 - Just Right (Quadratic model):

m=10:   J_train=0.9, J_cv=1.2  
m=100:  J_train=1.0, J_cv=1.1  
m=1000: J_train=1., J_cv=1.0

Yeh pattern kyon? Model capacity problem se match karti hai. Dono errors irreducible error $ϵ$ (data mein noise) par converge hoti hain. Gap puri tarah chhota rehta hai.

2. Training vs Validation Error Gap

Formula The Generalization Gap

First principles se derivation: Expected test error se shuru karein:

E [Test Error] = Bias^{2} + Variance + Irreducible Error

Yeh decomposition kyon? Error source ke hisaab se breakdown karein:

Bias²: Wrong model assumptions se error → training error ko affect karta hai
Variance: Training set ke sensitivity se error → train/test gap create karta hai
Irreducible: Data mein noise → minimum possible error

Training error mainly bias reflect karta hai (aur partial variance):

E [Train Error] \approx Bias^{2} + α \cdot Variance + Irreducible Error

(jahan $α < 1$ kyunki model is set par optimize kiya gaya hai)

Isliye, gap:

Gap = E [Test Error] - E [Train Error] \approx (1 - α) \cdot Variance

The diagnostic:

Large gap → High variance (overfitting)
Small gap + high errors → High bias (underfitting)

Worked example Gap Analysis: Image Classification

Scenario: Neural network se cat vs dog classifier banana

Experiment 1 - Too shallow (2 layers, 10 neurons each):

Training accuracy:   68%  →  Error = 32%
Validation accuracy: 65%  →  Error = 35%
Gap: 3%

Diagnosis: UNDERFITTING (high bias) Kyon? Dono errors high hain (human-level ~5% se bahut door), lekin gap chhota hai. Model training data ko bhi achhi tarah fit nahi kar sakta. Action: Capacity badhao (zyada layers/neurons)

Experiment 2 - Too deep (20 layers, no regularization):

Training accuracy:   99%  →  Error = 1%
Validation accuracy: 78%  →  Error = 22%
Gap: 21%

Diagnosis: OVERFITTING (high variance) Kyon? Training error excellent hai lekin validation error bahut zyada kharab hai. Model ne training peculiarities memorize kar li hain. Action: Regularization add karo (dropout, L2), zyada data lao, ya capacity reduce karo

Experiment 3 - Just right (8 layers, dropout 0.3, L2 reg):

Training accuracy:   94%  →  Error = 6%
Validation accuracy: 92%  →  Error = 8%
Gap: 2%

Diagnosis: Good fit Kyon? Dono errors human-level ke paas hain, gap minimal hai. Model ne generalizable features seekhe hain.

3. Error Components Analysis

Worked example Error Budget: Speech Recognition

Context: Speech-to-text system banana

Human-level error: $ϵ^{*} = 2%$ (professional transcribers)

Iteration 1:

Training error:    15%
Validation error:  18%
Avoidable bias = 15% - 2% = 13%  ← PRIMARY PROBLEM
Variance       = 18% - 15% = 3%

Diagnosis: Severe underfitting Bias par focus kyon? Aap sirf model limitations ki wajah se 13% chod rahe ho, jabki overfitting se sirf 3%. Action liya: Shallow RNN se deep Transformer par switch kiya (capacity badhaya)

Iteration 2 (capacity increase ke baad):

Training error:    4%
Validation error:  12%

Avoidable bias = 4% - 2% = 2%
Variance       = 12% - 4% = 8%   ← PRIMARY PROBLEM

Diagnosis: Ab overfitting bottleneck hai Shift kyon? High-capacity model ab training data achhi tarah fit kar leta hai (human-level ke paas) lekin generalize nahi karta. Action liya: Data augmentation add kiya (speed perturbation, noise injection), dropout apply kiya

Iteration 3 (regularization ke baad):

Training error:    3%
Validation error:  4%

Avoidable bias = 3% - 2% = 1%
Variance       = 4% - 3% = 1%

Diagnosis: Balanced, near-optimal Yeh achha kyon hai? Dono error sources roughly equal aur chhote hain. Aage gains ke liye better features ya zyada data chahiye hoga.

Diagnosis Decision Tree

Recall The Diagnosis Algorithm (Explain to a 12-year-old)

Socho tum darts phenkna seekh rahe ho: Step 1: Apna training board dekho (practice throws)

Kya tum bullseye se bahut zyada miss kar rahe ho? → Tumne basics nahi seekhe (underfitting)
Kya tum bullseye perfectly hit kar rahe ho? → Step 2 par jao

Step 2: Test board dekho (naya game)

Kya tum abhi bhi achhi tarah hit kar rahe ho? → Tum skilled ho! (good fit)
Kya tum buri tarah miss kar rahe ho? → Tumne woh specific practice board memorize kar liya, real dart skills nahi seekhi (overfitting)

The fix:

Agar step 1 fail hua: Zyada basics practice karo, apna stance improve karo (model capacity badhao)
Agar step 2 fail hua: Kai alag-alag boards par practice karo, sirf ek par nahi (zyada data lao, regularization use karo)

Formal decision tree:

START: Train your model
    ↓
Is J_train high (>> ε*)?
    ├─ YES → UNDERFITTING
    │   └─ Action: Increase capacity, remove regularization,
    │              engineer features, train longer
    │
    └─ NO → Is (J_cv - J_train) large?
        ├─ YES → OVERFITTING
        │   └─ Action: More data, regularization (L1/L2/dropout),
        │              reduce capacity, data augmentation
        │
        └─ NO → Is J_cv acceptable (≈ ε*)?
            ├─ YES → GOOD FIT (done!)
            └─ NO → HIGH IRREDUCIBLE ERROR
                └─ Action: Better features, cleaner data,
                           redefine problem

Common Mistakes

Common mistake Mistake 1: "Zyada training hamesha overfitting mein help karti hai"

Galat intuition: "Agar mera model bahut flexible hai aur overfit kar raha hai, toh mujhe ise stabilize karne ke liye zyada train karna chahiye."

Yeh sahi kyon lagta hai: Zyada train karna "zyada practice" jaisa lagta hai jo cheezein improve karni chahiye.

Sachchi baat: Same data par zyada train karna overfitting ko worse banata hai. Model ko training-set noise memorize karne ke liye zyada time milta hai.

Jo actually overfitting mein help karta hai:

Zyada data (same data par zyada epochs nahi)
Regularization (complexity par penalty)
Early stopping (jab validation error badhne lage, training rok do)

Yeh kyon kaam karta hai: Zyada data zyada constraints provide karta hai (model sab memorize nahi kar sakta). Regularization directly complexity ko penalize karta hai. Early stopping memorization dominate hone se pehle sweet spot pakad leta hai.

Common mistake Mistake 2: "Chhota train-test gap matlab achha model"

Galat intuition: "Meri training aur validation errors dono 40% hain, aur woh ek doosre ke karib hain. Yeh consistent hai, toh achha hai!"

Yeh sahi kyon lagta hai: Consistency = reliability, aur model overfit nahi kar raha (chhota gap).

Sachchi baat: ==Chhota gap lekin high errors = severe underfitting==. Aapka model "consistently bura" hai.

The fix: Errors ko baseline se compare karo:

If J_train ≈ J_cv but both >> ε*  → Underfitting (increase capacity)
If J_train≈ J_cv and both ≈ ε*   → Good fit (done!)

Example:

Task par human error: 2%
Aapka model: J_train=40%, J_cv=42%
Gap sirf 2% hai, lekin aap 38% table par chod rahe ho! Underfitting.

Common mistake Mistake 3: "Diagnose karne ke liye sirf validation accuracy use karo"

Galat intuition: "Agar validation accuracy 90% hai, toh mera model great hai!"

Yeh sahi kyon lagta hai: High accuracy akele mein achhi lagti hai.

Sachchi baat: Diagnose karne ke liye DONO training aur validation metrics chahiye. High validation accuracy ka matlab ho sakta hai:

Good fit (agar training accuracy bhi ~90% hai)
Potentially better solution ko underfit karna (agar training 95% hai, toh aapke paas better ke liye capacity hai)
Task easy hai (agar human-level 99% hai, toh aapka 90% underfitting hai)

The complete picture:

# Always check this trio:
baseline_error = 2%      # Human-level or theoretical minimum
train_error = 3%         # Your training error
val_error = 5%           # Your validation error
# Now diagnose:
if train_error >> baseline_error:
   print("Underfitting - model can't even fit training data")
elif (val_error - train_error) > threshold:
   print("Overfitting - big gap between train and val")
else:
   print("Good fit - near baseline with small gap")

Practical Diagnostic Checklist

Quick Diagnosis (30 seconds)

Compute: $J_{t r ain}$ , $J_{c v}$ , baseline $ϵ^{*}$
Check:
- Kya $J_{t r ain} ≫ ϵ^{*}$ hai? → Underfitting
- Kya $(J_{c v} - J_{t r ain})$ bada hai (>10% of $ϵ^{*}$ )? → Overfitting
Act: Upar diya decision tree dekho

Deep Diagnosis (jab quick check unclear ho)

Learning curves plot karo: Error vs $m$ (training set size)
- Underfitting: Dono curves high plateau karti hain
- Overfitting: Bada persistent gap
Validation curves plot karo: Error vs model complexity
- Underfitting: Complexity badhne par error decrease hoti hai
- Overfitting: Training error decrease hoti hai, validation error increase hoti hai (U-shape)

Error budget compute karo:

Avoidable bias = J_train - ε*
Variance = J_cv - J_train
Bade component par focus karo (80/20 rule)

Sanity checks:
- Kya aapka model ek single batch perfectly fit kar sakta hai? (Nahi → architecture issue)
- Kya validation error training ke dauran bilkul bhi decrease hoti hai? (Nahi → data/feature issue)
- Kya training loss abhi bhi decrease ho raha hai? (Haan → zyada training se benefit ho sakta hai)

Action Playbook

To Fix Underfitting (High Bias)

Action	Kyon Kaam Karta Hai	Kab Use Karein
Model capacity badhao	Model ko complex patterns represent karne deta hai	$J_{t r ain}$ high, gap chhota
Polynomial features add karo	Non-linear relationships capture karta hai	Curved data par linear model
Regularization reduce karo	Model flexibility par constraints hatata hai	$λ$ bahut zyada hai
Zyada train karo	Optimization ko converge hone deta hai	Loss abhi bhi decrease ho raha hai
Dropout/L2 hatao	Regularization reduce karne jaisa hi hai	Regularization bahut aggressive hai

To Fix Overfitting (High Variance)

Action	Kyon Kaam Karta Hai	Kab Use Karein
Zyada training data lao	Zyada constraints provide karta hai	$(J_{c v} - J_{t r ain})$ bada
Data augmentation	Artificially data diversity badhata hai	Real data nahi mil sakta
Regularization add karo (L1/L2)	Model complexity ko penalize karta hai	Gap bada, enough data
Dropout add karo	Redundant representations force karta hai	Deep neural networks
Model capacity reduce karo	Memorization ability limit karta hai	Severe overfitting
Early stopping	Memorization dominate hone se pehle rok deta hai	Validation error U-shaped
Ensemble methods	Individual model variance average out karta hai	Production system

Connections

2.6.01-Bias-variance-tradeoff - Theoretical foundation ki underfitting/overfitting kyon hoti hai
2.6.03-Learning-curves-interpretation - Curve patterns ka detailed analysis
2.6.04-Cross-validation-techniques - Diagnosis ke liye robust error estimation
2.6.05-Regularization-methods - Overfitting se ladne ke tools
3.2.01-Neural-network-capacity - Architecture fit ko kaise affect karta hai
4.1.02-Data-augmentation-strategies - Synthetic data se high variance address karna

Flashcards

#flashcards/ai-ml

Model generalization ke do failure modes kya hain? :: Underfitting (high bias) aur overfitting (high variance)

Underfitting ka mathematical signature kya hai?

High training error

J_{t r ain} ≫ ϵ^{*}

AUR chhota gap

(J_{c v} - J_{t r ain})

chhota hai

Overfitting ka mathematical signature kya hai?

Low training error

J_{t r ain} \approx 0

AUR bada gap

(J_{c v} - J_{t r ain}) ≫ 0

Learning curves mein kaunsa pattern underfitting indicate karta hai?

Dono training aur validation error curves HIGH value par plateau karti hain

Learning curves mein kaunsa pattern overfitting indicate karta hai?

Training (low) aur validation (high) error curves ke beech bada persistent gap, zyada data ke saath gap slowly decrease hoti hai

Avoidable bias ka formula kya hai?

Avoidable bias =

J_{t r ain} - ϵ^{*}

jahan

ϵ^{*}

human-level ya Bayes error hai

Error budget analysis mein variance ka formula kya hai?

Variance =

J_{c v} - J_{t r ain}

(the generalization gap)

Agar dono training aur validation errors 40% hain aur human-level 5% hai, toh diagnosis kya hai?

Underfitting (high bias) - chhota gap lekin dono errors baseline se bahut door hain

Agar training error 2% hai aur validation error 25% hai aur human-level 3% par hai, toh diagnosis kya hai?

Overfitting (high variance) - bada gap, training baseline ke paas lekin validation bahut upar

Underfitting fix karne ke teen tarike kya hain?

1) Model capacity badhao, 2) Polynomial/interaction features add karo, 3) Regularization reduce karo

Overfitting fix karne ke teen tarike kya hain?

1) Zyada training data lao, 2) Regularization add karo (L1/L2/dropout), 3) Model capacity reduce karo ya early stopping use karo

Zyada train karna overfitting mein help kyon nahi karta?

Same data par zyada train karna model ko noise memorize karne ke liye zyada time deta hai, overfitting worse ho jaata hai; tumhe zyada DATA chahiye na ki zyada epochs

Chhota train-test gap lekin high errors kya indicate karta hai?

Underfitting - model training data ko bhi fit karne mein "consistently bura" hai

Apne training error ko kis cheez se compare karna chahiye?

Ek baseline error

ϵ^{*}

(human-level performance ya theoretical minimum) se, avoidable bias measure karne ke liye

Error budget ke 80/20 rule ke according, effort kahan focus karna chahiye?

Jis par focus karo jo bada ho: avoidable bias (

J_{t r ain} - ϵ^{*}

) ya variance (

J_{c v} - J_{t r ain}

)