Learning curves analysis

2.6.12 · AI-ML › Model Evaluation & Selection

Core Concept

Ise guitar seekhne jaisa socho: agar tum mahino tak wahi teen chords practice karne ke baad bhi simple songs nahi baja paate (high bias), toh tumhe ek behtar learning method chahiye. Agar tum woh teen songs perfectly baja lete ho lekin kisi bhi nayi cheez par freeze ho jaate ho (high variance), toh tumhe zyada diverse material practice karna hoga.

Mathematical Foundation

Errors Is Tarah Behave Kyun Karti Hain

Training Error ka Behavior: Jaise jaise training set size $m$ badhti hai, training error typically badhti hai kyunki:

$J_{train}(m)\text{ increases with }m$

Kyun? Kam examples ke saath, ek simple model bhi unhe perfectly memorize kar sakta hai (zero training error). Jaise jaise aap zyada diverse examples add karte ho, model sab ko perfectly fit nahi kar sakta, toh training error badhti hai.

Validation Error ka Behavior: Jaise jaise training set size $m$ badhti hai, validation error typically ghat'ti hai kyunki:

$J_{val}(m)\text{ decreases with }m$

Kyun? Kam training examples ke saath, model underlying pattern ki ek poor representation seekhta hai—usse kam information milti hai. Zyada data use better generalize karne mein help karta hai, validation error ko reduce karta hai.

$J_{train}(m)=\frac{1}{m}{\sum }_{i=1}^{m}L(h_{\theta }(x^{(i)}),y^{(i)})$

$J_{val}(m)=\frac{1}{m_{val}}{\sum }_{i=1}^{m_{val}}L(h_{\theta }(x_{val}^{(i)}),y_{val}^{(i)})$

Jab $m\to \infty$:

• High bias: $J_{train}(m)\to {ϵ}_{high}$ aur $J_{val}(m)\to {ϵ}_{high}$, jahan dono high error par converge karte hain
• High variance: $J_{train}(m)\to {ϵ}_{low}$ lekin $J_{val}(m)\gg J_{train}(m)$ bade gap ke saath

Diagnostic Patterns

Pattern 1: High Bias (Underfitting)

Jo tum dekhte ho:

• Training error jaldi badhti hai aur ek high value par plateau karti hai
• Validation error ghat'ti hai lekin ek similarly high value par plateau karti hai
• Curves ke beech chhota gap
• Zyada data ke saath bhi dono errors high rehti hain

Ye kyun hota hai: Model bahut simple hai (jaise, non-linear data ke liye linear model). Ye underlying pattern capture nahi kar sakta, isliye:

1. Training data par bhi ye systematic errors karta hai (high training error)
2. Validation error similarly high hai kyunki wahi systematic errors hote hain
3. Zyada data help nahi karta—model mein capacity nahi hai

Plateau behavior ki derivation:

Bias error ${ϵ}_{bias}$ wale model ke liye, jaise $m$ badhta hai:

$J_{train}(m)\approx {ϵ}_{bias}+\frac{{\sigma }^2}{m}$

jahan ${\sigma }^2$ noise variance hai. Jab $m\to \infty$:

${\lim }_{m\to \infty }{J}_{train}(m)={ϵ}_{bias}$

Similarly validation ke liye:

${\lim }_{m\to \infty }{J}_{val}(m)={ϵ}_{bias}+{ϵ}_{variance}$

Agar ${ϵ}_{bias}$ bada hai (simple model, complex data), toh dono high plateau karte hain.

Fix: Model complexity badhao, features add karo, regularization kam karo

Pattern 2: High Variance (Overfitting)

Jo tum dekhte ho:

• Training error bahut low rehti hai (aksar near zero)
• Validation error kaafi zyada hai aur bada gap hai
• Gap persist karta hai ya slowly decrease hota hai jaise $m$ badhta hai
• Validation error end mein bhi decrease ho rahi hai (plateau nahi ki hai)

Ye kyun hota hai: Model bahut complex hai (jaise, high-degree polynomial, kam data ke saath deep network). Ye:

1. Training data ko perfectly memorize karta hai (low training error)
2. Generalize karne mein fail karta hai kyunki usne noise aur specifics seekhe (high validation error)
3. Zyada data model ko noise ki jagah true pattern seekhne mein help karta hai

Gap ki derivation:

Ek complex model ke liye, training error irreducible error ${ϵ}_{irreducible}$ (noise) ke paas pahunchti hai:

$J_{train}(m)\approx {ϵ}_{irreducible}$

Lekin validation error mein ek additional variance penalty hoti hai jo $m$ ke saath ghat'ti hai:

$J_{val}(m)\approx {ϵ}_{irreducible}+{ϵ}_{variance}\cdot \frac{k}{m}$

jahan $k$ model complexity se related hai. Gap:

$\text{Gap}(m)=J_{val}(m)-J_{train}(m)\approx {ϵ}_{variance}\cdot \frac{k}{m}$

Ye gap $m$ badhne ke saath shrink karta hai, lekin slowly agar $k$ (complexity) high hai.

Fix: Zyada training data lo, model complexity kam karo, regularization badhao, dropout/augmentation use karo

Pattern 3: Good Fit

Jo tum dekhte ho:

• Training aur validation errors ek low value par converge karti hain
• Curves ke beech chhota gap
• Dono errors plateau kar chuki hain

Ye kyun hota hai: Model complexity, data complexity se match karti hai. Ye noise ko memorize kiye bina true underlying pattern seekhta hai.

Degree 1 (Linear):

Training set size: [10, 30, 50, 70, 100]
Training error:    [45, 48, 50, 51, 52]  (in $1000s squared)
Validation error:  [50, 52, 53, 53, 54]

Analysis:

• Dono errors high (~$50K-54K squared)
• Chhota gap (2-3K)
• Jaldi plateau kar gaya
• Diagnosis: High bias—linear model housing data ke liye bahut simple hai

Ye step kyun? Hum final values aur gap size compare karte hain. High final values + chhota gap = underfitting.

Degree 10:

Training set size: [10, 30, 50, 70, 100]
Training error:    [0.1, 2 5, 8, 10]
Validation error:  [80, 60, 45, 35, 28]

Analysis:

• Training error bahut low (10K squared)
• Validation error kaafi zyada (28K squared)
• m=100 par bhi bada gap (18K)
• Validation error abhi bhi decrease ho rahi hai
• Diagnosis: High variance—polynomial overfit kar raha hai

Ye step kyun? Bada persistent gap + training error near zero + validation abhi bhi decrease ho rahi hai = zyada data help karega.

Optimal Degree 3:

Training set size: [10, 30, 50, 70, 100]
Training error:    [5, 12, 15, 16, 17]
Validation error:  [15, 18, 19, 18, 18]

Analysis:

• Dono ~17-18K squared par converge karte hain
• Chhota gap (~K)
• Dono plateau kar chuke hain
• Diagnosis: Good fit

500 images ke saath:

• Training accuracy: 99%
• Validation accuracy: 65%
• Diagnosis: High variance (34% gap). Action: Zyada data collect karo ya regularization add karo.

5000 images collect karne ke baad:

• Training accuracy: 95%
• Validation accuracy: 88%
• Diagnosis: Behtar! Gap 7% tak kam ho gaya. Agar validation accuracy acceptable hai, ruk jao. Agar higher accuracy chahiye AUR gap abhi bhi bada hai, toh zyada data collect karo.

Ye step kyun? Hum monitor karte hain ki additional data gap ko close karta hai aur validation performance improve karta hai ya nahi. 7% gap suggest karta hai ki mild overfitting remain hai lekin kaafi improve hua hai.

Depth = 1: Learning curves dono errors ko 40% par dikhate hain (accuracy = 60% dono sets par).

• Diagnosis: High bias—tree bahut shallow hai.
• Action: max_depth badhao.

Depth = 20 (unlimited): Learning curves training error ko 2% par lekin validation error ko 35% par dikhate hain.

• Diagnosis: High variance—tree training data memorize karta hai.
• Action: Depth limit karo, tree prune karo, ya ensemble methods use karo.

Ye step kyun? Tree depth directly model complexity control karta hai. Learning curves hume sahi complexity level tak guide karti hain.

Common Mistakes

Ye sahi kyun lagta hai: Zyada data aksar overfitting ke saath help karta hai, aur ye ek common recommendation hai.

Reality: Agar tumhe high bias hai (dono errors high plateau kiye hue), toh zyada data help nahi karega! Tumhare model mein pattern seekhne ki capacity nahi hai.

Example: Tum clearly non-linear data ke liye linear regression use karte ho. Curves dikhate hain:

• Training error: 50% (plateaued)
• Validation error: 52% (plateaued)

100x zyada data collect karne se dono errors 50-52% ke aas paas hi rahenge. Model simply us relationship ko represent nahi kar sakta.

Fix: Pehle learning curves se diagnose karo. Agar high bias hai, toh data collect karne ki jagah model complexity badhao.

Steel-man: Confusion isliye hoti hai kyunki "zyada data better hai" high variance ke liye sach hai. Key hai ki pehle diagnose karo ki tumhara konsa problem hai.

Ye sahi kyun lagta hai: Hum expect karte hain "zyada data = better performance," toh badhti error ulta lagti hai.

Reality: Training error ka data badhne ke saath badhna chahiye—ye healthy hai! Kam examples ke saath, tumhara model perfectly memorize kar sakta hai (zero error). Zyada data ke saath, ye sab ko perfectly fit nahi kar sakta aur training error badhti hai jo true difficulty ko reflect karta hai.

Jo matter karta hai: Validation error kahaan jaati hai. Agar training error badhne ke saath validation error ghat'ti hai, toh tum better generalizations seekh rahe ho.

Fix: Hamesha DONO curves plot karo. Validation error aur curves ke beech gap par focus karo, sirf training error par nahi.

Ye sahi kyun lagta hai: 1000 examples collect karne ke baad, tum results expect karte ho.

Reality: High variance ke saath, validation error curve aksar plateau nahi ki hoti. Ye abhi bhi significantly decrease ho rahi hoti hai, jo indicate karta hai ki zyada data continue help karega.

Kaise check karo: Validation curve ka slope dekho. Agar ye tumhare sabse bade $m$ par abhi bhi steeply decrease ho rahi hai, extrapolate karo: tumhe likely kaafi zyada data chahiye.

Fix:

1. Check karo ki validation error abhi bhi steeply decrease ho rahi hai
2. Extrapolate karo ki satisfactory error ke liye kitna data chahiye
3. Agar infeasible hai, toh model complexity kam karo

Decision Framework

Plot learning curves → Sabse bade m par pattern dekho:

Pattern A: Dono errors HIGH, CONVERGED, chhota gap
├→ Diagnosis: High bias
├→ Zyada data? NAHI
└→ Action: Complexity badhao, features add karo, regularization kam karo

Pattern B: Training error LOW, validation HIGH bada gap, validation abhi bhi decrease ho rahi hai
├→ Diagnosis: High variance  
├→ Zyada data? HAAN (agar feasible ho)
└→ Alternative: Complexity kam karo, regularization badhao, augmentation

Pattern C: Dono errors LOW, CONVERGED, chhota gap
├→ Diagnosis: Good fit
└→ Action: Ho gaya, ya hyperparameters optimize karo

Pattern D: Dono errors HIGH, lekin validation BELOW training shuru ke paas
├→ Diagnosis: Unusual—data leakage ya implementation bugs check karo

Practical Implementation

Learning Curves Kaise Generate Karein

Algorithm:

For each m in [10, 20, 50, 100, 200, 500, 1000, ...]:
    1. Training set se m examples sample karo
    2. In examples par model train karo
    3. SAARE validation examples par evaluate karo
    4. Un m training examples par J_train(m) compute karo
    5. Validation set par J_val(m) compute karo
    6. (m, J_train(m), J_val(m)) store karo
Dono curves ko m ke against plot karo

Ek hi set size ke liye multiple baar sample kyun karte hain? Chhote $m$ ke saath, performance depend karta hai ki tum kaun se examples sample karte ho. Stable curves paane ke liye har $m$ value ke liye multiple random samples ka average lo.

Computational Cost

$k$ alag alag training set sizes aur ek model ke liye jo $O(m^2)$ mein train hota hai:

$\text{Total cost}={\sum }_{i=1}^{k}O(m_i^2)=O(k\cdot m_{max}^2)$

Ye step kyun? Chhote training sets jaldi train hote hain, isliye curves generate karna aksar full dataset par kaafi baar train karne se faster hota hai.

Memory aid: "BIAS Both Are Stuck" (dono errors high par stuck hain), "VARIANCE Validation Awful, Still Reducing" (gap bada hai, validation improve ho rahi hai)

Scenario 1 (High Bias): Tum sirf stick figures use karte ho. Chahe tum 10 pictures par practice karo ya 1000 par, tumhare drawings practice aur nayi dono pictures par equally bure lagte hain. Tum stuck ho! Tumhe better drawing technique seekhni hai (more complex model), sirf zyada practice nahi.

Scenario 2 (High Variance): Tum practice pictures ko perfectly trace karte ho, lekin jab nayi pictures di jaati hain, tum unhe achhe se nahi bana paate. Tumne practice pictures memorize ki instead of faces generally kaise draw karte hain ye seekhne ke. Solution? AUR zyada alag alag pictures par practice karo taaki tum general skill seekho, sirf specific pictures nahi.

Scenario 3 (Good Fit): Tumhari practice drawings aur nayi drawings dono achhi lagrahi hain aur similar quality ki hain. Tumne skill seekh li!

Learning curves tumhe figure out karne mein help karti hain ki tum kaunse scenario mein ho, taaki tum jaano ki zyada practice karni hai (zyada data lena) ya better techniques seekhni hain (apna model change karna).

Connections

• Bias-Variance Tradeoff: Learning curves bias-variance ko directly visualize karti hain
• Train-Test Split: Curves plot karne ke liye proper validation set chahiye
• Cross-Validation: Zyada stable curves ke liye single validation set ki jagah CV scores use kar sakte hain
• Regularization: High variance diagnosis se regularization badhane ki zaroorat hoti hai
• Feature Engineering: High bias diagnosis se zyada/behtar features ki zaroorat hoti hai
• Model Complexity: Learning curves complexity selection guide karti hain
• Overfitting Detection: Overfitting detect karne ka primary tool
• Sample Size Determination: Predict karta hai ki tumhe kitna data chahiye

#flashcards/ai-ml

Learning curve kya plot karta hai? :: Ek learning curve training error aur validation error ko training set ke size ke against plot karta hai, dikhata hai ki zyada training examples add karne se model performance kaise change hoti hai.

High variance scenario mein training aur validation errors ka kya hota hai? :: Training error bahut LOW rehti hai (near zero), validation error kaafi zyada HIGHER hoti hai, ek bada gap create karta hai. Zyada data is gap ko close karne mein help karega.

Concept Map