Bias-variance tradeoff

2.6.1 · AI-ML › Model Evaluation & Selection

Bias-variance tradeoff machine learning ka fundamental tension hai: simple models important patterns miss karte hain (high bias), jabki complex models noise memorize kar lete hain (high variance). Har model is spectrum pe kahin na kahin hota hai, aur humara kaam woh sweet spot dhundhna hai jahan total error minimize ho.

Core Intuition

ML mein: nonlinear data pe linear model ka high bias hota hai (galat functional form) lekin low variance hota hai (datasets ke across stable predictions). Deep neural net ka low bias hota hai (koi bhi pattern fit kar sakta hai) lekin high variance hota hai (training data ki quirks ke liye sensitive). Tradeoff model complexity ko finite, noisy data ke against balance karne se arise karta hai: sirf ek limited, noise-corrupted sample ke saath, zyada flexible model noise chase karta hai (variance) jabki stiffer model real structure ignore karta hai (bias).

Mathematical Derivation from First Principles

• Variance of $\hat{f}$ at $x$: how much predictions change across different training sets $\text{Var}(\hat{f}(x))={\mathbb{E}}_{\mathcal{D}}\left[(\hat{f}(x)-{\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]{)}^2\right]$

• Irreducible error: data mein noise $ϵ$, $\mathbb{E}[{ϵ}^2]={\sigma }^2$

Deriving the Decomposition

Expected prediction error se shuru karo (training sets $\mathcal{D}$ aur noise $ϵ$ dono ke over expected) single point $x$ pe:

Setup: True model hai $y=f(x)+ϵ$ jahan $\mathbb{E}[ϵ]=0$, $\text{Var}(ϵ)={\sigma }^2$.

Goal: ${\mathbb{E}}_{\mathcal{D},ϵ}\left[(y-\hat{f}(x){)}^2\right]$ ko decompose karo

Step 1: Squared error expand karo ${\mathbb{E}}_{\mathcal{D},ϵ}\left[(y-\hat{f}(x){)}^2\right]$

Yeh step kyun? Hum samajhna chahte hain ki prediction error mein kya contribute karta hai.

Step 2: $y=f(x)+ϵ$ substitute karo $={\mathbb{E}}_{\mathcal{D},ϵ}\left[(f(x)+ϵ-\hat{f}(x){)}^2\right]$

Kyun? True signal $f(x)$ ko noise $ϵ$ se alag karo.

Step 3: ${\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$ (average prediction) add aur subtract karo $={\mathbb{E}}_{\mathcal{D},ϵ}\left[{\left(f(x)-{\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]+{\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]-\hat{f}(x)+ϵ\right)}^2\right]$

Kyun? Yeh algebraic trick humein systematic error (bias) ko random fluctuation (variance) se alag karne deti hai. Hum ek "center point" bana rahe hain jisse deviations measure karein.

Step 4: Square expand karo ($(A+B+C{)}^2$) $={\mathbb{E}}_{\mathcal{D},ϵ}\left[{\left(f(x)-{\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]\right)}^2+{\left({\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]-\hat{f}(x)\right)}^2+{ϵ}^2+\text{cross terms}\right]$

Kyun? Humein error ke teen distinct sources isolate karne hain.

Step 5: Dikhao ki cross terms vanish ho jaate hain

• ${\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)-{\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]]=0$ expectation ki definition se
• ${\mathbb{E}}_{ϵ}[ϵ]=0$ assumption se
• Cross terms jaise $2(f(x)-\mathbb{E}[\hat{f}])\cdot \mathbb{E}[ϵ]=0$

Kyun? Centered random variables ke constants ya independent variables ke saath zero expected cross products hote hain.

Step 6: Expectations lo (note karo ki $f(x)$ aur ${\mathbb{E}}_{\mathcal{D}}[\hat{f}(x)]$ $\mathcal{D}$ pe depend nahi karte jab expectation le li jaaye)

Physical Meaning:

• Bias²: Teri average prediction truth se kitni door hai? (systematic miss)
• Variance: Individual predictions teri average ke aas-paas kitna scatter karti hain? (instability)
• Irreducible error: Woh noise jo tum eliminate nahi kar sakte (fundamental limit)

Why the Tradeoff?

Model complexity badhana (zyada parameters, deeper networks, higher polynomial degree):

• Bias decrease karta hai: Zyada flexible function true $f(x)$ ko better fit kar sakta hai
• Variance increase karta hai: Zyada parameters = specific dataset mein noise overfit karne ke zyada tarike
• Total error = Bias² + Variance ek U-shaped curve se guzarta hai

Optimal complexity is U ke bottom pe hoti hai.

Worked Examples

Compare karne ke liye models:

1. Degree 0 (constant): $\hat{f}(x)=c$
2. Degree 1 (linear): $\hat{f}(x)=a+bx$
3. Degree 9 (high polynomial): $\hat{f}(x)={\sum }_{j=0}^9{a}_j{x}^j$

Analysis:

| Model | Bias | Variance | Kyun? | |----|----------|---| | Degree 0 | High | Low | Constant sine wave capture nahi kar sakta (systematic error), lekin prediction stable hai (sabhi datasets ke liye same $c$) | | Degree 1 | High | Low | Linear curvature miss karta hai, lekin phir bhi stable hai | | Degree 9 | Low | Very High | Sine wave perfectly fit kar sakta hai, lekin 10 points pe 10 parameters → noise memorize karta hai, alag samples pe wildly different fits |

Degree 9 ka high variance kyun hai? 10 datapoints aur 10 parameters ke saath, polynomial har training point ko exactly interpolate karta hai. Agar noise kisi ek point ko upar bump kare, toh poori curve us se guzarne ke liye bend ho jaati hai. Ek nayi dataset pe jisme different noise ho, tum ek bilkul alag curve paate ho. Kai datasets ke over average (bias) true sine ke close ho sakta hai, lekin individual fits (jo tumhe actually milti hai) wildly swing karti hain.

Optimal choice: Degree 3-5 polynomial ya ~3 terms wala Fourier basis dono ke beech balance karta hai.

k=1 (nearest neighbor):

• Bias: Low (piecewise constant regions se koi bhi boundary bana sakta hai)
• Variance: Very high (ek noisy point poore local region ko flip kar deta hai, boundaries datasets ke beech drastically change hoti hain)
• Symptom: Decision boundary jagged hai, har training point ko follow karti hai including outliers

k=n (sabhi neighbors → majority class):

• Bias: Very high (local structure ignore karta hai, har jagah global majority predict karta hai)
• Variance: Zero (sabhi datasets ke liye same prediction)
• Symptom: Har jagah same class predict karta hai, circular boundary miss karta hai

k=5-20 (moderate):

• Bias: Moderate (noise ko smooth karta hai lekin phir bhi circle approximate kar sakta hai)
• Variance: Moderate (local averaging individual points ke liye sensitivity reduce karta hai)
• Is problem ke liye Optimal.

Yeh step (k choose karna) kyun? Hum model flexibility control kar rahe hain. Small k = high flexibility = low bias, high variance. Large k = low flexibility = high bias, low variance. Yeh tradeoff hyperparameter mein explicit hai.

Common Mistakes & Misconceptions

Error: Zyada data variance reduce karta hai (har datapoint ka individual influence kam hota hai) lekin bias reduce nahi karta (nonlinear data pe linear model kitna bhi sample size ho, biased rehta hai). Tradeoff bana rehta hai—tumhe phir bhi appropriate complexity choose karni hai.

Fix: Zyada data optimal complexity ko rightward shift karta hai (zyada complex models "afford" karne deta hai) lekin bias aur variance balance karne ki zaroorat eliminate nahi karta. Infinite data ke saath, variance →0, lekin bias tab tak rehta hai jab tak model class true function include nahi karti.

Steel-man: Intuition partially sahi hai—zyada data ke saath, optimal model indeed zyada complex hota hai (tum zyada parameters "afford" kar sakte ho). Lekin us nayi optimal point pe phir bhi tradeoff face karte ho.

Error: Regularization bias increase karta hai (parameters ko zero ki taraf pull karta hai, systematic error add karta hai) variance reduce karne ke liye (parameter space shrink karta hai, predictions stabilize karta hai). Yeh bias reduce nahi karta—yeh variance ke liye trade karta hai.

Fix: Regularization ko ek "complexity knob" ki tarah socho jo bias-variance tradeoff tune karta hai. Stronger regularization = simpler effective model = higher bias, lower variance. Yeh optimal tradeoff point dhundhne ka tool hai, tradeoff escape karne ka nahi.

Error: Base learners (individual trees) tradeoff face karte hain. Bagging high-variance models ko average karta hai variance reduce karne ke liye unka low bias preserve karte hue. Lekin tumne multiple models ki computational cost chuki hai. Single model ke liye tradeoff bana rehta hai; ensembles models combine karke isse navigate karte hain.

Fix: Ensembles tradeoff violate nahi karte—woh isse cleverly exploit karte hain. Bagging isliye kaam karta hai kyunki averaging variance reduce karta hai ($n$ independent models ke liye $1/\sqrt{n}$ se) bias affect kiye bina. Boosting sequentially residuals fit karke bias reduce karta hai. Dono bias-variance framework ke andar operate karte hain.

Practical Strategy: The 80/20

80% results ke liye 20% effort:

1. Simple se shuru karo (linear/logistic regression, shallow tree)
2. Learning curves plot karo (training vs. validation error vs. dataset size)
   • Train/val ke beech gap → high variance (overfit) → regularize karo ya zyada data lo
   • Dono errors high → high bias (underfit) → complexity badhao
3. Cross-validation use karo bias-variance empirically estimate karne ke liye (dekho Cross-Validation)
4. Complex models regularize karo (dropout, L2, early stopping) unhe abandon karne se pehle

Key metric: Validation error bias + variance ka observable proxy hai. Tum single dataset pe bias aur variance separately measure nahi kar sakte, lekin unka sum generalization error ke roop mein appear hota hai.

Alternative: "The dartboard duality"—Bias = aim, Variance = scatter. Ya tum consistently off-target ho ya sab jagah bikhar rahe ho.

Active Recall Practice

Ab socho tum har shot improvise karne ki koshish karo, apne arms aur legs randomly adjust karo. Kabhi kabhi luck se lag jaata hai, lekin zyaadatar shots idhar-udhar jaate hain—left, right, short, long. Tum inconsistent ho. Yeh variance hai: tum unreliably galat ho.

Best shooters woh technique dhundhte hain jo dono sahi aim kare (low bias) aur repeatable ho (low variance). Lekin yahan catch hai: agar tum super flexible hone ki koshish karo (har chote factor ke hisaab se adjust karo—wind, lighting, tumhara mood), tum zyada scatter karoge. Agar tum consistent rehne ke liye rigid technique lock karo, tum ek flaw lock in kar sakte ho. Machine learning models ko usi dilemma face karni padti hai: simple models consistent hain lekin galat; complex models average pe sahi ho sakte hain lekin individual tries pe sab jagah hote hain.

#flashcards/ai-ml

What is the bias-variance tradeoff? :: Fundamental tension jahan simple models ka high bias hota hai (systematic error, underfitting) aur low variance (stable predictions), jabki complex models ka low bias hota hai lekin high variance (overfitting, sensitivity to training data). Total error = Bias² + Variance + Irreducible error.

Connections

• Cross-Validation: Generalization error (bias + variance) estimate karne aur model complexity select karne ka empirical method
• Regularization Techniques: L1/L2 penalties, dropout, early stopping—sab bias ke liye variance trade karte hain
• Learning Curves: Bias-variance issues visualize karne ka diagnostic tool
• Ensemble Methods: Bagging variance reduce karta hai, boosting bias reduce karta hai—dono tradeoff navigate karte hain
• Overfitting and Underfitting: Overfitting = high variance, underfitting = high bias
• Train-Validation-Test Split: Validation set generalization error estimate karta hai bias-variance problems detect karne ke liye
• Model Capacity: Fundamental quantity jo bias-variance spectrum pe position determine karti hai
• PAC Learning Theory: Bias-variance tradeoff ka formal framework approximation-estimation error decomposition ke through

Concept Map