Bias-variance trade-off

5.6.4 · Coding › Machine Learning (Aerospace Applications)

Overview

Bias-variance trade-off machine learning mein ek fundamental tension hai — model ki ability ke beech jo training data ko fit kare (low bias) aur nayi data par generalize kare (low variance). Yeh trade-off aerospace applications mein model selection govern karta hai, flight control prediction se lekar satellite image classification tak.

Core Concepts

Yeh kyun important hai: Ek linear model jo rocket thrust predict karta hai (jo fuel flow, pressure, temperature par nonlinearly depend karta hai) usme high bias hoga — uski assumptions galat hain.

Yeh kyun important hai: 20 flight data points par degree-50 polynomial fit, 20 alag flights se retrain karne par wildly different predictions dega — yeh specific training samples ke liye bahut sensitive hai.

jahan ${\sigma }^2$ irreducible error hai (data mein noise itself).

Derivation from scratch:

Maano $y=f(x)+ϵ$ jahan $ϵ$ random noise hai jiska mean 0 aur variance ${\sigma }^2$ hai. Hum chahte hain ki jab apne model $\hat{f}(x)$ se $y$ predict karein toh expected squared error kya hoga:

E[(y - \hat{f}(x))^2] &= E[(f(x) + \epsilon - \hat{f}(x))^2] \\ &= E[(f(x) - \hat{f}(x))^2 + 2\epsilon(f(x) - \hat{f}(x)) + \epsilon^2] \end{align}$$ **Yeh step kyun?** Hum square expand kar rahe hain aur expectation ki linearity use kar rahe hain. $$\begin{align} &= E[(f(x) - \hat{f}(x))^2] + 2E[\epsilon]E[f(x) - \hat{f}(x)] + E[\epsilon^2] \\ &= E[(f(x) - \hat{f}(x))^2] + 0 + \sigma^2 \end{align}$$ **Yeh step kyun?** $E[\epsilon] = 0$ by definition hai, aur $E[\epsilon^2] = \sigma^2$ noise variance hai. Ab pehle term par focus karein. $E[\hat{f}(x)]$ add aur subtract karein: $$\begin{align} E[(f(x) - \hat{f}(x))^2] &= E[(f(x) - E[\hat{f}(x)] + E[\hat{f}(x)] - \hat{f}(x))^2] \\ &= E[(f(x) - E[\hat{f}(x)])^2 + 2(f(x) - E[\hat{f}(x)])(E[\hat{f}(x)] - \hat{f}(x)) \\ &\quad + (E[\hat{f}(x)] - \hat{f}(x))^2] \end{align}$$ **Yeh step kyun?** Classic "add zero" trick hai — systematic error ko variability se alag karne ke liye. $$\begin{align} &= (f(x) - E[\hat{f}(x)])^2 + 2(f(x) - E[\hat{f}(x)])E[\hat{f}(x)] - \hat{f}(x)] \\ &\quad + E[(E[\hat{f}(x)] - \hat{f}(x))^2] \end{align}$$ **Yeh step kyun?** Pehle term mein koi randomness nahi (yeh constants hain). Middle term ka expectation zero hai kyunki $E[\hat{f}(x)] - \hat{f}(x)$ ka mean zero hota hai by definition. $$= (f(x) - E[\hat{f}(x)])^2 + E[(\hat{f}(x) - E[\hat{f}(x)])^2]$$ Pehla term $\text{Bias}^2[\hat{f}(x)]$ hai aur doosra $\text{Variance}[\hat{f}(x)]$ hai. Ho gaya! ∎ > [!formula] Total Error > $$\boxed{\text{Total Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}}$$ **Key insight**: Hum teeno ko simultaneously reduce nahi kar sakte. Bias reduce karna (model ko complex banana) variance badhata hai. Variance reduce karna (model simplify karna) bias badhata hai. ## The Trade-off Curve > [!intuition] U-Shape > Jaise model complexity badhti hai: > - **Bias ghatta hai**: zyada flexible model true pattern better fit kar sakta hai > - **Variance badhta hai**: zyada flexible model training data noise ke liye zyada sensitive hota hai > - **Total error U-curve follow karta hai**: ek sweet spot hota hai Aerospace mein: - **Underfitting** (high bias): Turbulent airflow ke liye linear model — critical nonlinear dynamics miss karta hai - **Just right**: Regularization ke saath neural network, hold-out flight data par validate kiya hua - **Overfitting** (high variance): Deep network jo actual patterns ki jagah sensor noise memorize kar leta hai ## Worked Examples > [!example] Example 1: UAV Wing ke liye Drag Coefficient mein Polynomial Regression > **Problem**: Ek UAV wing ke liye angle of attack $\alpha$ ke against drag coefficient $C_D$ predict karna. Hamare paas 15 wind tunnel measurements hain. **Setup**: True relationship roughly $C_D = 0.02 + 0.3\alpha + 0.5\alpha^2$ hai (simplified). Measurements mein noise $\sigma = 0.01$ hai. **Approach 1 — Degree 1 polynomial (linear)**: ```python # Model: C_D = a + b*alpha # This is high bias — we're forcing a linear fit on quadratic data ``` **Underfitting kyun hoti hai**: Haari model assumption (linear) reality se contradict karti hai (quadratic). Kitna bhi training data ho yeh fix nahi hoga — model literally truth represent hi nahi kar sakta. **Expected behavior**: - Training error: moderate (curve fit nahi ho sakti) - Test error: moderate (same systematic error) - Bias: **high** — hum middle angles par systematically underpredict karte hain - Variance: **low** — do alag datasets se bahut similar lines milti hain **Approach 2 — Degree 2 polynomial**: ```python # Model: C_D = a + b*alpha + c*alpha^2 # This matches the true form ``` **Yeh kyun kaam karta hai**: Model capacity problem complexity se match karti hai. Hamare paas 3-parameter truth ke liye 3 parameters hain, 15 data points ke saath (parameters se 5 guna). **Expected behavior**: - Training error: low - Test error: low - Bias: **low** — true function represent kar sakta hai - Variance: **moderate** — thodi sensitivity ki wajah se ki humne kaunse 15 points sample kiye - **Yeh sweet spot hai** **Approach 3 — Degree 10 polynomial**: ```python # Model: C_D = a + b*alpha + c*alpha^2 + ... + k*alpha^10 # This has 11 parameters for 15 data points ``` **Overfitting kyun hoti hai**: 11 parameters aur sirf 15 data points ke saath, model ke paas itni flexibility hai ki woh har measurement ke random noise ko "chase" kar sake. Woh noise fit karta hai, signal nahi. **Expected behavior**: - Training error: **bahut low** (almost saare points interpolate kar leta hai) - Test error: **high** (data points ke beech wild oscillations) - Bias: **low** — model kaafi flexible hai - Variance: **bahut high** — alag training sets se drastically different curves **Quantitative check** (simulated): ```python # Degree 1: Bias²≈ 0.04, Variance ≈ 0.001 # Degree 2: Bias² ≈ 0.001, Variance ≈ 0.005 # Degree 10: Bias² ≈ 0.0001, Variance ≈ 0.08 ``` Total error degree 2 par minimize hoti hai. > [!example] Example 2: Satellite Image Classification ke liye k-NN > **Problem**: Satellite pixels se terrain types classify karna (forests, urban, water). Har pixel mein 3 RGB values hain. **k=1 (ek nearest neighbor)**: - **Decision boundary**: Har training pixel ke around Voronoi cells - **Training error**: 0% — har training pixel apna khud ka nearest neighbor hai - **High variance kyun**: Agar hum thoda alag image sample se retrain karein, boundaries dramatically change ho jaati hain. Ek noisy "forest" pixel urban area mein ek isolated green island bana deta hai. - **Bias**: bahut low (arbitrarily complex boundaries represent kar sakta hai) - **Variance**: bahut high - **Result**: Overfitting — parking lot mein ek green car ke liye "forest" predict karta hai **k=100 (sau nearest neighbors)**: - **Decision boundary**: RGB space mein bahut smooth, almost linear - **High bias kyun**: 100 neighbors average karna over-smooth kar deta hai. River aur forest ke beech sharp boundary capture nahi ho sakti. - **Bias**: high (almost linear boundaries force karta hai) - **Variance**: low (training sets ke across stable) - **Result**: Underfitting — poori image ko "mixed terrain" classify kar deta hai **k=10 (das nearest neighbors)** — middle ground: - Noise pixels ignore karne ke liye kaafi smooth - Curved forest/urban boundaries follow karne ke liye kaafi flexible - **Moderate noise ke saath satellite imagery ke liye typically best** **Yeh step kyun important hai**: Aerospace mein satellite data expensive hai. Hum afford nahi kar sakte ki ek image set par overfit karein aur agli orbital pass par fail ho jaayein. > [!example] Example 3: Flight Control ke liye Neural Network Depth > **Problem**: Airspeed, altitude, pitch rate se elevator deflection predict karna. **1 hidden layer, 5 neurons** (simple): - Basic linear combinations seekh sakta hai - **High bias** agar control law nonlinear hai (jo hai — dynamic pressure $v^2$ ke saath jaata hai) - Low variance **3 hidden layers, 50 neurons each** (complex): - Kisi bhi continuous function approximate kar sakta hai (universal approximation theorem) - **High variance** chhote flight test dataset ke saath - General dynamics seekhne ki jagah specific test flights memorize karega **2 hidden layers, 20 neurons each + L2 regularization**: - Nonlinear aerodynamics ke liye kaafi capacity - Regularization overfitting penalize karta hai - **Sweet spot jo multiple flight regimes par cross-validation se validate hua** ## Common Mistakes > [!mistake] Mistake 1: "Zyada data hamesha overfitting fix karta hai" > **Kyun sahi lagta hai**: Zyada data matlab zyada examples, toh model better generalize karna chahiye, na? **Steel-man the intuition**: Yeh partially sach hai! Zyada data variance *reduce* karta hai (model zyada scenarios dekhta hai). Infinite data ke saath, variance zero ho jaata hai. **Fix**: Zyada data tabhi help karta hai **agar aapke model mein sahi capacity ho**. Agar aap 50 parameters wala degree-50 polynomial use kar rahe hain, toh 100 se 1000 data points tak jaana help karta hai, lekin phir bhi degree-2 model ke relative overfit ho raha hai. Aur zyada data **bias ke liye kuch nahi karta** — ek linear model quadratic data underfit karega chahe kitne bhi samples ho. **Sahi principle**: Model complexity ko match karein: (data size) × (signal-to-noise ratio) × (true complexity). > [!mistake] Mistake 2: "Low training error ka matlab accha model hai" > **Kyun sahi lagta hai**: Agar model training data achhi tarah fit karta hai, toh usne pattern seekh liya! **Steel-man**: Ek perfect world mein infinite data aur noise ke saath, zero training error matlab perfect learning hota. **Fix**: Training error bias measure karta hai, variance nahi. Degree-100 polynomial noise memorize karke zero training error le aata hai. Asli test **validation error** hai held-out data par. Aerospace mein, iska matlab hai un flights/images/trajectories par test karna jo model ne training ke dauran kabhi nahi dekha. **Best practice**: Hamesha data split karein: 60% train, 20% validation (hyperparameters tune karein), 20% test (final assessment). Test error report karein. > [!mistake] Mistake 3: "Regularization hamesha help karta hai" > **Kyun sahi lagta hai**: Regularization (jaise L2 penalty) specifically overfitting reduce karne ke liye design kiya gaya hai, model complexity penalize karke. **Steel-man**: Regularization variance *reduce* karta hai model constrain karke. **Fix**: Regularization **bias badhata hai**. Agar aapka model pehle se underfit kar raha hai (high bias), toh regularization add karna use aur bura kar deta hai. Example: Aap quadratic drag ke liye linear regression use kar rahe hain, aur strong L2 penalty add karte hain — ab model aur bhi constrained hai aur aur bhi bura fit karta hai. **Regularize kab karein**: Jab validation error training error se zyada ho (overfitting ka sign). Regularize mat karein agar dono errors high hain (underfitting ka sign — aapko kam capacity nahi, zyada chahiye). ## Practical Techniques ### Cross-Validation Bias aur variance dono estimate karne ke liye ==k-fold cross-validation== use karein: ```python from sklearn.model_selection import cross_val_score # Train model on different subsets, test on held-out fold scores = cross_val_score(model, X, y, cv=5) # Low mean score → high bias (underfitting) # High std of scores → high variance (overfitting) # Both low and stable → just right ``` **Yeh kyun kaam karta hai**: Alag subsets par train karke, hum directly measure karte hain ki predictions kitni vary karti hain (variance). Mean score systematic error (bias) batata hai. ### Learning Curves Training aur validation error ko dataset size ke against plot karein: **High bias signature**: - Training error: high, jaldi plateau ho jaata hai - Validation error: high, training error tak converge ho jaata hai - **Fix**: Model complexity badhao **High variance signature**: - Training error: low - Validation error: high, bada gap bana rehta hai - **Fix**: Zyada data, regularization, ya complexity kam karo ### Regularization Parameter Tuning Aerospace applications ke liye: ```python # Ridge regression (L2): penalizes sum of squared weights # Lasso regression (L1): penalizes sum of absolute weights # Elastic net: combines both from sklearn.linear_model import RidgeCV model = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0]) # Try different penalties model.fit(X_train, y_train) print(f"Best alpha: {model.alpha_}") # Chosen by cross-validation ``` **Aerospace mein kyun important hai**: Flight data expensive hai (wind tunnel time, test flights). Hamare paas typically chhote datasets hote hain → overfitting ka high risk → regularization critical hai. > [!mnemonic] BVIR Memory Aid > **B**ias: complexity se **B**ekhbar (model bahut simple) > **V**ariance: datasets ke across **V**olatile (model bahut sensitive) > **I**rreducible: **I**nherent noise (fix nahi ho sakta) > **R**elationship: Dono **R**everse directions mein chalte hain — complexity badhane se bias kam hota hai lekin variance badhta hai **Goldilocks Rule**: Na bahut simple (bias), na bahut complex (variance), bilkul sahi (total error minimize karo). ## Connections - [[5.6.01-Supervised-vsUnsupervised-Learning]] — Bias-variance dono paradigms par apply hota hai - [[5.6.02-Training-Validation-Test-Sets]] — Practice mein trade-off kaise measure karein - [[5.6.03-Overfitting-and-Regularization]] — Variance control karne ka tool hai regularization - [[5.6.05-Cross-Validation-Techniques]] — Sweet spot dhundhne ka empirical method - [[5.6.08-Hyperparameter-Tuning]] — Model complexity ek hyperparameter hai tune karne ke liye - [[3.4.07-Polynomial-Regression]] — Varying degree ke saath bias-variance ka classic example > [!recall]- Ek 12-saal ke bacche ko explain karo > Socho tum predict kar rahe ho ki basketball kahan giregi, iske hisaab se ki tum kitni zor se throw karte ho. Agar tum ek super simple rule use karo jaise "yeh hamesha 10 feet jaati hai," toh tum **biased** ho — tum dekh hi nahi rahe ki kitni zor se throw kiya! Yeh underfitting jaisa hai. Agar tum exactly memorize kar lo ki tumhari 5 practice shots mein ball kahan giri, tab jab thodi alag force se throw karo, tumhari prediction bahut galat hogi kyunki tumne exact throws memorize kiye instead of general pattern seekhne ke. Yeh **variance** hai — specific practice shots ke liye bahut sensitive hona. Yeh overfitting jaisa hai. Smart approach hai notice karna ki "zyada zor se throw → zyada door" lekin har chhoti wobble memorize mat karo. Tum pattern seekhte ho bina noise ke. Yeh sweet spot hai — low bias (tum asli rule seekh rahe ho) aur low variance (tum alag practice sessions mein stable ho). Machine learning mein rockets aur planes ke liye, hum wohi karte hain: woh model dhundho jo "bilkul sahi" ho — na bahut simple, na bahut memorizing wala, lekin asli physics capture karta ho. --- #flashcards/coding Machine learning mein bias kya hai? :: Learning algorithm mein galat assumptions se aane wala error; high bias matlab model true pattern capture karne ke liye bahut simple hai (underfitting). Mathematically: model prediction aur truth ke beech expected difference. Machine learning mein variance kya hai? ::: Specific training dataset ke liye model ki predictions ki sensitivity; high variance matlab model alag training data ke saath drastically change hota hai (overfitting). Mathematically: predictions ka unke mean se expected squared deviation. Bias-variance decomposition formula bolo ::: Expected error = Bias² + Variance + Irreducible error (noise). Yeh teen components additive hain aur systematic error, data ke liye sensitivity, aur inherent noise represent karte hain respectively. Hum bias aur variance simultaneously minimize kyun nahi kar sakte? ::: Unka ek inverse relationship hai. Model complexity badhane se bias kam hota hai (model better fit kar sakta hai) lekin variance badhta hai (model training data ke liye zyada sensitive ho jaata hai). Ek sweet spot hota hai jahan total error minimize hoti hai. Learning curves mein high bias ki signature kya hai? ::: Training error aur validation error dono high hain aur jaldi converge ho jaate hain. Unke beech gap chhota hota hai. Yeh underfitting indicate karta hai — model bahut simple hai. Learning curves mein high variance ki signature kya hai? ::: Training error low hai lekin validation error high hai bade persistent gap ke saath. Yeh overfitting indicate karta hai — model training data memorize karta hai lekin generalize nahi karta. K-fold cross-validation bias vs variance identify karne mein kaise help karta hai? ::: Low mean cross-validation score high bias indicate karta hai (systematic error). Scores ka high standard deviation folds ke across high variance indicate karta hai (predictions alag training subsets mein unstable hain). Zyada data hamesha overfitting kyun fix nahi karta? :: Zyada data variance reduce karta hai lekin bias ke liye kuch nahi karta. Agar tumhara model bahut simple hai (high bias), toh infinite data help nahi karega. Agar model bahut complex hai, toh tum sirf ek bade dataset par overfit kar rahe ho. Tumhe data size ke liye sahi model complexity chahiye. Regularization bias-variance trade-off ke saath kya karta hai? ::: Regularization bias badhata hai (model constrain karta hai) lekin variance kam karta hai (training data ke liye kam sensitive banata hai). Use karo jab overfitting ho (high variance), lekin yeh underfitting (high bias) ko aur bura karta hai. Polynomial regression mein degree-1 mein high bias kyun hota hai? ::: Kyunki true relationship often nonlinear hoti hai (e.g., quadratic drag). Ek linear model literally truth represent hi nahi kar sakta chahe kitna bhi data ho — uski assumptions fundamentally galat hain. k-NN mein k=1 mein high variance kyun hota hai? ::: Har training point apna khud ka nearest neighbor hota hai (0% training error), lekin predictions alag training sets ke saath dramatically change hoti hain. Ek noisy point isolated decision regions create karta hai. Model bahut flexible hai aur noise memorize karta hai. Irreducible error kya hai aur hum ise kyun eliminate nahi kar sakte? ::: Data mein inherent noise (σ²), jo measurement error, unmodeled variables, ya true randomness se aata hai. Koi bhi model noise predict nahi kar sakta — yeh prediction error ka theoretical lower bound hai. ## 🖼️ Concept Map ```mermaid flowchart TD BVT[Bias-Variance Trade-off] Bias[Bias] Variance[Variance] Under[Underfitting] Over[Overfitting] Decomp[MSE Decomposition] Noise[Irreducible Error sigma^2] Model[Model Selection] Aero[Aerospace Predictions] BVT -->|one side| Bias BVT -->|other side| Variance Bias -->|high causes| Under Variance -->|high causes| Over Bias -->|wrong assumptions| Model Variance -->|sensitivity to data| Model Decomp -->|sums| Bias Decomp -->|sums| Variance Decomp -->|adds| Noise Model -->|governs| Aero Under -->|too simple for| Aero Over -->|learns noise in| Aero ```