5.6.4 · D3 · HinglishMachine Learning (Aerospace Applications)

Worked examplesBias-variance trade-off

4,746 words22 min read↑ Read in English

5.6.4 · D3 · Coding › Machine Learning (Aerospace Applications) › Bias-variance trade-off

Yeh parent topic ka ek hands-on companion hai. Wahan humne derive kiya tha ki total error teen pieces mein kyun split hoti hai. Yahan hum har possible situation ko drill karte hain jo trade-off throw kar sakta hai — ek chhota model ek waqt — taaki koi bhi exam question ya real flight-data problem tumhe off guard na pakde.

Shuru karne se pehle: teen words jo hum har line pe use karenge.

Poora page ek equation pe tika hai, bias-variance decomposition. Yeh kehti hai ki kisi point pe expected squared error theek teen non-negative pieces mein split hoti hai:

Figure — Bias-variance trade-off

Upar ke chaar dartboards dekho. Har board equation ki ek row hai jo "on" hai. Inhe left se right padho:

  • Board 1 — low bias, low variance: cyan darts amber bullseye pe tightly cluster karte hain. aur dono chhote hain — woh goal jo hum pursue karte hain.
  • Board 2 — low bias, high variance: darts ka average centre pe aata hai (small bias) lekin wildly scatter karte hain (large variance). Yeh overfitting hai.
  • Board 3 — high bias, low variance: darts tightly cluster karte hain (small variance) lekin poora cluster off-centre hai (large bias). Yeh underfitting hai.
  • Board 4 — high bias, high variance: off-centre bhi aur scattered bhi — dono ka worst.

Yeh map dimaag mein rakho: neeche har example darts ko in chaar patterns mein se ek mein land karata hai aur batata hai ki decomposition ka kaun sa term zimmedaar hai. Board 4 (dono errors bade) Example 8 mein trap hai.


Scenario matrix

Har bias-variance problem in cells mein se ek hai. Is page pe hamaara kaam hai har row ke liye ek example work karna taaki test pe aane se pehle tumne yeh dekha ho.

Cell Kya vary ho raha hai Bias Variance Real-world flavour
A. Too simple model true complexity se kaafi neeche high low curve pe linear fit
B. Just right model capacity ≈ true complexity low low–moderate quadratic drag pe quadratic fit
C. Too flexible model true complexity se kaafi upar low high 15 points pe degree-10 fit
D. Degenerate: 1 point seekhne ke liye itna data nahi undefined/huge ek single measurement pe line fit karna
E. Limiting: infinite data unchanged variance vanish hoti hai, bias rehti hai
F. Zero-noise data can hit 0 error still can overfit perfect simulation output
G. Discrete knob (k-NN) neighbours small→large low→high high→low satellite pixel classifier
H. Word problem data budget ke andar model choose karna pick from curve naye route ke liye fuel-burn predictor
I. Exam twist "data add karo ya regularization?" reasoning reasoning kaun sa lever kaun si error move karta hai
J. Worst corner wrong shape AND too flexible high high galat features + tiny dataset

Hum A, B, C ko Example 1 mein cover karenge, D ko Example 2 mein, E ko Example 3 mein, F ko Example 4 mein, G ko Example 5 mein, H ko Example 6 mein, I ko Example 7 mein, aur J (dartboard 4) ko Example 8 mein. Har numeric answer neeche #Verification mein machine-checked hai.


Worked examples

Forecast (pehle guess karo!): Tumhare hisaab se kaun sa degree total error minimize karta hai? Aage padhne se pehle likh lo.

Figure — Bias-variance trade-off

Step 1 — Picture padho. Amber curve true quadratic hai. Blue straight line (degree 1) bend nahi kar sakti, isliye woh beech mein truth ke neeche aur ends pe upar baithti hai — ek systematic miss. Yeh step kyun? Bias exactly "systematic miss" hai, isliye yeh dekhna ki line kabhi curve ke belly ko nahi chhuti, humein bataata hai ki bias high hai kisi bhi arithmetic se pehle.

Step 2 — Degree 1 score karo (cell A). Ek line mein 2 knobs hain () lekin truth ko 3 chahiye (isme ek term hai). Koi bhi data missing curvature supply nahi kar sakta. Isliye bias high hai; lekin do alag random samples dono lagbhag same best-fit line dete hain, isliye variance low hai. Simulated numbers: , . Yeh step kyun? Hum "kya model shape match kar sakta hai?" (bias) ko "kya model samples ke beech hilta hai?" (variance) se alag karte hain.

Step 3 — Degree 2 score karo (cell B). Ab model mein exactly woh 3 knobs hain jo truth ko chahiye, aur hamare paas 15 points hain jo 3 knobs feed kar rahe hain — ek healthy 5-to-1 ratio. Bias collapse ho jaata hai; variance modest rehta hai. Simulated: , . Yeh step kyun? Model capacity ko true form se match karna wahi hai jo bias ko near-zero laata hai bina zyaada variance pay kiye — U-curve ke floor ki definition.

Step 4 — Degree 10 score karo (cell C). Degree-10 polynomial true quadratic ko ek special case ke roop mein contain karta hai ( ke coefficients zero set karo), isliye iska true bias exactly 0 hai — yeh truth represent kar sakta hai. Woh tiny jo hum quote karte hain, woh koi modelling limitation nahi hai; yeh leftover estimation residue hai: sirf 15 noisy points se averaged fit un zero high-order coefficients ko perfectly recover nahi karta, isliye simulation mein apparent bias ki ek whisker bachti hai. Jo dominate karta hai woh variance hai: wiggles sample-to-sample violently differ karte hain. Simulated: , . Yeh step kyun? Hum representational bias (kya shape match kar sakta hai? — yahan, haan, isliye ~0) ko tiny finite-sample estimation residue se distinguish karte hain, aur confirm karte hain ki over-flexibility ki real cost entirely variance term mein dikhti hai.

Step 5 — Inhe add karo (common ignore karo, yeh teeno mein equally add hota hai):

Yeh step kyun? Decomposition humein bas do model-dependent terms sum karne aur compare karne deti hai, kyunki saare models mein identical hai.

Verify: Degree 2 smallest total deta hai () — U-curve "just right" pe bottom out karti hai, parent note ke claim se match karta hai. Units: saare quantities (drag-coefficient) mein hain, dimensionless, isliye comparison valid hai. ✓


Forecast: Kya tum ek point se bhi unique line find kar sakte ho?

Step 1 — Knobs vs. equations count karo. Ek line mein 2 unknowns hain . Ek point 1 equation deta hai. Do unknowns, ek equation ⇒ infinitely many solutions. Yeh step kyun? Kya ek fit well-posed hai yeh sirf "equations ≥ unknowns" hai. Yahan , isliye problem underdetermined hai.

Step 2 — Variance kya ban jaata hai? Variance poochta hai "resampled datasets mein fitted line kitni change hoti hai?" Ek point ke saath, koi bhi tiny noise wiggle chosen line ko arbitrarily spin kar deti hai (solver infinitely many mein se koi line choose karta hai). Variance effectively unbounded / undefined hai jab tak koi tie-break rule pin nahi karte. Yeh step kyun? Ek underdetermined solver ke paas equally-valid answers ki poori space hoti hai, isliye data mein infinitesimal change use totally alag line pe jump kar sakti hai — exactly yahi unbounded variance ka matlab hai.

Step 3 — Bias kya ban jaata hai? Raw least-squares problem ke saath koi stable "average model" nahi hai truth se compare karne ke liye, isliye bias bhi undefined hai. Lekin standard conventions ise pin karte hain: (i) minimum-norm solution (jo numpy.linalg.lstsq aur ridge-at-zero return karta hai) coefficient space mein origin ke sabse paas wali single line choose karta hai; (ii) koi bhi regularization term jahan , solution ko unique bana deta hai. Dono conventions ke under well-defined hai — aur yeh heavily biased hoga, kyunki coefficients ko zero ki taraf kheenchna line ko true quadratic se door le jaata hai. Yeh step kyun? "Undefined" sirf naive solver ke liye true hai; minimum-norm / ridge tie-break ko name karna dikhata hai ki real libraries kaise ek definite (aur biased) answer recover karti hain, taaki reader ek false impossibility ke saath na rahe.

Step 4 — Fix. Data ke neeche model capacity reduce karo. Ek point ke saath, sirf ek degree-0 model (ek constant ) bina kisi tie-break ke well-posed hai: 1 knob, 1 equation. Iska huge bias hai (ek curve ke liye flat line) lekin finite variance.

Verify: degree-0 model ke knobs data points ki sankhya , isliye system exactly determined aur solvable hai. ✓ Lesson (cell D): kabhi bhi parameter count ko data count se zyaada mat hone do — parent ka "11 parameters, 15 points" pehle se is edge se flirt kar raha tha.


Forecast: Kya zyaada data high bias, high variance, dono, ya koi nahi — theek kar deta hai?

Step 0 — Assumptions state karo (woh law earn karti hain). Hum assume karte hain ki noise iid hai (independent, identically distributed) mean 0 aur constant variance ke saath, aur inputs ek fixed design distribution se aate hain (har baar same range). In standard ordinary-least-squares (OLS) assumptions ke under estimator ki variance ek point pe hai, aur proportionally ke saath grow karta hai, isliye uska inverse ki tarah shrink karta hai. Yeh step kyun? " law" free nahi hai — yeh exactly sirf iid noise aur fixed design ke under hold karta hai; un assumptions ko name karna hi woh hai jo neeche clean scaling ko license deta hai.

Step 1 — Variance ki tarah shrink karta hai. Step 0 se, . Data double karne se roughly variance half ho jaata hai. Yeh step kyun? ka matlab hai har extra point "information" mein equally add karta hai, aur linearly growing information ko invert karne se decay milta hai — law of large numbers OLS ke liye precise bana diya.

Step 2 — Numeric trace. pe degree-10 variance lo aur scale karo: pe yeh hai; pe, . Yeh step kyun? Concrete values ko law mein plug karna ek abstract rate ko aise numbers mein badalta hai jo check kar sako, variance ko 10× data pe 10× girte dikhaata hai.

Step 3 — Bias move nahi karta. Bias model shape ki property hai, sample size ki nahi. Degree-10 model quadratic ko exactly represent kar sakta hai, isliye uska bias pehle se ~0 tha aur ~0 rehta hai. Lekin ek linear model ka high bias hamesha high rehega, chahe kitna bhi data ho.

Step 4 — Conclusion. Zyaada data ek variance cure hai, bias cure nahi. Jab , degree-10 model ki variance aur woh "just right" model jitna achha ban jaata hai. Isliye big-data teams flexible models afford kar sakti hain.

Verify: pe variance hai, 100× data increase ke liye 100× drop — law se consistent. ✓


Forecast: Perfectly clean data ke saath, kya overfitting impossible hai?

Step 1 — Decomposition yaad karo. set karo. Achievable error ka floor ab zero hai — ek perfect model har point nail kar sakta hai. Yeh step kyun? Hum check kar rahe hain ki teen terms mein se actually kaun sa vanish karta hai. Sirf teesra karta hai.

Step 2 — Variance phir bhi nonzero ho sakta hai. Agar tumhare 15 simulator points alag regions se aate hain har run mein (maan lo alag grids), degree-10 fit phir bhi runs ke beech hilta rehta hai. Isliye variance necessarily 0 nahi hai chahe ho. Yeh step kyun? Variance kaun se points tune kiye this uski sensitivity measure karta hai, measurement noise ki nahi — isliye sirf sampling grid change karna variance ko alive rakh sakta hai chahe readings perfectly clean hon.

Step 3 — Overfitting ek variance issue hai, bias issue nahi. Overfitting ka matlab hai model samples ke beech wildly vary karta hai — yahi high variance hai. Clean data ke saath koi measurement noise nahi hai "chase" karne ke liye, isliye usual noise-driven variance largely disappear ho jaati hai; jo bachta hai woh sirf Step 2 ka milder sampling-grid variance hai. Agar grid fixed aur clean hai, degree-10 model exactly interpolate karta hai aur points ke beech well generalise karta hai. Yeh step kyun? Yeh vocabulary nail karta hai: overfitting variance term mein rehta hai. Noise remove karna () variance ko uske main fuel se starve karta hai, isliye clean data flexible models ke liye itna forgiving hota hai.

Step 4 — Takeaway. Isliye "high degree = bad" actually "high degree + noise = bad" hai. Zero-noise simulator data woh ek jagah hai jahan aerospace engineers safely bahut flexible models use kar sakte hain.

Verify: total error floor . Clean fixed-grid data pe degree-10 model ke liye, dono remaining terms , isliye total error — sirf isliye achievable kyunki . ✓


Forecast: High-variance kaun sa hoga — small ya large?

Figure — Bias-variance trade-off

Step 1 — Boundaries padho. (left) pe decision boundary jagged hai — har training pixel ek chhota cell own karta hai, isliye ek stray green pixel ek green island bana deta hai. (right) pe boundary almost straight hai — 100 votes kisi bhi single pixel ko drown out kar dete hain. Yeh step kyun? k-NN mein, hi complexity knob hai: small = flexible = complex; large = smooth = simple.

Step 2 — Bias/variance assign karo.

  • : bias low (koi bhi shape trace kar sakta hai), variance high (jagged, resample-sensitive). Cell C behaviour.
  • : bias high (over-smoothed, sharp river/forest edges miss karta hai), variance low. Cell A behaviour.
  • : middle — akele noisy pixels ignore karne ke liye enough smooth, curved coastlines follow karne ke liye sharp enough. Cell B. Yeh step kyun? Har pe decomposition se directly har board padhna humein batata hai kaun sa error term dominate karta hai koi choice commit karne se pehle.

Step 3 — Direction rule. Jab badhta hai, variance ghatta hai aur bias badhta hai — polynomial degree se opposite direction. Small ↔ high degree; large ↔ low degree. Yeh step kyun? Har algorithm mein ek complexity knob hota hai, lekin woh alag direction mein point karta hai; yeh jaanna ki kaun sa end "flexible" hai tumhe dial galat direction mein ghumane se rokta hai aur usi error ko worsen karne se jo tumne cut karna tha.

Step 4 — Choose karo. Moderate noise ke liye, dono ko balance karta hai. Yeh U-curve ke bottom pe baithta hai: akele noisy pixels ko average out karne ke liye itna bada (variance tame karte hue) phir bhi curved coastlines hug karne ke liye itna chhota (bias low rakhte hue). Yeh step kyun? Winner hamesha woh knob setting hoti hai jo dono terms ka sum minimize kare, aur wahan hai jahan na koi term bada ho — is problem ke liye U-curve floor.

Verify: monotonic check — variance ordering aur bias ordering . Dono orderings opposite ways mein chalti hain, trade-off confirm karte hue. ✓


Forecast: Zyaada layers = zyaada accuracy... sahi hai na?

Step 1 — Real risk identify karo. Test data (route 4) distributionally new hai. Khatraa yeh hai ki model 3 training routes memorise kar le — yahi high variance / overfitting hai. Yeh step kyun? Woh lever choose karo jo dominant error se ladte ho. Yahan unseen conditions mein generalisation, training fit se last drop nikalane se zyaada matter karta hai.

Step 2 — (c) rule out karo. Ek 12-layer net mein hazaron knobs hain; 200 samples unhe pin down nahi kar sakte (cell D ki echo: knobs ≫ data). Ek unseen route pe variance enormous hogi. Yeh step kyun? Jab knobs data se kaafi zyaada hote hain toh model underdetermined hota hai, exactly woh high-variance regime jisse hume bachna chahiye.

Step 3 — (a) rule out karo agar physics nonlinear hai. Fuel burn weight, altitude, aur headwind pe nonlinearly depend karta hai, isliye pure line mein high bias hai — yeh systematically miss karega. Yeh step kyun? Ek model jiska shape physics se match nahi kar sakta, ek irreducible bias carry karta hai jise koi data fix nahi kar sakta — (c) se opposite failure.

Step 4 — (b) choose karo. Ek modest degree-3 model main curvature capture karta hai (low-ish bias) aur regularization (dekho regularization) coefficients shrink karta hai variance tame karne ke liye. Yeh data budget ke andar U-curve ke minimum ke sabse paas baithta hai. Yeh step kyun? (b) sirf woh option hai jisme 200-sample budget ke liye bias aur variance dono control mein hain — U-curve ka middle.

Step 5 — Choice properly validate karo. Training error pe trust mat karo. Cross-validation use karo har fold mein ek poora route hold out karte hue, taaki score "ek naya route predict karo" ko mimic kare. Yeh step kyun? Random-point cross-validation folds mein similar flights leak kar deti; ek poora route hold out karna validation ko real deployment task se match karta hai.

Verify: (b) ke liye knobs-to-data ratio ≈ 4 knobs / 200 points = 0.02 (bahut safe, ≪ 1), jabki (c) even 2000 knobs ke saath 2000/200 = 10 deta hai (≫ 1, unsafe). Safe model (b) hai. ✓


Forecast: High training error — kya yeh bias problem hai ya variance problem?

Step 1 — Do error numbers se diagnose karo.

  • Training error high AUR test error high ⇒ model jo dekha hai woh bhi fit nahi kar sakta ⇒ high bias (underfitting).
  • Training error low lekin test error high ⇒ training set memorise kar liya ⇒ high variance (overfitting). Yahan dono high hain ⇒ yeh ek bias problem hai. Yeh step kyun? Train aur test error ke beech gap tell-tale hai: small gap + high error = bias; large gap = variance.

Step 2 — Question 1 ka jawab: zyaada data collect karo? Nahi. Example 3 ne prove kiya data variance ko 0 ki taraf drive karta hai lekin bias ko — ek model-shape property — untouched rehne deta hai. Ek too-simple model train aur test dono pe galat rehta hai chahe kitni bhi rows add karo. Yeh step kyun? Hum lever ko term se match karte hain: data sirf variance term move karta hai, isliye yeh pure bias problem mein help nahi kar sakta.

Step 3 — Question 2 ka jawab: capacity / features add karo? Haan. Model complexity badhana (higher degree, more features, more layers) directly bias lower karta hai, jo yahan bada wala term hai. Yeh correct lever hai. Yeh step kyun? Bias high exactly isliye hai kyunki model shape truth match karne ke liye too poor hai; capacity add karna us shape ko enrich karta hai.

Step 4 — Question 3 ka jawab: stronger regularization? Nahi — yeh cheezein worse karta hai. Regularization variance cut karne ke liye bias raise karta hai; yahan variance pehle se theek hai aur bias culprit hai, isliye hum galat term ko push up kar rahe hote. Yeh step kyun? Regularization variance ko bias ke saath trade karta hai; ise bias-limited model pe apply karna humein U-curve pe galat direction mein le jaata hai.

Step 5 — General map state karo.

Symptom Diagnosis Fix Mat karo
high train + high test high bias capacity add karo, features add karo, regularization reduce karo data add karo, regularize karo
low train + high test high variance zyaada data, regularize, simplify karo aur capacity add karo

Verify: Example 3 (data ↓ sirf variance) aur Example 5 (increasing ↑ bias) ke saath consistency: "more data" fix sirf high variance ke under appear hota hai, aur "add capacity" sirf high bias ke under. Koi contradictions nahi. ✓


Forecast: Kya ek model ek saath dono systematically galat aur wildly unstable ho sakta hai?

Step 1 — Bias source pakdo. Chosen feature (colour) thrust se almost unrelated hai, isliye coefficients ki koi bhi setting true thrust curve trace nahi kar sakti. Averaged prediction truth se door baithta hai ⇒ high bias. Yeh step kyun? Bias tab aata hai jab model truth represent nahi kar sakta; yahan failure upstream features mein hai, degree mein nahi — ek wajah jo students often miss karte hain.

Step 2 — Variance source pakdo. Degree 9, 11 points ke liye 10 knobs deta hai — knobs-to-data ratio , cell-D edge ke bilkul paas. Fit noise chase karta hai, isliye samples ke beech violently swing karta hai ⇒ high variance. Yeh step kyun? Variance data ke relative bahut zyaada flexibility se aati hai; near-1 ratio guarantee karta hai ki model wildly wiggle karega, feature problem se independent.

Step 3 — Combine karo. Dono terms ek saath large hain aur independent reasons se (bad features → bias; bahut zyaada knobs → variance). Yeh dartboard 4 hai: darts scattered bhi aur off-centre bhi. Yeh step kyun? Yeh dikhata hai ki chaar dartboards mutually exclusive labels nahi hain — ek single model dono error sources ek saath trigger kar sakta hai, jo woh corner hai jo hamare matrix (cell J) mein uncovered nahi rehna chahiye.

Step 4 — Fix ko do moves chahiye. Sirf degree cut karna (variance) phir bhi bad-feature bias chhod deta hai; achhe features add karna (bias) lekin degree 9 rakhna phir bhi variance chhod deta hai. Tumhe features fix karne AUR degree lower karne dono karne padenge escape karne ke liye. Yeh step kyun? Kyunki dono errors ke independent causes hain, ek lever dono fix nahi kar sakta — worst corner ka practical lesson.

Verify: knobs-to-data ratio (deep in the high-variance zone) aur feature–target relation ≈ 0 (high bias), isliye dono error terms ek saath large hain. ✓


Recall

Recall k-NN mein small

kaun se cell mein hai? Small ::: high variance / low bias (flexible, cell C) — high-degree polynomial ke same family mein.

Recall Kya zyaada data collect karna bias reduce karta hai?

Nahi ::: zyaada data variance ko 0 ki taraf drive karta hai (jaise ) lekin bias ko, jo ek model-shape property hai, unchanged chhod deta hai.

Recall Train error high aur test error high — kya galat hai, aur kya fix karta hai?

High bias / underfitting ::: capacity ya features add karke fix karo; data add mat karo ya regularize mat karo.

Recall Overfitting ek bias problem hai ya variance problem?

Variance ::: overfitting ka matlab hai model samples ke beech wildly jiggle karta hai, jo exactly decomposition ka variance term hai.

Recall Zero data noise (

) ke saath, kya overfitting phir bhi possible hai? Sirf sampling variance ke zariye ::: ek fixed clean grid pe ek flexible model perfectly interpolate kar sakta hai, isliye noise-driven overfitting largely disappear ho jaata hai.

Recall Ek model dartboard 4 (high bias AUR high variance) mein kaise land kar sakta hai?

Do independent faults ::: galat/uninformative features bias cause karte hain, jabki bahut kam data ke liye bahut zyaada knobs variance cause karte hain — tumhe dono fix karne padte hain.

Verification

Is page pe har numeric claim — degree-1/2/10 totals (Ex 1), one-point knob count (Ex 2), variance trace (Ex 3), zero-noise floor (Ex 4), k-NN orderings (Ex 5), knobs-to-data ratios (Ex 6, 8), aur lever-to-error map (Ex 7) — is note ke saath aane wale ===VERIFY=== block mein machine-checked hai. Har check apne symbols declare karta hai, quantity ko sympy ke saath recompute karta hai, aur stated result assert karta hai.

Yeh bhi dekho: Polynomial Regression, Training/Validation/Test splits, aur Hyperparameter Tuning knob automatically choose karne ke liye.