Bias-variance trade-off: Exercises — Hinglish deep dive

5.6.4 · D4 · Coding › Machine Learning (Aerospace Applications) › Bias-variance trade-off

Ye exercises recognising karne se lekar real aerospace models engineer karne tak jaati hain. Pehle har ek ko paper pe karo, phir solution dekho. Jo bhi tumhe chahiye wo parent note Bias-Variance Trade-off mein build kiya gaya hai; agar koi symbol unfamiliar lage, to wahan pe uski definition re-read karo pehle jhankne se.

Level 1 — Recognition

Kya tum pieces ko naam de sakte ho aur picture padh sakte ho?

Recall Solution

Numbers kya keh rahe hain. Bada training error matlab model us data ko bhi fit nahi kar sakta jो use dikhaya gaya tha — uske assumptions bahut rigid hain. Test error ka utna hi bada hona (zyada bada nahi) matlab model itna sensitive nahi hai ki usne kaun sa data dekha.

Training data pe badi error → pattern capture nahi kar sakta → high bias.
Train ≈ test (dono kharaab) → predictions alag-alag datasets mein bahut kam change hoti hain → low variance.

Answer: high bias → underfitting. Upar wali figure ke U-curve pe, tum left side pe ho (model bahut simple hai).

Recall Solution

$Bias^{2} \geq 0$ aur $Variance \geq 0$ hain (dono squares / averages of squares hain — ye kabhi negative nahi ho sakte). To total kabhi se kam nahi ho sakta: $Total_{m i n} = 0 + 0 + σ^{2} = 0.01.$ Claimed $0.004 < 0.01$ ke liye noise term ko shrink hona padega, lekin $σ^{2}$ irreducible hai — ye data mein rehta hai, model mein nahi. Koi bhi algorithm ise beat nahi kar sakta. Colleague ne measurement ya bookkeeping mein galti ki.

Level 2 — Application

Formula mein plug karo aur number nikalo.

Recall Solution

Formula ko Bias squared chahiye, raw bias nahi. Wo squaring hi wajah hai ki ek signed systematic error positive contribution ban jaati hai: $Bias^{2} = (- 0.03)^{2} = 0.0009.$ Phir teen pieces add karo: $Total = 0.0009 + 0.0004 + 0.0001 = 0.0014.$ Answer: $0.0014$ . Notice karo ki bias² ( $0.0009$ ) dominate kar raha hai — is model ki sabse badi problem ye hai ki ye bahut simple hai, bahut jumpy nahi.

Worked example Exercise 2.2 — Which degree wins?

Parent note ki UAV drag model ke liye simulated table use karke:

Degree	Bias²	Variance
1	0.04	0.001
2	0.001	0.005
10	0.0001	0.08

Noise $σ^{2} = 0.0001$ hai. Har degree ke liye total error compute karo aur winner chunno.

Recall Solution

Teeno columns row by row add karo:

Degree 1: $0.04 + 0.001 + 0.0001 = 0.0411$ .
Degree 2: $0.001 + 0.005 + 0.0001 = 0.0061$ .
Degree 10: $0.0001 + 0.08 + 0.0001 = 0.0802$ .

Winner: degree 2, total $= 0.0061$ . Ye U ka bottom hai: degree 1 bias-dominated hai (left side), degree 10 variance-dominated hai (right side). Dekho 3.4.07-Polynomial-Regression ki kyun higher degree flexibility laata hai stability ki cost pe.

Level 3 — Analysis

Mechanism explain karo, sirf compute mat karo.

Recall Solution

Knob $k$ = kitne neighbours vote karte hain.

$k = 1$ (kam voters): ek training pixel apne poore neighbourhood ka faisla karta hai → boundary jagged hai, har point ko wrap karti hai. Bias low (koi bhi shape trace kar sakta hai), variance high (ek noisy pixel poora region flip kar deta hai; alag image bahut alag map deti hai).
$k = 100$ (zyada voters): tum bahut badi crowd pe average karte ho → boundary smooth hai, almost straight. Bias high (sharp river/forest edge follow nahi kar sakta), variance low (averaging noise cancel karta hai, to alag images similar maps deti hain).

Rule: bada $k$ = simpler model = U pe left→right hona right→left ban jaata hai. Bada $k$ ↑bias ↓variance; chhota $k$ ↓bias ↑variance.

Recall Solution

(a) Zyada data. Variance measure karta hai ki predictions alag-alag training sets mein kitna wobble karti hain. Bahut zyada data ke saath, koi bhi akela noisy point kam weight carry karta hai, to re-sampling fit ko kam change karta hai → variance girta hai. Lekin agar model ki form pehle se truth represent karne ke liye kaafi flexible thi, to uski bias unchanged rehti hai — zyada data tumhare assumptions nahi badalta, sirf unhe kitna firmly pin kiya jaata hai. Data variance pe attack karta hai, bias pe nahi.

(b) Network shrink karo. Kam parameters = kam flexibility = variance girta hai lekin bias badhta hai (ab tum bahut simple ho sakte ho). Ye ek move mein trade-off hai.

Jab data available ho tab best: (a), kyunki ye variance ko bina kisi bias penalty ke girata hai — ek rare free lunch. Jab data expensive ho (satellites!), tum (b) ya regularization pe fall back karte ho (5.6.03-Overfitting-and-Regularization).

Level 4 — Synthesis

Ideas combine karo; ek procedure design karo.

Recall Solution

(i) Dono curves ek high plateau pe converge hoti hain aur karib-karib rehti hain. Close curves = low variance; high plateau = high error jis se model escape nahi kar sakta = high bias / underfitting.

(ii) Useful:

Model capacity badhao (deeper network, higher polynomial degree — 3.4.07-Polynomial-Regression).
Richer features add karo (jaise airspeed×pitch-rate jaise interaction terms).

Yahan useless: zyada data collect karna — curves pehle se flatten aur converge ho chuki hain, to extra samples kuch nahi badalta. Ye bias-limited model ki tell-tale sign hai.

Recall Solution

Recipe:

Data split karo → train / validation / test (5.6.02-Training-Validation-Test-Sets); test set ko vault mein lock karo.
Har candidate degree $d = 1, 2, \dots, 10$ ke liye: train+validation data pe k-fold cross-validation (5.6.05-Cross-Validation-Techniques) use karke error estimate karo. Cross-validation multiple splits pe average karta hai, jo seedha variance term estimate karta hai ye dekh ke ki folds mein fit kitna badalta hai.
Averaged CV error vs. degree plot karo — ye U-shaped hona chahiye.
U ke bottom pe degree chunno (yahan, degree 2). Ye 5.6.08-Hyperparameter-Tuning miniature mein hai: degree ek hyperparameter hai.
Sirf abhi locked test set ko ek baar touch karo, honest final error report karne ke liye.

Protecting tool: cross-validation — ye tumhe test set burn kiye bina generalisation error estimate karne deta hai kai baar.

Level 5 — Mastery

Prove ya derive karo; edge cases ke against defend karo.

Recall Solution

Parent note mein prove ki gayi decomposition se start karo: $E [(y - \hat{f} (x))^{2}] = \geq 0 Bias^{2} + \geq 0 Variance + σ^{2} .$

$Bias^{2} = (f (x) - E [\hat{f} (x)])^{2} \geq 0$ — ye ek real number squared hai.
$Variance = E [(\hat{f} (x) - E [\hat{f} (x)])^{2}] \geq 0$ — squares ka average hai.

$σ^{2}$ mein do non-negative quantities add karna use sirf badha sakta hai: $E [(y - \hat{f} (x))^{2}] \geq σ^{2} .$ Equality tab hogi jab $Bias^{2} = 0$ aur $Variance = 0$ simultaneously — matlab model true $f (x)$ ke average pe equal hai AND saare training sets mein perfectly stable hai. Ye "perfect estimator" practically almost unreachable hai, isliye $σ^{2}$ ko irreducible error kaha jaata hai. ∎

Recall Solution

Assumption used: noise $ϵ$ ka mean zero hai aur ye training-based estimate $\hat{f} (x)$ se independent hai (new test point $y$ pe noise wahi noise nahi hai jis pe model ne train kiya tha).

Independence se, expectation factorise hoti hai: $E [ϵ (f (x) - \hat{f} (x))] = E [ϵ] \cdot E [f (x) - \hat{f} (x)] .$ Kyunki $E [ϵ] = 0$ hai, poora cross term $2 \cdot 0 \cdot (\dots) = 0$ hai. ✔

Agar independence fail ho jaaye — jaise agar test point ka noise training mein leak ho jaaye (data leakage) — to $E [ϵ \hat{f} (x)] \neq = E [ϵ] E [\hat{f} (x)]$ hoga, cross term survive karega, aur clean "Bias² + Variance + noise" split collapse ho jaayegi. Reported error artificially low lagegi. Ye L4 test-set-leak trap ki mathematical shadow hai.

Recall Solution

Hum ek girte hue term ( $σ^{2} / k$ , variance) aur ek chadhte hue term ( $k^{2} / n^{2}$ , bias²) ke sum ko minimise kar rahe hain — exactly U-shape. Yahan calculus kyun? Kyunki "U ka bottom" matlab slope zero hai; derivative wo tool hai jo zero slope locate karta hai.

$k$ ke respect mein differentiate karo aur zero set karo:

\;\Longrightarrow\; \frac{2k}{n^2}=\frac{\sigma^2}{k^2} \;\Longrightarrow\; k^3=\frac{\sigma^2 n^2}{2}.$$ $$\boxed{\,k^\star=\left(\frac{\sigma^2 n^2}{2}\right)^{1/3}\,}$$ **Second derivative ki sanity:** $T''=2\sigma^2/k^3+2/n^2>0$, to ye genuinely ek minimum hai (U upar ki taraf khulta hai). ✔ **Limits:** - $k\to 1$: variance term $\sigma^2/1$ blow up karta hai → **overfitting corner**. - $k\to n$: bias² term $(n/n)^2=1$ dominate karta hai, model *sab kuch* average karta hai → flat prediction → **underfitting corner**. Optimum beech mein baitha hai, aur zyada data ke saath badhta hai ($k^\star \propto n^{2/3}$) — zyada flights ke saath tum ek bada, zyada stable neighbourhood afford kar sakte ho.

Recall Self-test cloze

Total expected error bias squared, variance, aur irreducible error mein split hoti hai.

High bias with train ≈ test ::: underfitting; model ko zyada complex banao ya features add karo. Low train error but high test error ::: overfitting (high variance); simplify karo, regularize karo, ya data add karo. Tool jo test set touch kiye bina generalisation error estimate karta hai ::: cross-validation. Wo ek error term jo koi model reduce nahi kar sakta ::: the irreducible noise $σ^{2}$ .