Bias-variance trade-off: Visual walkthrough — Hinglish deep dive

5.6.4 · D2 · Coding › Machine Learning (Aerospace Applications) › Bias-variance trade-off

Yeh page parent result — the error decomposition — ko bilkul zero se rebuild karta hai, ek sequence of pictures ke roop mein. Iske end tak tum dekh paoge ki kyun har prediction error exactly teen pieces mein split hoti hai, aur kyun tum teeno ko ek saath zero nahi kar sakte.

Koi bhi symbol aane se pehle, teen plain-word promises:

Ek target woh true number hai jo hum jaanna chahte hain.
Ek guess woh hai jo hamara trained model output karta hai.
Ek average ( $E [\cdot]$ , padho "expected value") matlab hai: poora experiment kai baar repeat karo aur brackets ke andar ki cheez ka mean lo. $E$ ka is page par bas yehi matlab hai.

Step 1 — Target ek true curve hai jisme thoda noise ka kaampna hai

KYA. Hum kehte hain ki real measured value hai $y = true value at x f (x) + random shiver ϵ$

$x$ — woh input jo hum feed karte hain (maan lo, angle of attack).
$f (x)$ — woh true, unknowable answer jo nature bina kisi measurement error ke deta.
$ϵ$ (epsilon) — ek tiny random wobble jo har baar measure karne par add hota hai: sensor jitter, gusts, rounding.

KYUN. Koi bhi real instrument perfect nahi hota. Agar hum pretend karein ki $y = f (x)$ exactly hai, toh hum pehle hi reality ke baare mein jhooth bol chuke hain. $y$ ko ek fixed part aur ek random part mein split karna honest starting point hai.

PICTURE. Smooth curve $f (x)$ hai. Scattered dots actual measurements $y$ hain — har dot curve ke thoda upar ya neeche $ϵ$ ke barabar baitha hai.

Step 2 — Hamara model ek guess hai jo guesses ki poori family se drawn hai

KYA. Hum apne trained model ko $\hat{f} (x)$ kehte hain (padho "f-hat"). Hat ka matlab hai estimate — $f (x)$ paane ki hamari koshish.

KYUN. Yahan woh crucial idea hai jo zyaadatar log miss karte hain: $\hat{f}$ is baat par depend karta hai ki hamein kaun sa training set mila. Aaj 15 wind-tunnel runs par train karo, kal 15 alag runs par, aur har baar ek alag curve milega. Toh $\hat{f} (x)$ khud random hai — prediction time par measurement noise ki wajah se nahi, balki us data ki wajah se jo ise raise kiya.

PICTURE. Kai pale training-set draws mein se har ek thoda alag fitted curve deta hai (thin lines). Unka average curve bold wala hai — woh average $E [\hat{f} (x)]$ likha jaata hai.

$\hat{f} (x)$ — ek thin curve (ek training set).
$E [\hat{f} (x)]$ — bold curve, saare possible training sets par average model. Yeh ek fixed, non-random reference hai jis par hum bahut zyaada lean karenge.

Step 3 — Woh quantity likhte hain jis ki humein actually parwah hai

KYA. Ek point $x$ par expected squared error: $E [(y - \hat{f} (x))^{2}]$

$(y - \hat{f} (x))$ — ek real measurement aur hamare guess ke beech ka gap.
ise square karna — har gap ko positive banata hai aur bade misses ko chhote misses se zyaada penalize karta hai.
$E [\cdot]$ iske around — us squared gap ko dono sources of randomness par average karo: noise $ϵ$ aur $\hat{f}$ mein training-set-driven wobble.

Squared kyun, absolute value kyun nahi? Squaring woh tool hai jo algebra ko cleanly split karne deta hai — cross-terms Steps 1–2 ke zero-mean rules ki wajah se vanish ho jaayenge. Absolute value is tarah factor nahi hoti. Woh clean split hi poora payoff hai.

PICTURE. Vertical red bar ek raw gap $y - \hat{f} (x)$ hai; hum ise square karte hain, phir poore dot-cloud par average karte hain.

Step 4 — "Add-zero" trick se noise ko alag karo, part one

KYA. $y = f (x) + ϵ$ substitute karo aur expand karo:

= E\big[(f(x)-\hat{f}(x))^2\big] + 2\,E[\epsilon]\,E[f(x)-\hat{f}(x)] + E[\epsilon^2]$$ Right side par term by term: - $E\big[(f(x)-\hat{f}(x))^2\big]$ — hamare model aur *true curve* ke beech ki doori (noise remove karne ke baad). - $2\,E[\epsilon]\,E[f(x)-\hat{f}(x)]$ — ek cross-term. Kyunki $E[\epsilon]=0$ (Step 1), yeh **poora term mar jaata hai**. - $E[\epsilon^2] = \sigma^2$ — raw noise variance, bina kisi change ke survive karta hai. **KYUN.** Hum reality ki apni randomness ($\epsilon$) ko model ke error se alag kar rahe hain. Noise model se independent hai, isliye iska cross-term average out ho jaata hai, aur sirf ek akela $\sigma^2$ reh jaata hai. **PICTURE.** Total gap ek chain hai: model→true-curve (blue) aur true-curve→measured-dot (yellow, woh $\epsilon$ hai). Blue aur yellow pieces correlate nahi karte, isliye unka cross-term vanish ho jaata hai. ![[deepdives/dd-coding-5.6.04-d2-s04.png]] Ab tak: $$E\big[(y-\hat{f}(x))^2\big] = \underbrace{E\big[(f(x)-\hat{f}(x))^2\big]}_{\text{still to split}} + \underbrace{\sigma^2}_{\text{irreducible}}$$ > [!definition] Irreducible error > ==$\sigma^2$== **irreducible error** hai — data mein baki hua noise. Koi bhi model, chahe kitna bhi clever ho, ise remove nahi kar sakta. Yeh ek floor set karta hai ki error kitni chhoti ho sakti hai. --- ## Step 5 — Phir zero add karo, is baar *average model* ke around **KYA.** Bacha hua term $E\big[(f(x)-\hat{f}(x))^2\big]$ lo aur $+E[\hat{f}(x)]-E[\hat{f}(x)]$ (ek fancy zero) insert karo: $$f(x)-\hat{f}(x) \;=\; \underbrace{\big(f(x)-E[\hat{f}(x)]\big)}_{\text{fixed offset}} + \underbrace{\big(E[\hat{f}(x)]-\hat{f}(x)\big)}_{\text{random wobble}}$$ - $f(x)-E[\hat{f}(x)]$ — *true curve* se *average model* tak ki doori. Yeh ek **constant** hai — isme koi randomness nahi bachi. - $E[\hat{f}(x)]-\hat{f}(x)$ — ek particular model apne average model se kitna bhatak jaata hai. Yeh **random wobble** hai, aur iska mean *by construction* zero hota hai. **KYUN.** Hum do bilkul alag failures alag kar rahe hain: (a) average model ka systematically galat hona, aur (b) individual models ka us average ke around scatter karna. Yahi do baaki bache hue villains hain. **PICTURE.** True point $f(x)$ se: bold average model tak ek fixed blue arrow, phir average se har ek individual thin model tak ek pink zig-zag. ![[deepdives/dd-coding-5.6.04-d2-s05.png]] --- ## Step 6 — Ise square karo aur dekho middle term kaise mar jaata hai **KYA.** Step 5 ke sum ko square karke $E[\cdot]$ lete hain: $$\big(f(x)-E[\hat{f}(x)]\big)^2 + 2\big(f(x)-E[\hat{f}(x)]\big)\,\underbrace{E\big[E[\hat{f}(x)]-\hat{f}(x)\big]}_{=\,0} + E\big[(E[\hat{f}(x)]-\hat{f}(x))^2\big]$$ - **Pehla term** — constant squared, average ke through seedha nikalta hai bina kisi change ke. - **Middle term** — wobble ka average $E[\hat f]-E[\hat f]=0$ hai, isliye yeh cross-term **vanish ho jaata hai** (doosra clean kill, Step 4 jaisi hi trick). - **Teesra term** — average squared wobble, survive karta hai. **KYUN.** Wahi zero-mean magic. Constant offset aur zero-mean wobble correlate karne se mana karte hain, isliye koi cross-term sum ko contaminate nahi karta. **PICTURE.** Teen pieces side by side rakhe hain: ek fixed blue square, ek pink cloud, aur vanishing cross-term jis par cross laga hua hai. ![[deepdives/dd-coding-5.6.04-d2-s06.png]] --- ## Step 7 — Dono survivors ko naam do **KYA.** Dono surviving pieces pehle se hi parent note ke definitions hain: $$\text{Bias}^2 = \big(f(x)-E[\hat{f}(x)]\big)^2 \qquad\qquad \text{Variance} = E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big]$$ - ==Bias== — **average** model truth ko kitna miss karta hai. Ek *systematic* aim error. Underfitting. - ==Variance== — ek **single** model apne average ke aas-paas kitna jiggle karta hai. Ek *stability* error. Overfitting. **KYUN.** Bias poochh ta hai "kya mera dartboard aim centered hai?" Variance poochh ta hai "kya mere darts tightly grouped hain?" Yeh genuinely alag questions hain, isliye inhein alag naam mile. **PICTURE.** Dartboard picture: high-bias/low-variance = off-center tight group; low-bias/high-variance = centered lekin scattered. ![[deepdives/dd-coding-5.6.04-d2-s07.png]] > [!formula] Assembled result > $$\boxed{\,E\big[(y-\hat{f}(x))^2\big] = \text{Bias}^2 + \text{Variance} + \sigma^2\,}$$ > Teen villains, koi leftover nahi. Yeh parent ke decomposition se exactly match karta hai. ∎ --- ## Step 8 — Degenerate cases (koi gap kabhi mat chhodna) Formula ko apne extreme corners mein survive karna chahiye. Sab check karo: > [!example] Corner cases > - **Perfect model, $\hat f = f$ always.** Toh $E[\hat f]=f$, isliye $\text{Bias}=0$ aur $\text{Variance}=0$. Error $=\sigma^2$ — noise floor. Tum **nahi** beat kar sakte ise. Isliye $\sigma^2$ ko irreducible kaha jaata hai. > - **Constant model, $\hat f(x)=c$** (chahe kuch bhi ho, wahi number predict karta hai). Variance $=0$ (yeh kabhi wobble nahi karta — har training set mein same output), lekin Bias bahut bada hai jahan bhi $f(x)\neq c$. Pure underfitting. > - **Interpolating model** (15 points ke through degree-10 polynomial, ya $k{=}1$ nearest neighbour). Bias $\approx 0$ (itna flexible ki kisi bhi shape ko hit kar sake), lekin Variance explode ho jaati hai — naaye dots par retrain karo aur curve thrash karne lagta hai. Pure overfitting. > - **Zero noise, $\sigma^2=0$.** Irreducible term disappear ho jaata hai; error $=\text{Bias}^2+\text{Variance}$ sirf. Yahan bhi tum finite data ke saath dono ko zero nahi kar sakte. **YEH KYUN MATTER KARTE HAIN.** Har real model constant aur interpolator ke beech kahin rehta hai. Dekho [[5.6.03-Overfitting-and-Regularization]] ki regularization kaise interpolator corner se wapas slide karta hai, aur [[5.6.05-Cross-Validation-Techniques]] ki hum variance corner ko test data dekhe bina kaise *measure* karte hain. **PICTURE.** U-curve: x-axis par complexity, y-axis par error. Bias girta hai, variance badhti hai, unka sum beech mein dip karta hai — woh sweet spot hai. Teen corner models usme pin kiye hue hain. ![[deepdives/dd-coding-5.6.04-d2-s08.png]] > [!mistake] Trap > "Zyaada complex = hamesha better." Nahi. U-curve ke bottom ke baad, bias mein har gain variance mein loss se *zyaada kaafi paid for* ho jaata hai. Parent ka degree-10 drag example yahan land karta hai: $\text{Bias}^2\approx0.0001$ lekin $\text{Variance}\approx0.08$ — variance sab par bhaari pad jaata hai. [[3.4.07-Polynomial-Regression]] compare karo aur degree ambition se nahi, validation se choose karo — [[5.6.08-Hyperparameter-Tuning]]. --- ## Ek-picture summary Sab kuch ek board par: true curve, noisy dots, average ke around scattered model family, aur teen arrows — **Bias** (average-model se truth tak), **Variance** (ek model se average-model tak), **Noise** (truth se dot tak) — boxed sum mein snap ho rahe hain. ![[deepdives/dd-coding-5.6.04-d2-s09.png]] > [!recall]- Feynman retelling — ek 12-saal ke bachchhe ko batao > Socho tum darts phenk rahe ho. Bullseye *true answer* hai. Lekin board khud thoda hawa mein kaampti hai — woh kaampna **noise** ($\sigma^2$) hai, aur koi bhi thrower ise fix nahi kar sakta. Ab, har baar jab tum apna model retrain karte ho tum ek dart phenktte ho. Kai baar karo: tumhare dart cluster ka *center* kahan hai? Agar woh center bullseye se off hai, toh woh **bias** hai — tumhara aim systematically galat hai (too-simple model). Aur cluster kitna *spread out* hai? Woh spread **variance** hai — ek over-eager model jo har baar retrain hone par kahin naya land karta hai. Tumhari total wrongness exactly aim-error-squared **plus** spread **plus** board ka kaampna hai. Poori derivation bas algebra hai jo prove karti hai ki yeh teen bina cross-talk ke add hote hain, kyunki noise aur wobble dono average to zero karte hain aur mix karne se mana karte hain. Punchline: apni grip tight karo (simpler model) aur cluster shrink ho jaata hai lekin center se drift karta hai; loose karo (complex model) aur tum re-center hote ho lekin scatter ho jaate ho. Sweet spot U ka bottom hai. > [!recall]- Apne aap check karo > Step 4 mein cross-term kyun vanish hota hai? ::: Kyunki $E[\epsilon]=0$ — noise zero-mean hai, isliye $2E[\epsilon]E[f-\hat f]=0$. > Step 6 mein cross-term kyun vanish hota hai? ::: Kyunki $E[E[\hat f]-\hat f]=E[\hat f]-E[\hat f]=0$ — average model ke around wobble by construction zero-mean hai. > $\sigma^2$ kya hai aur kya koi better model ise reduce kar sakta hai? ::: Yeh data mein hi irreducible noise variance hai; koi bhi model ise reduce nahi kar sakta — yeh error floor hai. > Joh model har input ke liye same constant output karta hai usmein kaun si error high hoti hai aur kaun si zero? ::: High bias, zero variance. > Bias *average* model ka miss measure karta hai ya *single* model ka? ::: Average model ka, $E[\hat f(x)]$.