Bias-variance trade-off: Question bank — Hinglish deep dive

5.6.4 · D5 · Coding › Machine Learning (Aerospace Applications) › Bias-variance trade-off

Yeh ek self-test page hai. Har line mein ::: ke baad ek reveal chupi hai. Prompt padho, apna jawab zor se ek reason ke saath bolo, phir reveal karo. Sirf "true"/"false" bolne se kuch nahi milega — reasoning hi asli point hai. Yahan har trap ek specific misconception ko target karti hai jo yeh topic reliably produce karta hai.

Agar koi bhi word unfamiliar lage, pehle parent note padho: Bias-Variance Trade-off (ya Hinglish version). Related ideas in jagahon par hain: 5.6.03-Overfitting-and-Regularization, 5.6.05-Cross-Validation-Techniques, 5.6.08-Hyperparameter-Tuning, aur 3.4.07-Polynomial-Regression.

Notation recap — pehle yeh padho

Kisi bhi trap se pehle, har symbol ko pin down kar lete hain taaki letters ke matlab par kabhi argument na ho.

Definition Woh chaar symbols jinpar yeh poori page tiki hai

$x$ ek input hai (jaise angle of attack). $y$ uske liye measured noisy output hai.
$f (x)$ true function hai — woh perfect answer jo nature bina noise ke deta. Hum ise actually kabhi nahi jaante, lekin yeh exist karta hai.
$\hat{f} (x)$ (padho "f-hat") hamare learned model ki prediction hai $x$ par. Hat ka matlab hamesha hai "ek training set se produce kiya gaya estimate." Hum ise $\hat{f} (x)$ likhenge har jagah — kabhi bare $\hat{f}$ nahi, kabhi plain $f$ nahi — confusion avoid karne ke liye.
$E [\cdot]$ expectation (average) operator hai. $E [Z]$ ka matlab hai "socho ki experiment hamesha repeat ho raha hai aur quantity $Z$ ko average karo." Yahan jo experiment repeat ho raha hai woh hai ek fresh training set draw karna aur re-train karna, toh $E [\hat{f} (x)]$ hai sabhi possible training sets par average prediction.

Recall

$σ^{2}$ ke peeche ka noise model (derivation traps ke liye zaruri) Hum assume karte hain $y = f (x) + ϵ$ , jahan $ϵ$ random noise hai jiske liye $E [ϵ] = 0$ aur $E [ϵ^{2}] = σ^{2}$ , aur importantly $ϵ$ training procedure se independent hai — ek naye test point par noise kuch aisa nahi hai jo hamara model $\hat{f}$ dekh sake ya seekh sake. Wahi independence hai jo $E [ϵ \cdot (f (x) - \hat{f} (x))] = E [ϵ] E [f (x) - \hat{f} (x)] = 0$ banati hai, cross-term ko khatam karti hai. Agar noise independent nahi hoti (jaise woh features mein leak ho jaati), toh woh factorization fail ho jaati aur clean three-term split toot jaata.

True or false — justify karo

True or false: Zero training error wale model ka bias low hona chahiye.

False. Zero training error ka sirf yeh matlab hai ki woh unhe dekhe gaye points fit karta hai; yeh high variance ka symptom hai (noise memorize karna), low bias ka nahi. Ek degree-10 polynomial 15 points se guzarti hui ~0 training error rakhti hai lekin generalize bahut buri tarah karti hai.

True or false: Agar zyaada training data collect karo, toh high bias eventually khatam ho jayega.

False. Step by step sochte hain: bias hai

E [\hat{f} (x)] - f (x)

. Jaise sample size

N \to \infty

hota hai,

\hat{f}

converge karta hai apni family ke andar best fit ki taraf — use

\hat{f}_{\infty}

bolte hain. Agar family hai "sabhi straight lines" lekin

f (x)

quadratic hai, toh

\hat{f}_{\infty}

ab bhi best straight line hai, aur

E [\hat{f}_{\infty}] - f (x) \neq = 0

. Zyaada data variance ko ~0 kar deta hai lekin yeh assumption-driven bias frozen chhod deta hai.

True or false: Zyaada training data variance ko reduce karta hai.

True. Variance hai

E [(\hat{f} (x) - E [\hat{f} (x)])^{2}]

, estimate ka scatter. Zyaada samples par average karne se koi bhi ek noisy point kam matter karta hai, toh fresh dataset par re-train karne se zyaada consistent predictions milti hain; bahut saare estimators ke liye variance

\sim σ^{2} / N

ki tarah scale karta hai,

N

badhne par khatam ho jaata hai.

True or false: Irreducible error

σ^{2}

ko ek achhe model se zero tak drive kiya ja sakta hai.

False.

σ^{2} = E [ϵ^{2}]

data mein hi noise hai (sensor jitter, unmeasured variables). Yeh total error mein $\hat{f}$ se regardless appear karta hai, toh koi bhi model — chahe kitna bhi clever ho — genuine randomness predict nahi kar sakta; yeh ek hard floor set karta hai.

True or false: High-variance model ka bias hamesha low hota hai.

Usually lekin guaranteed nahi. Extra flexibility bias ko lower karne ki tendency rakhti hai, lekin ek badly-designed flexible model (galat features, buri optimization) ek saath biased aur unstable dono ho sakta hai.

True or false: Regularization bias badhata hai.

True, aur yahi point hai. Regularization

\hat{f}

ke parameters ko zero ki taraf shrink karta hai,

E [\hat{f} (x)]

f (x)

se door kheeenchta hai (zyaada bias) lekin dart spread ko tight karta hai (kam variance) — yeh ek acha trade hai jab overfitting ho rahi ho.

True or false: Bias-variance decomposition

E [(y - \hat{f} (x))^{2}] = Bias^{2} + Variance + σ^{2}

classification accuracy ke liye utni hi tarah hold karta hai jitna squared error ke liye.

False. Woh clean three-term additive split squared-error loss ki ek special property hai. 0/1 ya cross-entropy loss ke under "decomposition" ab bhi ek systematic part aur ek variability part ko alag karti hai, lekin terms multiplicatively / non-additively combine hoti hain (bias variance ke flip predictions mein help bhi kar sakta hai), toh yeh simple sum nahi hai.

True or false: Identical test error wale do models ka bias aur variance identical hona chahiye.

False. Test error hai

Bias^{2} + Variance + σ^{2}

; ek high-bias/low-variance model aur ek low-bias/high-variance model bilkul opposite directions se same total par land kar sakte hain.

True or false: U-shaped total-error curve par minimum wahan hota hai jahan bias variance ke barabar hota hai.

False. Minimum wahan hota hai jahan slopes balance karte hain:

\frac{d ( Bias ^{2} )}{d c} + \frac{d ( Variance )}{d c} = 0

(complexity

c

), yaani jahan girta hua bias aur badhta hua variance equal-and-opposite rates par change karte hain. Yeh generally woh point nahi hai jahan do values equal hain — s02 figure dekho.

Error dhundho

"Mere model ki 99% training accuracy aur 99% test accuracy hai, toh yeh overfit ho raha hai."

Error: overfitting ek gap ke roop mein dikhti hai — high training, low test. Matching high scores ka matlab hai yeh achhe se generalize kar raha hai; kuch galat nahi hai.

"Main underfitting ko L2 regularization add karke fix karunga."

Error: regularization variance se ladne ke liye bias badhata hai. Underfitting ke liye (already high bias) tum opposite chahte ho — ek zyaada flexible model ya kam constraints.

"Variance error hai, bias sirf ek bonus term hai jise hum ignore kar sakte hain."

Error: dono genuine error contributions hain.

Bias^{2}

aur Variance dono expected test error mein add hote hain; bias ko ignore karna hi wajah hai ki linear models nonlinear data par fail hote hain chahe kuch bhi karo. Concrete example:

y = x^{2}

par fit ki gayi flat line near-zero variance rakhti hai lekin huge error — sab bias hai.

"k-NN ke liye, bada k zyaada overfitting ka matlab hai kyunki k badi number hai."

Error: yeh ulta hai. s03 figure dekho: k=1 ke saath boundary jagged hai aur har noisy point ke around wrap karti hai (overfit, high variance); jaise k badhta hai yeh zyaada neighbors average karta hai, nearly flat, high-bias boundary ki taraf smooth hota hai. Bada k = underfit.

"Truly quadratic data par ek degree-2 polynomial ka zero bias hai, toh uska test error zero hai."

Error: zero bias ke bawajood, Variance aur irreducible noise

σ^{2}

remain karte hain. Test error

σ^{2}

ke karib bottom out karta hai, zero par nahi.

"Cross-validation bias-variance trade-off hata deta hai."

Error: cross-validation generalization error ko zyaada reliably estimate karta hai (dekho 5.6.05-Cross-Validation-Techniques); yeh tumhein sweet spot locate karne mein help karta hai lekin trade-off ko khatam nahi karta.

"Noise term

ϵ

ka mean zero hai, toh yeh total error ko kabhi affect nahi karta."

Error:

E [ϵ] = 0

derivation mein cross term ko khatam karta hai, lekin

E [ϵ^{2}] = σ^{2}

survive karta hai aur directly total error mein land karta hai.

"Deeper neural network = hamesha lower bias, toh flight-control prediction ke liye deeper hamesha better hai."

Error: added depth bias ko lower karta hai lekin variance badhata hai aur sensor noise memorize kar sakta hai. Sweet spot ke baad, test error climb karta hai — U-curve depth par bhi apply hota hai.

"Squared error hi ek aisi loss hai jahan bias aur variance matter karte hain."

Error: yeh kisi bhi loss ke under matter karte hain. Squared error sirf woh hai jahan woh additively split hote hain. Cross-entropy / logistic loss ke liye bhi tumhare paas ek systematic term hai (tumhara average predicted probability off hai) aur ek variability term (training set ke saath predictions swing karte hain); algebra alag hai lekin shaky-vs-mis-aimed intuition identical hai.

Why questions

Decomposition mein middle cross-term

2 E [ϵ (f (x) - \hat{f} (x))]

kyun vanish ho jaata hai?

Kyunki test-point noise

ϵ

us training procedure se independent hai jisne

\hat{f} (x)

produce kiya, product ka expectation factor ho jaata hai:

E [ϵ (f - \hat{f})] = E [ϵ] E [f - \hat{f}]

. Phir

E [ϵ] = 0

ise zero kar deta hai, sirf

Bias^{2}

, Variance, aur

σ^{2}

bachte hain. Us independence ke bina product factor nahi hota aur term survive kar sakti thi.

Hum complexity tune karke bias aur variance ko simultaneously minimize kyun nahi kar sakte?

Kyunki complexity unhe ek dial par opposite directions mein move karti hai: zyaada flexibility bias lower karti hai lekin variance badhati hai. Sabse best jo tum kar sakte ho woh hai unka sum minimize karna, dono alag-alag nahi.

20 points par ek 50-degree polynomial har retrain par wildly different curves kyun deta hai?

Jab parameters

p

data points

N

se zyaada hote hain (

p > N

), model ke paas har dataset ki specific noise ko "chase" karne ki slack hoti hai; alag noise → drastically alag fits → high variance. Yeh

N < p

regime hai jo neeche edge cases mein discuss kiya gaya hai.

k=1 nearest-neighbor training par 0% error guarantee kyun karta hai lekin new data par often terrible hota hai?

Har training point apna khud ka nearest neighbor hota hai, toh yeh apna khud ka label perfectly reproduce karta hai — lekin iska matlab yeh bhi hai ki yeh har noisy label bhi faithfully reproduce karta hai, jagged, unstable boundaries deta hai (s03 dekho) jo fresh pixels par fail hoti hain.

Hum aerospace mein specifically is trade-off ki itni parwah kyun karte hain?

Aerospace data (flight tests, satellite passes) expensive hai aur operating conditions unseen aur safety-critical hain; ek model jo ek dataset overfit karta hai woh next real mission par catastrophically fail kar sakta hai.

Ek acha regularizer add karna often total test error lower kyun karta hai bias badhane ke bawajood?

Jo bias woh add karta hai woh thoda hai, lekin jo variance woh remove karta hai woh zyaada hai; kyunki total error

Bias^{2} + Variance

sum karta hai, ek chhota increase plus ek bada decrease net mein lower total deta hai.

True function

f (x)

Variance term mein kabhi appear kyun nahi karta?

Variance measure karta hai ki

\hat{f} (x)

apne khud ke average

E [\hat{f} (x)]

ke around kitna scatter karta hai; us average se truth ki doori separately bias mein capture hoti hai, in do concepts ko cleanly alag rakhte hue.

Edge cases

Ek model ka bias kya hai jo hamesha

f

ka exact true mean predict karta hai lekin uske around huge scatter rakhta hai?

Zero bias — uska average prediction truth par hit karta hai, toh

E [\hat{f} (x)] - f (x) = 0

— lekin scatter se bada variance, toh uska error entirely variance term se dominate hota hai.

Fixed data ke saath infinite model complexity ki limit mein bias aur variance ka kya hota hai?

Bias → near zero (yeh kuch bhi represent kar sakta hai), lekin variance blow up karta hai (yeh har noise wiggle fit karta hai); total error upar diverge karta hai — s02 figure mein U-curve ka right end.

Opposite limit mein kya hota hai — ek "model" jo

x

ignore karta hai aur ek fixed constant predict karta hai?

Variance essentially zero hota hai (training set se regardless same output), lekin bias maximum hota hai jab tak truth us constant na ho; yeh extreme-underfitting U-curve ka left end hai.

Agar data mein zero noise hai (

σ^{2} = 0

), kya total test error phir bhi nonzero ho sakta hai?

Haan. Irreducible noise na hone ke bawajood, ek mismatched model

Bias^{2}

aur finite-sample Variance contribute karta hai, toh error tabhi zero hai jab model bhi truth se perfectly match kare aur fully determined ho.

Jab data points

N

ki sankhya parameters

p

ke roughly barabar ho (

N \approx p

), toh variance mein typically kya hota hai?

Yeh typically spike karta hai. Har parameter ko pin down karne ke liye barely enough equations ke saath, fit noise-corrupted points se fully determined hota hai aur uske paas ise damp karne ke liye koi averaging nahi bachti — yeh notorious "interpolation threshold" hai jahan test error often peak karta hai (kuch over-parameterized models mein) phir se neeche aane se pehle.

Strict

N < p

regime mein (parameters se kam data points) kya hota hai?

Training system under-determined hota hai: infinitely many parameter settings data ko exactly fit karte hain (training error 0), aur tum kis par land karte ho woh entirely optimizer aur noise par depend karta hai — enormous variance, textbook overfitting.

Fixed model ke liye variance sample size

N

ke saath roughly kaise scale karta hai?

Fixed-complexity estimator ke liye variance

\sim σ^{2} / N

ki tarah shrink karta hai: data quadruple karo, prediction ka standard deviation roughly halve ho jaata hai. Bias, assumption-driven hone ke naate, yeh follow nahi karta aur wahan ruk jaata hai.

Agar tumhare paas infinitely many independent training sets hote aur tum unke saare models average kar lete, toh kaun sa error term tum eliminate kar lete?

Tum Variance wipe out kar lete (averaging dataset-to-dataset scatter cancel karta hai), sirf

Bias^{2} + σ^{2}

bachta — yahi exactly wajah hai ki ensembling/bagging high-variance models ki help karta hai.

Noiseless data par perfectly correct model ke liye theoretical total error kya hai?

Exactly zero:

Bias^{2} = 0

, Variance

= 0

, aur

σ^{2} = 0

. Yeh ideal real aerospace data mein essentially kabhi nahi hota, isliye trade-off hamesha bite karta hai.

Recall Lock in karne ke liye one-line summary

Bias $= E [\hat{f} (x)] - f (x)$ = on average galat (underfit); Variance $= E [(\hat{f} (x) - E [\hat{f} (x)])^{2}]$ = datasets ke across inconsistent (overfit); $σ^{2}$ = unbeatable noise floor. Total error unka sum hai, aur complexity pehle ko doosre ke liye trade karta hai.