Cross-validation — k-fold: Exercises — Hinglish deep dive

5.6.5 · D4 · Coding › Machine Learning (Aerospace Applications) › Cross-validation — k-fold

Poore note mein, $n$ = number of samples, $k$ = number of folds, $Score_{i}$ = fold $i$ par measure ki gayi metric jab woh test set hota hai, aur $CV_{k} = \frac{1}{k} \sum_{i = 1}^{k} Score_{i} .$

Level 1 — Recognition

Exercise 1.1 (L1)

Tum $5$ -fold cross-validation run karte ho. Total kitne models train hote hain, aur poori procedure mein har ek single data point kitni baar test set mein aata hai?

Recall Solution

Models trained: har fold ke liye ek, toh $== 5 ==$ models.
Test appearances per point: k-fold ka poora point yahi hai ki har sample test fold mein exactly ek baar aata hai. Toh har point $== 1 ==$ baar test hota hai (aur baaki $k - 1 = 4$ rounds mein training ke liye use hota hai).

Figure s01 dekho: koi bhi single row ko chaar columns ke across follow karo — woh exactly ek column mein "test" (pink) rang ka hai aur baaki teen mein "train" (blue).

Exercise 1.2 (L1)

Ek dataset mein $n = 200$ samples hain jo equal size ke $k = 4$ folds mein split hain. Loop ke ek round mein training set mein kitne samples hain aur test set mein kitne?

Recall Solution

Har fold mein $n / k = 200/4 === 50 ==$ samples hote hain.

Test set = ek fold = $50$ samples.
Training set = baaki $k - 1 = 3$ folds = $3 \times 50 === 150 ==$ samples.

Exercise 1.3 (L1)

Teen fold scores aate hain $[0.80, 0.86, 0.83]$ . $CV_{3}$ kya hai?

Recall Solution

$CV_{3} = \frac{0.80 + 0.86 + 0.83}{3} = \frac{2.49}{3} === 0.83 == .$ Hum mean lete hain kyunki independent estimates ko average karne se variance kam hota hai — kisi ek fold ki kismat dominant nahi hoti.

Level 2 — Application

Exercise 2.1 (L2)

Tumhare paas $n = 1000$ flight records hain aur tum $k = 5$ choose karte ho. Har fold ke liye training set ka size likho, aur confirm karo ki data ka kitna fraction har model train karta hai.

Recall Solution

Fold size $= 1000/5 = 200$ .

Test set $= 200$ , training set $= 1000 - 200 === 800 ==$ .
Training ke liye use hone wala fraction $= \frac{k - 1}{k} = \frac{4}{5} === 0.8 === 80%$ .

Issi liye k-fold estimates full-data performance ke kareeb hote hain: har model phir bhi $80%$ sab kuch dekhta hai, LOOCV ke tiny hold-out ke unlike (dekho 5.6.06-Leave-one-out-cross-validation-(LOOCV)).

Exercise 2.2 (L2)

Paanch folds ne F1 scores diye $[0.82, 0.79, 0.85, 0.81, 0.83]$ . CV mean aur sample standard deviation $σ_{CV}$ compute karo (denominator mein $k - 1$ use karo).

Recall Solution

Mean: $CV_{5} = \frac{0.82 + 0.79 + 0.85 + 0.81 + 0.83}{5} = \frac{4.10}{5} === 0.820 == .$ Mean se deviations (har $Score_{i} - 0.82$ ): $[0.00, - 0.03, 0.03, - 0.01, 0.01] .$ Squared: $[0, 0.0009, 0.0009, 0.0001, 0.0001]$ , sum $= 0.0020$ . Sample variance: $k - 1 = 4$ se divide karo: $\frac{0.0020}{4} = 0.0005.$ Std: $0.0005 \approx== 0.0224 ==$ . Toh report karo $F1 = 0.820 \pm 0.022$ .

Hum $k - 1$ se divide karte hain, $k$ se nahi, kyunki hum mean estimate karne mein pehle hi "ek degree of freedom spend" kar chuke hain; $k$ use karne se spread systematically underestimate hoga.

Exercise 2.3 (L2)

Tumhara dataset flight date ke hisaab se sorted hai. Tum plain KFold run karte ho shuffle=False ke saath. Yeh misleadingly low CV score kyun de sakta hai, aur kya ek-word fix yeh problem hata deta hai?

Recall Solution

shuffle=False ke saath, fold 1 = sabse purani flights, fold 5 = sabse nayi. Har round ek time-contiguous block par test karta hai jo training block se systematically alag ho sakta hai (purane engines, seasonal weather, alag routes). Model se kaha ja raha hai ki woh ek distribution shift par generalise kare jis par usne kabhi train nahi kiya, toh scores girte hain — lekin yeh split ko reflect karta hai, model ko nahi. Fix: shuffle (shuffle=True), jo har fold ko saare dates ka ek random sample bana deta hai. ⚠️ Exception: agar task genuinely ek forecast hai, toh shuffle mat karo — 8.5.02-Time-series-cross-validation use karo.

Level 3 — Analysis

Exercise 3.1 (L3)

Model A $5$ folds mein $0.82 \pm 0.02$ deta hai; Model B $0.85 \pm 0.09$ deta hai. Ek colleague kehta hai "B better hai, ship karo." Dono taraf se argument karo aur batao tum actually kya karoge.

Recall Solution

B ke liye: uska mean F1 ( $0.85$ ) A ka ( $0.82$ ) beat karta hai.
B ke khilaf: uska spread ( $\pm 0.09$ ) A ka ( $\pm 0.02$ ) se chaar baar se zyada hai. B ke liye worst-plausible fold $0.85 - 0.09 = 0.76$ ke aas paas hoga, jo A ke worst fold $0.82 - 0.02 = 0.80$ se neeche hai. B inconsistent hai — uska score heavily depend karta hai ki data ka kaun sa slice woh dekh raha hai, jo ek safety task ke liye red flag hai.
Kya karna chahiye: means sirf $0.03$ apart hain jabki B ka std $0.09$ hai; difference clearly significant nahi hai. Usi folds par ek paired comparison chalao, ya $k$ /repeats badhao. Aerospace mein, woh model prefer karo jiska lower bound safest ho — aksar A. Dekho 5.6.10-Bias-variance-tradeoff-in-model-selection.

Exercise 3.2 (L3)

Tum $k$ ko $5$ se $n$ (LOOCV) tak badhate ho. Estimate ke bias aur variance mein qualitative change predict karo, aur har ek ke mechanism ko explain karo.

Recall Solution

Bias ↓ (kam): har training set mein ab $n - 1$ samples hain, almost poora dataset, toh har model ki performance almost wahi hai jo full-data model achieve karta. Chhote- $k$ models sirf $\frac{k - 1}{k}$ data par train karte hain aur thoda under-perform karte hain, estimate ko pessimistically bias karte hain.
Variance ↑ (zyada): do mechanisms. (1) Har test set ek single sample hai, toh har $Score_{i}$ $0$ ya $1$ -ish hota hai — extremely noisy. (2) $n$ training sets apne $n - 2$ samples mein overlap karte hain, toh models almost identical hain aur unki errors correlated hain; correlated numbers ko average karne se variance barely kam hoti hai.
Net: LOOCV bias ke liye variance trade karta hai, aur $n$ model fits ka cost lagta hai. Issi liye $k = 5$ ya $10$ usual sweet spot hai.

Exercise 3.3 (L3)

$1000$ flights mein se $5%$ mein failures hain ( $50$ positives). Plain shuffled KFold aur $k = 5$ ke saath, ek fold ko sirf $2$ positive labels milte hain. Numerically explain karo ki yeh us fold ka F1 kyun barbad kar deta hai, aur fix ka naam batao.

Recall Solution

Har fold mein $200$ samples hain; fold mein expected positives $50/5 = 10$ hain. Sirf $2$ positives wale fold ka matlab hai:

Recall sirf $2$ true positives par measure hoti hai — ek miss karo aur recall $1.0$ se $0.5$ par aa jaati hai; ek bhi mistake score ko wildly swing kar deti hai.
Training folds mein ab $48$ positives concentraded doosri jagah hain, toh class balance round to round alag hoti hai, extra variance inject karte hue.

Fix: Stratified k-fold (5.6.04-Stratified-sampling) har fold ko $\approx 5%$ positives rakhne par force karta hai, yani $\approx 10$ per fold, har $Score_{i}$ ko stabilise karte hue.

Level 4 — Synthesis

Exercise 4.1 (L4)

Tumhe Random Forest ka max_depth $4$ candidate values par tune karna hai aur ek honest final performance estimate report karna hai. Ek junior engineer propose karta hai: "Har depth ke liye $5$ -fold CV chalao, best CV score chunlo, aur wahi best CV score final number ke roop mein report karo." Flaw identify karo aur correct procedure design karo.

Recall Solution

Flaw: CV score max_depth choose karne ke liye use hua tha. Wahi score final estimate ke roop mein report karna optimistically biased hai — tumne ek decision lene ke liye validation folds ko dekha, toh woh ab unseen nahi rahe. $4$ CV scores mein se best inflatedly select kiya gaya hai.
Correct design — nested CV (5.6.07-Nested-cross-validation):
- Outer loop ( $5$ folds): har outer training set ko ek inner CV ko pass kiya jaata hai jo max_depth tune karta hai (yeh grid search hai, 7.2.03-Hyperparameter-tuning-grid-search).
- Winning depth poore outer-training set par refit hota hai aur ek baar untouched outer-test fold par score hota hai.
- $5$ outer-test scores ka average → honest estimate.
Key principle: jo data model select karne ke liye use hua hai woh data jispe uski performance estimate ki jaati hai se disjoint hona chahiye.

Exercise 4.2 (L4)

Total compute budget: tum zyada se zyada $60$ model fits afford kar sakte ho. Tum chahte ho ki (a) $4$ hyperparameter settings tune ho aur (b) nested, unbiased estimate mile. Outer $k_{o}$ aur inner $k_{i}$ choose karo taaki total fits $\leq 60$ rahe, aur inhe count karo.

Recall Solution

Nested-CV fit count (final refits ignore karte hue, ya alag count karte hue): $fits = k_{o} \times (settings) \times k_{i} = k_{o} \cdot 4 \cdot k_{i} .$ Try karo $k_{o} = 3, k_{i} = 5$ : $3 \times 4 \times 5 === 60 == .$ Yeh exactly budget hit karta hai. Agar tum har outer fold ke liye ek refit bhi count karo ( $k_{o} = 3$ extra fits) toh exceed ho jaayega, toh $k_{i} = 4$ par aa jao: $3 \times 4 \times 4 = 48$ fits, room rehta hai. Dono $(k_{o}, k_{i}) = (3, 5)$ ceiling par ya $(3, 4)$ headroom ke saath defensible hain; apna assumption batao.

Level 5 — Mastery

Exercise 5.1 (L5)

Ek aerospace team ke paas $n = 240$ labelled flights hain 8 alag aircraft se, har ek se $30$ flights. Failures rare hain ( $6%$ ). Woh ek aisa CV estimate chahte hain jo (i) same physical aircraft ki flights ke beech information leak na kare, aur (ii) failure rate per fold preserve kare. Splitting scheme design karo aur har choice justify karo. Phir batao kitne folds natural hain.

Recall Solution

Leakage risk: same aircraft ki do flights ek hi serial-number-specific sensor quirks share karti hain. Agar aircraft #3 ki kuch flights training mein hain aur kuch test mein, toh model general failure signatures seekhne ki jagah us aircraft ko "pehchan" sakta hai — optimistic aur unsafe. Toh splits aircraft (group) level par hone chahiye: ek aircraft ki saari $30$ flights ya poori train ya poori test mein jaati hain. Yeh group k-fold hai.
Class balance: $6%$ failures ke saath hum bhi chahte hain ki har fold $6%$ ke kareeb ho — yeh stratification ke liye argue karta hai. Dono combine karne se stratified group k-fold milta hai: aircraft ke hisaab se partition karo jabki groups mein failure rate balance karo.
Natural $k$ : $8$ aircraft → up to $k = 8$ folds, har fold = ek held-out aircraft ki $30$ flights (aircraft par group-LOOCV). Agar $8$ bahut costly ya variable ho, $2$ aircraft per fold group karo $k = 4$ ke liye.
Yeh mastery answer kyun hai: yeh simultaneously ek saath un do independence assumptions ko respect karta hai jis par k-fold silently rely karta hai — ki (a) folds un units mein disjoint hain jo matter karte hain (aircraft, rows nahi), aur (b) har fold true label distribution represent karta hai.

Exercise 5.2 (L5)

Prove karo ki agar har fold ne identically score kiya ( $Score_{i} = c$ saare $i$ ke liye), toh $CV_{k} = c$ aur $σ_{CV} = 0$ , aur ek sentence mein explain karo ki real-world mein $σ_{CV} = 0$ (suspiciously) kya imply karta.

Recall Solution

Mean: $CV_{k} = \frac{1}{k} \sum_{i = 1}^{k} c = \frac{1}{k} (k c) === c == .$ Std: har deviation $Score_{i} - CV_{k} = c - c = 0$ , toh sum of squares $0$ hai aur $σ_{CV} = 0 === 0 == .$ Interpretation: practically fold scores kabhi exactly equal nahi hote; ek reported $σ_{CV} = 0$ almost hamesha ek bug signal karta hai — jaise same data har fold mein leak hua, ya metric training set par compute ho rahi hai. Perfect consistency ek trophy nahi, balki ek warning hai.

Recall Quick self-check (cloze)

k-fold == $k$ models train karta hai, aur har sample exactly once== test hota hai. Har model ke liye training fraction hai ::: $\frac{k - 1}{k}$ Fold scores ka sample std sum of squares ko divide karta hai ::: $k - 1$ se Rare-class folds ka fix ::: stratified k-fold Same-aircraft leakage ka fix ::: group (aircraft-level) k-fold Hyperparameters tune karte waqt honest estimate ::: nested cross-validation