Worked examples — Cross-validation — k-fold

5.6.5 · D3 · Coding › Machine Learning (Aerospace Applications) › Cross-validation — k-fold

Shuru karne se pehle, teen plain-word reminders taaki koi bhi symbol yahan begair wajah na ho:

Recall

$n$ aur $k$ ka matlab kya hai phir se? $n$ ::: tumhare dataset mein samples (rows) ki kul sankhya. $k$ ::: folds ki sankhya — kitne equal pile mein hum data ko kaatenge. Ek fold ::: un piles mein se ek, jo ek baar test set ki tarah use hoti hai jabki baaki $k-1$ piles model ko train karti hain.

Recall "Score" kya hota hai aur hum average kyun karte hain?

Ek fold ka score ::: ek single number (accuracy, F1, RMSE…) jo batata hai model ne us held-out pile pe kitna accha kiya. $k$ scores ko average kyun karte hain ::: ek split lucky ya unlucky ho sakta hai; $k$ splits ka mean true performance ka ek steady estimate hota hai.

Do metrics jinse hum actually score karenge

Is page par har fold ka "score" do numbers mein se ek hoga. Unhe ek baar yahan define karo, taaki neeche kuch bhi bina samjhe borrowed na ho.

---

Scenario matrix

Har k-fold problem neeche diye axes ka ek combination hai. Har cell ek alag cheez hai jo galat ho sakti hai ya alag behave kar sakti hai — isliye har ek ka apna worked example hai.

Cell	Case class	Kya cheez special hai	Covered by
A	Clean, balanced, $n$ divisible by $k$	Textbook case, har fold exactly same size	Example 1
B	$n$ not divisible by $k$ (remainder)	Folds ke sizes unequal hain — leftovers kaise distribute honge?	Example 2
C	Degenerate small: $k=n$ (LOOCV limit)	Test fold mein ek sample hai; score 0 ya 1 hoga	Example 3
D	Degenerate small: $k=2$	Sirf do folds — single split ke sabse kareeb	Example 4
E	Imbalanced classes (rare event)	Random folds class ratio bigaad dete hain → stratify karo	Example 5
F	Regression metric (RMSE) accuracy nahi	Error average karna, "correctness" nahi; units matter karti hain	Example 6
G	Time-ordered data (leakage trap)	Shuffling future chura leta hai → forward-chaining karo	Example 7
H	Exam twist: variance vs mean judgement	Do models, same mean, different std — kaun jeetega?	Example 8

---

Example 1 — Cell A: clean divisible case

Forecast: aage padhne se pehle mean guess karo — kya yeh $0.85$ se upar ya neeche hai?

Neeche ki figure rotation dikhati hai: har row paanch folds mein se ek hai, blue cells 8 training samples hain, aur orange block ke 2 cells us fold ka test set hain. Rows ke neeche trace karo aur dekho orange block left-to-right slide karta hai — har sample exactly ek baar test set mein aata hai, aur har row ka per-fold accuracy right side pe print hai.

Figure 1. Paanch stacked rows, ek per fold; ek do-cell orange test block rows ke neeche rightward slide karta hai jabki baaki aath blue cells train karte hain, aur fold ki accuracy ($0.90,0.80,1.00,0.80,0.90$) har row ke right edge pe print hai.

1. Fold size dhundho. $n/k=10/5=2$. Har fold mein exactly 2 samples hain (figure mein ek orange block), har model baaki 8 pe train karta hai (blue cells). Yeh step kyun? Hume confirm karna hai ki folds equal hain — Cell A tabhi valid hai jab $n$, $k$ se evenly divide ho.
2. Scores average karo (CV score). ${\text{CV}}_5=\frac{0.90+0.80+1.00+0.80+0.90}{5}=\frac{4.40}{5}=0.88$ Yeh step kyun? Parent formula ${\text{CV}}_k=\frac{1}{k}\sum {\text{Score}}_i$ — averaging hi 5 lucky/unlucky numbers ko ek estimate mein badalta hai.
3. Folds mein spread (sample standard deviation). ${\sigma }_{\text{CV}}=\sqrt{\frac{1}{k-1}{\sum }_i({\text{Score}}_i-0.88{)}^2}$ Squared deviations hain $0.0004,0.0064,0.0144,0.0064,0.0004$, sum $0.028$ hai. $k-1=4$ se divide karo: $0.007$. Square root: $0.0837$. Yeh step kyun? Hum $k-1$ se divide karte hain, $k$ se nahi, kyunki mean khud usi data se estimate hua tha — ek degree of freedom "use up" ho gaya. Yeh sample standard deviation hai.

Forecast check: tumse poochha gaya tha ki mean $0.85$ se zyada hai ya nahi. Hai — $0.88>0.85$. Eyeball intuition kaam karti hai kyunki paanch scores mein se teen ($0.90,1.00,0.90$) $0.85$ se upar hain aur average ko upar kheenchte hain; woh "count karo ki kitne line ke upar hain" trick woh mental shortcut hai jise forecast train kar raha tha.

Verify: paanch scores $[0.80,1.00]$ mein hain; mean $0.88$ us range ke andar hai ✓. Total $=4.40$ ko 5 equal shares of $0.88$ mein baanta ✓.

---

Example 2 — Cell B: $n$ not divisible by $k$

Forecast: $23/5=4.6$ — fold mein $4.6$ samples nahi ho sakte. Toh leftover ka kya hoga?

Neeche ki figure 23 samples ko paanch coloured bars ki tarah dikhati hai. Notice karo ki pehle teen bars last do se ek cell lambe hain: woh extra cell remainder hai jo earliest folds mein ek-ek karke distribute hota hai. Caption confirm karta hai ki sizes add hokar $n=23$ banta hai.

Figure 2. Paanch horizontal bars labeled fold 1–5 with lengths $5,5,5,4,4$; pehle teen ek cell lambe hain (remainder ek-per-fold distribute hua), aur ek caption confirm karta hai $5+5+5+4+4=23=n$.

1. Remainder split karo. $23=5\times 4+3$. Base size $4$ hai; remainder $3$ pehle teen folds mein ek-per-fold spread hota hai (teen lambe bars). Fold sizes: $\boxed{5,5,5,4,4}$. Yeh step kyun? scikit-learn ka KFold pehle n % k folds ko ek extra sample deta hai taaki har sample exactly ek baar use ho aur koi fold empty na rahe. Sizes "approximately equal" rehti hain, $|D_i|\approx n/k$ se match karti hain.
2. Total check karo. $5+5+5+4+4=23$ ✓ — har sample exactly ek baar test hota hai, koi twice nahi. Yeh step kyun? k-fold ki defining guarantee hai har point exactly ek baar test hota hai; unequal folds ko phir bhi $n$ mein sum karna chahiye.
3. RMSEs average karo. ${\text{CV}}_5=\frac{12+15+11+14+13}{5}=\frac{65}{5}=13\text{ kg}$ Yeh step kyun? RMSE ek per-fold number hai (upar $\sqrt{\frac{1}{m}\sum (y-\hat{y}{)}^2}$ ke roop mein define hua); CV estimate abhi bhi unka mean hai.

Forecast check: tumhari instinct sahi thi ki $4.6$ fold size nahi ho sakta — fix rounding nahi balki $3$ leftover samples ko distribute karna hai, jo mixed sizes $5,5,5,4,4$ deta hai. Yaad rakhne wala number hai n % k: yeh exactly count karta hai ki kitne folds ko bonus sample milega.

Verify: sizes sum hokar $n=23$ banta hai ✓. Mean $13$ min $11$ aur max $15$ ke beech hai ✓. Units kg rehte hain (humne kg values average kiye) ✓.

---

Example 3 — Cell C: LOOCV limit $k=n$

Forecast: sirf ek test point per fold ke saath, ek fold ki accuracy kaun si values le sakti hai?

1. Har test fold mein ek sample hai. $n/k=4/4=1$. $3$ pe train karo, $1$ pe test karo. Yeh step kyun? Yeh extreme $k\to n$ hai jo parent mein LOOCV ke naam se mention hai.
2. Har fold score 0 ya 1 hai. Ek sample ya toh sahi hoga ($1$) ya galat ($0$) — kuch beech mein nahi. Scores: $[1,1,0,1]$. Yeh step kyun? accuracy $=\frac{\text{correct}}{\text{tested}}=\frac{\{0\text{ ya }1\}}{1}$. Isliye LOOCV fold scores high variance / noisy hote hain — har ek coin-flip jaisa $0$ ya $1$ hai.
3. Average karo. ${\text{CV}}_4=\frac{1+1+0+1}{4}=0.75$ Yeh step kyun? $0/1$ scores ka mean equals overall fraction correct — exactly $3$ of $4$.

Forecast check: prompt ka answer hai "sirf $0$ ya $1$" — ek single test point ya toh poora sahi ho sakta hai ya poora galat. Isliye average ($0.75$) LOOCV mein informative number hai jabki kisi bhi single fold ka score akele almost useless hai.

Verify: $4$ mein se $3$ correct $=0.75$ ✓. Average kiye gaye har value $0$ ya $1$ thi, single-sample test folds confirm karta hai ✓.

---

Example 4 — Cell D: two-fold extreme $k=2$

Forecast: sirf 2 folds ke saath, har model data ka kitna hissa train karta hai?

1. Fold size aur training fraction. $n/k=8/2=4$. Har model $k-1=1$ fold $=4$ samples $=50\%$ data pe train karta hai. Yeh step kyun? Chhota $k$ matlab har model kam training data dekhta hai — yahan sirf aadha — isliye har score full-data model ka pessimistic (higher-bias) estimate hai.
2. Average karo. ${\text{CV}}_2=\frac{0.70+0.86}{2}=0.78$ Yeh step kyun? Wahi averaging rule; do folds abhi bhi ek-ek baar test karte hain ($4+4=8$). Yahan har F1 upar define kiya gaya balanced precision–recall blend hai.
3. Single split se connection interpret karo. $k=2$ ke saath, fold 1 fold 2 pe train karta hai aur vice-versa: yeh do mirror-image train/test splits hain. Unhe average karna barely ek split se zyada robust hai — isliye hum $k=5$ ya $10$ prefer karte hain.

Forecast check: "har model kitna train karta hai?" ka answer exactly aadha ($50\%$) hai. Woh low training fraction hi woh poori wajah hai ki $k=2$ biased-pessimistic hota hai — model tabhi judge kiya jaata hai jab woh us data se starved ho jo production mein uske paas hogi.

Verify: $4+4=8=n$ ✓. Mean $0.78\in [0.70,0.86]$ ✓.

---

Example 5 — Cell E: rare event → stratified k-fold

Forecast: $12$ failures ko $4$ folds mein split karte hue, har fold mein average kitne failures hone chahiye?

Figure do strategies ko "failures per fold" ke bar charts ke roop mein contrast karta hai. Left mein (random k-fold) bars jagged hain — ek fold red hai height $0$ pe, matlab usne koi failure nahi pakdi. Right mein (stratified) sare chaar bars flat hain target height $3$ pe, $6\%$ rate match karte hue. Dashed grey line woh target $3$ mark karti hai.

Figure 3. $k=4$ ke liye failures-per-fold ke do side-by-side bar charts. Left ("Random k-fold") jagged hai ek red bar height $0$ pe; right ("Stratified k-fold") chaar equal green bars dikhata hai height $3$ pe; ek dashed grey line target of $3$ per fold mark karti hai.

1. True class rate. $12/200=0.06=6\%$ failures. Yeh step kyun? Yeh baseline hai jo har fold ko mirror karna chahiye. Full dataset ratio stratified sampling ka target hai.
2. Expected failures per fold. $12/4=3$ failures per fold on average (figure mein dashed line). Yeh step kyun? Agar randomness kisi fold ko $0$ failures deal kar de (red bar), us fold mein koi positive detect karne ke liye nahi hai.
3. Zero-failure fold ka F1 kyun collapse karta hai. F1 ko true positives chahiye (uski upar wali formula mein $TP$). Test fold mein zero actual positives ke saath, recall ka denominator $TP+FN$ $0$ hai → F1 undefined hai (scikit-learn $0$ return karta hai). Us fold ka score meaningless hai aur average ko drag/spike karta hai. Yeh step kyun? Rare-event scoring tabhi valid hai jab har test fold mein positives exist karein.
4. Stratified fix. Stratified k-fold har fold ko $\approx 6\%$ failures hold karne par force karta hai — yaani $200/4\times 0.06=3$ failures per fold (flat right-hand chart), guarantee karta hai ki har fold real F1 compute kar sake.

Forecast check: "hona chahiye" wala number 3 per fold hai, aur exactly yahi value stratification har fold pe enforce karti hai (flat green bars). Random scheme ki failure thi ki woh is 3 se deviate hui — ek fold ko 0 mila. Tumhara forecast of 3 woh yardstick hai jo random scheme ko broken expose karta hai.

Verify: $6\%\times 50$ samples-per-fold $=3$ failures/fold, aur $3\times 4=12$ = total failures ✓. $12/200=0.06$ ✓.

---

Example 6 — Cell F: regression, RMSE units

Forecast: kya CV RMSE sirf inhe paanch ka mean hai, ya hume kuch square-and-root karna padega?

1. Fold RMSEs directly average karo. ${\text{CV}}_5=\frac{210+190+205+195+200}{5}=\frac{1000}{5}=200\text{ kg}$ Yeh step kyun? Har fold pehle se hi ek RMSE hai (upar definition se $\sqrt{\frac{1}{m}\sum (y-\hat{y}{)}^2}$). Classic recipe per-fold scores average karta hai — hum residuals ko re-pool nahi karte. (Residuals pool karna ek alag, valid variant hai; parent ka formula scores ka mean use karta hai.)
2. Folds mein spread. $200$ se deviations: $[10,-10,5,-5,0]$. Squares: $[100,100,25,25,0]$, sum $=250$. $k-1=4$ se divide karo: $62.5$. Root: $\approx 7.906$ kg. Yeh step kyun? Example 1 jaisi $\frac{1}{k-1}$ sample-std reasoning — mean ne ek degree of freedom use kiya.
3. Units state karo. Mean aur std dono kg mein hain (target ke same units), kyunki RMSE un final square root ke through target ke units inherit karta hai. Yahan "% accuracy" interpretation galat hogi. Yeh step kyun? Ek metric ke units decide karte hain ki tum uski values kaise interpret aur combine kar sakte ho; kg-error ko fractional accuracy ki tarah treat karna silently har downstream comparison ko corrupt kar dega.

Forecast check: "mean, ya square-and-root?" ka answer hai bas mean — kyunki har fold value pehle se hi ek RMSE hai (squaring-and-rooting har fold ke andar hua). Forecast jis trap se bachaa raha tha woh hai un numbers ko double-process karna jo pehle se error summaries hain.

Verify: sum $=1000$, mean $200$ ✓; std $=\sqrt{250/4}=\sqrt{62.5}\approx 7.906$ ✓; units kg ✓.

---

Example 7 — Cell G: time-ordered data, leakage trap

Forecast: $t_5$ pe train karke $t_2$ pe test karne mein kya galat hai?

Figure chaar forward-chaining folds ko rows ki tarah stack karta hai. Har row mein blue cells (training past) strictly left mein hain single orange cell (next-day test) ke, aur grey cells future ke din hain jo abhi available nahi hain. Bottom arrow time ko left-to-right flow karta dikhata hai — koi bhi orange test cell kabhi apne right mein koi blue training cell nahi rakhti, jo exactly "no future leaks backward" guarantee hai.

Figure 4. Chhe dino $t_1$–$t_6$ ki chaar stacked rows; har row mein blue cells (past = train) strictly ek orange cell (next day = test) ke left mein hain, grey cells unavailable future hain, aur ek bottom arrow time ko left-to-right flow karta dikhata hai.

1. Leakage spot karo. Shuffled folds model ko future days ($t_5,t_6$) pe train karne dete hain past day ($t_2$) predict karne ke liye. Deployment mein tumhare paas future kabhi nahi hota, isliye shuffled CV performance over-estimate karta hai. Yeh step kyun? k-fold ki fairness assume karti hai ki har test fold "unseen future" hai; time order us assumption ko todta hai. Yeh domain time-series CV ka hai.
2. $k=3$ shuffled kyun abandon karna hai, aur kya replace karta hai. Hum colleague ke $3$ shuffled folds bilkul nahi rakh sakte — shuffling wahi act hai jo future leak karta hai. Time-series CV mein, $k$ freely choose nahi kiya jaata; yeh derive hota hai ki tum kitne forward steps lete ho. Hum ek initial training window fix karte hain (yahan pehle $2$ din, koi bhi model fit karne ke liye zaroori) aur phir ek din ek baar aage badhte hain. $6$ dino aur $2$ ki window ke saath, yeh $6-2=4$ forward folds deta hai — effectively $k=4$, aur $k$ data length se dictate hota hai, convenience se nahi choose kiya jaata. Yeh step kyun? Yeh illegal shuffled splits ko ordered ones se replace karta hai aur stated $k=3$ se $4$ folds mein change explain karta hai: fold count n − (initial window) se aata hai, free choice se nahi.
3. Forward-chaining folds banao. Hamesha past (blue cells) pe train karo, next block (orange cell) pe test karo:
   • train $\{t_1,t_2\}$ → test $t_3$
   • train $\{t_1,t_2,t_3\}$ → test $t_4$
   • train $\{t_1,t_2,t_3,t_4\}$ → test $t_5$
   • train $\{t_1,\dots ,t_5\}$ → test $t_6$ Yeh step kyun? Har training set apne test point se time mein pehle aata hai — koi future backward leak nahi karta.
4. Test evaluations count karo. Points tested: $t_3,t_4,t_5,t_6$ → $4$ test points. Plain k-fold ke unlike, forward chaining har point ko test nahi karta: pehle do din ($t_1,t_2$) sirf earliest training window seed karte hain aur khud kabhi held out nahi hote. Isliye ordinary k-fold ki "har sample exactly ek baar tested" guarantee yahan deliberately give up ki jaati hai future peek karne se bachne ke exchange mein. Yeh step kyun? Recognize karna ki early samples train-only hain — yahi poora point hai — yeh woh cost hai jo time-series CV realism ke liye pay karta hai.

Forecast check: "$t_2$ predict karne ke liye $t_5$ pe train karne mein kya galat hai?" ka answer hai ki $t_5$, $t_2$ ke relative future hai — real deployment mein tumhare paas abhi tak woh nahi hota, isliye score dishonestly inflated hai. Forward-chaining scheme exactly woh arrangement hai jo aisi time-travel impossible banata hai.

Verify: $n=6$ aur initial train window $2$ ke saath, tested points $=n-2=4$ ✓. Har training set indices ka strict prefix hai, isliye max(train index) $<$ test index har fold mein ✓.

---

Example 8 — Cell H: exam twist, mean vs variance

Forecast: identical means — toh tie kya todega?

1. Confirm karo ki means match karte hain. $\bar{P}=\frac{0.83+0.81+0.82+0.82+0.82}{5}=\frac{4.10}{5}=0.82,\bar{Q}=\frac{0.90+0.70+0.95+0.72+0.83}{5}=\frac{4.10}{5}=0.82$ Yeh step kyun? Trap hai equal means assume karna ⇒ equal quality. Dono average pe tie karte hain, isliye mean akele decide nahi kar sakta.
2. Har ek ka spread compute karo (fold-to-fold std). P ke liye, deviations $[0.01,-0.01,0,0,0]$, squares sum $=0.0002$, $/4=0.00005$, root $\approx 0.00707$. Q ke liye, deviations $[0.08,-0.12,0.13,-0.10,0.01]$, squares $[0.0064,0.0144,0.0169,0.0100,0.0001]$ sum $=0.0478$, $/4=0.01195$, root $\approx 0.1093$. Yeh step kyun? Folds mein variance consistency measure karta hai — ek safety-critical concern jo mean chhupaata hai.
3. Decide karo. ${\sigma }_P\approx 0.007\ll {\sigma }_Q\approx 0.109$. Model P deploy karo: same expected F1 but data subsets mein bahut zyada steady. Model Q ka swing ($0.70$ se $0.95$ tak) matlab hai ki kuch real flight populations pe woh badly fail kar sakta hai. Yeh step kyun? Aerospace mein, predictable mediocrity erratic brilliance se behtar hai — ek model jo kabhi excellent hota hai aur kabhi poor, woh safety liability hai.

Forecast check: tie-breaker jise tum name karne ko bole gaye the woh standard deviation hai — "hamesha do numbers" pair mein se doosra. Equal means ne tumhe average se aage dekh ke consistency dekhne par majboor kiya, jo is exam-style cell ka poora lesson hai.

Verify: dono means $=4.10/5=0.82$ ✓; ${\sigma }_P=\sqrt{0.0002/4}\approx 0.00707$, ${\sigma }_Q=\sqrt{0.0478/4}\approx 0.10932$, aur ${\sigma }_P<{\sigma }_Q$ ✓.

---

Recall Kaun sa example kaun sa trap cover karta hai?

Unequal fold sizes ::: Example 2 (remainder pehle n % k folds mein spread hota hai). Single-sample test folds ::: Example 3 (LOOCV, scores 0/1 hote hain). Rare event / stratification ::: Example 5. Time leakage / forward chaining ::: Example 7 (sirf $n-\text{init}$ points tested hote hain). Same mean, std se choose karo ::: Example 8.

Parent: Cross-validation — k-fold · Hinglish: 5.6.05 Cross-validation — k-fold (Hinglish)