Cross-validation — k-fold: Visual walkthrough — Hinglish deep dive

5.6.5 · D2 · Coding › Machine Learning (Aerospace Applications) › Cross-validation — k-fold

Pehli line se pehle, teen simple words jinpe hum rely karenge:

Data point: tumhari table ki ek row — ek flight, uske sensor readings aur uska jaana-pehchaana answer (failed / did not fail).
Model: ek machine jo kuch data points dekhne ke baad, un data points ka answer guess karne ki koshish karti hai jo usne kabhi nahi dekhe.
Score: ek akela number jo kehta hai "unseen points par guess kitni baar sahi tha" (zyada = better).

Bas yahi vocabulary hai. Chalo draw karte hain.

Step 1 — Apna saara data ek row mein rakhho

KYA. Har data point ko ek coloured tile samjho. Hamare paas $n$ tiles hain — yaani $n = 1000$ flight records. Inhe left se right line up karo.

KYUN. Kuch bhi split karne se pehle, hume pure cheez ko ek object ki tarah dekhna hoga. Tiles haara raw material hai; baaki sab bas yeh choose karna hai ki kaun si tiles chhupani hain.

PICTURE. Neeche ek single strip dekho. Har tile ek flight hai. Abhi kuch bhi hidden nahi hai — model sab par train karta aur test karne ke liye kuch nahi bachta honestly. Yahi woh problem hai jo hum abhi fix karne wale hain.

Step 2 — Strip ko $k$ equal blocks mein kato

KYA. Strip ko $k$ side-by-side blocks mein slice karo same length ke. Har block ko fold kehte hain. Hum $k = 5$ choose karte hain, toh paanch blocks, har ek mein $n / k = 1000/5 = 200$ tiles.

KYUN. Hume unseen data ke kai alag pools chahiye, ek nahi. Agar hum hamesha wohi 200 tiles chhupate, toh ek lucky (ya unlucky) block humein bewakoof bana sakta. $k$ blocks mein kaatne se hume $k$ alag "unseen" pools milte hain baad mein try karne ke liye — ek at a time.

PICTURE. Strip ab 5 coloured blocks mein hai $D_{1}$ se $D_{5}$ tak label kiye. Yeh overlap nahi karte (koi tile do blocks mein nahi hai) aur milke sab kuch cover karte hain.

Step 3 — Round 1: pehla block chhupao, baaki par train karo

KYA. Block $D_{1}$ lo aur use test set ki tarah side mein rakh do — model ko yeh dekhna mana hai. Baaki chaar blocks ( $D_{2}, D_{3}, D_{4}, D_{5}$ ) training set ban jaate hain. Ek model train karo, use $M_{1}$ kaho, sirf un chaar blocks par.

KYUN. Model ko un tiles par score karna chahiye jo usne kabhi nahi dekhe, warna score ek memory test hai, generalization test nahi. $D_{1}$ chhupana guarantee karta hai ki $M_{1}$ ko uski koi memory nahi.

PICTURE. Neeche, block 1 glow kar raha hai (hidden test block) aur blocks 2–5 ek "training" region mein dim ho gaye hain with an arrow feeding model box $M_{1}$ ko.

Step 4 — Round 1 grade karo, phir rotate karo

KYA. $M_{1}$ se kaho ki hidden block $D_{1}$ ke answers guess kare, true answers se compare karo, aur ek number $Score_{1}$ record karo. Phir rotate karo: ab $D_{2}$ ko chhupao, fresh $M_{2}$ ko baaki chaar blocks par train karo, grade karo → $Score_{2}$ . Tab tak rotate karte raho jab tak har block ki turn na aa jaye hidden hone ki.

KYUN. Ek rotation ek opinion deta hai. Rotate karne ka matlab hai ki har tile eventually exactly ek baar unseen test tile ban jaati hai, toh data ka koi region ungraded nahi rehta aur koi do baar grade nahi hota. Yahi even coverage k-fold ka poora point hai.

PICTURE. Paanch rows, ek per round. Har row mein ek alag block lit hai (test block) jabki baaki training region hai. Top se bottom padho: lit block left→right march karta hai, toh "test" spotlight poore dataset ko sweep karta hai.

Is step ke end mein paanch numbers hain, jaise $[0.82, 0.79, 0.85, 0.81, 0.83]$ .

Step 5 — Paanch numbers ko ek mein compress karo: mean

KYA. Paanch scores add karo aur paanch se divide karo. Woh average hi k-fold cross-validation estimate hai.

KYUN. Har round ka score thoda wobble karta hai kyunki uska particular hidden block thoda easy ya hard tha. Average lene se yeh wobbles cancel ho jaate hain: high rounds upar kheenchte hain, low rounds neeche, aur middle kisi bhi single round se zyada steady guess hai true performance ka. Yeh law of large numbers hamare kaam aa raha hai.

PICTURE. Paanch dots heights $0.79$ – $0.85$ par, aur unke mean $0.82$ par ek bold horizontal line. Line ke around dashed band woh spread hai jise hum aage measure karte hain.

Step 6 — Wobble measure karo: standard deviation

KYA. Measure karo ki paanch dots apni mean line se average kitni door hain. Woh distance standard deviation $σ_{CV}$ hai.

KYUN. $0.82$ ka mean chhupa leta hai ki rounds agree kiye ya nahi. $0.82 \pm 0.02$ (dots line ke paas) matlab model consistent hai. $0.82 \pm 0.15$ (dots scattered) warn karta hai ki model fragile hai — uska score wildly swing karta hai depending on which tiles usne dekhe.

PICTURE. Wahi paanch dots jaise pehle, ab har dot se mean line tak ek vertical arrow jo uski gap $(Score_{i} - CV_{k})$ mark karta hai. Spread band ki half-width $σ_{CV}$ hai.

Intuition

$k - 1$ se divide kyun karte hain, $k$ se nahi? Standard deviation ke do flavors hain. Population version $k$ se divide karta hai — use karo jab tumhare numbers poori duniya hain. Sample version $k - 1$ se divide karta hai — use karo jab tumhare numbers ek sample hain jo ek wider truth guess karne ke liye use ho rahe hain. Hamare $k$ fold-scores ek chhota sample hain jo model ki performance over all possible data ka stand-in hai, toh hume sample version chahiye. $1$ ka subtraction spread ko gently inflate karta hai ek subtle cheat correct karne ke liye: humne har gap measure kiya mean $CV_{k}$ ke against, jo khud un same scores se compute hua tha, toh scores apne mean ke thoda zyada paas hote hain. $k - 1$ (ek chhota number) se divide karna us under-estimate ko undo karta hai. Code mein yeh hai np.std(scores, ddof=1) — ddof=1 hi woh "minus one" hai.

Final report: $0.820 \pm 0.022$ .

Step 7 — Edge cases: extreme mein $k$ kaisa dikhta hai

KYA. $k$ ke do degenerate choices jo balance tod dete hain.

KYUN / PICTURE. Neeche do panels dekho.

$k = 2$ (left panel), sabse chhoti legal value. Sirf do blocks, toh har model sirf aadhe data par train karta hai. Aadha dataset ek weak apprentice hai, toh scores pessimistically low aate hain — high bias. Lekin har test block bada hai (500 tiles), toh individual scores stable rehte hain — low variance.
$k = n$ (right panel), ek tile per block — yeh hai LOOCV. Har model $n - 1$ tiles par train karta hai (almost poora set → low bias), lekin har test block ek tile hai, toh har score ya toh "right" ya "wrong" hai — extremely noisy (high variance), aur saare $n$ training sets almost completely overlap karte hain, toh scores correlated hain. Tum $n$ poore models bhi train karte ho — brutally slow.

Ek-picture summary

Poori procedure ek single canvas par: strip 5 folds mein cut hai; paanch rows lit test block ko dataset mein rotate karti hain; har row ek score deti hai; scores mean $\pm$ spread mein collapse ho jaate hain.

Recall Feynman retelling — ise story ki tarah bolo

Main apni saari flights ek lambi row mein rakhta hoon. Row ko 5 equal blocks mein kaatta hoon. Round one: main block 1 ko haath se dhak leta hoon, model ko baaki chaar padhne deta hoon, phir block 1 uncover karta hoon aur dekhta hoon ki kitni flights sahi guess ki — yeh ek score hai. Haath block 2 par slide karta hoon aur fresh model ke saath sab dobara karta hoon. Paanch rounds, mera haath poori row mein sweep karta hai, toh har flight exactly ek baar cover hoti hai. Ab mere paas paanch scores hain. Main inhe average karta hoon — woh steady number mera real estimate hai ki model kitna achha hai. Main yeh bhi check karta hoon ki paanch kitne spread out the: ek saath tight matlab trustworthy, scattered matlab model jumpy hai. Agar bahut kam blocks use karun toh model bahut kam se seekhta hai aur worse dikhta hai jitna hai; agar utne blocks use karun jitni flights hain toh main forever train karta hoon aur scores noisy ho jaate hain. Paanch ya das blocks comfortable middle hai.

Recall Quick self-test

Folds disjoint kyun hone chahiye? ::: Taaki koi tile train aur test dono mein na ho — woh answer leak karega aur score inflate karega. $k$ scores average karne se kya milta hai? ::: Round-to-round wobble cancel hota hai, kisi bhi single split se zyada steady estimate milta hai. $k$ ki legal range kya hai? ::: $2 \leq k \leq n$ — kam se kam do folds (warna training set empty), zyada se zyada ek tile per fold (LOOCV). Jab $n$ , $k$ se evenly divide nahi hota toh kya hota hai? ::: Leftover tiles ek per fold spread karo, toh kuch folds mein ek extra tile hoti hai (ceil vs floor). Folding se pehle shuffle nahi kab karte? ::: Jab data time-ordered ho aur future forecast karna ho — time-series CV use karo.

Related build-ups: 5.6.01-Train-test-split-validation-set, 5.6.02-Holdout-method, 5.6.07-Nested-cross-validation, 7.2.03-Hyperparameter-tuning-grid-search.