Noesis
5.6.5 · D5 · HinglishMachine Learning (Aerospace Applications)

Question bank — Cross-validation — k-fold

1,940 words9 min read↑ Read in English

5.6.5 · D5 · Coding › Machine Learning (Aerospace Applications) › Cross-validation — k-fold

Shuru karne se pehle, ensure karo ki teen words tumhare liye kuch concrete matlab rakhte hain:

True or false — justify karo

True or false: 5-fold CV ek model train karta hai aur use paanch baar test karta hai.
False. Ye paanch alag-alag models train karta hai, har ek alag chaar-fold training set par; ek hi trained model ko paanch baar test karna matlab use apne hi training data par partly test karna hoga.
True or false: k-fold CV mein har data point exactly ek baar testing ke liye use hota hai.
True. Har row exactly ek fold mein hoti hai, aur har fold exactly ek baar test set banta hai — isliye har row ek baar test hoti hai aur baar training ke liye use hoti hai.
True or false: badhane se hamesha CV estimate zyada accurate hoti hai.
False. Zyada bias kam karta hai (bade training sets) lekin variance badhata hai (chhote, noisy test folds jinke overlapping training sets scores ko correlated banate hain), toh ye ek tradeoff hai, free win nahi. Dekho 5.6.10-Bias-variance-tradeoff-in-model-selection.
True or false: Final CV number ek specific trained model ki property hai.
False. Ye learning procedure ki performance estimate karta hai is size ke data par — models mein se koi bhi tumhara deliverable nahi hai. Tum baad mein saare rows par retrain karte ho.
True or false: CV mean ka matlab hai tumhara deployed model score karega.
Zyaadatar sach lekin sirf approximately: deployed model saare rows par retrain hota hai (thoda zyada data), toh uski true performance usually CV estimate se thodi behtar hoti hai, isliye CV ek halka underestimate hota hai.
True or false: Stratified k-fold sirf classes balance karne ke baare mein hai aur regression ke liye folds kabhi nahi badalta.
Standard tool ke liye True — stratification class proportions ko equal rakhti hai, jo sirf classification par apply hoti hai. Continuous target ke liye tumhe binned ya custom scheme chahiye hogi. Dekho 5.6.04-Stratified-sampling.
True or false: Sirf CV mean report karna model par trust karne ke liye kaafi hai.
False. Mean consistency chhupata hai; ek model trustworthy hai jabki signal karta hai ki score heavily depend karta hai us par ki kaunsi rows test fold mein gayi. Hamesha standard deviation bhi report karo.
True or false: k-fold CV overfitting eliminate karta hai.
False. Ye overfitting ko detect aur estimate karta hai training aur test-fold scores ka gap dikhake; ye model ko training folds memorize karne se nahi rokta.
True or false: shuffle=False ke saath, k-fold data ko contiguous blocks mein kaatta hai.
True. Ye row order mein slice karta hai, toh agar rows year ke hisaab se sorted hain, fold 1 saara early data hoga aur fold 5 saara late data — usually buri idea hai jab tak order ka matlab na ho.
True or false: Time-ordered flight logs ke liye, ordinary shuffled k-fold sahi default hai.
False. Shuffling model ko future par train karne aur past predict karne deta hai — temporal order ki leakage. Forward-chaining use karo: 8.5.02-Time-series-cross-validation.

Error pakdo

Error pakdo: "Maine saare features ko full dataset par scale kiya, phir 5-fold CV chalaya."
Scaler ne mean/std seekha test folds ko bhi use karke — ye leakage hai. Koi bhi preprocessing har fold ke andar sirf training portion par fit karo, phir use test fold par apply karo.
Error pakdo: "Maine hyperparameters apne CV score par tune kiye, phir wohi CV score model ki performance ke roop mein report kiya."
CV set model choose karne ke liye use hua, toh ab wo unseen nahi raha — ise report karna optimistic hai. Tumhe ek outer loop chahiye jo tuning se kabhi nahi chua: 5.6.07-Nested-cross-validation.
Error pakdo: "Mere classes 95/5 imbalanced hain, toh maine plain KFold use ki aur wildly varying fold scores mila."
Random folds mein 2% vs 10% failures aa sakti hain, toh har fold ek alag problem measure karta hai. StratifiedKFold use karo taaki har fold mein 5% rare-class rate steady rahe.
Error pakdo: "Maine minority class ko oversample kiya data balance karne ke liye, phir folds mein split kiya."
Synthetic/duplicated copies of same sample train aur test dono folds mein ja sakti hain — leakage. Splitting ke baad resample karo, sirf training fold use karke.
Error pakdo: "Mere paas same aircraft ke duplicate flight records hain, aur maine KFold ko unhe randomly split karne diya."
Ek aircraft ki near-identical rows train/test boundary par aa sakti hain, answer leak karke. Group-aware splitting use karo taaki ek aircraft ki saari rows ek hi fold mein rahein.
Error pakdo: "Maine KFold mein random_state=42 use kiya lekin RandomForest ke andar alag seed, phir kaha ki folds ne score change kiya."
Do alag randomness sources ek saath move kar rahe hain; tum variance ko folds ki wajah se attribute nahi kar sakte. Dono seeds fix karo taaki ek change ek cause se traceable ho.
Error pakdo: "Mera dataset sirf 40 rows ka hai toh maine safe rehne ke liye (LOOCV) choose kiya."
LOOCV yahan high-variance, correlated single-row scores deta hai aur 40 model fits hoti hain; chhote samples mein usually ek modest prefer karo. Dekho 5.6.06-Leave-one-out-cross-validation-(LOOCV).
Error pakdo: "Maine feature selection kiya (puri dataset par label ke saath correlation se top 20 sensors rakhe), phir CV kiya."
Selection ne har fold ke labels dekhe, toh chosen features already test rows ko 'jaante' hain — leakage. Selection ko har training fold ke andar redo karo.

Why questions

Hum fold scores ka average kyun lete hain instead of sirf best lene ke?
Best fold sabse lucky split hai, honest nahi; mean typical performance estimate karta hai aur averaging se kisi bhi single split ka noise damp hota hai.
Fold standard deviation mein se kyun nahi, se divide karte hain?
Scores ko apne hi sample mean se compare kiya jaata hai, jo ek degree of freedom use karta hai; se divide karne se spread ka unbiased estimate milta hai instead of systematically-too-small ek ke.
Chhota bias kyun badhata hai lekin bada variance kyun badhata hai?
Chhota matlab har model kam data par train hota hai, toh full-data model se bura lagta hai (pessimistic bias). Bada matlab test folds chhote aur noisy hain, aur heavily-overlapping training sets scores ko correlated banate hain, variance inflate karte hain.
Rare engine failures ke liye accuracy ki jagah F1 score kyun use kiya jaata hai?
5% failures ke saath, hamesha "no failure" predict karna 95% accuracy deta hai jabki ek bhi failure catch nahi hota. F1 precision aur recall combine karta hai, toh jab rare class miss hoti hai ye collapse karta hai.
Preprocessing fold loop ke andar kyun rehni chahiye?
Kyunki test fold genuinely unseen future data simulate karna chahiye; ek scaler ya imputer ko use dekhne dena matlab tumhara model test time par woh knowledge rakhega jo deployment mein nahi hogi.
k-fold CV final model ki performance ko thoda underestimate kyun karta hai?
Har fold model sirf data ke part par train hota hai, jabki deployed model saare rows par train hota hai, toh uske paas kisi bhi fold model se thoda zyada seekhne ko tha.
Scarce aerospace data ke liye single 80/20 holdout ki jagah 5-fold CV prefer kyun karein?
Holdout 20% precious labelled flights purely testing ke liye use karta hai aur ek number deta hai, jabki 5-fold har row ko dono roles ke liye reuse karta hai aur paanch numbers yield karta hai, variance expose karte hue. Contrast karo 5.6.02-Holdout-method aur 5.6.01-Train-test-split-validation-set se.

Edge cases

Edge case: 1003 rows aur — "equal" folds ka kya hoga?
Wo exactly equal nahi ho sakte; library kuch folds ko 201 rows deta hai aur kuch ko 200. Theek hai — folds design se approximately equal hote hain, exactly nahi.
Edge case: ek fold mein zero positive (failure) examples aate hain.
F1 (aur recall) us fold ke liye undefined ya zero ho jaate hain, mean ko poison karte hain. Stratified k-fold ise rokta hai har fold mein rare-class proportion force karke.
Edge case: tum set karte ho (ek row per fold).
Ye leave-one-out CV hai — almost full-size training sets (low bias) lekin har test score ek single, noisy prediction hai aur tum models fit karte ho. Dekho 5.6.06-Leave-one-out-cross-validation-(LOOCV).
Edge case: tum set karte ho.
Meaningless — tumhare paas ek fold hoga jo training aur testing dono ke liye use hoga, toh koi bhi unseen data nahi hoga. Sabse chhota sensible value hai.
Edge case: tum set karte ho.
Legal hai, lekin har model sirf aadhe data par train hota hai (high pessimistic bias) aur sirf do scores hain, toh std barely informative hai. Ye ek symmetric double holdout ki tarah behave karta hai.
Edge case: tumhari saari rows almost identical hain.
Har fold roughly same score karta hai, ek tiny std dete hue jo lagta hai great stability hai lekin actually data mein koi diversity nahi hai — CV tumhe ek aisa scenario nahi bataa sakta jo data mein absent hai.
Edge case: CV mean high hai lekin har fold ka training score bahut zyada high hai.
Saare folds mein bada train-minus-test gap overfitting ka fingerprint hai; model training folds memorize karta hai aur CV sahi tarah se weaker generalization report kar raha hai.
Edge case: tum CV run karte ho 100 hyperparameter combos ki grid search ke dauran aur best CV score choose karte ho.
100 tries ke saath, winning score partly luck hai (multiple-comparisons trap); ise 5.6.07-Nested-cross-validation ke zariye ek untouched outer fold par confirm karo trust karne se pehle. Dekho bhi 7.2.03-Hyperparameter-tuning-grid-search.
Recall Quick self-test

Over-optimistic CV score ka sabse common cause kya hai? ::: Data leakage — preprocessing, feature selection, resampling, ya tuning jo test fold ko test hone se pehle dekh chuka ho. Kya k-fold CV tumhe deploy karne ke liye ek model deta hai? ::: Nahi; ye procedure ki performance estimate karta hai, jiske baad tum saare rows par retrain karte ho.