K-fold cross-validation — Hinglish notes

2.6.4 · AI-ML › Model Evaluation & Selection

Problem: Limited Data & Unreliable Estimates

Jab hamare paas ek dataset hota hai, toh hum ek dilemma face karte hain:

KYUN? Humein apne model ko aise data pe evaluate karna hai jo training ke dauran usne dekha nahi
PROBLEM: Agar hum testing ke liye bahut zyada data hold out karte hain (maan lo 50%), toh hum sirf aadhe data pe train kar rahe hain → worse model
PROBLEM: Agar hum bahut kam hold out karte hain (maan lo 10%), toh hamara performance estimate high variance wala hoga → ek lucky/unlucky split hume mislead kar sakta hai

K-fold cross-validation iss problem ko solve karta hai har ek data point ko training AUR testing dono ke liye use karke (lekin kabhi ek saath nahi).

K-fold Algorithm: First Principles Se Derivation

Step-by-Step Derivation

HUMEIN KYA CHAHIYE: Generalization error $E [error]$ ka ek estimate jo haara sara data efficiently use kare.

HUMEIN KAISE MILEGA:

Chaliye define karte hain:

Dataset $D$ jisme $N$ samples hain
Performance metric $M$ (jaise accuracy, RMSE)
Model training procedure $T$

Step 1: Data ko partition karo

$D$ ko K disjoint subsets (folds) mein divide karo: $D = F_{1} \cup F_{2} \cup \dots \cup F_{K}$

Har fold mein approximately $\frac{N}{K}$ samples hote hain. KYUN equal-sized? Taaki har validation set data ka same proportion represent kare → fair comparison.

Step 2: K training/validation splits define karo

Iteration $i$ ke liye (jahan $i = 1, 2, \dots, K$ ):

Validation set: $V_{i} = F_{i}$
Training set: $T_{i} = D ∖ F_{i} = ⋃_{j \neq = i} F_{j}$

ISKA KYA MATLAB HAI: Har data point exactly 1 validation set mein aur K-1 training sets mein aata hai.

Step 3: K models train aur evaluate karo

Har fold $i$ ke liye:

$T_{i}$ pe model train karo: $θ_{i} = T (T_{i})$
$V_{i}$ pe evaluate karo: $score_{i} = M (θ_{i}, V_{i})$

Step 4: Results aggregate karo

K-fold CV estimate hai:

$CV_{K} = \frac{1}{K} \sum_{i = 1}^{K} score_{i}$

MEAN KYUN? Law of Large Numbers ke hisaab se, K estimates ko average karne se variance kam hota hai. Har $score_{i}$ true performance ka ek noisy estimate hai; averaging se humein zyada stable estimate milta hai.

Standard error humein reliability batata hai:

$SE = \frac{1}{K} \sum_{i = 1}^{K} (score_{i} - CV_{K})^{2}$

STANDARD ERROR KYU MATTER KARTA HAI: Agar SE bada hai, toh hamare folds bahut alag-alag scores dete hain → model training data ke baare mein sensitive hai (possible overfitting ya data issues).

Worked Examples

Worked example Example 1: 5-Fold CV with 100 Samples

Setup: 100 data points, K=5, accuracy as metric. Step 1: 5 folds banao, har ek mein 20 samples

$F_{1} =$ samples 1-20
$F_{2} =$ samples 21-40
$F_{3} =$ samples 41-60
$F_{4} =$ samples 61-80
$F_{5} =$ samples 81-100

YEH SPLIT KYUN? Random (ya stratified) assignment ensure karta hai ki har fold representative ho.

Step 2: Iteration 1

$F_{2} \cup F_{3} \cup F_{4} \cup F_{5}$ pe train karo (80 samples)
$F_{1}$ pe validate karo (20 samples)
$accuracy_{1} = 0.85$ milta hai

80 TRAINING SAMPLES KYUN? Hum $\frac{K - 1}{K} = \frac{4}{5} = 80%$ data training ke liye use karte hain.

Step 3: Saare folds ke liye repeat karo

Fold 2: $F_{1} \cup F_{3} \cup F_{4} \cup F_{5}$ pe train karo, $F_{2}$ pe validate karo → accuracy = 0.82
Fold 3: $F_{1} \cup F_{2} \cup F_{4} \cup F_{5}$ pe train karo, $F_{3}$ pe validate karo → accuracy = 0.88
Fold 4: $F_{1} \cup F_{2} \cup F_{3} \cup F_{5}$ pe train karo, $F_{4}$ pe validate karo → accuracy = 0.84
Fold 5: $F_{1} \cup F_{2} \cup F_{3} \cup F_{4}$ pe train karo, $F_{5}$ pe validate karo → accuracy = 0.86

Step 4: Mean aur SE calculate karo $CV_{5} = \frac{0.85 + 0.82 + 0.88 + 0.84 + 0.86}{5} = \frac{4.25}{5} = 0.850$

$SE = \frac{( 0.85 - ) ^{2} + ( 0.82 - 0.85 ) ^{2} + ( 0.88 - 0.85 ) ^{2} + ( 0.84 - 0.85 ) ^{2} + ( 0.86 - 0.85 ) ^{2}}{5}$ $= \frac{0 + 0.0009 + 0.0009 + 0.0001}{5} = 0.0004 = 0.02$

Interpretation: Model 85.0% ± 2.0% accuracy achieve karta hai. Kam SE matlab model folds mein consistent hai.

Worked example Example 2: K=3 vs K=10 Compare karna

Setup: 1000 samples, K ki do values compare kar rahe hain.

K=3 (3-fold CV):

Training set size: $\frac{2000}{3} \approx 667$ samples per fold
Har model ~66.7% data pe train hota hai
Bias: Zyada (model final model se kam data pe train hota hai)
Variance: Kam (average karne ke liye sirf 3 estimates)
Speed: Fast (sirf 3 model trainings)

ZYADA BIAS KYUN? Agar haara final model saare 1000 samples use karta hai, lekin CV models sirf 667 use karte hain, toh hum performance underestimate kar rahe hain. 667 aur 1000 ke beech ka gap pessimistic bias introduce karta hai.

K=10 (10-fold CV):

Training set size: 900 samples per fold
Har model 90% data pe train hota hai
Bias: Kam (final model ki training size ke zyada kareeb)
Variance: Zyada (10 noisier estimates average kar rahe hain)
Speed: Slow (10 model trainings)

KAM BIAS KYUN? 900/1000 samples pe training, final 1000 ke kaafi kareeb hai, isliye performance estimate zyada accurate hai.

KYA CHOOSE KAREIN?

K=5 ya K=10: Standard choice, accha bias-variance balance
K=3: Jab computation expensive ho (deep learning, large datasets)
K=N (LOOCV): Jab data bahut scarce ho (< 100 samples) aur minimum bias chahiye

Worked example Example 3: Imbalanced Data ke liye Stratified K-fold

Problem: Tumhare paas 100 samples hain jisme 90 class A aur 10 class B hain (90-10 imbalance). Regular K-fold ka risk: Random split saari 10 class B samples 1-2 folds mein daal sakta hai → kuch folds mein validation mein class B nahi → meaningless accuracy.

Solution: Stratified K-fold

Ensure karo ki har fold same class distribution maintain kare (90-10)
K=5 ke liye: har fold ko 18 class A + 2 class B milenge YEH KYU MATTER KARTA HAI: Har validation fold true data distribution ko represent karta hai.

IMPLEMENT KAISE KAREIN:

Data ko class ke hisaab se alag karo: $D_{A}$ (90 samples), $D_{B}$ (10 samples)
Har class pe alag-alag K-fold apply karo
Corresponding folds merge karo: $F_{i} = F_{i}^{A} \cup F_{i}^{B}$ Result: Har fold mein exactly 9:1 ratio hoga, overall dataset se match karta hua.

Common Mistakes & Fixes

Common mistake Mistake 1: Folds Mein Data Leakage

Log galat kya karte hain: Poore dataset ko PEHLE preprocess (normalize, scale) karte hain, fir folds mein split karte hain.

YEH SAHI KYUN LAGTA HAI: "Pehle apna data normalize karna hai, fir validation ke liye split karna hai."

YEH GALAT KYUN HAI: Jab tum full dataset pe mean/std compute karte ho, toh validation fold ne training folds ki information pehle se "dekh" li hoti hai. Yahi data leakage hai. Example:

# WRONG
X_normalized = (X - X.mean()) / X.std()  # Uses info from ALL data
for train_idx, val_idx in kfold.split(X_normalized):
    # Validation set has leaked info!

Fix: Preprocessing sirf training folds pe fit karo, validation fold pe apply karo:

# RIGHT
for train_idx, val_idx in kfold.split(X):
    X_train, X_val = X[train_idx], X[val_idx]
    scaler = StandardScaler().fit(X_train)  # Learn from training only
    X_train_scaled = scaler.transform(X_train)
    X_val_scaled = scaler.transform(X_val)  # Apply same transformation

Steel-man: Original intuition (pehle normalize karo) simple train/test split ke liye kaam karta hai. Galti yeh hai ki jab folds involve hote hain toh mental model update nahi hota.

Common mistake Mistake 2: Hyperparameter Tuning Ke Liye Seedha K-fold Use Karna

Log galat kya karte hain: Har hyperparameter combination ke liye K-fold CV run karte hain, best choose karte hain, aur woh K-fold score final performance batate hain.

YEH GALAT KYUN HAI: Ab tumne decisions (hyperparameter selection) lene ke liye validation folds use kar liye hain. K-fold score optimistically biased ho gaya hai.

Fix: Nested Cross-Validation

Outer loop: performance estimation ke liye K-fold
Inner loop: har outer fold ke andar hyperparameter selection ke liye K-fold Structure:

For each outer fold i:
    Training set = outer_traini
    Test set = outer_test_i
    For each hyperparameter config:
        Inner K-fold CV on outer_train_i → select best config
    Train final model with best config on outer_train_i
    Evaluate on outer_test_i → score_i
Report: mean(score_i)

YEH KYUN KAAM KARTA HAI: Outer test folds kabhi hyperparameter selection mein participate nahi karte, isliye score unbiased hota hai.

Common mistake Mistake 3: Time Dependencies Ignore Karna (Time Series)

Log galat kya karte hain: Time series data pe random K-fold apply karte hain.

YEH GALAT KYUN HAI: Tum future data pe train kar rahe ho aur past data pe test kar rahe ho → unrealistic, inflated performance.

Fix: Time Series Cross-Validation

Hamesha temporal order maintain karo
Expanding ya sliding window use karo Expanding window example (K=4):

Fold 1: Train [1:100], Test [101:125]
Fold 2: Train [1:125], Test [126:150]
Fold 3: Train [1:150], Test [151:175]
Fold 4: Train [1:175], Test [176:200]

YEH KYUN KAAM KARTA HAI: Tum sirf future predict karte ho, kabhi past nahi.

Bias aur Variance ke Peeche ka Math

Sawaal: Bade K wale K-fold mein lower bias lekin higher variance kyun hota hai?

Bias derivation:

Maano $θ_{N}$ woh model hai jo saare N samples pe trained hai (hamara target), aur $θ_{(K - 1) N / K}$ woh model hai jo $(K - 1) N / K$ samples pe trained hai (har fold).

Estimation bias is difference se aata hai: $Bias \propto ∣ E [error (θ_{(K - 1) N / K})] - E [error (θ_{N})] ∣$

BADA K BIAS KYUN KAM KARTA HAI: Jab $K \to N$ , $\frac{( K - 1 ) N}{K} \to N$ , isliye training set size full dataset ke kareeb pahunch jaati hai. Model ki performance waise hi ho jaati hai jaise full data pe train kiya gaya ho.

Variance derivation:

K-fold estimator ka variance hai: $Var (CV_{K}) = Var (\frac{1}{K} \sum_{i = 1}^{K} score_{i}) = \frac{1}{K ^{2}} \sum_{i = 1}^{K} Var (score_{i}) + \frac{2}{K ^{2}} \sum_{i < j} Cov (score_{i}, score_{j})$

$\frac{3}{4}$ $CV_{K} = \frac{1}{K} \sum_{i = 1}^{K} M (θ_{i}, F_{i})$ where each model θᵢ is trained on all folds except Fᵢ, and evaluated on Fᵢ.

Cross-validation mein bada K bias kyun kam karta hai?

Bada K matlab har training set size full dataset ke kareeb hoti hai (jaise K=10 mein 90% data use hota hai). Isse CV estimate uss model ki performance ke kareeb aa jaata hai jo saare data pe trained ho.

Cross-validation mein bada K variance kyun badhata hai?

Bada K chhote validation sets wale zyada folds banata hai (noisier estimates) aur training sets mein zyada overlap hota hai (scores ke beech positive correlation), dono hi variance badhate hain.

K ki typical choice kya hoti hai aur kyun?

K=5 ya K=10. Yeh bias (80-90% data pe training full dataset ke kareeb hai) aur variance (bahut zyada correlated estimates nahi) ko balance karta hai, aur computationally bhi reasonable hai.

K-fold CV mein data leakage kya hota hai?

Jab preprocessing (normalization, feature selection) poore dataset pe split karne se pehle ki jaati hai, toh validation folds mein training folds ki information aa jaati hai. Fix: preprocessing sirf training folds pe fit karo, phir validation pe apply karo.

Stratified K-fold kya hai aur kab zaroori hai?

Stratified K-fold har fold mein full dataset jaisi class distribution maintain karta hai. Imbalanced datasets ke liye zaroori hai taaki har validation fold representative ho.

Nested cross-validation kya hai?

Performance estimation ke liye outer K-fold loop, hyperparameter tuning ke liye inner K-fold loop. Validation data ko model selection mein use karne se hone wale optimistic bias ko rokta hai.

Time series ke liye regular K-fold kyun use nahi kar sakte?

Isse future data pe training aur past data pe testing hoti, jo temporal causality violate karta hai. Time series CV use karo expanding/sliding windows ke saath jo sirf past se future predict karte hain.

Folds mein high standard error kya indicate karta hai?

Model ki performance bahut sensitive hai iss baat ke liye ki use kis data pe train kiya gaya hai, jo possible overfitting, data quality issues, ya model ki robustness ki kami suggest karta hai.

Leave-One-Out Cross-Validation (LOOCV) kya hai? :: Woh extreme case jahan K=N (samples ki sankhya) hoti hai. Har model N-1 samples pe train hota hai aur 1 pe validate. Minimum bias lekin maximum variance aur computational cost.