Cross-validation — k-fold

5.6.5 · Coding › Machine Learning (Aerospace Applications)

Core Problem: Overfitting vs Generalization

Jab hum aerospace applications ke liye ML models train karte hain (engine failures predict karna, flight paths optimize karna, sensor data mein anomalies detect karna), hume ek fundamental tension face karna padta hai:

WHY hume validation ki zaroorat hai? Kyunki training accuracy ek jhoothi hoti hai. Ek model training data memorize kar sakta hai (100% training accuracy achieve karke) jabki naye data pe bahut bura perform karta hai. Aerospace mein yeh sirf bura nahi — yeh dangerous hai.

WHAT hai naive approach? Ek single test set hold out karo. Lekin iske teen fatal flaws hain:

1. Sample bias: Tumhara test set real-world conditions represent nahi kar sakta
2. Variance: Ek train-test split ek performance number deta hai — kya yeh reliable hai?
3. Data waste: Aerospace mein labeled data expensive hota hai (sensor failures, flight logs). Sirf testing ke liye 20% use karna precious information waste karta hai.

HOW k-fold CV yeh fix karta hai? Test set ko systematically rotate karke.

Cross-Validation Estimate Derive Karna

Chalte hain math scratch se build karte hain.

Given: Dataset $D$ with $n$ samples: $D=\{(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)\}$

Goal: Apne learning algorithm $\mathcal{A}$ ka true generalization error $E_{\text{true}}$ estimate karna.

Step 1: Data partition karo

$D$ ko $k$ disjoint subsets (folds) mein approximately equal size ke saath divide karo: $D=D_1\cup D_2\cup \cdots \cup D_k$

jahan $|D_i|\approx \frac{n}{k}$ aur $D_i\cap D_j=\varnothing$ for $i\ne j$.

Yeh step kyun? Hume independent test sets chahiye. Disjoint folds ensure karte hain ki koi data leakage na ho.

Step 2: k models train karo

Har fold $i\in \{1,2,\dots ,k\}$ ke liye:

• Training set: $D_{\text{train}}^{(i)}=D\setminus D_i$ (fold $i$ ko chhodkar sabhi data)
• Test set: $D_{\text{test}}^{(i)}=D_i$

Model train karo: $M_i=\mathcal{A}(D_{\text{train}}^{(i)})$

Yeh step kyun? Har model training ke liye data ka $(k-1)/k$ hissa dekhta hai, jo full dataset pe training ko approximate karta hai (ek single 80-20 split ke unlike).

Step 3: Har model evaluate karo

Held-out fold pe performance metric (accuracy, RMSE, F1, etc.) compute karo: ${\text{Score}}_i=\text{Metric}(M_i,D_{\text{test}}^{(i)})$

Yeh step kyun? Har fold ek alag "future data" scenario represent karta hai. Model ne training ke dauran fold $i$ nahi dekha, toh yeh ek fair test hai.

Step 4: Scores aggregate karo

k-fold CV estimate $k$ test scores ka mean hai:

k mein Bias-Variance Tradeoff

Chhota $k$ (e.g., $k=3$):

• Bias: Zyada. Har training set mein sirf data ka $\frac{2}{3}$ hissa hota hai, full dataset pe true performance underestimate karta hai.
• Variance: Kam. Average karne ke liye kam models, lekin har test set bada hai (zyada stable estimates).
• Computation: Faster (sirf 3 models train karne hain).

Bada $k$ (e.g., $k=n$, LOOCV):

• Bias: Kam. Training set mein $n-1$ samples hain, full dataset ke almost identical.
• Variance: Zyada. Test sets mein sirf 1 sample each hota hai; individual scores noisy hote hain. Saath hi, training sets heavily overlap karte hain (independent nahi), toh scores correlated hote hain.
• Computation: Sabse slow ($n$ models train karne hain).

$k=5$ ya $k=10$ kyun? Empirical studies (Kohavi 1995) dikhate hain ki yeh values zyaadatar datasets ke liye bias aur variance achhi tarah balance karti hain. Limited data wale aerospace applications mein, $k=5$ common hai.

Worked Example: Aircraft Engine Failure Predict Karna

Scenario: Tumhare paas 1000 labeled flight records hain. Har record mein 50 sensor features hain (temperature, vibration, pressure) aur ek binary label: engine 100 flight hours mein fail hua (1) ya nahi (0). Tum ek Random Forest classifier train kar rahe ho.

Task: Model performance estimate karne ke liye 5-fold CV use karo.

Implementation Steps

Step 1: Shuffle aur split karo

from sklearn.model_selection import KFold
import numpy as np
 
X = sensor_data  # Shape: (1000, 50)
y = failure_labels  # Shape: (1000,)
 
kf = KFold(n_splits=5, shuffle=True, random_state=42)
# shuffle=True ensure karta hai ki folds sirf sequential blocks na hon
# (e.g., 2019 ka sabhi data fold 1 mein, 2020 mein fold 2...)

Shuffle kyun? Agar data time ya flight route ke by sorted hai, toh sequential folds biased honge. Shuffling ensure karta hai ki har fold ek random sample ho.

Step 2: Train aur evaluate karo

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
 
scores = []
for train_idx, test_idx in kf.split(X):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]
    
    model = RandomForestClassifier(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    score = f1_score(y_test, y_pred)  # F1 for imbalanced classes
    scores.append(score)
 
print(f"Scores: {scores}")
# Example output: [0.82, 0.79, 0.85, 0.81, 0.83]

F1 score kyun? Engine failures rare hoti hain (class imbalance). Accuracy misleading hogi (sabke liye "no failure" predict karna 95% accuracy deta hai lekin sabhi failures miss kar deta hai). F1 precision aur recall ko balance karta hai.

Step 3: Mean aur std report karo

cv_mean = np.mean(scores)  # 0.82
cv_std = np.std(scores, ddof=1)  # 0.021
 
print(f"5-fold CV F1: {cv_mean:.3f} ± {cv_std:.3f}")
# Output: "5-fold CV F1: 0.820 ± 0.021"

Dono report kyun karein? Mean typical performance estimate karta hai. Std humein consistency batata hai. 0.82 ± 0.02 wala model 0.82 ± 0.15 se zyada reliable hai (high variance suggest karta hai ki model sensitive hai ki woh kaunsa data dekhta hai).

Results Interpret Karna

• Mean F1 = 0.82: Average pe, model unseen data pe 82% F1 achieve karta hai. Yeh real-world performance ka humara best estimate hai.
• Std = 0.021: Low variance matlab model alag-alag data subsets mein consistently generalize karta hai. Agar std 0.15 hoti, hum investigate karte: Kya kuch folds zyada mushkil hain? Kya data heterogeneous hai?

Imbalanced Classes ke liye Stratified k-fold

Aerospace mein, kai tasks mein rare events hote hain (failures, anomalies). Agar engine failures sirf 5% flights mein hoti hain, toh ek random fold mein 10% failures ho sakti hain jabki doosre mein 2%. Yeh variance badha deta hai.

Implementation:

from sklearn.model_selection import StratifiedKFold
 
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
 
for train_idx, test_idx in skf.split(X, y):
    # Har fold ka y_test mein ~5% failures hongi, y se match karte hue
    ...

Aerospace mein yeh kyun matter karta hai: Rare events safety drive karte hain. Agar ek fold mein accidentally 0 failures hain, toh us fold ka score meaningless hai. Stratification yeh prevent karta hai.

Common Mistakes aur Misconceptions

Computational Cost

$k$ models train karna cost $k$ factor se badhata hai: ${\text{Time}}_{\text{CV}}=k\cdot {\text{Time}}_{\text{train}}$

1000 samples wale Random Forest ke liye trees ke saath:

• Single train: ~5 seconds
• 5-fold CV: ~25 seconds
• 10-fold CV: ~50 seconds

Aerospace context: Large flight logs (millions of records) pe deep learning ke saath training ek model ke liye ghante le sakti hai. $k=5$ matlab 5× zyada time. Isliye GPU acceleration aur parallel fold execution matter karta hai.

Mitigation: Folds parallel mein train karne ke liye scikit-learn mein n_jobs=-1 use karo:

from sklearn.model_selection import cross_val_score
 
scores = cross_val_score(model, X, y, cv=5, scoring='f1', n_jobs=-1)
# 5 folds simultaneously multiple CPU cores pe train karta hai

Connections

• 5.6.01-Train-test-split-validation-set — k-fold basic split idea ko extend karta hai
• 5.6.02-Holdout-method — k-fold vs single holdout tradeoffs
• 5.6.04-Stratified-sampling — Folds mein class balance kaise maintain karein
• 5.6.06-Leave-one-out-cross-validation-(LOOCV) — Extreme case jahan $k=n$
• 5.6.07-Nested-cross-validation — Tuning aur evaluation dono ke liye k-fold use karna
• 5.6.10-Bias-variance-tradeoff-in-model-selection — k bias aur variance ko kyun affect karta hai
• 7.2.03-Hyperparameter-tuning-grid-search — CV grid search ke liye evaluation engine hai
• 8.5.02-Time-series-cross-validation — Aerospace-specific: temporal order respect karna

#flashcards/coding

k-fold cross-validation ka primary purpose kya hai? :: Model generalization performance ka ek zyada robust estimate paana test ke liye data subset ko rotate karke, yeh ensure karte hue ki har data point exactly ek baar test ho.

k-fold cross-validation score ka formula kya hai? :: ${\text{CV}}_k=\frac{1}{k}{\sum }_{i=1}^k{\text{Score}}_i$ jahan har Score_i fold i pe performance metric hai.

Concept Map