Cross-validation — k-fold

The Core Problem: Overfitting vs Generalization

When training ML models for aerospace applications (predicting engine failures, optimizing flight paths, detecting anomalies in sensor data), we face a fundamental tension:

WHY do we need validation? Because training accuracy is a liar. A model can memorize training data (achieving 100% training accuracy) while performing terribly on new data. In aerospace, this isn't just bad—it's dangerous.

WHAT is the naive approach? Hold out a single test set. But this has three fatal flaws:

1. Sample bias: Your test set might not represent real-world conditions
2. Variance: One train-test split gives one performance number—is it reliable?
3. Data waste: In aerospace, labeled data is expensive (sensor failures, flight logs). Using20% only for testing wastes precious information.

HOW does k-fold CV fix this? By systematically rotating the test set.

Deriving the Cross-Validation Estimate

Let's build the math from scratch.

Given: Dataset $D$ with $n$ samples: $D = \{(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)\}$

Goal: Estimate the true generalization error $E_{\text{true}}$ of our learning algorithm $\mathcal{A}$.

Step 1: Partition the data

Divide $D$ into $k$ disjoint subsets (folds) of approximately equal size: $D = D_1 \cup D_2 \cup \cdots \cup D_k$

where $|D_i| \approx \frac{n}{k}$ and $D_i \cap D_j = \emptyset$ for $i \neq j$.

Why this step? We need independent test sets. Disjoint folds ensure no data leakage.

Step 2: Train k models

For each fold $i \in \{1, 2, \ldots, k\}$:

• Training set: $D_{\text{train}}^{(i)} = D \setminus D_i$ (all data except fold $i$)
• Test set: $D_{\text{test}}^{(i)} = D_i$

Train model: $M_i = \mathcal{A}(D_{\text{train}}^{(i)})$

Why this step? Each model sees $(k-1)/k$ of the data for training, which approximates training on the full dataset (unlike a single 80-20 split).

Step 3: Evaluate each model

Compute performance metric (accuracy, RMSE, F1, etc.) on the held-out fold: $\text{Score}_i = \text{Metric}(M_i, D_{\text{test}}^{(i)})$

Why this step? Each fold represents a different "future data" scenario. The model hasn't seen fold $i$ during training, so this is a fair test.

Step 4: Aggregate scores

The k-fold CV estimate is the mean of the $k$ test scores:

Bias-Variance Tradeoff in k

Small $k$ (e.g., $k=3$):

• Bias: Higher. Each training set contains only $\frac{2}{3}$ of data, underestimating true performance on the full dataset.
• Variance: Lower. Fewer models to average, but each test set is larger (more stable estimates).
• Computation: Faster (only 3 models to train).

Large $k$ (e.g., $k=n$, LOOCV):

• Bias: Lower. Training set has $n-1$ samples, nearly identical to full dataset.
• Variance: Higher. Test sets have only1 sample each; individual scores are noisy. Also, training sets overlap heavily (not independent), so scores are correlated.
• Computation: Slowest ($n$ models to train).

Why $k=5$ or $k=10$? Empirical studies (Kohavi 1995) show these values balance bias and variance well for most datasets. In aerospace with limited data, $k=5$ is common.

Worked Example: Predicting Aircraft Engine Failure

Scenario: You have 1000 labeled flight records. Each record has50 sensor features (temperature, vibration, pressure) and a binary label: engine failed within 100 flight hours (1) or not (0). You're training a Random Forest classifier.

Task: Use5-fold CV to estimate model performance.

Implementation Steps

Step 1: Shuffle and split

from sklearn.model_selection import KFold
import numpy as np
 
X = sensor_data  # Shape: (1000, 50)
y = failure_labels  # Shape: (1000,)
 
kf = KFold(n_splits=5, shuffle=True, random_state=42)
# shuffle=True ensures folds aren't just sequential blocks
# (e.g., all 2019 data in fold 1, 2020 in fold 2...)

Why shuffle? If data is sorted by time or flight route, sequential folds would be biased. Shuffling ensures each fold is a random sample.

Step 2: Train and evaluate

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
 
scores = []
for train_idx, test_idx in kf.split(X):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]
    
    model = RandomForestClassifier(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    score = f1_score(y_test, y_pred)  # F1 for imbalanced classes
    scores.append(score)
 
print(f"Scores: {scores}")
# Example output: [0.82, 0.79, 0.85, 0.81, 0.83]

Why F1 score? Engine failures are rare (class imbalance). Accuracy would be misleading (predicting "no failure" for everything gives 95% accuracy but mises all failures). F1 balances precision and recall.

Step 3: Report mean and std

cv_mean = np.mean(scores)  # 0.82
cv_std = np.std(scores, ddof=1)  # 0.021
 
print(f"5-fold CV F1: {cv_mean:.3f} ± {cv_std:.3f}")
# Output: "5-fold CV F1: 0.820 ± 0.021"

Why report both? The mean estimates typical performance. The std tells us consistency. A model with 0.82 ± 0.02 is more reliable than 0.82 ± 0.15 (high variance suggests the model is sensitive to which data it sees).

Interpreting Results

• Mean F1 = 0.82: On average, the model achieves 82% F1 on unseen data. This is our best estimate of real-world performance.
• Std = 0.021: Low variance means the model generalizes consistently across different data subsets. If std were0.15, we'd investigate: Are certain folds harder? Is the data heterogeneous?

Stratified k-fold for Imbalanced Classes

In aerospace, many tasks involve rare events (failures, anomalies). If engine failures occur in only 5% of flights, a random fold might contain 10% failures while another has 2%. This inflates variance.

Implementation:

from sklearn.model_selection import StratifiedKFold
 
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
 
for train_idx, test_idx in skf.split(X, y):
    # Each fold's y_test has ~5% failures, matching y
    ...

Why this matters in aerospace: Rare events drive safety. If one fold accidentally has 0failures, that fold's score is meaningless. Stratification prevents this.

Common Mistakes and Misconceptions

Computational Cost

Training $k$ models increases cost by a factor of $k$: $\text{Time}_{\text{CV}} = k \cdot \text{Time}_{\text{train}}$

For a Random Forest on 1000 samples with trees:

• Single train: ~5 seconds
• 5-fold CV: ~25 seconds -10-fold CV: ~50 seconds

Aerospace context: Training on large flight logs (millions of records) with deep learning can take hours per model. $k=5$ means 5× longer. This is why GPU acceleration and parallel fold execution matter.

Mitigation: Use n_jobs=-1 in scikit-learn to train folds in parallel:

from sklearn.model_selection import cross_val_score
 
scores = cross_val_score(model, X, y, cv=5, scoring='f1', n_jobs=-1)
# Trains 5 folds simultaneously on multiple CPU cores

Connections

• 5.6.01-Train-test-split-validation-set — k-fold extends the basic split idea
• 5.6.02-Holdout-method — k-fold vs single holdout tradeoffs
• 5.6.04-Stratified-sampling — How to maintain class balance in folds
• 5.6.06-Leave-one-out-cross-validation-(LOOCV) — Extreme case where $k=n$
• 5.6.07-Nested-cross-validation — Using k-fold for both tuning and evaluation
• 5.6.10-Bias-variance-tradeoff-in-model-selection — Why k affects bias and variance
• 7.2.03-Hyperparameter-tuning-grid-search — CV is the evaluation engine for grid search
• 8.5.02-Time-series-cross-validation — Aerospace-specific: respecting temporal order

#flashcards/coding

What is the primary purpose of k-fold cross-validation? :: To get a more robust estimate of model generalization performance by rotating which subset of data serves as the test set, ensuring every data point is tested exactly once.

What is the formula for the k-fold cross-validation score? :: $\text{CV}_k = \frac{1}{k} \sum_{i=1}^{k} \text{Score}_i$ where each Score_i is the performance metric on fold i.

Concept Map

Hinglish (regional understanding)