Train - validation - test splitting

The Three-Way Split: Purpose & Philosophy

Training set is where the model learns patterns. Validation set is where we tune hyperparameters and make architecture decisions. Test set is the final exam—untouched until the very end, it measures real-world performance.

Why Three Sets Instead of Two?

The information leakage problem: If you use the same data to both select the best model AND evaluate it, your performance estimate is optimistically biased.

Derivation from first principles:

1. Goal: Estimate performance on unseen data $P_{\text{unseen}}$
2. Problem: We only have dataset $D$
3. Naive approach: Split into train/test, evaluate on test → But if we try100 models and pick the best test score, we've implicitly trained on the test set (through model selection)
4. Solution: Use validation set for model selection, keep test set completely hidden until final evaluation

This extends the holdout method (a two-way train/test split) by adding a validation set. Strictly speaking, "holdout" refers to holding out a single test set; the three-way version is often called holdout validation with a separate validation set to distinguish it from cross-validation.

The Splitting Process: Step-by-Step

Step 1: Shuffle (with stratification)

Why shuffle? Raw datasets often have temporal ordering or grouped structure. If first80% are all class A and last 20% are class B, a sequential split fails catastrophically.

Why stratify? For classification with imbalanced classes (e.g., 95% negative, 5% positive), random splitting might put all positives in training and none in validation.

Stratification math: For class $c$ with proportion $p_c$ in dataset: $n_c^{\text{train}} = \lfloor p_c \times n_{\text{train}} \rfloor$ $n_c^{\text{val}} = \lfloor p_c \times n_{\text{val}} \rfloor$ $n_c^{\text{test}} = \lfloor p_c \times n_{\text{test}} \rfloor$

This ensures each split has approximately the same class distribution as the original dataset.

Step 2: Assign indices (stratified)

To actually preserve class proportions, we must split within each class separately, then concatenate. A plain shuffle-and-slice does NOT guarantee stratification.

import numpy as np
 
def stratified_train_val_test_split(X, y, train_size=0.7, val_size=0.15,
                                    random_state=42):
    """
    Split data into train/val/test sets WITH true stratification.
    We split each class's indices separately, then combine, so that
    every split preserves the original class proportions.
    """
    rng = np.random.default_rng(random_state)
    train_idx, val_idx, test_idx = [], [], []
 
    for cls in np.unique(y):
        # Indices belonging to this class only
        cls_indices = np.where(y == cls)[0]
        rng.shuffle(cls_indices)               # shuffle within the class
 
        n_cls = len(cls_indices)
        train_end = int(train_size * n_cls)
        val_end = train_end + int(val_size * n_cls)
 
        train_idx.extend(cls_indices[:train_end])
        val_idx.extend(cls_indices[train_end:val_end])
        test_idx.extend(cls_indices[val_end:])
 
    # Shuffle the combined index lists so classes are interleaved
    train_idx = rng.permutation(train_idx)
    val_idx   = rng.permutation(val_idx)
    test_idx  = rng.permutation(test_idx)
 
    return (X[train_idx], y[train_idx],
            X[val_idx],   y[val_idx],
            X[test_idx],  y[test_idx])

Why this step? Working with indices instead of copying data is memory-efficient and preserves the ability to trace back to original samples.

Step 3: Fit-Transform Pattern (Critical!)

Data preprocessing must use ONLY training set statistics.

The leakage scenario:

# WRONG! Test data leaks into normalization
scaler = StandardScaler()
X_scaled = scaler.fit(X_all)  # Uses test set mean/std
X_train_scaled = X_scaled[:train_end]
X_test_scaled = X_scaled[train_end:]

Why wrong? The model indirectly "sees" test data through the scaling parameters. If test set has unusual values, they influence the mean/std used for training.

Correct approach:

# RIGHT! Only train statistics
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Compute μ, σ from train only
X_val_scaled = scaler.transform(X_val)          # Apply train μ, σ
X_test_scaled = scaler.transform(X_test)        # Apply train μ, σ

Derivation: For feature $x$, standardization is: $z = \frac{x - \mu}{\sigma}$

Where $\mu, \sigma$ must be computed from training set only: $\mu_{\text{train}} = \frac{1}{n_{\text{train}}} \sum_{i \in \text{train}} x_i$ $\sigma_{\text{train}} = \sqrt{\frac{1}{n_{\text{train}}} \sum_{i \in \text{train}} (x_i - \mu_{\text{train}})^2}$

Then apply these same $\mu_{\text{train}}, \sigma_{\text{train}}$ to validation and test sets.

Cross-Validation: When Simple Splitting Isn't Enough

For small datasets ($N < 1000$), splitting removes too much training data. K-fold cross-validation solves this.

How it works:

1. Split data into $K$ equal folds (typically $K=5$ or $K=10$)
2. For each fold $k$:
   • Train on $K-1$ folds
   • Validate on fold $k$
3. Average performance across all $K$ folds

Derivation of variance reduction: Single holdout uses $n_{\text{val}}$ samples for validation. K-fold cross-validation uses ALL $n$ samples for validation (each sample validated exactly once).

Variance of performance estimate: $\text{Var}(\text{holdout}) \approx \frac{\sigma^2}{n_{\text{val}}}$ $\text{Var}(\text{k-fold}) \approx \frac{\sigma^2}{n}$

Since $n > n_{\text{val}}$, k-fold gives more stable estimates.

Trade-off: Computational cost is $K \times$ higher (must train $K$ models instead of 1).

Time Series and Special Cases

Other special cases:

• Grouped data (multiple samples per patient/user): Use GroupKFold to ensure all samples from one group stay together
• Imbalanced classes (1% positive): Use StratifiedKFold to maintain class ratios
• Multiclass: Stratify on the target variable to balance all classes

Connections

• 2.1.11-Feature-scaling-normalization - Must be fit only on training set
• 2.1.13-Cross-validation - Advanced splitting strategy for small datasets
• 3.2.1-Overfitting-underfitting - Why we split: to detect overfitting
• 4.1.5-Early-stopping - Uses validation set, not test set
• 6.3.2-Data-leakage - What happens when splitting is done wrong

#flashcards/ai-ml

What are the three subsets in train/validation/test splitting and their purposes? :: Training set (learn patterns/fit parameters), Validation set (tune hyperparameters/select models), Test set (final performance evaluation on completely unseen data)

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)