Train - validation - test splitting — Hinglish notes

2.1.12 · AI-ML › Data Preprocessing & Feature Engineering

Teen-Way Split: Purpose & Philosophy

Training set wahan hai jahan model patterns seekhta hai. Validation set wahan hai jahan hum hyperparameters tune karte hain aur architecture decisions lete hain. Test set final exam hai—end tak bilkul untouched, yeh real-world performance measure karta hai.

Teen Sets Kyun, Do Kyun Nahi?

Information leakage problem: Agar tum same data use karo best model select karne aur evaluate karne dono ke liye, toh tumhara performance estimate optimistically biased hoga.

First principles se derivation:

Goal: Unseen data par performance estimate karna $P_{unseen}$
Problem: Hamare paas sirf dataset $D$ hai
Naive approach: Train/test mein split karo, test par evaluate karo → Lekin agar hum 100 models try karen aur best test score pick karen, toh humne implicitly test set par train kar liya hai (model selection ke through)
Solution: Validation set use karo model selection ke liye, test set ko bilkul chhupa ke rakho final evaluation tak

Yeh holdout method (ek do-way train/test split) ko extend karta hai ek validation set add karke. Strictly speaking, "holdout" ek single test set hold out karne ko refer karta hai; teen-way version ko often holdout validation with a separate validation set kaha jaata hai cross-validation se distinguish karne ke liye.

Formula Standard Split Ratios

Size $N$ ke dataset ke liye: $Train : Validation : Test = 60 : 20 : 20 or 70 : 15 : 15 or 80 : 10 : 10$ Trade-off ki derivation:

Bada training set → Seekhne ke liye zyada data → Better model parameters
Bada validation set → Zyada reliable hyperparameter tuning → Kam noisy estimates
Bada test set → Final performance estimate mein zyada confidence → Narrow confidence intervals

Fundamental tension: Validation/test mein har sample ek aisa sample hai jo training mein use nahi hua. Rule of thumb:

$N < 1000$ : Cross-validation use karo
$1000 < N < 100 k$ : 70/15/15 ya 80/10/10
$N > 100 k$ : 90:5:5 ya phir 95:2.5:2.5 (10,000 test samples good estimates ke liye kaafi hain)

Kyun? Statistical power: 10,000 test samples ke saath, accuracy ke liye 95% confidence interval width approximately $\pm 1.96 \frac{0.5 \times 0.5}{10000} \approx \pm 1%$ hoti hai

Figure — Train - validation - test splitting

Splitting Process: Step-by-Step

Step 1: Shuffle (stratification ke saath)

Shuffle kyun? Raw datasets mein often temporal ordering ya grouped structure hoti hai. Agar pehle 80% sab class A ke hain aur last 20% class B ke, toh sequential split catastrophically fail ho jaata hai.

Stratify kyun? Imbalanced classes wali classification ke liye (jaise 95% negative, 5% positive), random splitting saare positives training mein daal sakti hai aur validation mein koi nahi.

Stratification math: Dataset mein proportion $p_{c}$ wali class $c$ ke liye: $n_{c}^{train} = ⌊ p_{c} \times n_{train} ⌋$ $n_{c}^{val} = ⌊ p_{c} \times n_{val} ⌋$ $n_{c}^{test} = ⌊ p_{c} \times n_{test} ⌋$

Yeh ensure karta hai ki har split mein approximately original dataset jaisi class distribution ho.

Step 2: Indices assign karo (stratified)

Class proportions ko actually preserve karne ke liye, hume har class ke andar alag-alag split karni padegi, phir concatenate karna hoga. Plain shuffle-and-slice stratification guarantee nahi karta.

import numpy as np
 
def stratified_train_val_test_split(X, y, train_size=0.7, val_size=0.15,
                                    random_state=42):
    """
    Split data into train/val/test sets WITH true stratification.
    We split each class's indices separately, then combine, so that
    every split preserves the original class proportions.
    """
    rng = np.random.default_rng(random_state)
    train_idx, val_idx, test_idx = [], [], []
 
    for cls in np.unique(y):
        # Indices belonging to this class only
        cls_indices = np.where(y == cls)[0]
        rng.shuffle(cls_indices)               # shuffle within the class
 
        n_cls = len(cls_indices)
        train_end = int(train_size * n_cls)
        val_end = train_end + int(val_size * n_cls)
 
        train_idx.extend(cls_indices[:train_end])
        val_idx.extend(cls_indices[train_end:val_end])
        test_idx.extend(cls_indices[val_end:])
 
    # Shuffle the combined index lists so classes are interleaved
    train_idx = rng.permutation(train_idx)
    val_idx   = rng.permutation(val_idx)
    test_idx  = rng.permutation(test_idx)
 
    return (X[train_idx], y[train_idx],
            X[val_idx],   y[val_idx],
            X[test_idx],  y[test_idx])

Yeh step kyun? Data copy karne ki jagah indices ke saath kaam karna memory-efficient hai aur original samples tak trace back karne ki ability preserve karta hai.

Worked example scikit-learn ke saath Real-World Splitting

Scenario: Customer churn predict karna (binary classification) Dataset: 5000 customers, features = [age, tenure, monthly_charges, ...], target = churned (0/1)

from sklearn.model_selection import train_test_split
 
# First split: separate test set (80/20)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
# X_temp has 4000 samples, X_test has 1000 samples
# Second split: divide temp into train/val (87.5/12.5 of temp = 70/10 of original)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.125, stratify=y_temp, random_state=42
)
# X_train: 3500 samples (70%)
# X_val: 500 samples (10%)
# X_test: 1000 samples (20%)

Yeh sizes kyun?

3500 training samples: Customer behavior mein complex patterns seekhne ke liye kaafi
500 validation samples: 5-10 model variants reliably compare karne ke liye kaafi
1000 test samples: Accuracy estimates ke liye 95% CI width ≈ ±3%

Step 3: Fit-Transform Pattern (Critical!)

Data preprocessing mein SIRF training set statistics use honi chahiye.

Leakage scenario:

# WRONG! Test data leaks into normalization
scaler = StandardScaler()
X_scaled = scaler.fit(X_all)  # Uses test set mean/std
X_train_scaled = X_scaled[:train_end]
X_test_scaled = X_scaled[train_end:]

Galat kyun? Model indirectly test data "dekhta" hai scaling parameters ke through. Agar test set mein unusual values hain, toh woh training ke liye use hone wale mean/std ko influence karte hain.

Sahi approach:

# RIGHT! Only train statistics
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Compute μ, σ from train only
X_val_scaled = scaler.transform(X_val)          # Apply train μ, σ
X_test_scaled = scaler.transform(X_test)        # Apply train μ, σ

Derivation: Feature $x$ ke liye, standardization hai: $z = \frac{x - μ}{σ}$

Jahan $μ, σ$ sirf training set se compute hone chahiye: $μ_{train} = \frac{1}{n _{train}} \sum_{i \in train} x_{i}$ $σ_{train} = \frac{1}{n _{train}} \sum_{i \in train} (x_{i} - μ_{train})^{2}$

Phir yahi same $μ_{train}, σ_{train}$ validation aur test sets par apply karo.

Common mistake Common Splitting Mistakes

Mistake 1: Split se pehle saare data par fit karna

# WRONG!
X_normalized = normalize(X)
X_train, X_test = split(X_normalized)

Kyun sahi lagta hai: "Main sirf preprocessing kar raha hoon, model train nahi kar raha" Kyun galat hai: Test set statistics (min, max, mean, std) normalization parameters ke through training mein leak ho jaati hain Fix: Pehle split karo, phir sirf training set par preprocessing fit karo

Mistake 2: Early stopping ke liye test set use karna

# WRONG!
for epoch in range(100):
    train(model, X_train)
    if loss(model, X_test) < threshold:
        break  # Early stopping on test set

Kyun sahi lagta hai: "Main unseen data par best performing model chahta hoon" Kyun galat hai: Tum test performance ke basis par training decisions le rahe ho (kab rokna hai) → test set ek second validation set ban jaata hai Fix: Early stopping ke liye validation set use karo, test set ko final evaluation tak bilkul untouched rakho

Mistake 3: Imbalanced classes ko stratify na karna Scenario: Fraud detection jisme 99% legitimate, 1% fraud transactions hain Stratification ke bina: Random 70/30 split saare fraud cases training set mein daal sakta hai → validation/test mein 100% legitimate → misleading metrics Fix: Classification ke liye hamesha stratify=y use karo

Recall Ek 12-Saal Ke Bache Ko Samjhao

Socho tum alag-alag birds ko pehchanna seekh rahe ho. Tumhare paas 100 bird photos hain.

Galat tarika: Seekhte waqt saari 100 photos dekho. Phir... inhi 100 photos par apna test karo. Bahut accha karoge! Lekin kya tumne sach mein birds pehchanna seekha, ya sirf woh specific photos yaad kar li?

Behtar tarika:

70 photos lo → Train set (yeh tumhara study material hai)
15 photos chhupao → Validation set (inhe use karo apne aap ko quiz karne ke liye aur figure out karne ke liye ki zyada padhna hai ya approach change karni hai)
Aur 15 photos chhupao → Test set (FINAL exam, ekdum end mein sirf ek baar dekho)

Validation set practice quizzes jaisi hai—tum inhe dobara de sakte ho aur apni study strategy adjust kar sakte ho. Test set real final exam hai—sirf ek chance milta hai, aur yeh batata hai ki tumne sach mein seekha ya sirf lucky the.

Magic trick: Kuch photos ekdum end tak chhupa ke, tum ensure karte ho ki tum "robin kya hota hai" seekh rahe ho instead of sirf yeh memorize karne ke ki "photo #23 robin hai."

Cross-Validation: Jab Simple Splitting Kaafi Nahi Hoti

Chhote datasets ke liye ( $N < 1000$ ), splitting bahut zyada training data hata deti hai. K-fold cross-validation ise solve karta hai.

Kaise kaam karta hai:

Data ko $K$ equal folds mein split karo (typically $K = 5$ ya $K = 10$ )
Har fold $k$ ke liye:
- $K - 1$ folds par train karo
- Fold $k$ par validate karo
Saare $K$ folds mein performance average karo

Variance reduction ki derivation: Single holdout validation ke liye $n_{val}$ samples use karta hai. K-fold cross-validation validation ke liye SAARE $n$ samples use karta hai (har sample exactly ek baar validate hota hai).

Performance estimate ka variance: $Var (holdout) \approx \frac{σ ^{2}}{n _{val}}$ $Var (k-fold) \approx \frac{σ ^{2}}{n}$

Kyunki $n > n_{val}$ , k-fold zyada stable estimates deta hai.

Trade-off: Computational cost $K \times$ zyada hai (1 ki jagah $K$ models train karne padte hain).

Worked example Kaun Si Strategy Kab Use Karen

Scenario 1: Bada dataset (100,000 samples), deep learning

Simple 80/10 split use karo
Kyun? 10,000 validation samples reliable estimates ke liye kaafi hai, cross-validation bahut expensive hai Scenario 2: Medium dataset (5,000 samples), random forest
70/15/15 split ya 5-fold cross-validation for hyperparameter tuning + holdout test set use karo
Kyun? Training data size aur validation reliability ke beech accha balance

Scenario 3: Chhota dataset (500 samples), logistic regression

Model selection ke liye 10-fold cross-validation + final 80/20 holdout use karo
Kyun? Simple split ke saath validation mein 100 samples "waste" karna afford nahi kar sakte

Scenario 4: Bahut chhota dataset (100 samples), SVM

Leave-One-Out cross-validation use karo (LOOCV, jahan $K = N$ )
Kyun? Training data usage maximize karo, chhote $N$ ke liye computational cost acceptable hai

Time Series aur Special Cases

Formula Time Series Splitting Rules

Temporal data ke liye, KABHI random splitting use mat karo. Hamesha chronological order respect karo:

$Train \to Validation \to Test$

Kyun? Future se past predict karna causality violate karta hai. Random splits temporal leakage create karte hain.

Example split:

Train: Jan 2020 - Dec 2022 (3 saal)
Validation: Jan 2023 - Jun 2023 (6 mahine)
Test: Jul 2023 - Dec 2023 (6 mahine)

Walk-forward validation time series ke liye: $K$ folds ke liye, har fold time mein aage badhta hai:

Fold 1: Months 1-12 par train karo, month 13 par validate karo
Fold 2: Months 1-13 par train karo, month 14 par validate karo
Fold 3: Months 1-14 par train karo, month 15 par validate karo
...

Yeh realistic deployment simulate karta hai jahan tum periodically saare historical data par retrain karte ho.

Doosre special cases:

Grouped data (ek patient/user ke multiple samples): GroupKFold use karo ensure karne ke liye ki ek group ke saare samples saath rahen
Imbalanced classes (1% positive): Class ratios maintain karne ke liye StratifiedKFold use karo
Multiclass: Saari classes balance karne ke liye target variable par stratify karo

Connections

2.1.11-Feature-scaling-normalization - Sirf training set par fit hona chahiye
2.1.13-Cross-validation - Chhote datasets ke liye advanced splitting strategy
3.2.1-Overfitting-underfitting - Kyun split karte hain: overfitting detect karne ke liye
4.1.5-Early-stopping - Validation set use karta hai, test set nahi
6.3.2-Data-leakage - Kya hota hai jab splitting galat ki jaaye

#flashcards/ai-ml

Train/validation/test splitting mein teen subsets kya hain aur unke purposes kya hain? :: Training set (patterns seekhna/parameters fit karna), Validation set (hyperparameters tune karna/models select karna), Test set (bilkul unseen data par final performance evaluation)

Do-way train/test ki jagah alag validation set ki zaroorat kyun hai?

Model selection ke dauran information leakage rokne ke liye. Agar hum bahut saare models try karen aur best test performer pick karen, toh humne selection process ke through implicitly test set par train kar liya hota hai.

Stratification kya hai aur yeh kyun important hai?

Stratification ensure karta hai ki har split mein original dataset jaisi class distribution ho. Har class ko alag-alag split karke phir combine karke achieve hota hai. Imbalanced classes ke liye critical hai (jaise 95%/5%) taaki kisi split mein minority class ke samples na hon.

Preprocessing ka sahi order kya hai: scaler kis set par fit karein?

Preprocessing (scaler, imputer, encoder) SIRF training set par fit karo, phir unhi fitted parameters se train/val/test transform karo. Kabhi split se pehle saare data par fit mat karo.

Alag-alag dataset sizes ke liye standard split ratios kya hain?

N<1k: cross-validation use karo; 1k<N<100k: 70/15/15 ya 80/10/10; N>100k: 90:5:5 ya 95:2.5:2.5 (good estimates ke liye 10k test samples kaafi hain)

Time series data ko randomly split kyun nahi karna chahiye?

Random splitting temporal leakage create karta hai—model future dekhta hai past predict karne ke liye, jo causality violate karta hai. Chronologically split karna zaroori hai: Train→Validation→Test.

Fit-transform pattern kya hai?

Sirf training data par preprocessing fit karo (statistics jaise mean/std compute karo), phir unhi same statistics se train/val/test transform karo. Test data ka preprocessing parameters ke through leakage rokta hai.

Kya random shuffle-and-slice stratification guarantee karta hai?

Nahi. Random shuffling sirf on average splits balance karta hai. Imbalanced/chhote datasets ke liye phir bhi minority classes galat represent ho sakti hain. True stratification har class ko alag-alag split karta hai.

Holdout method aur three-way splitting mein kya fark hai?

"Holdout" strictly ek do-way train/test split ko refer karta hai. Teen-way split isme ek validation set add karke extend karta hai (holdout validation with separate validation set), jo cross-validation se alag hai.

Stratified splitting mathematically class distribution kaise preserve karti hai?

Proportion p_c wali class c ke liye, training set mein n_train^c = floor(p_c × n_train) samples allocate karo, similarly val/test ke liye. Splits mein proportions maintain hoti hain.

Simple split ki jagah K-fold cross-validation kyun use karen?

Chhote datasets ke liye, single split validation mein bahut zyada data "waste" karta hai. K-fold saare N samples validation ke liye use karta hai (har ek exactly ek baar validate hota hai), performance estimate ka variance σ²/n_val se σ²/n tak kam karta hai.