Training, validation, and test error

2.6.3 · AI-ML › Model Evaluation & Selection

Overview

Data ka three-way split — training, validation, aur test sets mein — honest model evaluation ki backbone hai. Har subset machine learning pipeline mein alag kaam karta hai, aur inhe confuse karne se overly optimistic (ya pessimistic) performance estimates nikalte hain.

Fundamental tension ye hai: tum jaanna chahte ho ki tumhara model bilkul nayi, unseen data pe kaisa perform karta hai, lekin development ke dauran improve karne ke liye feedback bhi chahiye. Iska solution hai compartmentalization — har dataset alag sawaal ka jawaab deta hai.

The Three Errors: Definitions & Roles

$E_{\text{train}}=\frac{1}{n_{\text{train}}}{\sum }_{i=1}^{n_{\text{train}}}L(y_i,\hat{f}(x_i;{\theta }^{\ast }))$

jahaan ${\theta }^{\ast }$ woh parameters hain jo is sum ko minimize karte hain. Ye error fit measure karta hai, generalization nahi.

YE KYU EXIST KARTA HAI: Tumhe optimization guide karne ke liye ek metric chahiye. Gradient descent ko descend karne ke liye ek function chahiye.

YE KYA BATATA HAI: Model training data ko kitna memorize karta hai. Low training error ≠ achha model (overfitting).

ISKO KAISE USE KAREIN: Training ke dauran monitor karo ensure karne ke liye ki learning ho rahi hai. Agar training error decrease nahi ho raha, to learning rate, initialization, ya data preprocessing check karo.

$E_{\text{val}}=\frac{1}{n_{\text{val}}}{\sum }_{j=1}^{n_{\text{val}}}L(y_j,\hat{f}(x_j;{\theta }^{\ast }))$

YE KYU EXIST KARTA HAI: Development ke dauran test set ko "peek" kiye bina generalization estimate karne ke liye. Tum hyperparameters (learning rate, regularization, architecture) ko validation errors compare karke tune karte ho.

YE KYA BATATA HAI: Tune kiye gaye options mein se kaun sa configuration best generalize karta hai.

ISKO KAISE USE KAREIN: Multiple models alag-alag hyperparameters ke saath train karo, jo lowest $E_{\text{val}}$ wala ho use pick karo. Jab $E_{\text{val}}$ decrease hona band ho jaye to training rok do (early stopping).

$E_{\text{test}}=\frac{1}{n_{\text{test}}}{\sum }_{k=1}^{n_{\text{test}}}L(y_k,\hat{f}(x_k;{\theta }^{\ast }))$

YE KYU EXIST KARTA HAI: Real-world data pe generalization ka unbiased estimate dene ke liye. Ye tumhari "competition performance" hai.

YE KYA BATATA HAI: Nayi data pe true expected error (sampling noise aur distribution shift tak).

ISKO KAISE USE KAREIN: Bilkul ek baar compute karo, ekdum end mein, jab saare decisions finalize ho jaayein. Agar tum test error ke basis pe iterate karte ho, to ye ek second validation set ban jaata hai (data leakage).

Derivation: Validation Error Generalization Ko Kyun Approximate Karta Hai

True generalization error (data distribution pe expected loss) se shuru karo:

$E_{\text{true}}={\mathbb{E}}_{(x,y)\sim P_{\text{data}}}[L(y,\hat{f}(x))]$

Hum ye compute nahi kar sakte — humare paas full distribution $P_{\text{data}}$ nahi hai. Lekin agar validation set ${(x_j,y_j)}_{j=1}^{n_{\text{val}}}$ $P_{\text{data}}$ se i.i.d. draw kiya gaya hai, to Law of Large Numbers ke anusaar:

$E_{\text{val}}=\frac{1}{n_{\text{val}}}{\sum }_{j=1}^{n_{\text{val}}}L(y_j,\hat{f}(x_j))\stackrel{n_{\text{val}}\to \infty }{\to }{\mathbb{E}}_{P_{\text{data}}}[L(y,\hat{f}(x))]=E_{\text{true}}$

Ye step kyun? Har validation sample us point pe true loss ka unbiased estimate contribute karta hai. Independent unbiased estimates ka average expectation ki taraf converge karta hai.

Sabse important baat, validation set training process se independent hona chahiye. Agar model ne gradient updates ke dauran validation data dekha, to estimate downward biased ho jaata hai (model ne validation patterns "memorize" kar liye).

Isi tarah, $E_{\text{test}}$ bhi $E_{\text{true}}$ ko approximate karta hai, lekin ek key difference ke saath: test error us model pe compute hota hai jiske hyperparameters validation error use karke choose kiye gaye the. Agar tumhare paas separate validation set nahi hai aur tum test set pe tune karte ho, to $E_{\text{test}}$ $E_{\text{true}}$ ko underestimate karta hai (kyunki tumne woh model pick kiya jo uss test set pe chance se achha karta tha).

Typical Dataset Split Ratios

• Large datasets ($N>100,000$):

  • Train: 98%, Val: 1%, Test: 1%
  • Kyun? 100k samples ke saath, 1% = 1000 samples — stable error estimates ke liye kaafi hai.

• Medium datasets ($10,000<N<100,000$):

  • Train: 80%, Val: 10%, Test: 10%
  • Kyun? Kaafi training data hone aur reliable validation/test estimates ke beech balance banta hai.

• Small datasets ($N<10,000$):

  • Iske bajaay k-fold cross-validation use karo: $k$ folds mein split karo, rotate karo kaunsa fold validation hai, results average karo. Phir ek held-out test set pe ek baar evaluate karo.
  • Kyun? Chhote data ke saath, ek single validation split mein high variance hota hai. Cross-validation ise reduce karta hai.

Fold size ki derivation: Maano $N=1000$, $k=5$. Har fold mein $N/k=200$ samples hain. Tum $4\times 200=800$ samples pe train karte ho aur 200 pe validate karte ho. 5 baar repeat karo, har baar alag fold validation ke roop mein use karo. 5 validation errors ko average karke ek stable estimate lo.

The Training-Validation Gap

$\Delta =E_{\text{val}}-E_{\text{train}}$

Interpretation:

• $\Delta \approx 0$: Model achhi tarah generalize karta hai (good fit, no overfitting).
• $\Delta \gg 0$: Overfitting — model ne training data memorize kiya lekin nayi data pe fail ho jaata hai.
• $\Delta <0$: Rare hai, lekin possible hai agar training set unrepresentative ho ya label noise ho.

$\Delta$ model complexity ke saath kyun badhta hai?

Ek aisa model socho jisme $p$ parameters hain aur $n$ samples pe train hua hai. Jaise-jaise $p\to n$, model har training point ko exactly fit kar sakta hai (interpolation). Training error $\to 0$. Lekin model ne apni saari capacity noise memorize karne mein "kharch" kar di, isliye validation error high rehta hai.

Bias-variance decomposition (informal): $E_{\text{val}}\approx \text{(irreducible noise)}+{\text{(bias)}}^2+\text{(variance)}$

• Bias: galat model assumptions se error (underfitting).
• Variance: training set fluctuations ke sensitivity se error (overfitting).

Jaise model complexity badhti hai:

• Bias $\downarrow$ (zyada expressive model)
• Variance $\uparrow$ (overfit karne ke zyada tarike)
• Training error $\downarrow$ (better fit)
• Validation error: $\downarrow$ phir $\uparrow$ (beech mein sweet spot)

Gap $\Delta$ primarily variance se driven hai. Regularization (L2, dropout, data augmentation) variance reduce karta hai, gap ko shrink karta hai.

Worked Examples

Step 1: Training ko 45,000 train, 5,000 validation mein split karo.

Ye step kyun? Hyperparameters compare karne ke liye tumhe validation set chahiye. Test set bilkul untouched rehna chahiye.

Step 2: 45k images pe alag-alag $(lr,\lambda )$ pairs ke saath 10 models train karo.

| Config | $lr$ | $\lambda$ | $E_{\text{train}}$ | $E_{\text{val}}$ | |-----|------|-------------------|----------------| | A | 0.01 | 0.001 | 0.15 | 0.42| | B | 0.001| 0.01 | 0.08 | 0.38 | | C | 0.1 | 0 | 0.02 | 0.55 |

Dono kyun compute karein? Training error akela nahi batata ki kaunsa model generalize karta hai. Config C ka training error sabse low hai lekin validation error sabse high hai (overfitting).

Step 3: Config B pick karo (lowest $E_{\text{val}}$).

Step 4: Config B ko saare 50k training images pe retrain karo (train+val combine karke) taaki maximum data use ho. 10k test set pe ek baar evaluate karo.

Retrain kyun? Tune karne ke liye sirf 5k samples hold out kiye the. Ab tuning ho gayi hai, final model improve karne ke liye unhe use karo.

Result: $E_{\text{test}}=0.40$. Ye tumhara generalization ka unbiased estimate hai.

Split: 70 train, 15 validation, 15 test.

Training Results (MSE):

| Degree $d$ | $E_{\text{train}}$ | $E_{\text{val}}$ | $E_{\text{test}}$ | |----------|---------------------|----------------| | 1 | 2.8 | 3.1 | (abhi compute nahi hua)| | 2 | 0.26 | 0.29 | (abhi compute nahi hua)| | 5 | 0.18 | 0.52 | (abhi compute nahi hua)| | 10 | 0.09 | 1.24 | (abhi compute nahi hua)|

Ye step kyun? Degree 1 (linear) underfit karta hai — high bias, dono errors large. Degree 2 true function se match karta hai — low bias, low variance. Degree 5 overfit karna shuru karta hai — training error girta hai, validation error badhta hai. Degree 10 severely overfit karta hai — noise memorize karta hai.

Decision: $d=2$ pick karo (lowest $E_{\text{val}}$).

Test evaluation: $d=2$ model ke liye $E_{\text{test}}=0.31$. $E_{\text{val}}$ ke close, jo achhi generalization confirm karta hai.

Agar training error ke basis pe pick karte: $d=10$ choose karte (lowest $E_{\text{train}}=0.09$), phir discover karte ki $E_{\text{test}}\approx 1.2$ — disaster!

Common Mistakes

Ye sahi kyun lagta hai: Test set "unseen" hai, to kya uss pe performance generalization reflect nahi karta?

The fix: Jab tum test performance dekh ke decisions lete ho (jaise "model A ne model B ko beat kiya, A ko aur tweak karte hain"), to test set ab independent nahi raha. Tum implicitly uss specific test set ke liye optimize kar rahe ho. Ye data leakage hai.

Steel-man: Confusion isliye hoti hai kyunki dono validation aur test sets "held out" hote hain. Key distinction ye hai ki tum unhe kab use karte ho. Validation development-time decisions ke liye hai. Test final reporting ke liye hai.

Sahi approach: Test error ko bilkul end tak kabhi mat dekho. Saari tuning ke liye validation use karo. Agar test error dekhne ke baad aur iterate karna hai, to ek naya test set collect karo.

Ye sahi kyun lagta hai: High accuracy achha lagtaa hai, aur ye wahi metric hai jise tumne optimize kiya.

The fix: Training accuracy memorization measure karta hai, generalization nahi. Ek model overfitting karke 100% training accuracy achieve kar sakta hai (jaise ek lookup table). Hamesha validation ya test performance report karo.

Example: Ek 10-degree polynomial 10 noisy points ko perfectly fit kar sakta hai ($E_{\text{train}}=0$) lekin points ke beech wildly oscillate karta hai ($E_{\text{val}}\gg 0$).

Steel-man: Beginners training error pe focus karte hain kyunki gradient descent directly usi ko minimize karta hai. "Iska matlab model achha hai" — ye leap natural hai lekin galat hai.

Ye sahi kyun lagta hai: Model pehle se train ho gaya hai — phir se kyun train karein?

The fix: Jo model tumne tune kiya woh sirf training subset pe train hua tha (validation exclude karke). Performance maximize karne ke liye, chosen hyperparameters ke saath final testing se pehle train+val combined pe retrain karo. Validation data sirf decisions lene ke liye hold out kiya tha, isliye nahi ki woh contaminated hai.

Kab skip karein: Agar training bahut expensive hai (GPU time ke kaafi din), to tum retraining skip kar sakte ho aur thodi suboptimal performance accept kar sakte ho. Lekin zyaatar cases mein, retrain karo.

Active Recall Questions

#flashcards/ai-ml

What are the three types of dataset splits in ML and their purposes? :: Training set (optimize model parameters), Validation set (tune hyperparameters and model selection), Test set (unbiased final evaluation).

1. Practice problems (training): Tum inhe baar baar solve karte ho, techniques seekhte ho. Apne answers check karte ho aur improve karte ho. Ye tumhare liye ab easy hain kyunki tumne inhe memorize kar liya hai.

2. Rehearsal problems (validation): Nayi problems jo tumne practice nahi ki, lekin tum inhe test karne ke liye use karte ho ki kaun si technique best hai. "Kya main diagram banaaun ya table?" Dono rehearsal problems pe try karo aur dekho kaun si zyada sahi milti hai. Best technique pick karo.

3. Competition problems (test): Actual competition mein bilkul nayi problems. Inhein tumne pehle kabhi nahi dekha. Yahaan tumhara score batata hai ki tum sachchi mein kitne achhe ho, sirf practice ki hui problems mein nahi.

Agar competition problems practice ke dauran dekh lete, to sochte ki tum zyada achhe ho jitne ho — ye cheating hai! Machine learning mein hum test set ko exactly isi wajah se end tak secret rakhte hain.

Ya socho: TV Test — tum ek TV appearance ke liye rehearse karte ho (training), dress rehearsal hoti hai (Validation), phir live broadcast (Test — ek shot, koi redo nahi).

Connections

• Bias-Variance Tradeoff: Training-validation gap variance (overfitting) se driven hai. High-bias models mein gap chhota hota hai lekin overall errors high hote hain.
• Overfitting and Regularization: Regularization techniques (L2, dropout, early stopping) validation error monitor karke choose ki jaati hain.
• Cross-Validation: Chhote datasets ke liye single train-val split ka alternative; zyada robust error estimates deta hai.
• Model Selection: Model architectures, hyperparameters, aur feature sets mein se choose karne ka criterion validation error hai.
• Learning Curves: Training aur validation error ke plots training set size ya epochs ke saath; underfitting vs overfitting diagnose karte hain.
• Generalization Error: Test error data distribution pe true generalization error ka humara best estimate hai.
• Data Leakage: Test data kisi bhi decision ke liye use karna (indirect bhi) estimates ko upward bias karta hai — practice mein ek critical mistake.

Concept Map