Visual walkthrough — Train - validation - test splitting

---

Step 1 — What is "the data", and what do we actually want?

WHAT. Picture a bag of dots. Each dot is one example the model can learn from — one customer, one photo, one house. Call the number of dots $N$. That's it: $N$ is just how many dots are in the bag.

WHY. Before we split anything, we must be honest about the real goal. We do not care how well the model does on dots we already have. We care about new dots we have never seen — tomorrow's customer, next week's photo. Call the model's true skill on brand-new dots $P_{\text{unseen}}$. The symbol $P$ means "performance" (say, accuracy); the subscript unseen means "measured on dots outside the bag".

PICTURE. The bag holds all $N$ dots we have. The cloud floating above holds the infinite dots we will never see but must succeed on. The whole game is: estimate the cloud using only the bag.

---

Step 2 — The naive idea: learn on everything, grade on everything

WHAT. The simplest thing: let the model learn from all $N$ dots, then measure its score on those same $N$ dots. Call that score $P_{\text{train}}$ — performance measured on the training dots.

WHY. It sounds efficient — use every dot twice! But look at the picture: the model is graded on the exact dots it studied. A model that just memorizes every dot scores a perfect $P_{\text{train}}$ while learning nothing general. This is overfitting — great on the bag, useless on the cloud.

PICTURE. The blue arrow (learn) and the yellow arrow (grade) both point at the same dots. When study-set and exam-set overlap, the grade is a lie about the cloud.

---

Step 3 — The first honest cut: hold some dots back

WHAT. Split the bag into two piles. A big pile of $n_{\text{train}}$ dots the model may study, and a hidden pile of $n_{\text{test}}$ dots it never sees during learning. Their sizes add up: $n_{\text{train}}+n_{\text{test}}=N$.

WHY. Grade on the hidden pile. Because those dots were never studied, scoring them mimics the cloud — a memorizing model now fails them, exactly as it would fail future data. This score, $P_{\text{test}}$, is our first honest estimate of $P_{\text{unseen}}$. This two-pile idea is the holdout method.

PICTURE. One wall now divides the bag. The blue (learn) arrow only touches the left pile; the yellow (grade) arrow only touches the right, walled-off pile. No overlap → the grade is trustworthy.

Each symbol, right where it lives:

$\underset{\text{dots to learn from}}{\underbrace{n_{\text{train}}}}+\underset{\text{dots hidden for the exam}}{\underbrace{n_{\text{test}}}}=\underset{\text{all dots we own}}{\underbrace{N}}$

---

Step 4 — Why two piles secretly break: the model-picking trap

WHAT. In practice we don't build one model. We build many — different depths, different learning rates. These knobs we choose by hand are hyperparameters. We build, say, 100 candidate models and keep the one with the best $P_{\text{test}}$.

WHY. Here's the trap. Every time we peek at the test score to choose a model, that choice is influenced by the test pile. After 100 peeks and one "pick the best", the test pile has quietly shaped our final model — we have implicitly trained on it through selection. The best-of-100 test score is now optimistically biased, just like Step 2 in disguise. This is a form of data leakage.

PICTURE. Many models fire their arrows at the test wall. We keep whichever arrow scored highest. The dashed "selection" arrow feeding back means test information leaked into the choice — the wall has a crack.

---

Step 5 — Fix the crack: a THIRD pile

WHAT. Cut the study pile again. Now three piles: $n_{\text{train}}$ to learn parameters, $n_{\text{val}}$ to compare and choose models, and $n_{\text{test}}$ kept sealed until the very end. They sum to everything:

$n_{\text{train}}+n_{\text{val}}+n_{\text{test}}=N$

WHY. Do all the peeking, comparing, and knob-turning against the validation pile ($P_{\text{val}}$). The test pile stays sealed — touched exactly once, at the finish. Because we never selected anything on it, $P_{\text{test}}$ is finally an unbiased estimate of the cloud $P_{\text{unseen}}$. The validation pile absorbs all the selection bias so the test pile can stay pure. (The early-stopping decision "when to stop training" is also a validation-set job.)

PICTURE. Two walls, three rooms. Blue arrow → left room (learn). Yellow arrow → middle room (choose). The right room stays behind a padlock, opened once.

---

Step 6 — How big should each pile be? The trade-off, drawn

WHAT. We choose fractions that add to one. Writing them as a ratio: $\text{Train}:\text{Val}:\text{Test}$, e.g. $70:15:15$ or $80:10:10$.

WHY. Every dot placed in val or test is a dot stolen from training. Bigger train → better parameters. Bigger val → less noisy model choice. Bigger test → a tighter, more confident final number. You cannot maximise all three; they share the same $N$ dots. This is the same "spend your data wisely" tension behind cross-validation, which recycles dots when $N$ is small.

PICTURE. A sliding bar of length $N$. As you drag the walls, one region grows only by shrinking another. The three coloured widths must always sum to the full bar.

How confident is the test number? With $n_{\text{test}}$ test dots and an accuracy near $50\%$ (the hardest case to pin down), the $95\%$ confidence half-width is

$\pm \underset{\text{95\% z-value}}{\underbrace{1.96}}\times \sqrt{\frac{\stackrel{p}{\overbrace{0.5}}\times \stackrel{1-p}{\overbrace{0.5}}}{\underset{\text{test dots}}{\underbrace{n_{\text{test}}}}}}$

• $1.96$ — the multiplier that captures the middle $95\%$ of a bell curve.
• $0.5\times 0.5$ — the worst-case spread of a yes/no accuracy (biggest when the model is a coin flip).
• $n_{\text{test}}$ — more test dots → smaller fraction → tighter interval.

Plug in $n_{\text{test}}=10000$: the half-width is about $\pm 0.0098\approx \pm 1\%$. That's why once $N$ is huge you can afford tiny test fractions — $5\%$ of a million is still 50000 dots, plenty.

---

Step 7 — The degenerate cases you must never hit

WHAT. Three ways the split silently breaks, each with its own fix.

WHY & PICTURE. These are the corners the parent note warned about — here they are visually, side by side.

1. Ordered data, sliced without shuffling. If the bag is sorted (all class-A first, then class-B, or oldest-to-newest), a straight cut hands one class entirely to one pile. Fix: shuffle first.
2. Imbalanced classes, shuffled but not stratified. With $90\%$ healthy, $10\%$ sick, a plain shuffle can leave a pile with almost no sick dots — the proportions wobble. Fix: stratify — split each class separately so every pile carries the same $10\%$ by construction (this is exact, not "on average").
3. Tiny $N$. If $N<1000$, any single test pile is too small to trust ($\pm$ interval is huge). Fix: use cross-validation instead of one fixed split.

Stratified counts, symbol by symbol — for a class with proportion $p_c$:

$n_c^{\text{train}}=\lfloor \underset{\text{class share}}{\underbrace{p_c}}\times \underset{\text{train pile size}}{\underbrace{n_{\text{train}}}}\rfloor$

The $\lfloor \rfloor$ (floor) just rounds down to a whole number of dots — you can't have half a dot.

---

Step 8 — The one rule that keeps the walls solid: fit on train only

WHAT. Any preprocessing that learns numbers from data — like the mean $\mu$ and spread $\sigma$ used in feature scaling — must compute those numbers from the training pile alone, then apply them to val and test.

WHY. If you compute $\mu ,\sigma$ over the whole bag, the test dots have already whispered their values into the scaling. The model then "feels" the test data indirectly — a leak straight through the wall you built in Step 5. This is exactly data leakage.

PICTURE. The $\mu ,\sigma$ measuring cup dips only into the train room. It then pours the same $\mu ,\sigma$ onto the val and test rooms — never scooping from them.

---

The one-picture summary

Everything above, compressed into a single flow: one bag → shuffle & stratify → three walled rooms → fit stats on train → learn / choose / final-exam → an honest estimate of the cloud.

Recall Feynman retelling — say it like a story

We own a bag of dots but truly care about dots we'll never see. If we grade the model on the same dots it studied, it can cheat by memorizing — the score lies. So we hide some dots and grade on those: honest, until we start choosing between many models by peeking at the hidden pile — then that pile gets used up and lies again. Cure: a middle pile for all the choosing, and a final pile locked away, opened exactly once. Split sizes are a tug-of-war over the same dots — more test means a tighter final number but less to learn from. Watch the corners: shuffle so sorted data doesn't dump one class into one pile; stratify so rare classes appear in every pile exactly, not just on average; and if the bag is tiny, switch to cross-validation. Finally, whatever you measure for scaling — the mean and spread — measure it from the train pile only, or the test dots leak through the wall you worked so hard to build.

Recall Quick self-check

Why three piles, not two? ::: The validation pile absorbs all the model-selection bias so the test pile stays untouched and gives an unbiased final estimate. Why compute $\mu ,\sigma$ on train only? ::: Using test values leaks test information into training through the scaling — a data leak. Difference between shuffle and stratify? ::: Shuffle balances piles on average; stratify splits each class separately so proportions are exact. When do you drop the fixed split entirely? ::: When $N$ is small (roughly $N<1000$) — use cross-validation.

---

Parent: 2.1.12 Train - validation - test splitting (Hinglish)