Worked examples — Train - validation - test splitting

Before anything, six plain-word anchors so no symbol is unearned:

---

The scenario matrix

Every splitting problem you will meet lives in one of these cells. The last column names the worked example that covers it.

Cell	Situation	What breaks if you're naive	Covered by
A	Balanced, medium $N$	nothing — the baseline	Example 1
B	Imbalanced classes	minority class vanishes from a split	Example 2
C	Rounding leftovers	floors don't add up to $N$	Example 3
D	Tiny $N$ (< 1000)	splits too small to trust	Example 4
E	Huge $N$ (> 100k)	wasting data on oversized test set	Example 5
F	Temporal / ordered data	future leaks into the past	Example 6
G	Leakage via scaling	test stats sneak into training	Example 7
H	Exam twist: two-step split ratios	second split size is not the final ratio	Example 8

Each example says which cell it hits. Together they fill the whole table.

---

Example 1 — Cell A: the balanced baseline

Forecast: guess the three counts per class before reading on. (Hint: symmetry should make spam and not-spam identical.)

1. Find the target sizes of each split. $n_{\text{train}}=0.70\times 1000=700,n_{\text{val}}=0.15\times 1000=150,n_{\text{test}}=0.15\times 1000=150$ Why this step? Fractions describe intent; models need whole counts. $700+150+150=1000$, clean.

2. Split within each class using proportions. Here $p_{\text{spam}}=0.5$. $n_{\text{spam}}^{\text{train}}=\lfloor 0.5\times 700\rfloor =350,n_{\text{spam}}^{\text{val}}=\lfloor 0.5\times 150\rfloor =75,n_{\text{spam}}^{\text{test}}=75$ Why this step? Stratifying = keeping $p_c$ identical in every bucket, so we allocate each class separately (see the parent's Step 2).

3. Not-spam is identical by symmetry: 350 / 75 / 75.

Verify: Spam total $350+75+75=500$ ✓. Not-spam $=500$ ✓. Grand total $700+150+150=1000$ ✓. Every split is exactly 50 % spam — stratification holds perfectly.

---

Example 2 — Cell B: severe imbalance

Forecast: with only 100 diseased images, how many make it to the 150-image test set? Guess before computing.

1. Split sizes: 700 / 150 / 150 (same as Example 1). Why? Split sizes depend only on $N$, not on class balance.

2. Allocate the minority class using $p_{\text{diseased}}=0.1$: $n_{\text{dis}}^{\text{train}}=\lfloor 0.1\times 700\rfloor =70,n_{\text{dis}}^{\text{val}}=\lfloor 0.1\times 150\rfloor =15,n_{\text{dis}}^{\text{test}}=15$ Why this step? Splitting within the 100 diseased rows guarantees each bucket gets its 10 % share — a plain shuffle could dump all 100 into training.

3. Majority class fills the rest: $700-70=630$, $150-15=135$, $150-15=135$ healthy.

The figure below stacks each bucket as a bar: cyan = healthy, amber = diseased. What to look for: the amber slice sits at exactly 10 % of every bar's height — that constant amber fraction across all three bars is the whole visual point of stratification. The absolute amber counts (70, 15, 15) shrink with the bucket, but the ratio never moves.

Verify: Diseased $70+15+15=100$ ✓, healthy $630+135+135=900$ ✓, total $1000$ ✓. Each split is exactly 10 % diseased.

---

Example 3 — Cell C: the rounding leftover

Forecast: $0.70+0.15+0.15=1.00$ exactly — so surely nothing is left over? Guess, then watch the floors bite.

1. Apply floors: $n_{\text{train}}=\lfloor 0.70\times 1003\rfloor =\lfloor 702.1\rfloor =702$ $n_{\text{val}}=\lfloor 0.15\times 1003\rfloor =\lfloor 150.45\rfloor =150,n_{\text{test}}=150$ Why this step? Each split independently rounds down, so each drops a fractional row.

2. Count the assigned rows: $702+150+150=1002$. One row is homeless. Why this step? Three downward roundings almost always lose 1–2 rows even when the fractions sum to 1.

3. Assign the remainder to the last split: $n_{\text{test}}\to 150+1=151$. $n_{\text{test}}\to 150+1=151$ Why this step? We must place every row somewhere, and the standard convention (what scikit-learn's slicing does) is that the last split — test, the tail of the array — absorbs leftovers, since it is "everything after the val cut."

Verify: $702+150+151=1003$ ✓. Ratios become $69.99\%/14.96\%/15.05\%$ — negligibly off, which is exactly why leftovers are harmless. The lesson: never assert your counts sum to $N$ without checking.

---

Example 4 — Cell D: tiny dataset

Forecast: how many samples land in the validation set — and would you trust a model comparison made on that many?

1. Split sizes: $n_{\text{val}}=\lfloor 0.15\times 120\rfloor =18$, $n_{\text{test}}=18$, $n_{\text{train}}=84$. Why? Same floor arithmetic.

2. Estimate the confidence-interval width for test accuracy. The parent's formula: for accuracy $\hat{a}$ on $n$ test samples, the 95 % interval half-width is $\Delta =1.96\sqrt{\frac{\hat{a}(1-\hat{a})}{n}}.$ Why this tool and not another? Accuracy is a mean of 0/1 outcomes; the standard error of a proportion is $\sqrt{\hat{a}(1-\hat{a})/n}$, and $1.96$ is the number of standard errors covering 95 % of a normal curve. It answers "how fuzzy is my reported accuracy?"

3. Plug in worst case $\hat{a}=0.5$, $n=18$: $\Delta =1.96\sqrt{\frac{0.25}{18}}=1.96\times 0.1179\approx \mathrm{0.231.}$ Why worst case? $\hat{a}(1-\hat{a})$ is largest at $0.5$, giving the widest (most pessimistic) interval.

Verify: $\Delta \approx \pm 23\%$. A test accuracy of "80 %" really means "somewhere between 57 % and 100 %" — useless. Conclusion: with $N<1000$, obey the parent's rule and use 2.1.13-Cross-validation instead of a single holdout.

---

Example 5 — Cell E: huge dataset

Forecast: how much narrower is the confidence interval for the giant split, and is the difference worth it?

1. Test sizes: $n_{\text{test}}^{15\%}=0.15\times 2000000=300000,n_{\text{test}}^{1\%}=20000.$ Why? We're comparing what fraction is "spent" on evaluation.

2. CI half-width (worst case $\hat{a}=0.5$) for each: ${\Delta }_{300k}=1.96\sqrt{\frac{0.25}{300000}}\approx 0.00179,{\Delta }_{20k}=1.96\sqrt{\frac{0.25}{20000}}\approx \mathrm{0.00693.}$ Why this step? To see whether shrinking the test set actually hurt precision.

3. Interpret: both are already tiny — $\pm 0.18\%$ vs $\pm 0.69\%$. Meanwhile training rows jump from $1.4$M to $1.96$M. Why this step? $\pm 0.7\%$ is plenty for a final estimate, so the extra $560000$ training rows are pure gain.

Verify: Training rescued $=0.98\times 2000000-0.70\times 2000000=560000$ rows ✓. Both intervals under $\pm 1\%$ ✓. Lesson: at huge $N$, absolute test size (not percentage) is what matters — 20k is enough.

---

Example 6 — Cell F: temporal data (ordering trap)

Forecast: which month should be the first row of the test set?

1. Diagnose the leak. Random shuffling scatters future months into training and past months into test. The model trains on Dec-Year2 to predict, say, Mar-Year1 — using the future to explain the past. Why this step? The whole point of a test set is to imitate deployment, where you only ever have the past. See 6.3.2-Data-leakage.

2. Cut chronologically, no shuffle. $\text{train}=\text{months }1-\lfloor 0.70\times 24\rfloor =1-16,$ $\text{val}=\text{months }17-(16+\lfloor 0.15\times 24\rfloor )=17-19,$ $\text{test}=\text{months }20-24.$ Why floors again? $0.15\times 24=3.6\to 3$ val months.

The figure below draws the 24 months as a left-to-right timeline, coloured by bucket: cyan = train, amber = val, white = test. What to look for: the three colours never interleave — every cyan bar is strictly to the left of every amber bar, which is strictly left of every white bar. That "no colour ever jumps backward" property is exactly what stops the future from leaking into the past; a random split would speckle the three colours all across the timeline.

3. Count: train 16, val 3, test $24-19=5$ months. Leftovers land in test (Cell C convention).

Verify: $16+3+5=24$ ✓. Test months (20–24) are strictly later than every training month (1–16) — no future information reaches training. The ratio drifted to 66.7 / 12.5 / 20.8 because 24 is small; that is acceptable for time series, where order matters more than exact percentages.

---

Example 7 — Cell G: scaling leakage (the classic bug)

Forecast: does fitting on all-data make the test point look more or less extreme?

1. Wrong way — fit on all 5 values. Mean ${\mu }_{\text{all}}=\frac{10+20+30+40+100}{5}=40.$ Why this step? This is exactly what scaler.fit(X_all) does — the test value 100 pulls the mean up.

2. Wrong-way std. Deviations from ${\mu }_{\text{all}}=40$ are $-30,-20,-10,0,+60$, so ${\sigma }_{\text{all}}=\sqrt{\frac{(-30{)}^2+(-20{)}^2+(-10{)}^2+0^2+60^2}{5}}=\sqrt{\frac{900+400+100+0+3600}{5}}=\sqrt{1000}\approx \mathrm{31.62.}$ Why this step? We need the leaky $\sigma$ to finish the leaky standardization and see how tame the outlier becomes.

3. Right way — fit on train only. Mean ${\mu }_{\text{train}}=\frac{10+20+30+40}{4}=25.$ Deviations from $25$ are $-15,-5,+5,+15$, so ${\sigma }_{\text{train}}=\sqrt{\frac{(-15{)}^2+(-5{)}^2+(5{)}^2+(15{)}^2}{4}}=\sqrt{\frac{225+25+25+225}{4}}=\sqrt{125}\approx \mathrm{11.18.}$ Why this tool? Standardization $z=(x-\mu )/\sigma$ (see 2.1.11-Feature-scaling-normalization) needs a $\mu ,\sigma$; using train-only ones keeps the test set genuinely unseen.

4. Standardize the test value 100 both ways with $z=(x-\mu )/\sigma$. $z_{\text{correct}}=\frac{100-25}{11.18}\approx 6.71,z_{\text{leaky}}=\frac{100-40}{31.62}\approx \mathrm{1.90.}$ Why this step? Putting the two $z$-scores side by side is the punchline: the leak literally changes how anomalous the point appears.

Verify: ${\mu }_{\text{all}}=40\ne 25={\mu }_{\text{train}}$, and ${\sigma }_{\text{all}}=\sqrt{1000}\ne \sqrt{125}={\sigma }_{\text{train}}$ ✓. $z_{\text{correct}}=(100-25)/\sqrt{125}\approx 6.71$; $z_{\text{leaky}}=(100-40)/\sqrt{1000}\approx 1.90$ ✓. Lesson: the correct pipeline flags 100 as a huge outlier ($z\approx 6.7$); the leaky one tames it to a mild $z\approx 1.9$, hiding the anomaly. Always fit_transform on train, transform on val/test.

---

Example 8 — Cell H: exam twist, two-step ratios

Forecast: the second split needs 70 : 10 — is test_size=0.10 correct? (It is a trap — guess before reading.)

1. First split: hold out 20 % test. This leaves X_temp with $80\%$ of $N$: $n_{\text{temp}}=0.80\times 5000=4000,n_{\text{test}}=0.20\times 5000=1000.$ Why this step? The test set must be carved off first and then never touched again, so all later splitting happens on the remaining 4000 rows.

2. Work out what "10 % of $N$" means as a fraction of X_temp. We want val to be $10\%$ of the original 5000 = 500 rows, but the second train_test_split only sees the 4000 temp rows, so: $\text{test\_size}=\frac{n_{\text{val}}}{n_{\text{temp}}}=\frac{0.10\times 5000}{4000}=\frac{500}{4000}=\mathrm{0.125.}$ Why this step? train_test_split's test_size is always relative to the array it receives. Passing $0.10$ would take 10 % of 4000 = 400 rows (only 8 % of $N$) — the trap. We rescale via $\frac{0.10}{0.80}=0.125$.

3. Convert the fractions back into row counts to confirm we land on 70 / 10 / 20: $n_{\text{val}}=0.125\times 4000=500,n_{\text{train}}=4000-500=3500,n_{\text{test}}=1000.$ Why this step? Turning the fraction back into counts is the final check that the $0.125$ trick actually produces the intended split, not just plausible-looking algebra.

Verify: $3500/5000=70\%$ ✓, $500/5000=10\%$ ✓, $1000/5000=20\%$ ✓. And $0.10/0.80=0.125$ ✓. The trap — passing test_size=0.10 — would give val $=400$ (8 %), silently wrong.

---

Recall Self-test

For $N=1003$, 70/15/15 with floors, how many test samples after leftovers? ::: 151 (702 + 150 + 150 = 1002, remaining 1 goes to test) Two-step split for final 70/10/20, second test_size? ::: 0.125, i.e. $0.10/0.80$ Correct standardized value of test point 100 given train $\{10,20,30,40\}$? ::: $\approx 6.71$ using $\mu =25,\sigma =\sqrt{125}$ Leaky standardized value of the same point (fit on all data)? ::: $\approx 1.90$ using $\mu =40,\sigma =\sqrt{1000}$ — the outlier gets hidden Why is $N=120$ with an 18-sample test set untrustworthy? ::: 95% CI half-width $\approx \pm 23\%$ — too wide; use cross-validation

Related: 3.2.1-Overfitting-underfitting (why the val set exists), 4.1.5-Early-stopping (a val-set consumer), 2.1.13-Cross-validation (the small-$N$ fix).