Exercises — Train - validation - test splitting

Compare on the validation set. The test set stays sealed until the single final measurement. Why: picking the better of two models is a decision made from data. Any set used to make a decision has been "trained on" in a soft sense, so it can no longer give an honest estimate of unseen performance. The test set is the only set that has influenced zero decisions — that is what makes its number trustworthy.

(a) Train — weights are parameters, learned by the fitting procedure. (b) Validation — the learning rate is a hyperparameter, chosen by comparing runs. (c) Test — the last, untouched measurement.

$0.80\times 5000=4000$ train, $0.10\times 5000=500$ val, $0.10\times 5000=500$ test. Check the total: $4000+500+500=5000$. ✓ Why check the sum? Splits must be mutually exclusive and exhaustive — every sample belongs to exactly one set, none is lost or duplicated.

$0.70\times 5000=3500$, $0.15\times 5000=750$, $0.15\times 5000=750$. Sum: $3500+750+750=5000$. ✓

After removing $20\%$ test, the remaining pool is $0.80\times 5000=4000$. We want the val set to be $10\%$ of the original $=0.10\times 5000=500$ samples. That is a fraction $\frac{500}{4000}=0.125=12.5\%$ of the remaining pool. Why the number changes: $10\%$ of the whole is a bigger slice of the smaller leftover — the denominator shrank from $5000$ to $4000$, so the same 500 samples take up a larger share. This is why the scikit-learn code passes test_size=0.125 on the second call, not 0.10.

Stratification splits each class separately so proportions are exact by construction. Diseased count $=0.10\times 1000=100$.

• Train: $0.70\times 100=70$
• Val: $0.15\times 100=15$
• Test: $0.15\times 100=15$ Check: $70+15+15=100$. ✓ Each split is $10\%$ diseased, matching the whole. See the figure — each colour band is cut in the same ratio.

$1.96\sqrt{\frac{0.5\times 0.5}{2500}}=1.96\sqrt{\frac{0.25}{2500}}=1.96\times 0.01=0.0196\approx \pm 1.96\%.$ Why $p=0.5$ is "worst case": the product $p(1-p)$ is largest at $p=0.5$ (a coin flip is the noisiest possible outcome), so this gives the widest interval — a safe upper bound on your uncertainty.

Set $1.96\sqrt{\frac{0.25}{n}}=0.005$ and solve. $\sqrt{\frac{0.25}{n}}=\frac{0.005}{1.96},\frac{0.25}{n}={\left(\frac{0.005}{1.96}\right)}^2,n=\frac{0.25}{(0.005/1.96{)}^2}.$ $n=\frac{0.25\times 1.96^2}{0.005^2}=\frac{0.25\times 3.8416}{0.000025}=38416$. So you need about $\approx 38,416$ test samples. Why so many? Halving the width from $\pm 1\%$ to $\pm 0.5\%$ requires 4× the data, because error falls as $1/\sqrt{n}$: to divide error by 2 you multiply $n$ by $2^2=4$. This is exactly why the parent note stops adding test data past a point — the returns shrink brutally.

Flaw: the scaler's mean $\mu$ and standard deviation $\sigma$ were computed from all rows — including test rows. The training pipeline has now indirectly seen the test set through those statistics. This is data leakage (6.3.2-Data-leakage), and it inflates the test score. Corrected:

scaler.fit(X_train)               # statistics from TRAIN only
X_train_s = scaler.transform(X_train)
X_test_s  = scaler.transform(X_test)   # apply train's μ, σ

Related: 2.1.11-Feature-scaling-normalization explains what $\mu$ and $\sigma$ do; here the point is when to compute them.

Positives total $=0.12\times 5000=600$. Validation is $10\%$ of the data, so validation positives $=0.10\times 600=60$. Validation size $=0.10\times 5000=500$, with $60$ positives. Enough? For comparing a handful of models, yes — you only need to distinguish which is better, and $500$ samples give a val-accuracy half-width around $1.96\sqrt{0.25/500}\approx \pm 4.4\%$ at worst case. That resolves models that differ by more than a few percent. But with only $60$ positives, any metric that depends on the minority class (like recall) is noisy — treat close calls with suspicion and consider 2.1.13-Cross-validation for small pools.

Val and test are each $10\%$: $0.10\times 300=30$ samples each, containing $0.10\times 15=1.5\to 1$ positive each (after floor). So validation and test each hold one positive example. Measuring recall on one positive means your metric can only be $0\%$ or $100\%$ — it carries almost no information, and its confidence interval is enormous. Better strategy: for $N<1000$ use ==$k$-fold cross-validation== (2.1.13-Cross-validation) with stratified folds, so every example gets to be tested exactly once and the minority class is spread across folds. The parent's rule of thumb ("$N<1000$ → cross-validation") exists precisely to avoid single-digit test classes.

Mechanism: with 100 variants, the best validation score is partly luck — you selected the model that happened to fit the validation set's noise best (this is validation overfitting, a cousin of 3.2.1-Overfitting-underfitting). The maximum of many noisy numbers is biased upward. The test set, untouched by that selection, reveals the truth. Honest estimate: $0.89$. The test set is the only number that had zero influence on which model you chose. $0.94$ is optimistically biased by the selection process — exactly the information-leakage-through-model-selection argument the parent note derives. Connected idea: if you also use validation to decide when to stop training (4.1.5-Early-stopping), the same caution applies — the stopping point is chosen from val, so val is not a clean generalization estimate either.

$1.96\sqrt{\frac{0.5\times 0.5}{10000}}=1.96\sqrt{\frac{0.25}{10000}}=1.96\times 0.005=0.0098\approx \pm 0.98\%\approx \pm 1\%.✓$ This is why past $\sim 100$k samples you can drop the test fraction to $5\%$ or less — $10$k test rows already pin accuracy to within a percent.

The set used to choose hyperparameters ::: validation set The set that may influence zero decisions until the end ::: test set To halve a CI half-width you multiply $n$ by ::: 4 Preprocessing statistics ($\mu ,\sigma$) must be computed from ::: the training set only For $N<1000$, prefer this over a fixed split ::: cross-validation