Exercises — Benchmark design and evaluation rigor

The number is a point estimate: it hides the uncertainty (spread) of performance across seeds/splits — we can't tell signal from noise. Picking the "best" run is selection bias (cherry-picking the right tail of the distribution), which over-estimates true capability.

$\text{SEM}=\frac{s}{\sqrt{n}}=\frac{2.0}{\sqrt{4}}=\frac{2.0}{2}=1.0$ With $n=16$: $\text{SEM}=2.0/\sqrt{16}=2.0/4=0.5$. Because SEM scales as $1/\sqrt{n}$, quadrupling runs ($4\to 16$) halves the uncertainty. This is why "4× more runs to halve error" is the rule of thumb.

This is a negative control. If the label is now random noise, no real reasoning can help, so a properly-behaving model should collapse to chance = $1/4=25\%$. Scoring well above chance on a scrambled task is a red flag for spurious correlations (surface cues leaking the answer).

First the SEM: $\text{SEM}=s/\sqrt{n}=1.5/\sqrt{5}=1.5/2.2360679...=\mathrm{0.6708...}$ ${\text{CI}}_{95}=\bar{P}\pm t_{0.975,4}\cdot \text{SEM}=91.8\pm 2.776\times 0.6708=91.8\pm 1.862$ So ${\text{CI}}_{95}=[89.94,93.66]$ (percent). Why $t$, not $1.96$? With only $5$ runs the central-limit "$z=1.96$" shortcut under-covers; Student's $t$ with $n-1=4$ degrees of freedom is fatter-tailed and honestly wider.

The pooled standard deviation: $s_{\text{pool}}=\sqrt{\frac{(n_A-1)s_A^2+(n_B-1)s_B^2}{n_A+n_B-2}}=\sqrt{\frac{9(4)+9(4)}{18}}=\sqrt{4}=2.0$ $d=\frac{{\bar{P}}_A-{\bar{P}}_B}{s_{\text{pool}}}=\frac{90-88}{2.0}=1.0$ Since $|d|=1.0\ge 0.8$, this is a large effect. Note: effect size measures "how many std-devs apart", and is independent of $n$ — that's what distinguishes it from a $p$-value.

(a) Pairs of 20: $\left(\genfrac{}{}{0ex}{}{20}{2}\right)=\frac{20\times 19}{2}=190$ comparisons. (b) Bonferroni: $\alpha /m=0.05/190=\mathrm{0.000263...}$ per test. (c) Ignoring it, expected false positives $=m\alpha =190\times 0.05=9.5\approx 10$. So you'd "discover" ~10 significant differences that are pure chance — hence the correction.

The standard error of the difference combines both noises: ${\text{SE}}_{\text{diff}}=\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}=\sqrt{\frac{0.25}{5}+\frac{0.49}{5}}=\sqrt{0.05+0.098}=\sqrt{0.148}=0.3847$ $t=\frac{{\bar{P}}_A-{\bar{P}}_B}{{\text{SE}}_{\text{diff}}}=\frac{0.5}{0.3847}=1.30$ Now the degrees of freedom — don't hand-wave "about 8". The Welch–Satterthwaite formula (from the parent note) tells us how many "effective independent measurements" the noisy variances are worth: $\nu =\frac{{\left(\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}\right)}^2}{\frac{(s_A^2/n_A{)}^2}{n_A-1}+\frac{(s_B^2/n_B{)}^2}{n_B-1}}$ Plug in $s_A^2/n_A=0.05$ and $s_B^2/n_B=0.098$: $\nu =\frac{(0.05+0.098{)}^2}{\frac{0.05^2}{4}+\frac{0.098^2}{4}}=\frac{0.148^2}{\frac{0.0025+0.009604}{4}}=\frac{0.021904}{0.003026}=7.24$ So $\nu \approx 7.24$ — round down to $7$ df to stay conservative. That is why it lands near 8: five runs each would give $8$ df only if the variances were equal; unequal variances shrink the effective count slightly. The two-tailed critical value at $\alpha =0.05$, $\nu =7$ is $t_{\text{crit}}\approx 2.36$. Since $|t|=1.30<2.36=t_{\text{crit}}$, we fail to reject $H_0$. Conclusion: not convincing. The $0.5\%$ lead is inside the noise. (The parent note used per-run SEMs of 0.5 and 0.7 giving $t\approx 0.58$; here we use per-run std with $n=5$, so the signal-to-noise is larger but still short of significance — either way, don't celebrate.)

"Best of 20" is the maximum order statistic — the largest of 20 draws sits deep in the right tail, so A's reported number is systematically inflated. Where does the inflation number come from? For $m$ independent standard-normal draws, the expected maximum grows roughly like $\mathbb{E}[\max ]\approx \sqrt{2\ln m}$ (a standard extreme-value approximation). For $m=20$: $\sqrt{2\ln 20}=\sqrt{2\times 2.9957}=\sqrt{5.9915}=2.448$ standard deviations above the mean in the crude form; the sharper Blom estimate ${\Phi }^{-1}\left(\frac{m-0.375}{m+0.25}\right)$ gives about $1.87$ standard deviations, which is the value we quote. Either way it is positive and large. In real units: $\mathbb{E}[\text{best of }20]\approx \bar{P}+1.87s=90+1.87\times 2=90+3.74=\mathrm{93.74.}$ B's honest mean sits at $\approx 90$. They are the same model, yet A "wins" by ~3.7 points purely from selection bias. Fix: compare means (with CIs), never a cherry-picked max against an honest mean. This is a distribution-level illusion, not a real capability gap.

The test set is deliberately out-of-distribution relative to training. It probes generalization / systematic reasoning, not memorization. A model that fit the training distribution tightly (low bias on train, effectively high variance w.r.t. the true task) has learned surface statistics of "1–2 object" scenes; on 3–5 objects those statistics no longer hold, so accuracy drops. This mirrors overfitting: great on seen data, poor under shift — exactly why 2.4.02-Regularization-techniques and held-out distribution shift matter for honest evaluation.

(a) Split by generator parameter, not by random row. Random row-splitting leaks: near-identical templates land in both train and test, so the model "remembers" rather than reasons. Instead hold out entire ranges of a controlled parameter — e.g. train on object-count $\in \{1,2\}$, test on $\{3,4,5\}$; and hold out specific material combinations entirely. (b) This forces the test to measure compositional generalization: the model must apply the physical rule to configurations it literally never saw. Add a negative control subset (scrambled answers) to confirm the un-leaked test still yields chance-level for a non-reasoner. This is the "programmatic generation → provable required reasoning" idea from Example 3 in the parent note.

We want $\text{SEM}=s/\sqrt{n}\le 0.5$. Solve for $n$: $\frac{2.0}{\sqrt{n}}\le 0.5\Rightarrow \sqrt{n}\ge \frac{2.0}{0.5}=4\Rightarrow n\ge 16.$ So $n=16$ runs minimum. Notice the price of precision: to get another halving to $0.25\%$ you'd need $n=64$ — quadratic cost, the $1/\sqrt{n}$ tax again.

Recall from the glossary: $H_0$ is "no true difference" (${\mu }_A-{\mu }_B=0$), and $t_{\text{crit}}$ is the two-tailed threshold at $\alpha =0.05$ that $|t|$ must exceed to reject $H_0$. (a) Means. ${\bar{P}}_A=(92.5+91.9+92.8+92.1+92.7)/5=462.0/5=92.40$ ${\bar{P}}_B=(91.4+92.0+91.1+91.8+91.7)/5=458.0/5=91.60$ (b) Sample stds (divide by $n-1=4$). $s_A=\sqrt{\frac{1}{4}\sum (P_i-92.40{)}^2}$. Deviations: $0.1,-0.5,0.4,-0.3,0.3$; squares sum $=0.01+0.25+0.16+0.09+0.09=0.60$; $s_A^2=0.15$, $s_A=0.3873$. $s_B$: deviations $-0.2,0.4,-0.5,0.2,0.1$; squares $=0.04+0.16+0.25+0.04+0.01=0.50$; $s_B^2=0.125$, $s_B=0.3536$. (c) Welch $t$. ${\text{SE}}_{\text{diff}}=\sqrt{\frac{0.15}{5}+\frac{0.125}{5}}=\sqrt{0.03+0.025}=\sqrt{0.055}=0.2345$ $t=\frac{92.40-91.60}{0.2345}=\frac{0.80}{0.2345}=3.41$ Welch–Satterthwaite df: $\nu =\frac{0.055^2}{\frac{0.03^2}{4}+\frac{0.025^2}{4}}=\frac{0.003025}{0.0003906}=7.74\approx 7$–$8$, so the two-tailed $t_{\text{crit}}$ at $\alpha =0.05$ is about $2.31$–$2.36$. (d) $|t|=3.41>2.31=t_{\text{crit}}$, so we reject $H_0$: the difference is statistically significant here (unlike the earlier $0.5\%$ example, this $0.8\%$ gap has tight, consistent runs). Effect size for context: $s_{\text{pool}}=\sqrt{(0.60+0.50)/8}=\sqrt{0.1375}=0.3708$, so $d=0.80/0.3708=2.16$ — large. Claim supported statistically; also flag it should still pass a negative control before "SOTA" is earned.

No — not in any useful sense. With enormous $n$, the SEM shrinks so far that any microscopic difference becomes "significant". This is statistical significance ≠ practical significance. Report the effect size and a practically meaningful threshold: a $0.05\%$ gain is almost certainly below the noise floor of deployment (data drift, prompt variation). The honest write-up: "statistically distinguishable but practically negligible ($d\approx 0$)." Significance tests guard against fooling yourself with noise; they cannot tell you whether a real-but-tiny gap matters. The figure below makes this visible.

What does SEM measure, and how does it scale with $n$? ::: The uncertainty of the mean estimate; it scales as $1/\sqrt{n}$ (4× runs → half the error). Why use Student's $t$ instead of $1.96$ for small $n$? ::: For few runs the normal approximation under-covers; $t$ with $n-1$ df has fatter tails and gives an honestly wider interval. What is $H_0$ when comparing two models? ::: The null hypothesis that the two true means are equal, ${\mu }_A-{\mu }_B=0$ — we only claim a winner if the data make it implausible. What is $t_{\text{crit}}$ and what three things set it? ::: The threshold $|t|$ must exceed to reject $H_0$; it depends on $\alpha$, the degrees of freedom, and one- vs. two-tailed choice. Two-tailed vs. one-tailed — which is the safe default and why? ::: Two-tailed, because you rarely pre-commit to the winner's direction; one-tailed is only honest if the direction was fixed before seeing data. What is the danger of reporting "best of $k$ runs"? ::: Selection bias — the max order statistic sits in the right tail, inflating the reported score above the true mean. Why Bonferroni-correct with many comparisons? ::: Each test has a false-positive chance; $m$ tests at $\alpha$ give $\approx m\alpha$ false positives, so use $\alpha /m$ to control family-wise error. Significance vs. practical significance? ::: Significance = "unlikely to be noise given $n$"; practical = "large enough to matter." Big $n$ can make trivial gaps significant.