Benchmark design and evaluation rigor: Visual walkthrough

This page rebuilds the parent result — Benchmark design and evaluation rigor — from the ground up, entirely in pictures. The central claim we are chasing:

When a paper says "Model A scores 92.3%", that single number is one draw from a cloud of possible scores. Rigor is the art of measuring that cloud, not the dot inside it.

We build every symbol from zero. A smart 12-year-old who has never seen a mean, a square root, or the letter $σ$ can follow from line one.

Step 1 — A score is a dart throw, not a fixed point

WHAT. Suppose you train the same model design five times. You change nothing you care about — same data, same architecture. But each run uses a fresh random seed: the coin-flips that set the starting weights, the order the data is shuffled in, which neurons dropout switches off. Result: five different accuracy numbers, e.g. $91.2%, 92.4%, 90.8%, 92.9%, 91.7%$ .

WHY. If one training run could land anywhere in a spread, then reporting a single run is like throwing one dart and calling that spot "the bullseye." We need to see the whole dartboard first.

PICTURE. Each run is a dot on the accuracy axis. They scatter. There is no single true score you can observe — only samples of it.

Figure — Benchmark design and evaluation rigor

Step 2 — The mean: where the cloud sits

WHAT. Add all the dots up and divide by how many there are. That balance point is the sample mean, written $\overset{ˉ}{P}$ (read "P-bar" — the bar means "averaged").

$\overset{ˉ}{P} = share out over n runs \frac{1}{n} stack all run scores together i = 1 \sum n P_{i}$

$n$ = how many runs (here $n = 5$ ).
$\sum_{i = 1}^{n} P_{i}$ is shorthand for " $P_{1} + P_{2} + \dots + P_{n}$ ." The tall $\sum$ (Greek capital sigma, chosen because it starts the word sum) just says add these up, sweeping $i$ from $1$ to $n$ .
Multiplying the total by $\frac{1}{n}$ = dividing by $n$ = fair-sharing the total equally.

WHY. We want one representative number, but chosen honestly — the centre of the cloud, not the luckiest dot on its edge. Picking the best run instead is selection bias: you'd be reading the right-hand tail and calling it typical.

PICTURE. The mean is the fulcrum where the see-saw of dots balances.

Step 3 — The spread: how wide is the cloud?

WHAT. Knowing the centre is not enough — a cloud can be a tight pinprick or a wide smear. We measure the width with the sample standard deviation $s_{P}$ .

$s_{P} = average, with a "-1" honesty tax \frac{1}{n - 1} squared distance of each dot from centre i = 1 \sum n (P_{i} - \overset{ˉ}{P})^{2}$

Read it inside-out:

$P_{i} - \overset{ˉ}{P}$ = how far run $i$ landed from the centre (a signed gap, left or right).
$(\cdot)^{2}$ = square it. Why square? To make every gap positive (so left and right gaps don't cancel), and to punish far-away dots more than near ones.
$\sum \dots$ = add all those squared gaps.
$\frac{1}{n - 1}$ = average them. We divide by $n - 1$ , not $n$ , because $\overset{ˉ}{P}$ was itself estimated from the same dots — that "steals" one degree of freedom, so we compensate. This is the honesty tax.
= square-root at the end, to undo the squaring and return to the original units (percent, not percent-squared).

WHY. Two models can share the same average and be wildly different: one reliable, one a coin-flip. $s_{P}$ is the number that separates them.

PICTURE. Two clouds, same fulcrum, different width — the tight one is trustworthy, the wide one is a gamble.

Step 4 — Where is the true centre? The standard error

WHAT. $\overset{ˉ}{P}$ is our guess for the model's true average capability. But $\overset{ˉ}{P}$ is itself a random thing — run five more times and you'd get a slightly different $\overset{ˉ}{P}$ . How jittery is the guess? That jitter is the standard error of the mean:

$SEM = \frac{s _{P}}{n}$

$s_{P}$ = width of the raw cloud (Step 3).
$n$ in the denominator = the payoff for doing more runs. More runs → averages steady out → the guess jitters less.

WHY this $\frac{1}{n}$ and not $\frac{1}{n}$ ? Because errors partly cancel when you average, but only in a square-root way — random pushes left and right don't fully undo each other. The consequence, drawn below: to halve your uncertainty you need four times as many runs ( $4 = 2$ ). Cheap precision does not exist.

PICTURE. The cloud of individual runs (wide, $s_{P}$ ) versus the tighter cloud of the mean itself (narrow, SEM), shrinking as $n$ grows.

Step 5 — Draw the honest claim: the confidence interval

WHAT. Instead of a dot, report a bar around $\overset{ˉ}{P}$ that plausibly contains the true centre. For $n \geq 30$ runs:

$CI_{95} = \overset{ˉ}{P} \pm "how many SEMs wide" 1.96 \cdot SEM from Step 4 \frac{s _{P}}{n}$

$\pm$ = "plus and minus" — the bar reaches equally left and right.
$1.96$ = the magic width that captures the central $95%$ of a bell-shaped cloud. (Change it and you change your confidence level.)

Intuition The normality assumption — the floor you're standing on

Picture the whole calculation as a person standing on a wooden floor. That $1.96$ is one plank of the floor — it only holds your weight if the cloud of means is Normal (a symmetric bell curve). If the floor is missing, you fall through: the number lies to you. Two facts keep the plank in place:

The Central Limit Theorem (CLT): the average of many runs tends toward a Normal shape even when a single run's scores are not Normal — provided the runs are independent and there are enough of them. " $n \geq 30$ " is a rough rule of thumb for when the CLT has kicked in and $s_{P}$ is a stable enough estimate of the true spread to treat $1.96$ as reliable.
For small $n$ the CLT hasn't fully smoothed things and $s_{P}$ is itself shaky, so the Normal bell is too thin in the tails. We must widen it — that is precisely why the t-distribution below exists. If the raw scores are wildly non-Normal (heavy tails, hard floors/ceilings like accuracy pinned near 100%), neither approximation is safe and you should prefer distribution-free tools (e.g. bootstrap resampling).

For small $n$ (our case, $n = 5$ ), the cloud has fatter tails, so $1.96$ is too optimistic. We swap in a bigger number from Student's t-distribution with $n - 1$ degrees of freedom (df).

$CI_{95} = \overset{ˉ}{P} \pm t_{0.975, n - 1} \cdot \frac{s _{P}}{n}$

For $n = 5$ (so df $= 4$ ), $t_{0.975, 4} = 2.776$ — noticeably wider than $1.96$ , exactly because five dots tell you less than thirty.

WHY. This bar is the honest deliverable. "The true score lies in $[89.86, 93.74]$ with 95% confidence" is a claim that survives someone else re-running you.

PICTURE. The point estimate becomes a whiskered bar; the fewer the runs, the longer the whiskers.

Step 6 — Comparing two models: do the clouds overlap?

WHAT. To ask "is A really better than B?", overlay their two clouds and measure the gap between centres in units of the combined jitter. That ratio is the t-statistic:

$t = \frac{P ˉ _{A} - P ˉ _{B} signal: gap between centres}{noise: combined SEM of the two \frac{s _{A}^{2}}{n _{A}} + \frac{s _{B}^{2}}{n _{B}}}$

Before reading the formula, meet its symbols — each is just a Step-2/Step-3 quantity computed separately for each model:

Now the formula reads plainly:

Numerator $\overset{ˉ}{P}_{A} - \overset{ˉ}{P}_{B}$ = how far apart the two means sit (the effect you're excited about).
Denominator = how much both means jitter, added in quadrature (squares add, then square-root — the same "errors combine in a square-root way" rule from Step 4). Each model contributes its own SEM-squared, $s_{A}^{2} / n_{A}$ and $s_{B}^{2} / n_{B}$ .
Big $∣ t ∣$ → gap dwarfs the jitter → real difference. Small $∣ t ∣$ → the clouds smear into each other → could be pure luck.

This t-statistic is compared against the same Student-t distribution from Step 5, so it too leans on the near-normality of the two clouds — safe-ish by the CLT when each has enough independent runs, shakier for very small or very skewed samples.

How many degrees of freedom? With two samples of possibly different spreads we can't just write $n_{A} + n_{B} - 2$ . Welch's method uses the Welch–Satterthwaite approximation:

$ν = \frac{( \frac{s _{A}^{2}}{n _{A}} + \frac{s _{B}^{2}}{n _{B}} ) ^{2}}{\frac{( s _{A}^{2} / n _{A} ) ^{2}}{n _{A} - 1} + \frac{( s _{B}^{2} / n _{B} ) ^{2}}{n _{B} - 1}}$

The top squares the total noise term; the bottom shares the blame between the two samples, each penalised by its own df.
Why bother? Because Model A might be far more seed-sensitive than Model B (unequal variances). Welch does not assume equal spreads, so it stays honest. For our example numbers this formula yields $ν \approx 7$ .

WHY. Eyeballing "92.3 vs 91.8, so A wins" ignores the width. The t-statistic forces you to divide the signal by the noise before believing anything.

PICTURE. Two overlapping bell clouds; $t$ is literally "how many noise-widths separate their peaks."

Step 7 — The degenerate cases (never leave the reader stranded)

WHAT & WHY. Every formula above breaks somewhere. Here is the full atlas of edge cases, each with what happens and what to do.

Situation	What breaks	What it means / the fix
$n = 1$ (single run)	$s_{P}$ needs $n - 1 = 0$ in the denominator → undefined	You have zero spread information. No CI, no comparison possible. This is exactly Paper A's sin.
$n = 2$	$s_{P}$ defined but $t_{0.975, 1} = 12.7$ — a huge whisker	The bar is so wide it's almost useless. Two runs barely constrain anything.
$s_{P} = 0$ (all runs identical)	$t \to \infty$ , denominator vanishes	Suspicious! Usually means the seed was not actually varied — a reproducibility bug, not perfect stability.
non-Normal scores	$1.96$ and $t$ both lose their meaning	Heavy tails or a ceiling near 100% break the bell assumption — use bootstrap resampling instead.
Testing $m$ model pairs	many chances to get a fluke $p < 0.05$	Use Bonferroni (explained just below the table): demand $p < α / m$ . 20 models → 190 pairs → naive threshold yields $\approx 10$ false wins by chance.
Tiny but "significant" gap	huge $n$ shrinks the SEM until any nonzero gap crosses $p < 0.05$	Significance is not the same as importance. A $0.05%$ improvement can be "significant" yet useless in practice. Report Cohen's $d$ (below) to judge whether the gap is big relative to the noise, not just detectable.

Bonferroni correction — worked out. Suppose you compare $20$ models against each other. The number of distinct pairs is

$m = (2 20) = \frac{20 \times 19}{2} = 190 comparisons.$

Reading the formula: $(2 20)$ ("20 choose 2") counts how many ways to pick $2$ models out of $20$ to pit against each other; it equals $\frac{20 \times 19}{2}$ because there are $20$ choices for the first and $19$ for the second, then divide by $2$ so that "A vs B" and "B vs A" aren't double-counted.

If each test uses the naive threshold $α = 0.05$ , then by chance alone you expect

$m \times α = 190 \times 0.05 = 9.5 \approx 10 false "wins".$

Bonferroni's fix: shrink each test's threshold to $α / m = 0.05/190 \approx 0.00026$ . Now a comparison must clear a far higher bar before you believe it, and the whole family's false-alarm rate stays under $α$ . The figure below shows the lottery: 190 random p-values, the greedy line at $0.05$ catching flukes, the strict Bonferroni line at $α / m$ letting almost none through.

Cohen's $d$ rescales the gap by the pooled spread, so it answers "is the difference big relative to the noise?", independent of how many runs you ran:

$d = \frac{P ˉ _{A} - P ˉ _{B}}{s _{pooled}}, s_{pooled} = \frac{( n _{A} - 1 ) s _{A}^{2} + ( n _{B} - 1 ) s _{B}^{2}}{n _{A} + n _{B} - 2}$

$s_{pooled}$ = one combined width, blending both samples' spreads. Each sample is weighted by its own degrees of freedom $n_{A} - 1$ and $n_{B} - 1$ , and the whole thing is normalised by the total df $n_{A} + n_{B} - 2$ , then square-rooted back into original units.
Reading $d$ : $∣ d ∣ < 0.2$ negligible, $0.2 \leq ∣ d ∣ < 0.5$ small, $0.5 \leq ∣ d ∣ < 0.8$ medium, $∣ d ∣ \geq 0.8$ large.

PICTURE. The failure gallery: the $n = 1$ point with no whisker, the collapsed $s_{P} = 0$ spike, and the Bonferroni "lottery of false positives."

The one-picture summary

Everything above is one story: turn a dot into a cloud, measure the cloud's centre and width, shrink the uncertainty with more runs, then compare clouds — not dots.

Recall Feynman retelling — say it to a friend

Imagine you throw a dart at a scoreboard. It lands at 92. Does the dart-thrower always hit 92? No — throw again, you get 91, then 93. So one dart tells you almost nothing. First, throw many darts (many random seeds). Now you have a little cloud of holes. Where is the cloud? Find its balance point — that's the mean, $\overset{ˉ}{P}$ . Don't brag about your single best hole; that's cheating. How wide is the cloud? Measure how far the holes typically sit from the centre — that's the standard deviation $s_{P}$ . A tight cloud is a reliable thrower; a wide one is a gambler. How sure are you about the centre? Even the balance point wobbles if you re-throw the whole set. But it wobbles less than the raw holes, and shrinks like one-over-root-of-how-many-throws. Four times the darts, half the wobble. That shrunk wobble is the SEM, and it sets the length of your error bar — but only if the cloud is bell-shaped, which the Central Limit Theorem grants you once you have enough independent throws. Is your model better than mine? Put both clouds side by side. If they smear into each other, the "win" is just luck — that's what a small $t$ means. If both models faced the same seeds, pair them up so the shared bad-luck cancels. And if you compare lots of models, remember someone always wins the lottery by chance, so raise the bar (Bonferroni). Finally, even a real gap might be too tiny to care about — that's what Cohen's $d$ checks. That's the whole of evaluation rigor: measure the cloud, not the dot.

Recall

What does the "-1" in $\frac{1}{n - 1}$ pay for? ::: The honesty tax for having estimated $\overset{ˉ}{P}$ from the same data — it corrects the systematic under-estimate of spread. Why divide by $n$ and not $n$ in the SEM? ::: Random errors only partially cancel when averaged, in a square-root way — so precision improves like $1/ n$ . Why is the subscript $0.975$ (not $0.95$ ) on the t-quantile for a 95% CI? ::: A 95% interval leaves 5% outside split into two 2.5% tails, so the right edge sits at 97.5%. What assumption lets us use $1.96$ or the t-quantile at all? ::: That the cloud of means is (near-)Normal — granted by the Central Limit Theorem for enough independent runs; otherwise use bootstrap. When should you use a paired t-test instead of Welch's? ::: When both models were evaluated on the same seeds, so runs are matched and shared noise cancels; df becomes (pairs − 1). Paper A "92.3" vs Paper B " $91.8 \pm 1.94$ " — is A better? ::: No — 92.3 lies inside B's confidence interval, so the gap is consistent with noise. When is $s_{P}$ undefined and why? ::: At $n = 1$ , since $n - 1 = 0$ ; a single run carries zero spread information. How many pairwise comparisons among 20 models, and how many false wins at $α = 0.05$ ? ::: 190 comparisons; about $190 \times 0.05 \approx 10$ false wins — hence Bonferroni's $α / m$ threshold. Statistical vs practical significance? ::: A big enough $n$ makes any gap "significant"; Cohen's $d$ tells you if the gap is actually large relative to the noise.

See also: Evaluation metrics · Regularization techniques · Bias–variance tradeoff · Interpretability and AI safety · Emerging architectures