Benchmark design and evaluation rigor: Worked examples

This page is the "every case" companion to the parent topic. In statistics you can be fooled in many different ways, and each way needs its own worked drill. Below we first build a scenario matrix — a grid of every situation the topic can throw at you — then work through examples that together fill every cell.

Before we touch a single formula, three words, defined in plain English, anchored to a picture.

Figure — Benchmark design and evaluation rigor

Look at figure s01: two clouds of dots (two models). The red bracket is the SEM — the fuzziness of each average. If the red brackets overlap a lot, you cannot honestly say one model beat the other.

The scenario matrix

Every cell below is a class of situation. The worked examples that follow are tagged with the cell they cover, and by the end every cell is filled.

#	Cell (situation class)	What makes it tricky	Covered by
C1	Overlapping intervals — small gap, small $n$	Gap looks real, noise is bigger	Ex 1
C2	Non-overlapping / clearly separated	Real effect — but is it big?	Ex 2
C3	Zero / degenerate input ( $n = 1$ , or $s = 0$ )	Formulas divide by zero or $n - 1$	Ex 3
C4	Limiting behaviour — $n \to \infty$	Tiny gap becomes "significant"	Ex 4
C5	Many comparisons — false positives by chance	20 tests, one "wins" by luck	Ex 5
C6	Negative control fails — leakage / shortcut	High score, wrong reason	Ex 6
C7	Real-world word problem — deployment decision	Statistical ≠ practical significance	Ex 7
C8	Exam twist — sign of $t$ , one- vs two-sided	Direction of the claim matters	Ex 8

Prerequisite links: this all rests on bias–variance intuition and the definitions in evaluation metrics.

The one formula engine

Everything uses the same machine. We name each part before using it.

Worked examples

Worked example Example 1 — Cell C1: overlapping intervals, small

$n$ Paper A: $92.3% \pm 0.5$ (SEM), $n_{A} = 5$ . Paper B: $91.8% \pm 0.7$ (SEM), $n_{B} = 5$ . Is A really better?

Forecast: guess now — does the $0.5%$ lead survive?

Signal (top): $\overset{ˉ}{P}_{A} - \overset{ˉ}{P}_{B} = 92.3 - 91.8 = 0.5$ . Why this step? The gap is the only thing that could count as an improvement; isolate it first.
Noise (bottom): SEM values are already given, so combine them: $0. 5^{2} + 0. 7^{2} = 0.25 + 0.49 = 0.74 = 0.860$ . Why this step? Two fuzzy averages compared → their uncertainties add in quadrature (squares), because independent noises don't line up.
Ratio: $t = 0.5/0.860 = 0.581$ . Why this step? This is the signal-to-noise number. Under $H_{0}$ (no real difference), $∣ t ∣$ above ~2 is surprising; $0.58$ is not.
Read the verdict: with $ν \approx 7$ degrees of freedom, $t = 0.58$ gives $p \approx 0.58$ — nowhere near $0.05$ . Why this step? $p$ is "chance of a gap this big when there's truly no difference." $58%$ means: happens all the time by luck.

Verdict: cannot conclude A > B. The lead is noise.

Verify: $0.74 = 0.8602$ , $0.5/0.8602 = 0.5813$ ✓. The 95% CI of the difference is $0.5 \pm 2.365 \cdot 0.860 = 0.5 \pm 2.03$ , i.e. $[- 1.53, 2.53]$ — it straddles $0$ , confirming "no significant difference."

Worked example Example 2 — Cell C2: clearly separated clouds

Model A: $88.0 \pm 0.3$ (SEM), $n = 5$ . Model B: $84.0 \pm 0.4$ (SEM), $n = 5$ . Real? And is it big?

Forecast: four points of gap, tiny fuzz — will it also be a large effect?

Signal: $88.0 - 84.0 = 4.0$ .
Noise: $0. 3^{2} + 0. 4^{2} = 0.09 + 0.16 = 0.25 = 0.5$ . Why this step? Same quadrature rule; the clouds are tight so this is small.
Ratio: $t = 4.0/0.5 = 8.0$ . Enormous → clearly significant ( $p ≪ 0.001$ ).
Effect size (Cohen's d): convert SEM back to spread: $s = SEM \cdot n = 0.3 \cdot 5 = 0.6708$ and $0.4 \cdot 5 = 0.8944$ . Pooled spread $= \frac{( 4 ) ( 0.670 8 ^{2} ) + ( 4 ) ( 0.894 4 ^{2} )}{8} = \frac{1.8 + 3.2}{8} = 0.625 = 0.7906$ . Then $d = 4.0/0.7906 = 5.06$ . Why this step? $t$ says "real"; $d$ says "how big in human terms." $d > 0.8$ is already "large" — $5.06$ is gigantic.

Verify: $0.25 = 0.5$ ; $4/0.5 = 8$ ; $d = 5.06$ , firmly $> 0.8$ ✓. This is the rare honest win: separated and large. Contrast with regularization gains, which are often real but small.

Worked example Example 3 — Cell C3: degenerate input (

$n = 1$ , and $s = 0$ ) Paper C reports a single run: $92.3%$ , no error bar. Paper D ran 4 times and got exactly $91.0, 91.0, 91.0, 91.0$ . Handle both edge cases.

Forecast: what does the formula do when there's only one dot, or when all dots stack on one spot?

$n = 1$ case: SEM $= s / 1$ , but $s$ needs $n - 1 = 0$ in its denominator — undefined. You cannot compute spread from one point. Why this step? A single number has no measurable fuzziness; the honest answer is "uncertainty unknown," never "uncertainty zero."
Consequence: any comparison against Paper C is impossible — treat its claim as a point with unknown error. Report it as such.
$s = 0$ case (Paper D): $s = \frac{1}{3} \sum (91.0 - 91.0)^{2} = 0$ . Then SEM $= 0/ 4 = 0$ . Why this step? Identical runs means the seed changed nothing — usually a sign of a frozen seed or cached result, not true determinism. Treat $s = 0$ as suspicious, not as certainty.
If you plug $s_{D} = 0$ into the $t$ engine comparing D to any noisy model, the denominator uses only the other model's noise — legal, but the $p$ -value is now over-confident. Report the $s = 0$ explicitly.

Verify: with data ${91, 91, 91, 91}$ , mean $= 91$ , variance $= 0$ , so $s = 0$ and SEM $= 0$ ✓. $n = 1$ gives $n - 1 = 0$ → division by zero → undefined ✓.

Worked example Example 4 — Cell C4: the

$n \to \infty$ limit (significance inflation) Two models truly differ by $0.1%$ , each with spread $s = 1.0$ . How many runs $n$ make this "significant"?

Forecast: if the real gap is a measly $0.1%$ , can raising $n$ manufacture significance?

Set up (equal $n$ , equal $s$ ): $t = \frac{0.1}{2 s ^{2} / n} = \frac{0.1}{1.0} n /2 = 0.1 n /2$ . Why this step? Freeze the true gap and spread; let only $n$ vary, so we see $t$ 's growth law.
Solve $t = 1.96$ (the $95%$ threshold): $0.1 n /2 = 1.96 \Rightarrow n /2 = 19.6 \Rightarrow n /2 = 384.16 \Rightarrow n = 768.3$ . Why this step? This is the exact $n$ where a real-but-microscopic gap crosses the "significant" line.
Limit reading: as $n \to \infty$ , $t \to \infty$ . Any nonzero gap eventually becomes significant. Why this step? Significance measures "is the gap nonzero," not "is the gap useful."

Verdict: with ~ $770$ runs, a $0.1%$ difference is "significant" and still meaningless — always pair $p$ with Cohen's $d$ .

Verify: $0.1 768.32/2 = 0.1 \cdot 384.16 = 0.1 \cdot 19.6 = 1.96$ ✓. Effect size $d = 0.1/1.0 = 0.1 < 0.2$ → "negligible" no matter how big $n$ is ✓.

Worked example Example 5 — Cell C5: many comparisons (false positives)

A leaderboard has $20$ models. You run all pairwise comparisons at $α = 0.05$ . How many "wins" appear by pure luck if no model is truly better?

Forecast: everyone's equal — how many false "significant" results should you brace for?

Count comparisons: $(2 20) = \frac{20 \cdot 19}{2} = 190$ . Why this step? Each pair is one test; the count grows like $n^{2}$ , fast.
Expected false positives: $190 \times 0.05 = 9.5$ . Why this step? At $α = 0.05$ , each null test fires falsely $5%$ of the time; expectation is count $\times$ rate.
Fix — Bonferroni: use $α^{'} = 0.05/190 = 0.000263$ . Why this step? Shrinking each test's threshold keeps the family-wide false-alarm rate at $5%$ .

Verdict: without correction, expect ~ $9$ – $10$ fake winners; with Bonferroni, only tests with $p < 0.000263$ survive.

Verify: $(2 20) = 190$ ; $190 \cdot 0.05 = 9.5$ ; $0.05/190 = 0.0002632$ ✓.

Worked example Example 6 — Cell C6: negative control fails (shortcut, not skill)

A "reasoning" benchmark: model scores $89%$ . Control 1 — shuffle answer choices randomly (should drop to chance $= 25%$ for 4 options): model gets $27%$ . Control 2 — replace distractors with logically impossible ones (a reasoner should stay near $89%$ ): model drops to $52%$ . Diagnose.

Forecast: which control number is the smoking gun?

Control 1 reading: $27%$ vs chance $25%$ — gap of $2%$ , essentially chance. Why this step? Confirms the model isn't exploiting the label ordering — good, that pathway is clean.
Control 2 reading: true reasoning shouldn't care that distractors are impossible, so it should hold at ~ $89%$ . It collapsed to $52%$ . Why this step? The drop of $89 - 52 = 37$ points exposes reliance on surface plausibility of options, not logic.
Fraction of "skill" that was shortcut: $\frac{89 - 52}{89 - 25} = \frac{37}{64} = 0.578$ . Why this step? Of the headroom above chance ( $64$ points), $37$ vanished under the honest control — ~ $58%$ of the apparent skill was a spurious correlation.

Verdict: the $89%$ is inflated; the benchmark measured pattern-matching. See interpretability for why models find these shortcuts.

Verify: $89 - 52 = 37$ ; $89 - 25 = 64$ ; $37/64 = 0.5781$ ✓.

Worked example Example 7 — Cell C7: real-world deployment word problem

A fraud detector currently catches $91.8%$ of fraud. A new model tests at $92.3%$ , statistically significant ( $p = 0.01$ , $n = 100$ runs). Deployment costs $ $2$ M in engineering. Each $1%$ of fraud caught saves $ $40$ k/year. Should you ship?

Forecast: significant and an improvement — is that enough to spend $2M?

Practical gain: $92.3 - 91.8 = 0.5$ percentage points.
Annual value: $0.5 \times \$ 40{,}000 = $20{,}000$/year. Why this step? Convert the statistical gap into money — the units that actually decide deployment.
Payback time: $\$ 2{,}000{,}000 / $20{,}000 = 100$ years. Why this step? Compare cost to benefit in the same units (years) — the honest decision axis.

Verdict: do not ship. The gain is real (significant) but practically insignificant — a 100-year payback. Significance answered "is it nonzero"; the money answered "does it matter."

Verify: $0.5 \cdot 40000 = 20000$ ; $2000000/20000 = 100$ ✓.

Worked example Example 8 — Cell C8: exam twist (sign of

$t$ , one- vs two-sided) Claim to test: "Model A is strictly better than B" (directional). Data: $\overset{ˉ}{P}_{A} = 90.0$ , $\overset{ˉ}{P}_{B} = 90.6$ , combined noise $= 0.4$ , $ν = 30$ . The two-sided $5%$ critical value is $2.04$ ; the one-sided is $1.70$ .

Forecast: A's average is lower — can a one-sided test still rescue the claim "A is better"?

Compute $t$ : $t = \frac{90.0 - 90.6}{0.4} = \frac{- 0.6}{0.4} = - 1.5$ . Why this step? Keep the sign: negative means A scored below B — the direction is already against the claim.
One-sided test for "A > B": we'd reject $H_{0}$ only if $t > + 1.70$ . Our $t = - 1.5$ is on the wrong side entirely. Why this step? A one-sided test only accepts evidence in the claimed direction; wrong-sign evidence can never support "A better."
Two-sided reading: $∣ t ∣ = 1.5 < 2.04$ , so even "A differs from B" fails. Why this step? Shows the mistake of dropping the sign — some students report $∣ t ∣ = 1.5$ and forget A is actually worse.

Verdict: claim rejected; if anything B trends higher. Never discard the sign of $t$ .

Verify: $(90.0 - 90.6) /0.4 = - 1.5$ ; $- 1.5 < 1.70$ and $1.5 < 2.04$ ✓.

Recall Self-test

Why divide the gap by the noise instead of just reporting the gap? ::: Because a raw gap is unitless-meaningless — "is 5 big?" only has an answer relative to the noise. The ratio $t$ makes it scale-free. With $n = 1$ , why can't you compute SEM? ::: $s$ needs $n - 1 = 0$ in its denominator; one point has no measurable spread, so uncertainty is unknown, not zero. A $0.1%$ gap becomes "significant" at large $n$ — good or bad? ::: Neither — significance only means "nonzero." Check Cohen's $d$ ; here $d = 0.1$ is negligible. 20 models, all pairwise, $α = 0.05$ — expected false wins? ::: $(2 20) \cdot 0.05 = 190 \cdot 0.05 = 9.5$ .

Related frontier reading: emerging architectures keep breaking old benchmarks — the exact reason rigor matters.