Benchmark design and evaluation rigor: Question bank

This page assumes you have met the ideas built in the parent note: the mean $\overset{ˉ}{P}$ (the average score over runs), the standard error of the mean $SEM = s / n$ (how wobbly that average is), a confidence interval (a range that plausibly contains the true score), a p-value (how surprised $H_{0}$ would be by your data), and negative controls (tests a model should fail). Every trap below leans on one of those.

True or false — justify

TF1. A benchmark is just a dataset; the metric is a separate concern.

False — a benchmark is a measurement instrument = dataset plus task spec plus metric plus protocol. Swap the metric and you can turn the same dataset into a different (sometimes meaningless) measurement.

TF2. If Model A scores 92.3% and Model B scores 91.8%, Model A is better.

False — you cannot compare point estimates without their spread. If B's confidence interval covers 92.3%, the gap may be pure noise, so no ordering is justified.

TF3. Reporting the best of 20 runs is fine as long as you say "best".

False — the max of many runs is a biased estimate of true capability (it samples the lucky right tail). Honesty about how you cherry-picked does not undo the selection bias.

TF4. A statistically significant improvement (

p < 0.05

) is always worth deploying.

False — significance only says the difference is unlikely to be zero, not that it is large. With enough runs a meaningless 0.1% gap becomes "significant" yet has no practical value; check effect size too.

TF5. Getting near-chance accuracy on a negative control is bad news for your model.

False — that is exactly what you want. A negative control is designed so a properly-reasoning model fails it; near-chance there confirms the benchmark isn't trivially exploitable.

TF6. Increasing the number of seeds

n

shrinks the confidence interval linearly.

False — the interval shrinks with

1/ n

, not

1/ n

. To halve your uncertainty you need 4× the runs, not 2×.

TF7. If two 95% confidence intervals overlap, the models are definitely not significantly different.

False — overlapping intervals often mean no significance, but a proper two-sample t-test can still find significance for modest overlaps. Non-overlap is sufficient for significance, but overlap is not sufficient for the reverse.

TF8. A benchmark that every new model saturates near 100% is a great, mature benchmark.

False — saturation means the benchmark has stopped discriminating between models. It has become useless as a measurement tool precisely because it can no longer separate good from great.

TF9. Fixing hyperparameters before touching the test set is over-cautious bureaucracy.

False — tuning on the test set leaks test information into your model choice, inflating the reported score. Freezing hyperparameters first is what keeps the test set an honest estimate of generalization.

TF10. Welch's t-test is only needed when the two models have equal variance.

False — it's the opposite. Welch's version exists precisely because we do not assume equal variances (Model A may be far more seed-sensitive than Model B).

Spot the error

SE1. "We ran the experiment once, got 94%, so our model reaches 94% accuracy."

The error is treating a single draw as the truth. One run is a point from a distribution with no spread reported — you cannot separate a real 94% capability from a lucky run around a true 90%.

SE2. "We tested 20 architectures pairwise at

α = 0.05

and found 9 significant winners."

No multiple-comparison correction was applied. 20 models give ~190 pairwise tests, so ~

190 \times 0.05 \approx 10

false positives are expected by chance alone — use a Bonferroni level

α / m

or the "wins" are noise.

SE3. "Our model beats the baseline by 0.3% with

p = 0.001

, so it's a major advance."

Conflating statistical with practical significance. A tiny effect can be highly significant given many runs; report Cohen's d to show the 0.3% is likely negligible (

∣ d ∣ < 0.2

SE4. "We built the benchmark by scraping physics textbooks — guaranteed correct answers."

Correct answers, wrong risk. Scraped text is likely inside pretraining corpora, so models can memorize rather than reason. Provenance overlap with training data is a leakage bug, not a feature.

SE5. "We used the same 5 seeds for both models and just compared the two means directly."

Comparing raw means ignores the noise. You must divide the mean difference by the standard error of the difference, i.e. run a t-test; otherwise you have no idea if

\overset{ˉ}{P}_{A} - \overset{ˉ}{P}_{B}

exceeds the wobble.

SE6. "n=5 runs, so we used the 1.96 z-value for our 95% CI."

With small

n

the central limit approximation is too optimistic. Use Student's

t_{0.975, n - 1}

(here

t_{0.975, 4} \approx 2.776

), which is wider than 1.96 and honestly reflects small-sample uncertainty.

SE7. "Random chance is 25%, our model scores 27% on the shuffled control, so it's still reasoning."

27% is essentially chance — the 2% is within noise. That result diagnoses that the model was pattern-matching, not reasoning; the author read a failure signal as success.

SE8. "Train and test are both drawn i.i.d. from the same generator, so the split is clean."

An i.i.d. split can still be too easy: it never tests distribution shift. A robust split changes generation parameters (more objects, rougher surfaces) so the test probes generalization, not interpolation.

Why questions

WY1. Why is a single reported number without uncertainty scientifically incomplete?

Because performance is a random variable over seeds, splits, and hyperparameters; a lone number is one draw with no way to tell signal from noise or to compare against another model.

WY2. Why does Welch's t-test use that awkward

ν

formula instead of

n_{A} + n_{B} - 2

degrees of freedom?

Because unequal variances distort the true sampling distribution of

t

; the Welch–Satterthwaite

ν

adjusts the degrees of freedom downward so the p-value stays honest when one model is noisier.

WY3. Why do we build negative controls that the model should fail?

To prove the benchmark isn't trivially solvable by a shortcut. If a model that lacks the real capability still passes, the benchmark is measuring a spurious correlation, not the intended skill.

WY4. Why specify a task qualitatively ("does the ball go up or down?") instead of quantitatively?

A qualitative task can't be solved by memorizing formulas or plugging numbers, so it isolates understanding of physical common sense rather than symbolic recall.

WY5. Why does saturation of a benchmark make it worse as models improve, even though the scores look impressive?

Once the ceiling is hit, the benchmark loses discriminative power — differences shrink into the noise band, so it can no longer rank the newer, stronger models it was meant to measure.

WY6. Why prefer procedurally generated scenarios over hand-collected ones for a reasoning benchmark?

Because you control and can prove exactly what reasoning each item requires, eliminating hidden ambiguity and guaranteeing the answer key follows from stated rules — and reducing overlap with scraped training data.

WY7. Why does the standard error carry a

n

in the denominator rather than

n

Averaging

n

independent noisy scores reduces variance by a factor

n

, and standard deviation is the square root of variance, so the spread of the mean shrinks as

n

WY8. Why link benchmark overfitting to the bias–variance idea and regularization?

Because tuning against a fixed test set is functionally low-bias/high-variance fitting to that specific set; it inflates apparent scores the same way an unregularized model memorizes training noise instead of generalizing.

Edge cases

EC1. What does a 95% confidence interval mean when you have exactly

n = 1

run?

It is undefined — with one point the sample standard deviation

s

has no denominator (

n - 1 = 0

). You have an estimate of the mean but zero information about its spread.

EC2. If a negative-control model scores above chance by a large margin, what does that reveal about the benchmark?

The benchmark has an exploitable shortcut leaking the answer (e.g., distractor phrasing or label artifacts). It is measuring surface cues, so real-model scores on it are inflated and untrustworthy.

EC3. Two models each report identical means but Model A has

s = 0.5

and Model B has

s = 4.0

. Are they equivalent?

Not for deployment. Equal means hide unequal reliability: Model B swings wildly across seeds, so its worst-case runs are far worse — variance is a first-class result, not a footnote.

EC4. Your test-set metric is accuracy, but the classes are 99%/1% imbalanced. What breaks?

A "predict majority" baseline scores 99% while learning nothing. Accuracy is the wrong instrument here; you need metrics aligned to the minority signal (see evaluation metrics like precision/recall/F1).

EC5. You run 40 seeds and the CI is razor-thin, but the mean sits below random chance. What happened?

Precision is not correctness. A tight interval around a bad mean just means you've reliably measured a broken model, mislabeled data, or a scoring bug — narrow uncertainty never rescues a wrong point estimate.

EC6. A benchmark ages: what is the earliest sign it is being memorized rather than solved?

A sudden jump when a new model family (trained on newer web scrapes) appears, especially if it fails matched held-out or paraphrased variants — contamination shows up as brittle, item-specific success.

EC7. You apply Bonferroni across 190 comparisons and now nothing is significant. Is that a failure?

No — it's the correction doing its job. It signals your evidence is too thin to survive family-wise control; the honest conclusion is "insufficient runs / effect sizes," not "no differences exist." Consider more runs or a less conservative correction.

Recall Self-check before you leave

One-line SEM vs SD — which one shrinks with more runs? ::: SEM ( $= s / n$ ) shrinks with $n$ ; the sample SD $s$ estimates the underlying spread and does not shrink — it just gets measured more accurately. One-line: overlap of CIs implies no significance? ::: No — overlap is not sufficient to rule out significance; run the actual two-sample t-test. One-line: what does a good negative control score look like? ::: Near chance — a properly reasoning model should fail the control, confirming the task can't be shortcut.

Related: 6.5.01-Emerging-architectures · 6.5.02-Interpretability-and-AI-safety.