Foundations — Benchmark design and evaluation rigor

Before you can read the parent note Benchmark design and evaluation rigor, you need the language it speaks. Below is every symbol and idea it assumes, built from nothing, in an order where each one leans on the one before it.

0. The picture we keep returning to: a cloud of dots

Imagine you train the same model five times. Because of random shuffling and random starting weights, you get five different accuracy numbers: maybe $91.2,92.4,91.1,92.9,91.4$. Plot them on a number line and you get a little scatter of dots — a cloud.

Everything that follows is about describing this cloud with numbers instead of eyeballing it. Look at the figure: the centre of the cloud and the width of the cloud are the two things we will name.

1. Performance $P$ — a random variable

• Plain words: $P$ (for Performance) is the accuracy you get from one training run.
• Picture: one dot in the cloud of figure s01.
• Why the topic needs it: if $P$ were a fixed constant, a single number would tell the whole truth. Because $P$ jumps around each run, one number is only a sample — a single dot, not the cloud. This is why the parent note refuses to trust "1 run, best hyperparameters."

The randomness comes from three knobs the parent note lists:

Each different $(s,D,\theta )$ gives a possibly different dot.

2. Subscripts and the running index $i$

• Plain words: $P_1$ is run number 1, $P_2$ is run number 2, and so on. The letter $i$ is a placeholder that walks through $1,2,3,\dots$
• Picture: number the dots left-to-right in figure s01; $P_i$ points at the $i$-th dot.
• Why the topic needs it: to talk about "all the runs at once" without writing them all out, we say $P_1,P_2,\dots ,P_n$.

The letter $n$ is simply how many runs you did (five, in our example).

3. The summation sign $\sum$ — "add them all up"

• Plain words: the big stretched-out "S" (Greek capital sigma) is shorthand for a long addition.
• Picture: ${\sum }_{i=1}^n{P}_i$ = slide along the cloud, scoop up every dot's height, pile them together.
• Why this tool and not "just write + + +": with 5 runs you could write it out; with 500 you can't. $\sum$ scales to any $n$.

${\sum }_{i=1}^3{P}_i=P_1+P_2+P_3$

4. The sample mean $\bar{P}$ — the centre of the cloud

• Plain words: add up all the runs, divide by how many there are — the plain average.
• Picture: the balance point of the cloud in figure s01 (the pale-yellow tick). If the dots were weights on a see-saw, $\bar{P}$ is where it balances.
• The bar on top ($\stackrel{ˉ}{}$) is read "P-bar" and always means "the average of the P's."
• Why the topic needs it: this is the centre — our single best guess for the model's true capability. But a centre alone hides how spread out the cloud is, which is the next symbol.

5. The sample standard deviation $s_P$ — the width of the cloud

This is the first place a tool is chosen for a reason, so we build it slowly.

The question: "How far do the dots typically sit from the centre?"

Naive attempt: average the distances $P_i-\bar{P}$. But those distances are positive for dots above the centre and negative for dots below — they cancel to zero every time. Useless.

The fix: square each distance first (squares are always positive), average the squares, then take a square root to get back to the original units (percent, not percent-squared).

• Picture: figure s02 shows each dot's distance to the centre as a chalk-blue bar; $s_P$ is the typical length of those bars.
• Why divide by $n-1$ and not $n$? A subtle but real point: you already "spent" one piece of information computing $\bar{P}$ from the same data. Dividing by $n-1$ (called degrees of freedom) corrects a slight underestimate. For your intuition, treat it as "almost the average, nudged to be honest."
• Why the topic needs it: a small $s_P$ = tight cloud = trustworthy number. A big $s_P$ = fat cloud = the model is moody and your single run means little. This is the variance side of the bias–variance picture, seen through evaluation.

6. The true mean $\mu$ — the target we never see

• Plain words: $\bar{P}$ is what your finite runs show; $\mu$ is the truth they point toward.
• Picture: in figure s01 the dashed line is $\mu$ — the real centre of the infinite cloud. Your five dots' average $\bar{P}$ lands near it, rarely exactly on it.
• The key distinction: roman letters ($\bar{P}$, $s_P$, $n$) = things you measure. Greek letters ($\mu$, later $\sigma$, $\nu$, $\alpha$) = the hidden truth or fixed constants. The parent note's hypothesis test is entirely about guessing ${\mu }_A$ vs ${\mu }_B$ from measured ${\bar{P}}_A$ and ${\bar{P}}_B$.

7. Standard error $\text{SEM}$ — uncertainty about the centre

Do not confuse this with $s_P$.

• $s_P$ = how spread the individual runs are.
• $\text{SEM}$ = how spread the average $\bar{P}$ would be if you repeated the whole 5-run experiment again and again.

• Picture: figure s02 (right side) shows a narrow cloud-of-averages sitting inside the fat cloud-of-runs.
• Why the $\sqrt{n}$: averaging cancels noise. More runs $\Rightarrow$ steadier average $\Rightarrow$ smaller SEM. But it shrinks like $1/\sqrt{n}$, so 4× the runs only halves the uncertainty — the parent note's warning.

8. The confidence interval — the honest "$\pm$"

• Picture: figure s03 draws $\bar{P}$ with whiskers stretching left and right. Two overlapping whisker-bars = "can't tell these models apart."

Two versions appear in the parent note:

${\text{CI}}_{95}=\bar{P}\pm 1.96\cdot \text{SEM}(\text{many runs}){\text{CI}}_{95}=\bar{P}\pm t_{0.975,n-1}\cdot \text{SEM}(\text{few runs})$

• The number $1.96$: for a big-enough pile of runs the cloud-of-averages is bell-shaped, and $95\%$ of a bell sits within $1.96$ SEMs of the centre. It is a fixed constant, not a mystery.
• $t_{0.975,n-1}$: the same idea but wider when you have few runs, because a small cloud gives you a shaky $s_P$. The $n-1$ is the same degrees-of-freedom count from §5. With 5 runs, $n-1=4$ and this number is $2.776$ — noticeably bigger than $1.96$, meaning "be more humble with less data."

9. Hypothesis testing symbols: $H_0$, $t$, $p$, $\nu$, $\alpha$

The parent note asks: "is Model A really better than Model B, or are their clouds just overlapping?"

• Picture: figure s03's two whisker-bars; $H_0$ says "these are secretly the same cloud."

The test statistic $t$ — a signal-to-noise ratio: $t=\frac{{\bar{P}}_A-{\bar{P}}_B}{\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}}$

• Top (signal): how far apart the two centres are.
• Bottom (noise): the combined wobble of both clouds' averages.
• Read it as: "how many noise-widths apart are the two centres?" A big $|t|$ means the gap dwarfs the wobble — probably a real difference.

The $p$-value — if $H_0$ were true, how surprising is a gap this big?

• A small $p$ (below $\alpha =0.05$, the significance level) means "too surprising to be luck" $\Rightarrow$ reject $H_0$.
• $\alpha$ is your chosen risk of a false alarm.

$\nu$ (Greek "nu"), the degrees of freedom — a count of how much independent information feeds the test, controlling how wide/humble the $t$-based conclusion is. Welch's ugly formula for $\nu$ just handles the case where the two clouds have different widths.

10. Effect size (Cohen's $d$) — "how big, in cloud-widths?"

• Plain words: the gap between centres, measured in units of typical spread.
• Picture: in figure s03, $d$ = "how many cloud-widths separate the two centres." Two centres one full width apart look obviously different; a tenth of a width apart look merged.
• Why separate from $t$: $t$ grows just by adding runs; $d$ does not. $d$ answers practical size, $t$ answers is-it-real.

The parent note's ladder: $|d|<0.2$ negligible, up to $|d|\ge 0.8$ large.

11. Selection bias & spurious correlations — the traps

• Picture: selection bias = drawing your conclusion from the single top-right dot in figure s01. Spurious correlation = a "reasoning" score that collapses the moment you shuffle the answer choices (the parent note's negative control).
• Why they close the loop: these are exactly the failures that all the machinery above (averaging over runs, confidence intervals, negative controls) is built to catch. They connect to regularization and evaluation metrics — a metric that rewards a shortcut causes spurious success.

How these foundations feed the topic

Equipment checklist

Test yourself — reveal only after answering.

Ready? Return to Benchmark design and evaluation rigor and the Hinglish companion. Related frontier notes: 6.5.01-Emerging-architectures, 6.5.02-Interpretability-and-AI-safety.