Foundations — A - B testing for models

This page assumes you have seen none of the notation in the parent note. We will earn every symbol — one at a time, each anchored to a picture — before it is ever used.

1. A single user is a coin flip — the symbol $X_i$

Picture one user arriving at your site. They either convert (buy, click, sign up) or they don't. That is a yes/no event: exactly like flipping a coin that lands heads or tails.

We write that outcome as $X_i$ — the little $i$ is just a name-tag ("user number $i$"). We agree:

$X_i=\{\begin{array}{ll}1& \text{if user }i\text{ converts (heads)}\\ 0& \text{if not (tails)}\end{array}$

In figure s01 you see ten users drawn as bars: amber bars stand at height $1$ (converted), cyan bars at height $0$ (didn't). The number printed above each bar is that user's $X_i$. Look at how their average — the box on the right — is simply the count of amber bars divided by ten.

This is the Bernoulli random variable. "Random variable" just means a number whose value we won't know until the event happens. A single yes/no coin flip like $X_i$ is called a Bernoulli trial.

2. Adding the flips up — the count $K$ and the sample size $n$

Before we sum anything, we must name how many users we watched.

Once we have $n$ independent coin flips, the total number of conversions is their sum:

$K={\sum }_{i=1}^n{X}_i=X_1+X_2+\cdots +X_n.$

3. The true rate $p$ vs. the measured rate $\hat{p}$

The coin has a secret bias: some hidden fraction of all possible users would convert. Call that hidden fraction $p$ (a number between $0$ and $1$).

You cannot see $p$. What you can do is watch $n$ real users and count. That gives you an estimate, and we mark estimates with a little hat: $\hat{p}$ ("p-hat").

The symbol ${\sum }_{i=1}^n{X}_i$ is a summation: the tall "S" ($\Sigma$, Greek letter sigma) says "add these up". $i=1$ underneath and $n$ on top say "run $i$ from $1$ to $n$". So ${\sum }_{i=1}^n{X}_i$ just means $X_1+X_2+\cdots +X_n$ — the total conversion count $K$ from the previous section.

4. Variance — how much the measurement wobbles

If $\hat{p}$ jumps around every time you re-run, how much does it jump? That "typical amount of wobble" is what variance and its square root, standard error, measure.

In figure s02 the amber curve plots the variance of one coin flip, $p(1-p)$, against $p$. Follow it: the hill peaks at $p=0.5$ and touches zero at both ends. The two cyan arrows mark the "rigged coin" corners where a coin always does the same thing and never surprises you.

For one coin flip $X_i$ the variance is $p(1-p)$ — biggest at $p=0.5$ (a fair coin is the most unpredictable) and $0$ at $p=0$ or $p=1$.

5. The Normal curve and the Central Limit Theorem

A single $\hat{p}$ is a lumpy thing. But here is the magic: average enough independent coin flips and the wobble of the average always takes the same bell shape, no matter how weird one flip was. That is the Central Limit Theorem (CLT).

In figure s03 the cyan curve is the standard bell; the vertical dashed line marks a point $z$ on the $x$-axis, and the amber shaded region is everything to its left. That shaded area is exactly the number $\Phi (z)$ we define in the next section.

6. Two groups: the symbols $A$, $B$, and their per-group notation

Now the whole point: we run two models side by side and compare them.

The question "is B better?" is a question about the gap ${\hat{p}}_B-{\hat{p}}_A$. To judge whether that gap is bigger than luck, we need its wobble — its standard error. Since the two groups are separate people, their variances add:

In figure s04 the same amber gap is drawn on two bells. On the narrow (large-$n$) bell it juts far out — many wobbles wide, convincing. On the wide (small-$n$) bell the identical gap is buried in the fuzz — just noise. Same gap, opposite verdict — that is why sample size decides everything.

7. Standardizing — the $z$-score and then $\Phi$

Now the punchline symbol. Suppose the new model's rate beats the old by some gap. Is that gap big? "Big" only means something relative to the wobble. The $z$-score answers exactly that — it re-expresses the gap in units of standard error:

$z=\frac{{\hat{p}}_B-{\hat{p}}_A}{{\mathrm{SE}}_0}=\frac{\text{observed gap}}{\text{standard error under }{H}_0}=\frac{\text{signal}}{\text{noise}}$

Only now, with $z$ defined as the standard Normal variable, does $\Phi$ earn its keep:

8. The decision vocabulary: $H_0$, $\alpha$, $\beta$, $\delta$, and the p-value

These are the words the Hypothesis Testing framework wraps around the $z$-score.

A confidence interval then reports the plausible range of the true gap — not just "different: yes/no" but "different by roughly this much." It is built with the unpooled SE from §6, because there we're estimating the actual gap rather than testing "no gap".

How the pieces feed the topic

Each foundation flows into the two-proportion $z$-test and the sample-size formula in the parent note. Neighbouring ideas — Shadow Deployment, Canary Releases, Multi-Armed Bandits, Offline vs Online Evaluation, and Model Monitoring & Drift — reuse this same signal-vs-noise machinery.

Equipment checklist

Cover each answer and test yourself before moving to the parent note.