A - B testing for models: Visual walkthrough — AI-ML deep dive

Prerequisites we lean on (each one is built here too, in case you skipped it): Bernoulli and Binomial Distributions, Central Limit Theorem, Confidence Intervals, Hypothesis Testing.

Step 1 — One user is a coin flip

WHAT. Picture a single user meeting model A. They do exactly one of two things: they convert (buy, click, sign up) or they don't. Nothing in between. We write down a number for that user: $1$ if they convert, $0$ if they don't. Call it $X_{1}$ .

WHY. Our raw data are yes/no events. The honest mathematical name for "a single yes/no event that comes up 'yes' with probability $p$ " is a Bernoulli trial. We cannot skip to averages before we have named the atom the averages are made of.

PICTURE. Look at the two-branch tree in red. From one user, probability $p_{A}$ flows to the "convert = 1" branch and probability $1 - p_{A}$ flows to "no = 0". That is the entire random behaviour of one user.

Two facts about this one coin, both readable off the tree:

$average outcome E [X_{i}] = value if convert 1 \cdot its chance p_{A} + value if not 0 \cdot (1 - p_{A}) = p_{A}$

The mean of a Bernoulli is just $p_{A}$ — sensible: if $5%$ convert, the average of a pile of $0$ s and $1$ s is $0.05$ .

$Var (X_{i}) = spread of one flip p_{A} (1 - p_{A})$

Step 2 — Averaging many users shrinks the wobble

WHAT. We run $n_{A}$ users through A, get outcomes $X_{1}, \dots, X_{n_{A}}$ , and compute the sample conversion rate

$\overset{p}{^}_{A} = \frac{1}{n _{A}} \sum_{i = 1}^{n_{A}} X_{i} = \frac{x _{A}}{n _{A}}$

where $x_{A}$ is simply the count of $1$ s (the number who converted). The little hat $^$ means "estimated from data" — it is our guess at the true, hidden $p_{A}$ .

WHY. We can never see $p_{A}$ directly. $\overset{p}{^}_{A}$ is the closest thing we can measure. The whole test is about comparing two of these hats and asking whether their gap is real.

PICTURE. The figure shows three dot-clouds of $\overset{p}{^}_{A}$ : for $n = 10$ , $n = 100$ , $n = 1000$ users, each cloud centred on the same true $p_{A}$ (red line). The cloud narrows as $n$ grows. That narrowing is the single most important visual on this page.

Here is the algebra that draws that narrowing, term by term:

$Var (\overset{p}{^}_{A}) = Var (\frac{1}{n _{A}} \sum X_{i}) = \frac{1}{n _{A}^{2}} indep. users: variances add i = 1 \sum n_{A} Var (X_{i}) = \frac{1}{n _{A}^{2}} \cdot n_{A} p_{A} (1 - p_{A}) = \frac{p _{A} ( 1 - p _{A} )}{n _{A}}$

Read the chain:

Pulling $\frac{1}{n _{A}}$ out of a variance squares it → $\frac{1}{n _{A}^{2}}$ (variance is in squared units).
Variances of independent things add, so a sum of $n_{A}$ identical wobbles is $n_{A} p_{A} (1 - p_{A})$ .
The $n_{A}$ on top cancels one $n_{A}$ on the bottom, leaving $\frac{p _{A} ( 1 - p _{A} )}{n _{A}}$ .

Step 3 — We test the difference, so its noise is the sum of two noises

WHAT. We don't care about A alone or B alone; we care about the gap $D = \overset{p}{^}_{B} - \overset{p}{^}_{A}$ . Since A-users and B-users are different people (independent groups), the wobbles of the two hats add:

$Var (\overset{p}{^}_{B} - \overset{p}{^}_{A}) = noise from A \frac{p _{A} ( 1 - p _{A} )}{n _{A}} + noise from B \frac{p _{B} ( 1 - p _{B} )}{n _{B}}$

WHY. A minus sign inside a variance still adds the two variances (subtracting a wobbly thing is just as wobbly as adding it). The gap we measure inherits noise from both clouds — so it is noisier than either hat alone. We must beat that combined noise.

PICTURE. Two bell-shaped clouds side by side — A centred at $p_{A}$ , B centred at $p_{B}$ — with a red bracket showing the gap $D$ between their centres, and a shaded band showing the combined wobble of that gap.

Step 4 — The Central Limit Theorem turns the mess into one bell curve

WHAT. Even though each $X_{i}$ is a jagged $0/1$ , the average of many of them piles up into a smooth, symmetric bell curve (a Normal distribution). So both $\overset{p}{^}_{A}$ and $\overset{p}{^}_{B}$ — and their difference $D$ — are approximately Normal once $n$ is large.

WHY this tool and not another. We need a way to convert "how far apart are the clouds?" into "how surprising is that gap?" A bell curve is the one shape whose surprise is fully described by a single number: how many standard errors from the centre you are. The Central Limit Theorem is what earns us the right to use a bell curve here, even though our data are lumpy coins.

PICTURE. The jagged $0/1$ spikes (top) morph, via CLT, into a smooth red bell (bottom). The horizontal axis is now measured in standard errors, and the marker at distance $z$ from the centre shows "how many SEs out is our gap $D$ ."

Now we imagine the world of the null hypothesis $H_{0}$ : "A and B are secretly the same", i.e. $p_{A} = p_{B}$ , so the true gap is $0$ . We ask: if the true gap were zero, how many standard errors is our observed gap? That count is the z-statistic:

$z = \frac{( p ^ _{B} - p ^ _{A} ) observed gap D - 0 gap if H _{0} true}{typical noise size SE ( D )} = \frac{p ^ _{B} - p ^ _{A}}{SE ( D )}$

A big $∣ z ∣$ means "our gap is many noise-widths out — hard to explain by luck."

Step 5 — Pool the rate, because under $H_{0}$ there is only one true $p$

WHAT. To compute $SE (D)$ we need $p_{A}$ and $p_{B}$ — which we don't know. Under $H_{0}$ they are equal, so we estimate that one shared rate using all the conversions from both groups pooled together:

$\overset{p}{^} = \frac{x _{A} + x _{B}}{n _{A} + n _{B}}$

and plug it into the standard error:

$z = \frac{p ^ _{B} - p ^ _{A}}{one shared wobble p ^ ( 1 - p ^ ) split over both groups ( \frac{1}{n _{A}} + \frac{1}{n _{B}} )}$

WHY pool. We are testing the null world where both arms share one true $p$ . The most accurate estimate of that single number uses every user we have, not just half. This gives the sharpest, fairest standard error for the "no difference" hypothesis.

PICTURE. Two buckets of conversions ( $x_{A}$ and $x_{B}$ ) being poured into one big bucket, whose fill-level is $\overset{p}{^}$ . The red level line is the pooled rate.

Decide. Convert $z$ to a two-sided p-value $= 2 (1 - Φ (∣ z ∣))$ , where $Φ$ is the area under the bell to the left of a point. If p-value $< α$ (usually $0.05$ ), the gap is too surprising to be luck → reject $H_{0}$ . See it worked in Step 7.

Step 6 — WHY it takes so long: signal must climb out of noise

WHAT. Before running, we ask: how many users $n$ per group do we need to reliably catch a lift of size $δ = p_{B} - p_{A}$ ? The classic answer:

$n \approx \frac{( z _{1 - α /2} + z _{1 - β} ) ^{2} how sure we demand [ p _{A} ( 1 - p _{A} ) + p _{B} ( 1 - p _{B} ) ] how noisy the metric is}{how tiny the effect is δ ^{2}}$

WHY. Read it as a tug-of-war between signal $δ$ and noise. To see $δ$ , it must poke above the noise band. The $δ^{2}$ in the denominator is the killer: halve the effect you want to catch and $n$ quadruples.

PICTURE. The signal $δ$ (red arrow, fixed length) against a noise band that shrinks with $n$ . Where the noise band's edge drops below the signal, the effect becomes detectable — that crossing point is your required $n$ .

Step 7 — The whole machine on real numbers

WHAT. Run the pipeline on the parent's Example 1.

PICTURE. Our observed $z = 1.894$ (red) sits just inside the $\pm 1.96$ rejection thresholds — you can literally see it fall short. The p-value is the shaded tail area beyond $\pm 1.894$ .

The one-picture summary

Everything above, compressed: raw $0/1$ coins → averaged into hats that narrow with $n$ → their gap $D$ standardized against pooled noise into $z$ → compared to the bell's thresholds → verdict.

Recall Feynman retelling — the whole walkthrough in plain words

Two lemonade recipes. Each kid who visits either buys again (a $1$ ) or doesn't (a $0$ ) — that's one coin flip (Step 1). Count the fraction who came back at each stand; with few kids that fraction bounces around wildly, with many kids it settles down (Step 2). You care about the difference between the two stands' fractions, and because both fractions bounce, the difference bounces even more — its jitter is the two jitters added (Step 3). Pile up enough kids and that difference forms a smooth hill; measure how many "jitter-widths" the two stands are apart — that's $z$ (Step 4). To judge fairly whether the recipes are truly the same, you pretend they are and pool all the kids to estimate the one shared come-back rate (Step 5). If your measured gap is more than about two jitter-widths out, it's too weird to be luck — the new recipe really is better. And the reason you had to hand out lemonade for weeks? To detect a small improvement, the jitter must shrink below it, and jitter only shrinks with the square root of the crowd — so tiny gains need enormous crowds (Step 6). Example 1 landed at $1.9$ jitter-widths — almost convincing, but not past the $1.96$ line, so you keep pouring (Step 7).

Related deep context: Confidence Intervals (the flip side of this test), Model Monitoring & Drift, Offline vs Online Evaluation, Canary Releases, Shadow Deployment, Multi-Armed Bandits.