A - B testing for models: Worked examples — AI-ML deep dive

This is the hands-on companion to A - B testing for models. The parent note built the two-proportion $z$ -test and the sample-size formula. Here we run the machine on every kind of input it can receive — a clear win, a clear loss, a "too small to trust" case, a degenerate zero-conversion group, the huge- $n$ trap, a real-world word problem, and an exam twist. If a scenario can happen in a real experiment, it lives in the table below and gets its own worked example.

Before we start, one reminder of the three symbols we lean on, in plain words:

We reject the null (declare a real difference) when the p-value drops below $α = 0.05$ , i.e. when $∣ z ∣ > 1.96$ .

The scenario matrix

Every A/B test outcome falls into one of these cells. The examples below are labelled by cell so you can see the whole space is covered.

Cell	What's special about the input	Sign of $\overset{p}{^}_{B} - \overset{p}{^}_{A}$	Decision it forces
C1 Clear win	large $n$ , gap well above noise	$+$	reject $H_{0}$ , ship candidate
C2 Clear loss	large $n$ , B clearly worse	$-$	reject $H_{0}$ , keep A
C3 Underpowered	real-looking gap but $n$ too small	$+$	fail to reject — need more data
C4 Null / A-A	B $=$ A, sanity check	$\approx 0$	must NOT reject (else pipeline bug)
C5 Degenerate zero	one group has $0$ conversions	either	formula still works; watch $\overset{p}{^} = 0$
C6 Huge- $n$ trap	tiny gap, millions of users	$+$	significant but practically useless
C7 Sample size	plan the run before data	—	compute $n$ from MDE
C8 Word problem	messy real numbers + guardrail	$+$	significance AND guardrail check

We will hit all eight.

The picture above is the mental model for the whole page: a bell curve of "gaps we'd see by pure luck if A and B were identical." The red shaded tails past $\pm 1.96$ noise-widths hold $5$ of the area — that is $α$ . Every example just asks: where does our measured $z$ land relative to those red tails?

The reusable recipe

For a two-proportion test, every example runs the exact same five lines. Pin them once:

Here $Φ$ is the "area-so-far under the bell curve" function (from Central Limit Theorem and Confidence Intervals). $Φ (1.96) \approx 0.975$ , so a $z$ of $1.96$ leaves $2.5$ in each tail, $5$ total. That single fact is why $1.96$ is the magic threshold.

Worked examples

C1 — A clear win

Worked example Statement (Cell C1)

A: $n_{A} = 20, 000$ , $x_{A} = 1, 000$ . B: $n_{B} = 20, 000$ , $x_{B} = 1, 150$ .

Forecast: B looks like $5.0$ vs $5.75$ . Guess: is that gap wide enough to trust? Jot yes/no before reading.

$\overset{p}{^}_{A} = 1000/20000 = 0.050$ , $\overset{p}{^}_{B} = 1150/20000 = 0.0575$ . Why this step? Turn raw counts into rates so the two groups are comparable.
Pooled $\overset{p}{^} = \frac{1000 + 1150}{40000} = 0.05375$ . Why this step? Under $H_{0}$ ("same true rate") the best estimate of that one rate uses all $40, 000$ users.
$SE = 0.05375 \cdot 0.94625 \cdot (1/20000 + 1/20000) \approx 0.002255$ . Why this step? This is the noise-width — the wobble we'd expect in the gap from luck alone.
$z = \frac{0.0575 - 0.050}{0.002255} \approx 3.326$ . Why this step? The gap is over three noise-widths — deep past the red tail at $1.96$ .
p-value $= 2 (1 - Φ (3.326)) \approx 0.00088$ . Why this step? Far below $0.05$ → reject $H_{0}$ , B is genuinely better.

Verify: $0.00088 ≪ 0.05$ and $∣ z ∣ = 3.33 > 1.96$ agree. Sanity: a $0.75$ -point lift on $20$ k users should be findable — it is.

C2 — A clear loss

Worked example Statement (Cell C2)

A: $n_{A} = 20, 000$ , $x_{A} = 1, 200$ . B: $n_{B} = 20, 000$ , $x_{B} = 1, 000$ .

Forecast: B is $5.0$ vs A's $6.0$ — B looks worse. Will $z$ come out negative, and does that change the decision? Guess.

$\overset{p}{^}_{A} = 0.060$ , $\overset{p}{^}_{B} = 0.050$ . Why this step? Same first move — measured rates.
Pooled $\overset{p}{^} = 2200/40000 = 0.055$ .
$SE = 0.055 \cdot 0.945 \cdot (2/20000) \approx 0.002280$ .
$z = \frac{0.050 - 0.060}{0.002280} \approx - 4.386$ . Why this step? The sign just flipped — negative $z$ means B fell below A. The two-sided test uses $∣ z ∣$ .
p-value $= 2 (1 - Φ (4.386)) \approx 1.15 \times 1 0^{- 5}$ . Why this step? Tiny p-value → reject $H_{0}$ , but the direction says keep A.

Verify: $∣ - 4.386∣ = 4.386 > 1.96$ , and the negative sign correctly points to A as the winner. A two-sided test flags "different"; you read the sign to know which way.

C3 — The underpowered look-alike

This is exactly Example 1 from the parent, re-derived as the "looks real but isn't proven" cell.

Worked example Statement (Cell C3)

A: $n_{A} = 10, 000$ , $x_{A} = 500$ . B: $n_{B} = 10, 000$ , $x_{B} = 560$ .

Forecast: $5.0$ vs $5.6$ — a nice-looking $+ 0.6$ lift. Ship it? Guess before computing.

$\overset{p}{^}_{A} = 0.050$ , $\overset{p}{^}_{B} = 0.056$ .
Pooled $\overset{p}{^} = 1060/20000 = 0.053$ .
$SE = 0.053 \cdot 0.947 \cdot (2/10000) \approx 0.003168$ .
$z = \frac{0.006}{0.003168} \approx 1.894$ . Why this step? $1.894$ sits just short of the $1.96$ red tail — inside the "could be luck" zone.
p-value $= 2 (1 - Φ (1.894)) \approx 0.0582$ . Why this step? $0.0582 > 0.05$ → do not reject. Same real-looking gap as C1 but half the users → not enough power.

Verify: $0.0582 > 0.05$ and $1.894 < 1.96$ agree — a hair from significant, which is precisely why peeking (see the parent's mistake list) is so tempting and so dangerous here. Contrast with C1: same lift proportionally, but larger $n$ shrank SE and crossed the line.

C4 — The null / A-A test

Worked example Statement (Cell C4)

Split traffic between the same model twice (an A/A test). A: $n_{A} = 15, 000$ , $x_{A} = 750$ . B: $n_{B} = 15, 000$ , $x_{B} = 759$ .

Forecast: There is no real difference by construction. What should the test say? Guess the decision.

$\overset{p}{^}_{A} = 0.050$ , $\overset{p}{^}_{B} = 0.0506$ .
Pooled $\overset{p}{^} = 1509/30000 = 0.05030$ .
$SE = 0.05030 \cdot 0.94970 \cdot (2/15000) \approx 0.002524$ .
$z = \frac{0.0506 - 0.050}{0.002524} \approx 0.2378$ . Why this step? A tiny gap gives a tiny $z$ , nowhere near $1.96$ — right in the fat middle of the bell curve.
p-value $= 2 (1 - Φ (0.2378)) \approx 0.812$ . Why this step? $0.812 ≫ 0.05$ → correctly fail to reject.

Verify: A/A tests should almost always fail to reject. If your pipeline flags an A/A split as "significant," the bug is in the splitter or logging, not the model — this is the sanity check the mnemonic "RUN A SAFE" bakes in as the A/A step. See Shadow Deployment and Offline vs Online Evaluation for related pre-launch sanity checks.

C5 — The degenerate zero-conversion group

Worked example Statement (Cell C5)

A brand-new feature converts nobody yet. A: $n_{A} = 2, 000$ , $x_{A} = 40$ . B: $n_{B} = 2, 000$ , $x_{B} = 0$ .

Forecast: $\overset{p}{^}_{B} = 0$ . Does the formula break? Does dividing by $\overset{p}{^} (1 - \overset{p}{^})$ blow up? Guess.

$\overset{p}{^}_{A} = 40/2000 = 0.020$ , $\overset{p}{^}_{B} = 0/2000 = 0.000$ . Why this step? $\overset{p}{^}_{B} = 0$ is a legal rate, not an error — nobody converted.
Pooled $\overset{p}{^} = 40/4000 = 0.010$ . Why this step? Crucially the pooled rate is not zero — it borrows A's conversions. This is exactly why we pool: it keeps the SE well-defined even when one arm is empty.
$SE = 0.010 \cdot 0.990 \cdot (2/2000) \approx 0.003146$ . Why this step? No division-by-zero, because $\overset{p}{^} = 0.01 \neq = 0$ . The individual $\overset{p}{^}_{B} = 0$ never enters the SE — only the pooled rate does.
$z = \frac{0 - 0.020}{0.003146} \approx - 6.356$ .
p-value $= 2 (1 - Φ (6.356)) \approx 2.1 \times 1 0^{- 10}$ . Why this step? Overwhelming evidence B is worse — reject $H_{0}$ , keep A.

Verify: The pooled-SE trick is what saves us: had we used $\overset{p}{^}_{B} (1 - \overset{p}{^}_{B}) = 0$ in an unpooled SE, the B-term would vanish and inflate $z$ . Because we pool, $SE = 0.003146$ stays finite and honest. Degenerate input handled.

C6 — The huge- $n$ significance trap

Worked example Statement (Cell C6)

A giant site. A: $n_{A} = 5, 000, 000$ , $x_{A} = 250, 000$ . B: $n_{B} = 5, 000, 000$ , $x_{B} = 251, 500$ .

Forecast: $5.000$ vs $5.030$ — a lift of just $0.03$ points. Significant? Worth shipping? Guess both.

$\overset{p}{^}_{A} = 0.05000$ , $\overset{p}{^}_{B} = 0.05030$ .
Pooled $\overset{p}{^} = 501500/10000000 = 0.05015$ .
$SE = 0.05015 \cdot 0.94985 \cdot (2/5000000) \approx 0.00013805$ . Why this step? With millions of users the noise-width is microscopic.
$z = \frac{0.0003}{0.00013805} \approx 2.173$ . Why this step? Even a $0.03$ -point gap clears $1.96$ once the noise is this small.
p-value $= 2 (1 - Φ (2.173)) \approx 0.0298$ . Why this step? Below $0.05$ → statistically significant. But a $0.03$ -point lift may be commercially meaningless.

Verify: $0.0298 < 0.05$ (significant) yet the lift is $0.6$ relative. This is the parent's warning "significant $\neq =$ important" made concrete — always pair the p-value with a confidence interval on the lift and ask if the effect size pays for the migration cost.

C7 — Planning the sample size

Worked example Statement (Cell C7)

Before launching, we want to catch a lift from $p_{A} = 0.05$ to $p_{B} = 0.055$ (MDE $δ = 0.005$ ), with $α = 0.05$ ( $z_{1 - α /2} = 1.96$ ) and power $0.8$ ( $z_{1 - β} = 0.84$ ).

Forecast: Guess the per-group size — hundreds? thousands? tens of thousands?

Sum of $z$ 's squared: $(1.96 + 0.84)^{2} = 2. 8^{2} = 7.84$ . Why this step? This term encodes "how strict + how sensitive" — bigger confidence and power push it up.
Variance term: $p_{A} (1 - p_{A}) + p_{B} (1 - p_{B}) = 0.05 \cdot 0.95 + 0.055 \cdot 0.945 = 0.0475 + 0.051975 = 0.099475$ . Why this step? The total per-user noise the effect must climb over.
$n = \frac{7.84 \cdot 0.099475}{0.00 5 ^{2}} = \frac{0.77988}{2.5 \times 1 0 ^{- 5}} \approx 31, 195$ . Why this step? The $δ^{2} = 2.5 \times 1 0^{- 5}$ in the denominator is why small effects need huge samples.

Verify: $\approx 31, 200$ per group, so $\approx 62, 400$ total. Halving the MDE to $0.0025$ would quadruple $n$ (the $1/ δ^{2}$ law) — the single most important intuition of experiment planning. If you can't afford this many users, consider Multi-Armed Bandits or a larger MDE.

C8 — Word problem with a guardrail (exam twist)

Worked example Statement (Cell C8)

A checkout team ships model B. Over one week: A served $30, 000$ carts, $2, 400$ completed a purchase; B served $30, 000$ carts, $2, 640$ purchased. But B's p99 latency rose $80 ms \to 240 ms$ . Is B a win?

Forecast: The purchase lift looks real. Does the latency change the answer? Guess.

$\overset{p}{^}_{A} = 2400/30000 = 0.080$ , $\overset{p}{^}_{B} = 2640/30000 = 0.088$ . Why this step? Convert cart counts to purchase rates — the OEC.
Pooled $\overset{p}{^} = 5040/60000 = 0.084$ .
$SE = 0.084 \cdot 0.916 \cdot (2/30000) \approx 0.002265$ .
$z = \frac{0.088 - 0.080}{0.002265} \approx 3.532$ . Why this step? $3.53$ noise-widths — well past $1.96$ .
p-value $= 2 (1 - Φ (3.532)) \approx 0.000412$ . Why this step? Highly significant conversion lift → the OEC says "ship."
Guardrail check: latency tripled. A $200 ms$ p99 regression usually reduces conversions elsewhere and hurts UX. Why this step? A single OEC never decides alone — guardrails can veto a statistically real win (see the parent's Example 3 and Model Monitoring & Drift).

Verify: Numerically B wins the OEC ( $p \approx 0.0004$ ), yet the correct engineering decision is hold or fix latency first. The exam trap is to stop at step 5. Significance is necessary, not sufficient.

Recall Which cell am I in? — quick router

Big gap, big $n$ , positive ::: C1, ship. Big gap, big $n$ , negative ::: C2, keep A. Nice gap but $∣ z ∣ < 1.96$ ::: C3, underpowered — collect more data, don't peek-and-stop. Same model twice ::: C4, A/A — must NOT reject; if it does, pipeline bug. One group has $0$ conversions ::: C5, pooled SE stays finite — formula holds. Significant but tiny lift on millions ::: C6, check effect size + CI, maybe skip. No data yet, planning ::: C7, use the $n \approx \frac{( z + z ) ^{2} [ \cdot ]}{δ ^{2}}$ formula. Win on OEC but a guardrail regressed ::: C8, guardrail can veto — don't ship.