A - B testing for models: Exercises — AI-ML deep dive

Two constants recur, so let us anchor them once, in plain words, before any symbol is used.

Level 1 — Recognition

Exercise 1.1 — Name the pieces

In a test, the currently deployed model serves half the users and a new candidate serves the other half. The team splits traffic by hashing the user ID. They watch conversion rate and reject the "no difference" idea only if the false-alarm rate stays under $5%$ .

Identify: (a) control, (b) treatment, (c) randomization unit, (d) $α$ , (e) the null hypothesis in symbols.

Recall Solution 1.1

(a) Control (A) = the currently deployed model.
(b) Treatment (B) = the new candidate model.
(c) Randomization unit = the user (they hash the user ID).
(d) $α = 0.05$ — the tolerated false-positive rate.
(e) $H_{0} : p_{A} = p_{B}$ — the two models have the same true conversion probability.

Exercise 1.2 — Read the sample-size formula

Look at $n \approx \frac{( z _{1 - α /2} + z _{1 - β} ) ^{2} [ p _{A} ( 1 - p _{A} ) + p _{B} ( 1 - p _{B} ) ]}{δ ^{2}}$ . Without computing anything, say what happens to $n$ if you halve the MDE $δ$ .

Recall Solution 1.2

$n$ depends on $1/ δ^{2}$ . Halving $δ$ means $δ^{2}$ becomes one quarter, so $1/ δ^{2}$ becomes four times larger. $n$ quadruples. Tiny effects are enormously expensive to detect — this is the single most important intuition in test design.

Level 2 — Application

Exercise 2.1 — Compute a pooled rate and standard error

Group A: $n_{A} = 8000$ , $x_{A} = 400$ conversions. Group B: $n_{B} = 8000$ , $x_{B} = 448$ conversions. Compute (a) $\overset{p}{^}_{A}$ , (b) $\overset{p}{^}_{B}$ , (c) the pooled rate $\overset{p}{^}$ , (d) the standard error $SE = \overset{p}{^} (1 - \overset{p}{^}) (1/ n_{A} + 1/ n_{B})$ .

Recall Solution 2.1

(a) $\overset{p}{^}_{A} = 400/8000 = 0.050$ .
(b) $\overset{p}{^}_{B} = 448/8000 = 0.056$ .
(c) $\overset{p}{^} = \frac{400 + 448}{16000} = \frac{848}{16000} = 0.053$ . Why pooled? under $H_{0}$ both arms share one true rate, so use all the data to estimate it.
(d) $1/ n_{A} + 1/ n_{B} = 0.000125 + 0.000125 = 0.00025$ . Then $SE = 0.053 \cdot 0.947 \cdot 0.00025 = 1.25478 \times 1 0^{- 5} \approx 0.003542.$

Exercise 2.2 — Finish the z-test

Using Exercise 2.1's numbers, compute the z-statistic $z = (\overset{p}{^}_{B} - \overset{p}{^}_{A}) / SE$ and the two-sided p-value $2 (1 - Φ (∣ z ∣))$ . At $α = 0.05$ , do you reject $H_{0}$ ?

Recall Solution 2.2

$\overset{p}{^}_{B} - \overset{p}{^}_{A} = 0.056 - 0.050 = 0.006$ .
$z = 0.006/0.003542 \approx 1.694$ .
Two-sided p-value $= 2 (1 - Φ (1.694)) \approx 2 (1 - 0.9549) = 2 (0.0451) \approx 0.0902$ .
Since $0.0902 > 0.05$ , do NOT reject $H_{0}$ . The $0.6%$ lift is not yet significant — same story as the parent note's Example 1, but with fewer users the noise is even larger.

Level 3 — Analysis

Exercise 3.1 — Design the sample size

You want to detect a lift from $p_{A} = 0.04$ to $p_{B} = 0.045$ (so $δ = 0.005$ ) with $α = 0.05$ ( $z_{1 - α /2} = 1.96$ ) and power $0.8$ ( $z_{1 - β} = 0.84$ ). Compute the required $n$ per group.

Recall Solution 3.1

$n \approx \frac{( 1.96 + 0.84 ) ^{2} [ 0.04 \cdot 0.96 + 0.045 \cdot 0.955 ]}{0.00 5 ^{2}} .$

$(1.96 + 0.84)^{2} = 2. 8^{2} = 7.84$ .
$0.04 \cdot 0.96 = 0.0384$ ; $0.045 \cdot 0.955 = 0.042975$ ; sum $= 0.081375$ .
$δ^{2} = 0.00 5^{2} = 2.5 \times 1 0^{- 5}$ .
$n \approx \frac{7.84 \cdot 0.081375}{2.5 \times 1 0 ^{- 5}} = \frac{0.6379...}{2.5 \times 1 0 ^{- 5}} \approx 25, 519$ per group.

So $\approx 25, 519$ users each side — round up to be safe, e.g. $25, 600$ .

Exercise 3.2 — Confidence interval on the lift

Return to Exercise 2.1's data. Build a $95%$ confidence interval for the true lift $p_{B} - p_{A}$ . Here you must use the un-pooled standard error $SE_{diff} = \frac{p ^ _{A} ( 1 - p ^ _{A} )}{n _{A}} + \frac{p ^ _{B} ( 1 - p ^ _{B} )}{n _{B}}$ and the multiplier $1.96$ . Does the interval contain $0$ , and does that agree with Exercise 2.2?

Recall Solution 3.2

$\frac{0.05 \cdot 0.95}{8000} = \frac{0.0475}{8000} = 5.9375 \times 1 0^{- 6}$ .
$\frac{0.056 \cdot 0.944}{8000} = \frac{0.052864}{8000} = 6.608 \times 1 0^{- 6}$ .
Sum $= 1.25455 \times 1 0^{- 5}$ ; $SE_{diff} = 1.25455 \times 1 0^{- 5} \approx 0.003542$ .
Interval: $0.006 \pm 1.96 \cdot 0.003542 = 0.006 \pm 0.006943$ , i.e. $[- 0.000943, 0.012943]$ .
The interval contains $0$ . This agrees with Exercise 2.2 (we did not reject $H_{0}$ ): whenever a two-sided $95%$ CI on the difference covers $0$ , the two-sided test at $α = 0.05$ fails to reject. See Confidence Intervals for why these two views are the same coin.

Level 4 — Synthesis

Exercise 4.1 — Peeking inflates false positives

A team runs an A/A test (both arms are the same model, so the true lift is exactly $0$ ). They peek $k = 5$ times during the run, each time doing an independent-looking test at $α = 0.05$ , and they stop and declare a winner the first time any peek shows $p < 0.05$ . If the peeks were independent, what is the probability they falsely declare a winner at least once? Compare to the intended $5%$ .

Recall Solution 4.1

The probability a single peek does not trigger is $1 - 0.05 = 0.95$ . For $5$ independent peeks all staying quiet: $P (no false alarm) = 0.9 5^{5} \approx 0.7738.$ So $P (at least one false alarm) = 1 - 0.7738 \approx 0.2262$ , i.e. $\approx 22.6%$ — over four times the intended $5%$ .

(Real peeks are correlated because they share accumulated data, so the true inflation is somewhat less than this independent upper bound — but it is always above $5%$ , and it grows with the number of looks.) Fix: fix $n$ in advance, or use sequential/alpha-spending methods. This is exactly the peeking mistake in A - B testing for models.

Exercise 4.2 — Guardrails override significance

Model B lifts conversion from $5.0%$ to $5.15%$ , and with $n = 200, 000$ per arm this is statistically significant. But B's p99 latency rises $80 ms \to 260 ms$ and error rate goes $0.2% \to 0.5%$ . The OEC is conversion; latency and errors are guardrails that must not regress by more than $10%$ relative. Decide: ship or not, and justify with numbers.

Recall Solution 4.2

Conversion lift (relative) $= (0.0515 - 0.050) /0.050 = 0.03 = + 3%$ — significant and positive. ✅
Latency guardrail: relative change $= (260 - 80) /80 = 2.25 = + 225%$ — vastly exceeds the $10%$ regression budget. ❌
Error-rate guardrail: relative change $= (0.005 - 0.002) /0.002 = 1.5 = + 150%$ . ❌

Decision: do NOT ship. A single OEC win cannot buy back massive guardrail regressions — statistical significance $\neq =$ shippability. Investigate why B is slower/erroring (perhaps a heavier model needing optimization) before re-running. Relatedly, watch Model Monitoring & Drift post-launch.

Level 5 — Mastery

Exercise 5.1 — Full experiment design from a brief

You currently convert at $p_{A} = 0.10$ . Product wants to ship any model that lifts conversion by at least $1$ absolute percentage point ( $δ = 0.01$ , so $p_{B} = 0.11$ ). You get $50, 000$ eligible users per day, split $50/50$ . Use $α = 0.05$ and power $0.8$ . (a) Required $n$ per group. (b) How many days must the test run? (c) Name one reason you should still run at least a full week even if the math says fewer days.

Recall Solution 5.1

(a) Sample size: $n \approx \frac{( 1.96 + 0.84 ) ^{2} [ 0.10 \cdot 0.90 + 0.11 \cdot 0.89 ]}{0.0 1 ^{2}} .$

$7.84$ as before.
$0.10 \cdot 0.90 = 0.09$ ; $0.11 \cdot 0.89 = 0.0979$ ; sum $= 0.1879$ .
$δ^{2} = 1 \times 1 0^{- 4}$ .
$n \approx \frac{7.84 \cdot 0.1879}{1 0 ^{- 4}} = \frac{1.473136}{1 0 ^{- 4}} \approx 14, 731$ per group.

(b) You get $50, 000$ /day split into two arms $= 25, 000$ per arm per day. Days $= 14, 731/25, 000 \approx 0.59$ day. So in raw traffic terms under one day suffices.

(c) Still run $\geq 7$ days because: weekly seasonality (weekend users behave differently from weekday users) and novelty/primacy effects (users react to change itself in the first days). A sub-day test would sample only one slice of behavior and could mislead. See Offline vs Online Evaluation for why online timing matters.

Exercise 5.2 — When A/B is the wrong tool

For each scenario, choose A/B test vs. an alternative from the vault, and justify in one sentence. (i) You must rank $8$ candidate creatives and keep earning while learning which is best. (ii) You want to run the new model on live traffic but never show its output to users, just to check it doesn't crash. (iii) You want to expose the new model to $2%$ of users first, then ramp to $100%$ if healthy.

Recall Solution 5.2

(i) Multi-armed bandit (Multi-Armed Bandits) — it adaptively shifts traffic toward winners, minimizing regret while exploring, which fits "keep earning while learning" better than a fixed-split A/B.
(ii) Shadow deployment (Shadow Deployment) — the model scores real traffic but its outputs are discarded, so you validate stability and latency with zero user risk. A/B would expose users.
(iii) Canary release (Canary Releases) — a staged ramp ( $2% \to \dots \to 100%$ ) with automatic rollback on guardrail breach, which is about safe rollout, not statistical comparison.

A/B testing answers "is B statistically better on the OEC?" — a different question from "is B safe to roll out?" (canary/shadow) or "how do I optimally allocate traffic across many arms?" (bandit).