A - B testing for models: Question bank — AI-ML deep dive

The symbols before we trap you

Every trap below uses a handful of symbols. This page is standalone, so here they are in plain words — nothing appears in a question that isn't defined here first.

Definition The vocabulary (read once, then the traps make sense)

$\overset{p}{^}_{A}, \overset{p}{^}_{B}$ — the measured conversion rate in arm A (old model) and arm B (new model): (users who converted) $\div$ (users in that arm). The hat means "estimated from data," not the true unknown rate $p$ .
$n_{A}, n_{B}$ — how many users landed in each arm.
$x_{A}, x_{B}$ — the raw count of conversions (the number of users who said "yes") in arm A and arm B. So by definition $\overset{p}{^}_{A} = x_{A} / n_{A}$ and $\overset{p}{^}_{B} = x_{B} / n_{B}$ .
SE (standard error) — the typical wobble of $\overset{p}{^}_{B} - \overset{p}{^}_{A}$ if you re-ran the same experiment many times. Think "the width of the noise cloud." Small SE = trustworthy number.
$z$ — the test statistic: how many SEs the observed gap $\overset{p}{^}_{B} - \overset{p}{^}_{A}$ sits away from zero. $z = 2$ means "two noise-widths out," which is surprising under "no difference."
$z_{1 - α}$ and $z_{1 - β}$ — Normal quantiles: the point on the standard bell curve with that much area to its left. $z_{1 - α}$ is "how many SEs out you must be before you cry significance" (e.g. $z_{0.975} = 1.96$ for two-sided $α = 0.05$ ); $z_{1 - β}$ is "how many SEs of headroom you need for the desired power" (e.g. $z_{0.8} = 0.84$ ). They are the dials that trade confidence and power for sample size.
$H_{0}$ — the null hypothesis: "A and B have the same true rate, $p_{A} = p_{B}$ ." The boring world we try to rule out.
$α$ — the false-positive budget: the chance we're willing to accept of shouting "B is better!" when it truly isn't. Usually $0.05$ (5%).
$β, 1 - β$ — $β$ is the chance of missing a real effect (false negative); $1 - β$ is the power, the chance of catching it. Usually power $= 0.8$ .
$δ$ (delta) — the effect size $p_{B} - p_{A}$ we hope to see. The smallest such $δ$ worth acting on is the MDE (Minimum Detectable Effect).
OEC — Overall Evaluation Criterion: the single business metric you decide on (e.g. conversion rate).
Guardrail metrics — safety metrics (latency, error rate, revenue-per-user) that must not get worse, even if the OEC improves.

Where SE $\propto 1/ n$ comes from (so you can defend it)

A single user is a coin flip: convert (1) or not (0), a Bernoulli trial with variance $p (1 - p)$ . Averaging $n$ independent flips, variances add and the $1/ n$ from averaging squares in: $Var (\overset{p}{^}) = \frac{1}{n ^{2}} \sum_{i = 1}^{n} Var (X_{i}) = \frac{n p ( 1 - p )}{n ^{2}} = \frac{p ( 1 - p )}{n} \Rightarrow SE = \frac{p ( 1 - p )}{n} \propto \frac{1}{n} .$ So the noise shrinks like $1/ n$ — quadruple the users to halve it. See Central Limit Theorem (the reason that average is even Normal-shaped) and Bernoulli and Binomial Distributions (the reason a user's variance is $p (1 - p)$ ).

Building $z$ , and why we pool under $H_{0}$

Two independent arms → variances add: $Var (\overset{p}{^}_{B} - \overset{p}{^}_{A}) = \frac{p ( 1 - p )}{n _{A}} + \frac{p ( 1 - p )}{n _{B}}$ . Under $H_{0}$ both arms share one rate $p$ , so our best estimate uses all the data — the pooled rate $\overset{p}{^} = \frac{x _{A} + x _{B}}{n _{A} + n _{B}}$ , i.e. total conversions divided by total users. Then $z = \frac{p ^ _{B} - p ^ _{A}}{p ^ ( 1 - p ^ ) ( \frac{1}{n _{A}} + \frac{1}{n _{B}} )} .$ Pooling is optimal for the null world because if $p_{A} = p_{B}$ is true, splitting the estimate wastes data and inflates the SE. See Hypothesis Testing for why we always model the null world first.

Where $n \propto 1/ δ^{2}$ comes from

For the sample-size calculation we no longer live in the null world — we plan for a world where the effect is real, so the two arms genuinely have different rates $p_{A}$ and $p_{B}$ . That is why the variance term here is the unpooled sum $p_{A} (1 - p_{A}) + p_{B} (1 - p_{B})$ : each arm keeps its own Bernoulli variance ( $p (1 - p)$ per user) and, because the arms are independent, their variances add (exactly as in the "Building $z$ " step, but now with two different rates because we expect a difference). To flag such an effect we need $z$ to clear a threshold: roughly $δ \geq (z_{1 - α} + z_{1 - β}) \cdot SE$ . Substitute $SE \propto 1/ n$ , square both sides, solve for $n$ : $n \approx \frac{( z _{1 - α} + z _{1 - β} ) ^{2} [ p _{A} ( 1 - p _{A} ) + p _{B} ( 1 - p _{B} ) ]}{δ ^{2}} .$ The $δ^{2}$ in the denominator is the punchline: halving the effect you chase quadruples the users — small effects are expensive.

One-sided vs two-sided (a whole family of traps lives here)

True or false — justify

A test that reaches

p < 0.05

proves B is better than A.

False. A small p-value means the data are unlikely under $H_{0}$ ; it never "proves" the alternative and says nothing about effect size — it could be a tiny, worthless lift on millions of users. See Hypothesis Testing.

If the 95% confidence interval for the lift includes

0

, the result is not statistically significant at

α = 0.05

True. The two-sided CI and the two-sided test are two views of the same computation; a CI covering

0

is exactly the case where we fail to reject

H_{0}

. See Confidence Intervals.

Doubling both group sizes halves the standard error of

\overset{p}{^}_{B} - \overset{p}{^}_{A}

False. Since

SE \propto 1/ n

, doubling

n

divides SE by

2 \approx 1.41

, not by

2

. You need four times the users to halve the noise.

Under the null hypothesis you should still estimate each group's variance with its own separate rate.

False. Under

H_{0}

both groups share one true

p

, so the pooled rate gives the most accurate standard error for the "no difference" world — that's the whole reason pooling exists.

A statistically significant result guarantees the same effect will appear when you ship to 100% of users.

False. Novelty effects, seasonality, and a shifting population (Model Monitoring & Drift) can all make the launched effect shrink or vanish; the experiment estimates the effect during the test window, not forever.

The p-value is the probability that

H_{0}

is true.

False. It's the probability of seeing data at least this extreme assuming $H_{0}$ is true — a conditional on the data, not a probability of the hypothesis. Confusing the two is the classic inversion error.

An A/A test (both arms run the identical model) should still reject

H_{0}

about 5% of the time at

α = 0.05

True. With no real difference, false positives occur at exactly the rate

α

— that's what

α

means. An A/A test that rejects far more often reveals a broken pipeline or bad randomization.

Higher power always means a smaller required sample size.

False. Higher power (

1 - β

) means detecting real effects more reliably, which requires a larger

z_{1 - β}

term and therefore more users, not fewer.

A one-sided test at

α = 0.05

needs a larger

z

to reach significance than a two-sided test at the same

α

False. One-sided puts all

0.05

in one tail (critical

z = 1.645

); two-sided splits it (

z = 1.96

). The one-sided bar is lower, which is exactly why post-hoc switching to it is cheating.

If B wins on the primary OEC, guardrail metrics can be ignored.

False. A conversion win with a latency or error-rate regression is often a net loss; guardrails must not regress even when the OEC improves.

The Central Limit Theorem lets us use a z-test even though individual outcomes are just 0/1 Bernoulli values.

True. The sum/average of many independent Bernoulli trials is approximately Normal for large

n

, which is precisely what the z-test needs — the individual values needn't be Normal. See Central Limit Theorem.

If B beats A in every user segment separately, it must beat A overall.

False. Segment sizes can differ between arms, so the pooled totals can flip the winner — Simpson's paradox. Always check that randomization balanced the segments before trusting the aggregate.

Spot the error

"Users kept switching between A and B mid-session but each visit is one independent sample, so it's fine."

The error is broken independence. Per-request splitting correlates one user's outcomes and mixes their experience; randomize by user ID hashed to a sticky bucket so each user stays in one arm.

"We hit significance on day 2, so we stopped early to ship faster."

This is the peeking / multiple-comparisons error. Every extra look is another chance for noise to cross the line, inflating the true false-positive rate above

α

. Fix the sample size in advance or use sequential/alpha-spending methods.

H_{0}

was not rejected, so we've proven A and B are equal."

Failing to reject is not evidence of equality — it may just mean the test was underpowered to see a real effect. Absence of evidence is not evidence of absence; report the CI so readers see how large an effect you could have missed.

"We set the MDE tiny (

δ = 0.0001

) to be safe, and the sample size came out to millions."

The error is treating "safe" as "smaller MDE." Because

n \propto 1/ δ^{2}

, an unnecessarily small MDE explodes the sample size; the MDE should be the smallest lift worth acting on, not the smallest imaginable.

"We used the separate (unpooled) SE for the test statistic and the pooled rate for the sample-size formula, mixing them freely."

The test statistic under

H_{0}

should use the pooled SE (both arms share

p

under the null); the sample-size formula uses the anticipated

p_{A}, p_{B}

because it reasons about a world where the effect is real. Swapping them gives wrong significance.

"The metric is average revenue per user, which is roughly Bernoulli, so a two-proportion z-test fits."

Revenue is a continuous, often heavy-tailed variable, not a 0/1 outcome — a two-proportion test is only for conversion-style Bernoulli metrics. Use a t-test / Mann–Whitney or a CLT-based mean comparison instead. See Bernoulli and Binomial Distributions.

"Overall B looked worse, but we sliced by device and B won on mobile and won on desktop, so B is actually better — the overall number is a bug."

Not a bug — it's Simpson's paradox. Unequal segment sizes across arms can make each slice favour B while the aggregate favours A; fix the imbalance (or report segment-weighted lift), don't dismiss the total.

Why questions

Why does the required sample size explode as the effect

δ

shrinks?

Because signal-to-noise scales like

δ / SE

and SE shrinks only as

1/ n

; solving for the

n

that keeps a tiny

δ

above the noise gives

n \propto 1/ δ^{2}

, so halving

δ

quadruples the users.

Why randomize at all instead of just deploying B to everyone and comparing to last week's numbers?

Randomization removes confounders (seasonality, marketing, drift) so the only systematic difference between arms is the model — that's what turns a correlation into causal evidence that B caused the change.

Why can a result be statistically significant yet practically worthless?

With enough users the SE becomes tiny, so even a negligible lift (say

0.01%

) clears the significance threshold; significance measures "is it non-zero," while practical importance needs the effect size and CI.

Why prefer a confidence interval over reporting only "significant / not significant"?

The interval shows the range of plausible effects and its practical scale, whereas a single verdict hides whether the effect is huge, trivial, or uncertain. It answers "how much" not just "any at all." See Confidence Intervals.

Why does a shadow or canary rollout not replace an A/B test?

Shadow Deployment and Canary Releases check that B runs safely (latency, errors, no crashes) on real traffic, but they don't randomize users to measure a business-metric lift — you still need the controlled experiment for causal comparison.

Why might a multi-armed bandit be chosen instead of a fixed A/B split?

A bandit shifts traffic toward the better arm during the experiment to reduce regret, trading clean fixed-horizon inference for faster exploitation — useful when the cost of showing the worse arm is high.

Why choose a two-sided test by default even when you "only care if B is better"?

Because a surprising regression (B much worse) is also information you must not ignore, and defaulting two-sided removes the temptation to pick the easier one-sided bar after peeking. See Hypothesis Testing.

Edge cases

n_{A} = n_{B}

but

\overset{p}{^}_{A} = \overset{p}{^}_{B}

exactly, what is

z

z = 0

: the numerator

\overset{p}{^}_{B} - \overset{p}{^}_{A}

is zero, so no evidence against

H_{0}

regardless of sample size — the p-value is

1

What happens to the pooled SE when the observed conversion rate is

0

in both arms?

With

x_{A} = x_{B} = 0

the pooled rate is

\overset{p}{^} = 0

, so

\overset{p}{^} (1 - \overset{p}{^}) = 0

, making the SE

0

and

z

undefined (

0/0

). The Normal approximation fails at the boundary; use an exact (Fisher/binomial) test for rare events.

What about the symmetric boundary — every user converts, so

\overset{p}{^} = 1

in both arms?

With

x_{A} = n_{A}

and

x_{B} = n_{B}

the pooled rate is

\overset{p}{^} = 1

, so

\overset{p}{^} (1 - \overset{p}{^}) = 0

again, SE

= 0

and

z

undefined. Near

p = 1

the noise cloud is one-sided (rates can't exceed 1), so the symmetric Normal approximation is invalid — use an exact test.

The CLT-based z-test assumes "large

n

" — when does it break down?

When

n \overset{p}{^}

n (1 - \overset{p}{^})

is small (a common rule: fewer than

\sim 5

–

10

expected successes or failures), the sampling distribution is too skewed for the symmetric Normal to fit — see the skew in the figure below.

What if a real effect exists but you set power to only

0.5

You'll miss the effect half the time — a coin-flip chance of a false negative — so a "not significant" outcome is nearly uninformative. Low power wastes the experiment even when B truly is better.

If you run the test over a period shorter than one full week, what bias can appear?

Weekly seasonality: weekday and weekend users behave differently, so a partial-week window over- or under-samples one behaviour and biases the estimate. Cover whole weeks to average out the cycle.

If offline evaluation already showed B has higher AUC, is the online A/B test redundant?

No. Offline metrics are measured on frozen history and on a proxy loss, not the live business OEC under drift; only the online test gives causal evidence on the metric that pays the bills.

Recall One-line self-check

Cover this and state each in your own words: (1) why pooling under $H_{0}$ , (2) why $n \propto 1/ δ^{2}$ , (3) why peeking is dangerous, (4) why per-user randomization, (5) why significant $\neq =$ shippable, (6) how Simpson's paradox flips a segmented result. If you can justify all six, you've internalised the traps.