The core danger we defend against: seeing a difference that is just random luck. If B looks 2% better on 100 users, that could easily be noise. Statistics tells us how many users we need and how confident we can be.
Suppose the metric is a conversion rate (each user either converts or not → a Bernoulli trial). Group A has nA users with sample conversion rate p^A; group B has nB, p^B.
Step 1 — Model one user. Each user in group A converts with unknown probability pA. A single conversion Xi∼Bernoulli(pA) has mean pA and variance pA(1−pA).
Why this step? Because our raw data are yes/no outcomes; Bernoulli is the honest model of a coin-flip event.
Step 2 — Average over users. The sample rate is p^A=nA1∑Xi. By linearity, E[p^A]=pA. Because the users are independent,
Var(p^A)=nApA(1−pA).
Why this step? Variance of a sum of independent variables adds, and dividing by nA scales variance by 1/nA2 — this is why more users shrink noise.
Step 3 — Difference of two groups. We care about p^B−p^A. For independent groups variances add:
Var(p^B−p^A)=nApA(1−pA)+nBpB(1−pB).
Step 4 — Standardize (Central Limit Theorem). For large n, p^B−p^A is approximately Normal. Under H0 the true difference is 0, so the test statistic is
Why pooled? Under the null hypothesis they have the samep, so our best estimate of that shared p uses all the data. This gives the most accurate standard error for the "no difference" world.
Step 5 — Decide. Compute the two-sided p-value=2(1−Φ(∣z∣)). If p-value<α, reject H0 → B is significantly different. Combine with a confidence interval on the lift to judge practical significance.
To detect a lift δ=pB−pA (the MDE) with significance α and power 1−β, we need the signal δ to stand out above the noise. The classic per-group formula:
Read it as intuition: n grows when you demand more confidence (z terms up), when the metric is noisier (variance up), and explodes as δ→0 (tiny effects need huge samples — the 1/δ2).
Imagine two lemonade stands on the same street. Stand A uses Grandma's recipe, Stand B uses your new recipe. You send random kids to each stand and count how many come back to buy again. If way more come back to B — but only after lots of kids visited both — you can trust the new recipe is really tastier and not just that a few thirsty kids happened to pick B. The "lots of kids" part is the math: small crowds can fool you by luck, big crowds tell the truth.
Dekho, A/B testing ka matlab hai — apna purana model (A, yani control) aur naya model (B, yani treatment) dono ko live traffic pe chalao, lekin users ko randomly baant do. Fir dekho ki B se business metric (jaise conversion rate ya click) sach mein badha ya sirf luck tha. Offline test set pe AUC accha hona kaafi nahi hota, kyunki real duniya me users alag behave karte hain aur jo metric paisa deta hai wo aksar training loss se alag hota hai.
Core statistics simple hai: har user ek coin-flip jaisa hai (convert hua ya nahi = Bernoulli). Jitne zyada users, utna noise kam hota hai — variance p(1−p)/n hoti hai, isliye n badhne pe error shrink hota hai. Do groups ka difference nikaal ke usko standard error se divide karo, wo z statistic ban jaata hai. Agar p-value <0.05, tab bolo ki B genuinely alag hai.
Ek important cheez: sample size pehle hi calculate karo, kyunki chhota effect (δ) detect karne ke liye 1/δ2 ke hisaab se bahut saare users chahiye. Aur peeking mat karo — roz roz check karke jaise hi p<0.05 dikhe stop kar dena galti hai, kyunki har baar dekhne se false-positive chance badhta hai. Randomization hamesha user ke basis pe karo, request pe nahi, warna ek hi user A aur B dono me ghusega.
Aur haan, sirf significant hona kaafi nahi — guardrail metrics (latency, error rate, revenue) bhi check karo. Agar conversion 3% badha par latency 80ms se 300ms ho gayi, to B ship mat karo. Yaad rakho: statistical significance ka matlab "sach me change hua" hai, "important change hua" nahi.
