A - B testing for models — Hinglish notes

5.3.14 · AI-ML › MLOps & Deployment

WHY karte hain hum A/B testing models ki?

Woh core danger jisse hum bachte hain: ek aisa difference dekhna jo sirf random luck hai. Agar B 100 users par 2% better lagta hai, woh easily noise ho sakta hai. Statistics humein batata hai ki kitne users chahiye aur hum kitne confident ho sakte hain.

WHAT hain iske pieces?

HOW decide karte hain? — Test ko scratch se derive karo

Maan lo metric ek conversion rate hai (har user ya toh convert karta hai ya nahi → ek Bernoulli trial). Group A mein $n_{A}$ users hain jinka sample conversion rate $\overset{p}{^}_{A}$ hai; group B mein $n_{B}$ hain, $\overset{p}{^}_{B}$ .

Step 1 — Ek user ko model karo. Group A mein har user unknown probability $p_{A}$ se convert karta hai. Ek single conversion $X_{i} \sim Bernoulli (p_{A})$ ka mean $p_{A}$ aur variance $p_{A} (1 - p_{A})$ hai.

Yeh step kyun? Kyunki haara raw data yes/no outcomes hain; Bernoulli ek coin-flip event ka honest model hai.

Step 2 — Users par average nikalo. Sample rate hai $\overset{p}{^}_{A} = \frac{1}{n _{A}} \sum X_{i}$ . Linearity se, $E [\overset{p}{^}_{A}] = p_{A}$ . Kyunki users independent hain, $Var (\overset{p}{^}_{A}) = \frac{p _{A} ( 1 - p _{A} )}{n _{A}} .$

Yeh step kyun? Independent variables ke sum ka variance add hota hai, aur $n_{A}$ se divide karne par variance $1/ n_{A}^{2}$ se scale hota hai — yahi reason hai ki zyada users noise ko shrink karte hain.

Step 3 — Do groups ka difference. Hum $\overset{p}{^}_{B} - \overset{p}{^}_{A}$ ki parwah karte hain. Independent groups ke liye variances add hote hain: $Var (\overset{p}{^}_{B} - \overset{p}{^}_{A}) = \frac{p _{A} ( 1 - p _{A} )}{n _{A}} + \frac{p _{B} ( 1 - p _{B} )}{n _{B}} .$

Step 4 — Standardize karo (Central Limit Theorem). Bade $n$ ke liye, $\overset{p}{^}_{B} - \overset{p}{^}_{A}$ approximately Normal hota hai. $H_{0}$ ke under true difference $0$ hai, isliye test statistic hai

Pooled kyun? Null hypothesis ke under dono ka same $p$ hota hai, toh us shared $p$ ka best estimate saara data use karke milta hai. Yeh "no difference" wali duniya ke liye sabse accurate standard error deta hai.

Step 5 — Decide karo. Two-sided p-value $= 2 (1 - Φ (∣ z ∣))$ compute karo. Agar $p -value < α$ hai, $H_{0}$ reject karo → B significantly different hai. Practical significance judge karne ke liye lift par confidence interval ke saath combine karo.

Sample size derive karna (WHY experiment itna lamba lagta hai)

Significance $α$ aur power $1 - β$ ke saath lift $δ = p_{B} - p_{A}$ (the MDE) detect karne ke liye, signal $δ$ ko noise se upar stand out karna hoga. Classic per-group formula:

Ise intuition se padho: $n$ tab badhta hai jab aap zyada confidence maango ( $z$ terms upar), jab metric noisier ho (variance upar), aur blast ho jaata hai jab $δ \to 0$ (tiny effects ko huge samples chahiye — woh $1/ δ^{2}$ ).

Worked examples

Common mistakes (Steel-manned)

Recall Feynman: ek 12-saal ke bachche ko explain karo

Socho ek hi street par do lemonade stands hain. Stand A Grandma ki recipe use karta hai, Stand B aapki nayi recipe. Aap random bachche dono stands par bhejte ho aur count karte ho kitne wapas kharidne aate hain. Agar bahut zyada B wapas aate hain — lekin sirf tab jab kaafi saare bachche dono par gaye hon — tab aap trust kar sakte ho ki nayi recipe sach mein zyada tasty hai, na ki sirf kuch pyaase bachche B par chale gaye. "Kaafi saare bachche" wala part math hai: chhoti crowds aapko luck se bewakoof bana sakti hai, badi crowds sach batati hai.

Flashcards

Model A/B test mein control group kya hota hai?

Currently-deployed baseline model (A), jiske against naye candidate (B) ko compare kiya jaata hai.

Request ke bajaye user ke hisaab se randomize kyun karte hain?

Har user ko consistent experience dene ke liye aur observations independent rakhne ke liye; per-request splitting ek user ke liye arms mix kar deti hai aur outcomes correlate ho jaate hain.

Two-proportion z-test mein pooled rate

\overset{p}{^}

kyun use karte hain?

H_{0}

ke under dono groups same true conversion rate share karte hain, isliye saara data pool karna common

p

ka best estimate standard error ke liye deta hai.

Two-proportion z statistic batao.

z = \frac{p ^ _{B} - p ^ _{A}}{p ^ ( 1 - p ^ ) ( 1/ n _{A} + 1/ n _{B} )}

jahan

\overset{p}{^}

pooled rate hai.

Required sample size minimum detectable effect

δ

ke saath kaise scale karta hai?

1/ δ^{2}

ki tarah — jis effect ko detect karna ho usse aadha karne par roughly sample size chaar guna ho jaata hai.

Statistical power kya hoti hai?

1 - β

, specified size ke true effect ko correctly detect karne ki probability; usually

0.80

target ki jaati hai.

'Peeking aur early stopping' problem kyun hai?

Repeated significance checks har baar false-positive ka mauka add karte hain, overall Type-I error ko

α

se upar inflate kar dete hain; fixed samples ya sequential methods use karo.

Guardrail metrics kya hote hain?

Secondary metrics (latency, error rate, revenue) jo regress nahi hone chahiye, chahe primary OEC improve ho.

Significant par tiny lift — ship karein ya nahi?

Automatically nahi; practical significance effect size / confidence interval aur guardrails se check karo, kyunki significance ≠ importance.

A/A test kis kaam aata hai?

Ek sanity check hai jo same model par identical traffic split karta hai, yeh validate karne ke liye ki experiment pipeline koi false difference nahi deta.

Connections

Hypothesis Testing · Confidence Intervals · Central Limit Theorem
Bernoulli and Binomial Distributions
Model Monitoring & Drift · Shadow Deployment · Canary Releases
Multi-Armed Bandits (fixed A/B ka adaptive alternative)
Offline vs Online Evaluation