Bayesian statistics — prior, likelihood, posterior (intro): Exercises

4.9.24 · D4 · Maths › Probability Theory & Statistics › Bayesian statistics — prior, likelihood, posterior (intro)

Shuru karne se pehle, hamare chaar characters ki ek reminder (hum inhe bina kisi apology ke use karte rahenge):

Level 1 — Recognition

Goal: pieces ko naam do. Ek fraction se aage koi arithmetic nahi.

Exercise 1.1. Is sentence mein "I believed a coin was fair, then I flipped 8 heads in 10, and now I think it's biased," label karo ki kaun sa clause prior hai, kaun sa likelihood-driven evidence hai, aur kaun sa posterior hai.

Recall Solution 1.1

Prior = "I believed a coin was fair" — data se pehle belief, to $P (θ)$ $θ = 0.5$ par peaked tha.
Likelihood = "I flipped 8 heads in 10" — yeh data $D$ hai, aur poochna ki har $θ$ ke liye yeh kitna probable hai woh hai $P (D ∣ θ)$ .
Posterior = "now I think it's biased" — data ke baad updated belief, $P (θ ∣ D)$ bade $θ$ ki taraf shift ho gayi.

Exercise 1.2. Inme se kaun sa $θ$ par ek probability distribution hai ( $θ$ mein $1$ tak integrate hoti hai): prior $P (θ)$ , likelihood $P (D ∣ θ)$ , ya posterior $P (θ ∣ D)$ ? Jo bhi apply hote hain sab chuno.

Recall Solution 1.2

Prior $P (θ)$ aur posterior $P (θ ∣ D)$ $θ$ par genuine distributions hain — dono $1$ tak integrate hote hain. Likelihood $P (D ∣ θ)$ nahi hai: yeh $θ$ ka ek function hai lekin data ki probability hai. Iska $θ$ par $1$ tak integrate hone ka koi reason nahi. (Yeh poore subject mein sabse common confusion hai — frequentist cousin ke liye Maximum Likelihood Estimation note dekho jo bas is function ko maximise karta hai.)

Exercise 1.3. Cloze fill karo. Identity ==posterior $\propto$ likelihood $\times$ prior== kaun sa term drop karti hai, aur drop karna kyun allowed hai?

Recall Solution 1.3

Yeh evidence $P (D)$ drop karti hai. Drop karna allowed hai kyunki $P (D)$ $θ$ par depend nahi karta — $θ$ mein posterior ki shape dhundhne ke purpose ke liye, yeh bas ek constant hai. Tum isse bilkul end mein normalise karke wapas laate ho.

Level 2 — Application

Goal: Bayes' theorem mein numbers sahi se plug karo.

Exercise 2.1 (discrete, two hypotheses). Ek factory mein do machines hain. Machine A $60%$ parts banati hai $2%$ defect rate ke saath; Machine B $40%$ banati hai $5%$ ke saath. Ek random part defective hai. $P (A ∣ defective)$ kya hai?

Recall Solution 2.1

Maano $θ \in {A, B}$ , $D = defective$ . Priors: $P (A) = 0.60, P (B) = 0.40$ . Likelihoods: $P (D ∣ A) = 0.02, P (D ∣ B) = 0.05$ . Evidence (Law of Total Probability se): $P (D) = 0.02 (0.60) + 0.05 (0.40) = 0.012 + 0.020 = 0.032.$ Posterior: $P (A ∣ D) = \frac{0.02 \times 0.60}{0.032} = \frac{0.012}{0.032} = 0.375.$ To A ke zyada parts banane ke bawajood, ek defective part B se aane ki zyada likelihood hai ( $P (B ∣ D) = 0.625$ ) kyunki B zyada dirty hai.

Exercise 2.2 (odds form). 2.1 ke same numbers. Posterior odds of A versus B compute karo aur confirm karo ki woh match karte hain. (Posterior odds $=$ prior odds $\times$ likelihood ratio.)

Recall Solution 2.2

Prior odds $A : B = 0.60 : 0.40 = 1.5$ . Likelihood ratio $= \frac{P ( D ∣ A )}{P ( D ∣ B )} = \frac{0.02}{0.05} = 0.4$ . Posterior odds $= 1.5 \times 0.4 = 0.6$ , i.e. $A : B = 0.6 : 1$ . Convert to probability: $P (A ∣ D) = \frac{0.6}{0.6 + 1} = \frac{0.6}{1.6} = 0.375$ . ✓ 2.1 se match karta hai — evidence $P (D)$ ratio mein poori tarah cancel ho jaata hai, aur exactly isliye odds form convenient hai.

Exercise 2.3 (pehle wala disease test ek twist ke saath). Prevalence $2%$ ; test $95%$ sensitive hai ( $P (+ ∣ sick) = 0.95$ ) aur $90%$ specific ( $P (- ∣ healthy) = 0.90$ ). Tumhara test positive aata hai. $P (sick ∣ +)$ nikalo.

Recall Solution 2.3

$P (sick) = 0.02$ , $P (+ ∣ sick) = 0.95$ , $P (+ ∣ healthy) = 1 - 0.90 = 0.10$ . $P (+) = 0.95 (0.02) + 0.10 (0.98) = 0.019 + 0.098 = 0.117.$ $P (sick ∣ +) = \frac{0.019}{0.117} \approx 0.1624.$ Lagbhag 16% — phir se chhota prior ise kam rakhta hai, halanki parent note ke $1%$ case jitna brutally nahi.

Level 3 — Analysis

Goal: machinery ke baare mein reason karo, bas use mat chalao.

Exercise 3.1 (sequential updating = order matter nahi karta). Prior $P (θ)$ se shuru karo. Tum data $D_{1}$ dekhte ho, posterior update karte ho, phir independent data $D_{2}$ dekhte ho aur phir update karte ho. Algebraically dikhao ki final posterior wahi hai jo $D_{1}$ aur $D_{2}$ par ek saath update karne se milta, aur $D_{1}, D_{2}$ ka order matter nahi karta.

Recall Solution 3.1

$D_{1}$ ke baad: $P (θ ∣ D_{1}) \propto P (D_{1} ∣ θ) P (θ)$ . Ise naya prior use karo aur $D_{2}$ par update karo ( $θ$ given hone par independent, to $P (D_{2} ∣ θ, D_{1}) = P (D_{2} ∣ θ)$ ): $P (θ ∣ D_{1}, D_{2}) \propto P (D_{2} ∣ θ) kal ka posterior P (D_{1} ∣ θ) P (θ) .$ Dono par batch update: $P (θ ∣ D_{1}, D_{2}) \propto P (D_{1}, D_{2} ∣ θ) P (θ) = P (D_{1} ∣ θ) P (D_{2} ∣ θ) P (θ) .$ Bilkul same. Aur kyunki multiplication commute karti hai, $D_{1} \leftrightarrow D_{2}$ swap karne se kuch nahi badlega — aaj ka posterior kal ka prior hai, aur arrival order irrelevant hai (figure dekho).

Figure — Bayesian statistics — prior, likelihood, posterior (intro)

Exercise 3.2 (mean kyun shrink hota hai). Ek coin $n$ tosses mein $h$ heads deti hai $Beta (α, β)$ prior ke saath. Posterior hai $Beta (α + h, β + n - h)$ jiska mean hai $\frac{α + h}{α + β + n}$ . Dikhao ki yeh mean hamesha prior mean $\frac{α}{α + β}$ aur data fraction $\frac{h}{n}$ ka ek weighted average hai, aur batao weights kya hain.

Recall Solution 3.2

Maano $m = α + β$ (prior "strength") aur likho: $\frac{α + h}{m + n} = w_{prior} \frac{m}{m + n} \cdot \frac{α}{m} + w_{data} \frac{n}{m + n} \cdot \frac{h}{n} .$ Check karo: right side expand karo $= \frac{α}{m + n} + \frac{h}{m + n} = \frac{α + h}{m + n}$ . ✓ Weights $\frac{m}{m + n}$ aur $\frac{n}{m + n}$ sum karke $1$ dete hain. Interpretation: prior $m$ "pseudo-observations" count karta hai. Jab $n \to \infty$ , $w_{data} \to 1$ aur data jeet jaata hai — Frequentist vs Bayesian Inference se connection. (Yeh Beta Distribution ko Binomial Distribution ke conjugate ke roop mein use karta hai.)

Exercise 3.3 (numerical shrinkage). Prior $Beta (2, 2)$ (mean $0.5$ ) aur data $10$ mein se $7$ heads ke saath, posterior compute karo, uska mean nikalo, aur confirm karo ki mean strictly $0.5$ aur $0.7$ ke beech hai.

Recall Solution 3.3

Posterior $= Beta (2 + 7, 2 + 3) = Beta (9, 5)$ . Mean $= \frac{9}{9 + 5} = \frac{9}{14} \approx 0.6429$ . Weights: $w_{prior} = \frac{4}{14} \approx 0.286$ , $w_{data} = \frac{10}{14} \approx 0.714$ . Check: $0.286 (0.5) + 0.714 (0.7) = 0.1429 + 0.5 = 0.6429$ . ✓ Aur waqai $0.5 < 0.6429 < 0.7$ .

Level 4 — Synthesis

Goal: kai ideas ko ek coherent argument mein combine karo.

Exercise 4.1 (do positive tests). Disease prevalence $1%$ ; ek test $99%$ sensitive aur $95%$ specific hai ( $P (+ ∣ healthy) = 0.05$ ). Tum do independent tests mein do baar positive aate ho. Pehle posterior ko doosra prior use karke $P (sick ∣ + +)$ nikalo.

Recall Solution 4.1

Pehla update (parent note se): $P (sick ∣ +) = \frac{0.99 ( 0.01 )}{0.99 ( 0.01 ) + 0.05 ( 0.99 )} = \frac{0.0099}{0.0594} = 0.1 \overline{6}$ . Doosra update — naya prior $P (sick) = 0.1 \overline{6}$ , to $P (healthy) = 0.8 \overline{3}$ : $P (+ + ’s 2nd + ∣ sick) = 0.99, P (+ ∣ healthy) = 0.05.$ $P (sick ∣ + +) = \frac{0.99 ( 0.16 6 )}{0.99 ( 0.16 6 ) + 0.05 ( 0.83 3 )} = \frac{0.165}{0.165 + 0.041 6} = \frac{0.165}{0.206 6} \approx 0.7984.$ Do positives belief ko $17%$ se $\approx 80%$ tak push karte hain. Likelihood ratio $\frac{0.99}{0.05} = 19.8$ do baar apply hota hai: prior odds $\frac{0.01}{0.99} \times 19. 8^{2} \approx 3.96$ , giving $\frac{3.96}{4.96} \approx 0.798$ . ✓

Exercise 4.2 (teen-hypothesis coin). Ek drawer mein teen coins hain: fair ( $θ = 0.5$ ), biased-heads ( $θ = 0.75$ ), two-headed ( $θ = 1$ ), priors $0.5, 0.3, 0.2$ ke saath choose ki gayi hain. Tum ek baar flip karte ho aur heads milta hai. Teeno coins par posterior nikalo.

Recall Solution 4.2

Likelihoods $P (H ∣ θ) = θ$ : $0.5, 0.75, 1$ . Unnormalised posteriors (likelihood × prior): $0.5 (0.5) = 0.25, 0.75 (0.3) = 0.225, 1 (0.2) = 0.20.$ Evidence $P (H) = 0.25 + 0.225 + 0.20 = 0.675$ . Posterior: $P (fair ∣ H) = \frac{0.25}{0.675} \approx 0.3704, P (0.75 ∣ H) = \frac{0.225}{0.675} \approx 0.3333, P (2 -headed ∣ H) = \frac{0.20}{0.675} \approx 0.2963.$ Ek heads probability mass ko heads-favouring coins ki taraf nudge karta hai, lekin fair coin ka bada prior use (mushkil se) lead mein rakhta hai. Yeh discrete-hypothesis pattern exactly woh engine hai jo ek Naive Bayes Classifier ke andar hota hai.

Level 5 — Mastery

Goal: kuch general prove/derive karo aur limits interpret karo.

Exercise 5.1 (ek poore continuous prior ke saath posterior — koi conjugacy shortcut nahi). Ek coin ke bias ka prior density $P (θ) = 2 θ$ hai $[0, 1]$ par (ek $Beta (2, 1)$ : tum heads ki taraf lean karte ho). Tum 1 toss mein 1 head observe karte ho. Khud evidence integrate karke posterior density exactly derive karo, phir posterior mean do.

Recall Solution 5.1

Likelihood of "1 head" hai $P (H ∣ θ) = θ$ . Unnormalised posterior: $P (H ∣ θ) P (θ) = θ \cdot 2 θ = 2 θ^{2}$ . Evidence (saare $θ$ par integrate karo, continuous Law of Total Probability): $P (H) = \int_{0}^{1} 2 θ^{2} d θ = 2 \cdot \frac{1}{3} = \frac{2}{3} .$ Posterior: $P (θ ∣ H) = \frac{2 θ ^{2}}{2/3} = 3 θ^{2} on [0, 1] .$ Check karo ki yeh density hai: $\int_{0}^{1} 3 θ^{2} d θ = 1$ . ✓ Yeh $Beta (3, 1)$ hai — conjugacy ke saath consistent ( $Beta (2, 1) + 1 head \to Beta (3, 1)$ ). Posterior mean $= \int_{0}^{1} θ \cdot 3 θ^{2} d θ = \int_{0}^{1} 3 θ^{3} d θ = \frac{3}{4} = 0.75$ .

Exercise 5.2 (vague prior ka limiting behaviour). $Beta (α, β)$ prior ke liye $h$ heads in $n$ tosses ke data ke saath, dikhao ki jab $α, β \to 0^{+}$ (ek increasingly "vague" prior), posterior mean pure data fraction $h / n$ ki taraf jaata hai — Maximum Likelihood Estimation answer. Woh degenerate case batao jahan yeh toot jaata hai.

Recall Solution 5.2

Posterior mean $= \frac{α + h}{α + β + n}$ . Maano $α, β \to 0^{+}$ : $lim_{α, β \to 0^{+}} \frac{α + h}{α + β + n} = \frac{h}{n} .$ To vague-prior Bayesian mean frequentist MLE $\hat{θ} = h / n$ se coincide karta hai — Frequentist vs Bayesian Inference mein bridge. Degenerate case: agar additionally $n = 0$ (koi data nahi), to limit $\frac{0}{0}$ hai — undefined, kyunki vague prior aur koi observations nahi hone par tumhare paas literally koi information nahi ek mean banane ke liye. Aur agar chhote $α, β$ ke saath $h = 0$ ya $h = n$ ho, to posterior lagbhag saara mass ek endpoint ( $θ = 0$ ya $1$ ) par dhakelta hai, giving ek over-confident "coin certainly two-tailed/two-headed hai" — chhote extreme samples par ML-style estimates ka jaana-maana khataraa.

Exercise 5.3 (prior-vs-data tug of war, quantitative). Data $h = 7, n = 10$ ke saath, tum chahte ho ki posterior mean exactly $0.6$ par ho. $Beta (α, β)$ prior use karte hue jiska mean $0.5$ ho (to $α = β$ ), woh prior strength $α$ nikalo jo ise achieve kare.

Recall Solution 5.3

$α = β$ ke saath, posterior mean $= \frac{α + 7}{2 α + 10} = 0.6$ . Solve karo: $α + 7 = 0.6 (2 α + 10) = 1.2 α + 6$ , to $1 = 0.2 α$ , giving $α = 5$ . Prior $Beta (5, 5)$ (strength $m = 10$ pseudo-observations, yaani real $n = 10$ jitna hi weight). Check: posterior $= Beta (12, 8)$ , mean $= \frac{12}{20} = 0.6$ . ✓ Data fraction $0.7$ se prior mean $0.5$ ki taraf estimate ko aadha kheenchne ke liye, prior ko exactly utne hi pseudo-counts carry karne chahiye jitne tumhare paas real data hai — Exercise 3.2 ka tug-of-war concrete roop mein.

Recall Jane se pehle ek-line self-audit

Kya tum, bina dekhe bata sakte ho: (1) prior/likelihood/posterior mein se kaun sa $θ$ par distribution nahi hai; (2) disease posterior itna low kyun hai; (3) sequential aur batch updates kyun agree karte hain; (4) "prior strength" kya control karta hai? (1) likelihood ::: (2) rare disease ⇒ false positives true positives ko daba dete hain ::: (3) likelihoods multiply hoti hain aur multiplication commute karti hai, to order-independent aur joint update ke barabar ::: (4) prior kitne pseudo-observations ke barabar hai — yaani use hilane ke liye kitna data chahiye.