Bayesian statistics — prior, likelihood, posterior (intro): Exercises

Before we start, one reminder of our four characters (we will use these words with no further apology):

Level 1 — Recognition

Goal: name the pieces. No arithmetic beyond a fraction.

Exercise 1.1. In the sentence "I believed a coin was fair, then I flipped 8 heads in 10, and now I think it's biased," label which clause is the prior, which is the likelihood-driven evidence, and which is the posterior.

Recall Solution 1.1

Prior = "I believed a coin was fair" — belief before data, so $P (θ)$ peaked at $θ = 0.5$ .
Likelihood = "I flipped 8 heads in 10" — this is the data $D$ , and asking how probable it is for each $θ$ is $P (D ∣ θ)$ .
Posterior = "now I think it's biased" — updated belief after data, $P (θ ∣ D)$ shifted toward larger $θ$ .

Exercise 1.2. Which of these is a probability distribution over $θ$ (integrates to $1$ in $θ$ ): the prior $P (θ)$ , the likelihood $P (D ∣ θ)$ , or the posterior $P (θ ∣ D)$ ? Pick all that apply.

Recall Solution 1.2

The prior $P (θ)$ and the posterior $P (θ ∣ D)$ are genuine distributions over $θ$ — each integrates to $1$ . The likelihood $P (D ∣ θ)$ is not: it is a function of $θ$ but a probability of the data. It has no reason to integrate to $1$ over $θ$ . (This is the single most common confusion in the whole subject — see the Maximum Likelihood Estimation note for the frequentist cousin that just maximises this function.)

Exercise 1.3. Fill the cloze. The identity ==posterior $\propto$ likelihood $\times$ prior== drops which term, and why is dropping it allowed?

Recall Solution 1.3

It drops the evidence $P (D)$ . Dropping is allowed because $P (D)$ does not depend on $θ$ — for the purpose of finding the shape of the posterior in $θ$ , it is just a constant. You put it back at the very end by normalising.

Level 2 — Application

Goal: plug numbers into Bayes' theorem correctly.

Exercise 2.1 (discrete, two hypotheses). A factory has two machines. Machine A makes $60%$ of parts with a $2%$ defect rate; Machine B makes $40%$ with a $5%$ defect rate. A random part is defective. What is $P (A ∣ defective)$ ?

Recall Solution 2.1

Let $θ \in {A, B}$ , $D = defective$ . Priors: $P (A) = 0.60, P (B) = 0.40$ . Likelihoods: $P (D ∣ A) = 0.02, P (D ∣ B) = 0.05$ . Evidence (by the Law of Total Probability): $P (D) = 0.02 (0.60) + 0.05 (0.40) = 0.012 + 0.020 = 0.032.$ Posterior: $P (A ∣ D) = \frac{0.02 \times 0.60}{0.032} = \frac{0.012}{0.032} = 0.375.$ So despite A making most parts, a defective part is more likely from B ( $P (B ∣ D) = 0.625$ ) because B is dirtier.

Exercise 2.2 (odds form). Same numbers as 2.1. Compute the posterior odds of A versus B and confirm they match. (Posterior odds $=$ prior odds $\times$ likelihood ratio.)

Recall Solution 2.2

Prior odds $A : B = 0.60 : 0.40 = 1.5$ . Likelihood ratio $= \frac{P ( D ∣ A )}{P ( D ∣ B )} = \frac{0.02}{0.05} = 0.4$ . Posterior odds $= 1.5 \times 0.4 = 0.6$ , i.e. $A : B = 0.6 : 1$ . Convert to probability: $P (A ∣ D) = \frac{0.6}{0.6 + 1} = \frac{0.6}{1.6} = 0.375$ . ✓ Matches 2.1 — the evidence $P (D)$ cancels entirely in the ratio, which is exactly why odds form is convenient.

Exercise 2.3 (repeat the earlier disease test with a twist). Prevalence $2%$ ; test is $95%$ sensitive ( $P (+ ∣ sick) = 0.95$ ) and $90%$ specific ( $P (- ∣ healthy) = 0.90$ ). You test positive. Find $P (sick ∣ +)$ .

Recall Solution 2.3

$P (sick) = 0.02$ , $P (+ ∣ sick) = 0.95$ , $P (+ ∣ healthy) = 1 - 0.90 = 0.10$ . $P (+) = 0.95 (0.02) + 0.10 (0.98) = 0.019 + 0.098 = 0.117.$ $P (sick ∣ +) = \frac{0.019}{0.117} \approx 0.1624.$ About 16% — again the small prior keeps it low, though less brutally than the $1%$ case in the parent note.

Level 3 — Analysis

Goal: reason about the machinery, not just crank it.

Exercise 3.1 (sequential updating = order doesn't matter). Start with prior $P (θ)$ . You see data $D_{1}$ , update to a posterior, then see independent data $D_{2}$ and update again. Show algebraically that the final posterior equals what you'd get updating on $D_{1}$ and $D_{2}$ together, and that the order of $D_{1}, D_{2}$ does not matter.

Recall Solution 3.1

After $D_{1}$ : $P (θ ∣ D_{1}) \propto P (D_{1} ∣ θ) P (θ)$ . Use this as the new prior and update on $D_{2}$ (independent given $θ$ , so $P (D_{2} ∣ θ, D_{1}) = P (D_{2} ∣ θ)$ ): $P (θ ∣ D_{1}, D_{2}) \propto P (D_{2} ∣ θ) yesterday’s posterior P (D_{1} ∣ θ) P (θ) .$ Batch update on both at once: $P (θ ∣ D_{1}, D_{2}) \propto P (D_{1}, D_{2} ∣ θ) P (θ) = P (D_{1} ∣ θ) P (D_{2} ∣ θ) P (θ) .$ Identical. And since multiplication commutes, swapping $D_{1} \leftrightarrow D_{2}$ changes nothing — today's posterior is tomorrow's prior, and the arrival order is irrelevant (see the figure).

Figure — Bayesian statistics — prior, likelihood, posterior (intro)

Exercise 3.2 (why the mean shrinks). A coin gives $h$ heads in $n$ tosses with a $Beta (α, β)$ prior. The posterior is $Beta (α + h, β + n - h)$ with mean $\frac{α + h}{α + β + n}$ . Show this mean is always a weighted average of the prior mean $\frac{α}{α + β}$ and the data fraction $\frac{h}{n}$ , and say what the weights are.

Recall Solution 3.2

Let $m = α + β$ (prior "strength") and write: $\frac{α + h}{m + n} = w_{prior} \frac{m}{m + n} \cdot \frac{α}{m} + w_{data} \frac{n}{m + n} \cdot \frac{h}{n} .$ Check: expand the right side $= \frac{α}{m + n} + \frac{h}{m + n} = \frac{α + h}{m + n}$ . ✓ The weights $\frac{m}{m + n}$ and $\frac{n}{m + n}$ sum to $1$ . Interpretation: the prior counts as $m$ "pseudo-observations." As $n \to \infty$ , $w_{data} \to 1$ and the data wins — the connection to Frequentist vs Bayesian Inference. (This uses the Beta Distribution as conjugate to the Binomial Distribution.)

Exercise 3.3 (numerical shrinkage). With prior $Beta (2, 2)$ (mean $0.5$ ) and data $7$ heads in $10$ , compute the posterior, its mean, and confirm the mean lies strictly between $0.5$ and $0.7$ .

Recall Solution 3.3

Posterior $= Beta (2 + 7, 2 + 3) = Beta (9, 5)$ . Mean $= \frac{9}{9 + 5} = \frac{9}{14} \approx 0.6429$ . Weights: $w_{prior} = \frac{4}{14} \approx 0.286$ , $w_{data} = \frac{10}{14} \approx 0.714$ . Check: $0.286 (0.5) + 0.714 (0.7) = 0.1429 + 0.5 = 0.6429$ . ✓ And indeed $0.5 < 0.6429 < 0.7$ .

Level 4 — Synthesis

Goal: combine several ideas into one coherent argument.

Exercise 4.1 (two positive tests). Disease prevalence $1%$ ; a test is $99%$ sensitive and $95%$ specific ( $P (+ ∣ healthy) = 0.05$ ). You test positive twice with two independent tests. Find $P (sick ∣ + +)$ by using the first posterior as the second prior.

Recall Solution 4.1

First update (from the parent note): $P (sick ∣ +) = \frac{0.99 ( 0.01 )}{0.99 ( 0.01 ) + 0.05 ( 0.99 )} = \frac{0.0099}{0.0594} = 0.1 \overline{6}$ . Second update — new prior $P (sick) = 0.1 \overline{6}$ , so $P (healthy) = 0.8 \overline{3}$ : $P (+ + ’s 2nd + ∣ sick) = 0.99, P (+ ∣ healthy) = 0.05.$ $P (sick ∣ + +) = \frac{0.99 ( 0.16 6 )}{0.99 ( 0.16 6 ) + 0.05 ( 0.83 3 )} = \frac{0.165}{0.165 + 0.041 6} = \frac{0.165}{0.206 6} \approx 0.7984.$ Two positives push belief from $17%$ to $\approx 80%$ . The likelihood ratio $\frac{0.99}{0.05} = 19.8$ applied twice: prior odds $\frac{0.01}{0.99} \times 19. 8^{2} \approx 3.96$ , giving $\frac{3.96}{4.96} \approx 0.798$ . ✓

Exercise 4.2 (three-hypothesis coin). A drawer has three coins: fair ( $θ = 0.5$ ), biased-heads ( $θ = 0.75$ ), two-headed ( $θ = 1$ ), chosen with priors $0.5, 0.3, 0.2$ . You flip once and get heads. Find the posterior over the three coins.

Recall Solution 4.2

Likelihoods $P (H ∣ θ) = θ$ : $0.5, 0.75, 1$ . Unnormalised posteriors (likelihood × prior): $0.5 (0.5) = 0.25, 0.75 (0.3) = 0.225, 1 (0.2) = 0.20.$ Evidence $P (H) = 0.25 + 0.225 + 0.20 = 0.675$ . Posterior: $P (fair ∣ H) = \frac{0.25}{0.675} \approx 0.3704, P (0.75 ∣ H) = \frac{0.225}{0.675} \approx 0.3333, P (2 -headed ∣ H) = \frac{0.20}{0.675} \approx 0.2963.$ One heads nudges probability mass toward the head-favouring coins, but the fair coin's larger prior keeps it (barely) in the lead. This discrete-hypothesis pattern is exactly the engine inside a Naive Bayes Classifier.

Level 5 — Mastery

Goal: prove/derive something general and interpret limits.

Exercise 5.1 (posterior with a full continuous prior — no conjugacy shortcut). A coin's bias has prior density $P (θ) = 2 θ$ on $[0, 1]$ (a $Beta (2, 1)$ : you lean toward heads). You observe 1 head in 1 toss. Derive the posterior density exactly by integrating the evidence yourself, then give the posterior mean.

Recall Solution 5.1

Likelihood of "1 head" is $P (H ∣ θ) = θ$ . Unnormalised posterior: $P (H ∣ θ) P (θ) = θ \cdot 2 θ = 2 θ^{2}$ . Evidence (integrate over all $θ$ , the continuous Law of Total Probability): $P (H) = \int_{0}^{1} 2 θ^{2} d θ = 2 \cdot \frac{1}{3} = \frac{2}{3} .$ Posterior: $P (θ ∣ H) = \frac{2 θ ^{2}}{2/3} = 3 θ^{2} on [0, 1] .$ Check it's a density: $\int_{0}^{1} 3 θ^{2} d θ = 1$ . ✓ This is $Beta (3, 1)$ — consistent with conjugacy ( $Beta (2, 1) + 1 head \to Beta (3, 1)$ ). Posterior mean $= \int_{0}^{1} θ \cdot 3 θ^{2} d θ = \int_{0}^{1} 3 θ^{3} d θ = \frac{3}{4} = 0.75$ .

Exercise 5.2 (limiting behaviour of a vague prior). For a $Beta (α, β)$ prior with data $h$ heads in $n$ tosses, show that as $α, β \to 0^{+}$ (an increasingly "vague" prior), the posterior mean approaches the pure data fraction $h / n$ — the Maximum Likelihood Estimation answer. State the degenerate case where this breaks.

Recall Solution 5.2

Posterior mean $= \frac{α + h}{α + β + n}$ . Let $α, β \to 0^{+}$ : $lim_{α, β \to 0^{+}} \frac{α + h}{α + β + n} = \frac{h}{n} .$ So the vague-prior Bayesian mean coincides with the frequentist MLE $\hat{θ} = h / n$ — the bridge in Frequentist vs Bayesian Inference. Degenerate case: if additionally $n = 0$ (no data), the limit is $\frac{0}{0}$ — undefined, because with a vague prior and no observations you have literally no information to form a mean. Also if $h = 0$ or $h = n$ with tiny $α, β$ , the posterior piles almost all mass at an endpoint ( $θ = 0$ or $1$ ), giving an over-confident "the coin is certainly two-tailed/two-headed" — the well-known danger of ML-style estimates on extreme small samples.

Exercise 5.3 (prior-vs-data tug of war, quantitative). With data $h = 7, n = 10$ , you want the posterior mean to land exactly at $0.6$ . Using a $Beta (α, β)$ prior whose mean is $0.5$ (so $α = β$ ), find the prior strength $α$ that achieves it.

Recall Solution 5.3

With $α = β$ , posterior mean $= \frac{α + 7}{2 α + 10} = 0.6$ . Solve: $α + 7 = 0.6 (2 α + 10) = 1.2 α + 6$ , so $1 = 0.2 α$ , giving $α = 5$ . Prior $Beta (5, 5)$ (strength $m = 10$ pseudo-observations, i.e. as much weight as the real $n = 10$ ). Check: posterior $= Beta (12, 8)$ , mean $= \frac{12}{20} = 0.6$ . ✓ To pull the estimate halfway from the data fraction $0.7$ toward the prior mean $0.5$ , the prior must carry exactly as many pseudo-counts as you have real data — the tug-of-war of Exercise 3.2 made concrete.

Recall One-line self-audit before you leave

Can you, without looking, state: (1) which of prior/likelihood/posterior is not a distribution over $θ$ ; (2) why the disease posterior is so low; (3) why sequential and batch updates agree; (4) what "prior strength" controls? (1) likelihood ::: (2) rare disease ⇒ false positives swamp true positives ::: (3) likelihoods multiply and multiplication commutes, so order-independent and equal to the joint update ::: (4) how many pseudo-observations the prior is worth — i.e. how much data it takes to move it.