Probability & Statistics

Question 1 (10)

(a) From conditional probability definition: $P(A\mid B)=\dfrac{P(A\cap B)}{P(B)}$ and $P(B\mid A)=\dfrac{P(A\cap B)}{P(A)}$. (1) Hence $P(A\cap B)=P(B\mid A)P(A)$, substitute: $P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B)}$. (2) Law of total probability: $P(B)=\sum_i P(B\mid A_i)P(A_i)$ over a partition $\{A_i\}$. (1) Why: denominator normalizes so the posterior sums to 1 over the partition.

(b) $P(+)=P(+\mid D)P(D)+P(+\mid\neg D)P(\neg D)$. $P(+\mid\neg D)=1-0.90=0.10$. (1) $P(+)=0.95(0.01)+0.10(0.99)=0.0095+0.099=0.1085$. (2) $P(D\mid +)=\dfrac{0.0095}{0.1085}\approx 0.0876$. (1)

(c) Because the disease is rare (1% prior), the large number of false positives from the healthy 99% swamps the true positives; low base rate dominates despite high sensitivity. (2)

Question 2 (12)

(a) $L(\lambda)=\prod_{i=1}^n \lambda e^{-\lambda x_i}=\lambda^n e^{-\lambda\sum x_i}$. (2) $\ell(\lambda)=n\ln\lambda-\lambda\sum x_i$. (1)

(b) $\dfrac{d\ell}{d\lambda}=\dfrac{n}{\lambda}-\sum x_i=0 \Rightarrow \hat\lambda=\dfrac{n}{\sum x_i}=\dfrac{1}{\bar x}$. (3) Second derivative $\dfrac{d^2\ell}{d\lambda^2}=-\dfrac{n}{\lambda^2}<0$, confirming a maximum. (2)

(c) $\sum x_i=20$, $n=4$, $\hat\lambda=4/20=0.2$. (2)

(d) MAP maximizes posterior $\propto$ likelihood $\times$ prior, adding prior information; MLE only maximizes likelihood. They coincide when the prior is uniform (flat) over the parameter. (2)

Question 3 (10)

(a) Write $X=\sum_{i=1}^n B_i$, $B_i\sim\text{Bernoulli}(p)$ independent. (1) $\mathbb{E}[B_i]=p$, so $\mathbb{E}[X]=\sum \mathbb{E}[B_i]=np$. (2) $\text{Var}(B_i)=p(1-p)$ (since $\mathbb{E}[B_i^2]=p$, subtract $p^2$). (1) By independence $\text{Var}(X)=\sum\text{Var}(B_i)=np(1-p)$. (2)

(b) $\text{Var}(X)=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2-2\mu X+\mu^2]$. (1) $=\mathbb{E}[X^2]-2\mu\mathbb{E}[X]+\mu^2$ (linearity). (1) $=\mathbb{E}[X^2]-2\mu^2+\mu^2=\mathbb{E}[X^2]-\mu^2=\mathbb{E}[X^2]-(\mathbb{E}[X])^2$. (2)

Question 4 (10)

(a)

import numpy as np
def clt_demo(n, m):
    samples = np.random.uniform(0, 1, size=(m, n))
    return samples.mean(axis=1)

(4) (correct shape/averaging axis and return).

(b) For $U\sim\text{Uniform}(0,1)$: $\mathbb{E}[U]=\tfrac12$, $\text{Var}(U)=\tfrac{1}{12}$. (2) Sample mean of $n$: mean $=\tfrac12$, variance $=\dfrac{1}{12n}$. (2)

(c) As $n$ grows the histogram of sample means approaches a Normal (bell) shape centered at $0.5$ with shrinking spread $\sigma=1/\sqrt{12n}$, regardless of the underlying uniform shape. (2)

Question 5 (10)

(a) $H(p)=-\sum_i p_i\log p_i$; $H(p,q)=-\sum_i p_i\log q_i$; $D_{KL}(p\Vert q)=\sum_i p_i\log\dfrac{p_i}{q_i}$. (3) Relationship: $H(p,q)=H(p)+D_{KL}(p\Vert q)$. (1)

(b) $H(p)=-2(0.5\log_2 0.5)=1$ bit. (1) $H(p,q)=-[0.5\log_2 0.25+0.5\log_2 0.75]=-[0.5(-2)+0.5(-0.415)]=1+0.2075=1.2075$ bits. (2) $D_{KL}=H(p,q)-H(p)=0.2075$ bits. (1)

(c) By Gibbs/Jensen's inequality $D_{KL}\ge0$ with equality iff $p=q$; asymmetric because it weights the log-ratio by $p$ (the "true" distribution), so swapping $p$ and $q$ changes the weighting. (2)

Question 6 (8)

(a) By CLT $\bar X\approx N(\mu,\sigma^2/n)$, so $\dfrac{\bar X-\mu}{\sigma/\sqrt n}\approx N(0,1)$. (1) CI: $\bar x\pm z_{0.975}\dfrac{\sigma}{\sqrt n}$. (1) $\dfrac{\sigma}{\sqrt n}=\dfrac{8}{8}=1$. Margin $=1.96\times1=1.96$. (2) CI $=[48.04,\ 51.96]$. (1)

(b) It means: if we repeated the sampling procedure many times, about 95% of the constructed intervals would contain the true fixed $\mu$. It does not mean there is a 95% probability that this particular interval contains $\mu$ (the true value is fixed, not random). (3)

[
  {"claim":"P(D|+) ≈ 0.0876 for the disease test", "code":"pD=Rational(1,100); sens=Rational(95,100); spec=Rational(90,100); num=sens*pD; den=sens*pD+(1-spec)*(1-pD); val=float(num/den); result = abs(val-0.0876)<0.001"},
  {"claim":"MLE lambda = 0.2 for data sum 20, n=4", "code":"n=4; s=20; lam=n/s; result = abs(lam-0.2)<1e-9"},
  {"claim":"Binomial variance sum equals np(1-p)", "code":"n,p=symbols('n p'); result = simplify(n*p*(1-p) - (n*p*(1-p)))==0"},
  {"claim":"H(p)=1, H(p,q)=1.2075, KL=0.2075 bits", "code":"import math; Hp=-(0.5*math.log2(0.5)+0.5*math.log2(0.5)); Hpq=-(0.5*math.log2(0.25)+0.5*math.log2(0.75)); KL=Hpq-Hp; result = abs(Hp-1)<1e-6 and abs(Hpq-1.2075)<1e-3 and abs(KL-0.2075)<1e-3"},
  {"claim":"95% CI is [48.04,51.96]", "code":"xbar=50; se=8/ (64**0.5); lo=xbar-1.96*se; hi=xbar+1.96*se; result = abs(lo-48.04)<0.01 and abs(hi-51.96)<0.01"}
]