Level 5 — MasteryProbability & Statistics

Probability & Statistics

⏱ 90 minutes60 marksprintable — key stays hidden on paper

Chapter: 1.3 Probability & Statistics Difficulty: Level 5 — Mastery (cross-domain: math + inference + coding) Time limit: 90 minutes Total marks: 60

Instructions: Answer all THREE questions. Show every derivation. Where code is requested, pseudo-code or Python/NumPy is acceptable but must be correct and runnable in spirit. Use $...$ for mathematics.

Question 1 — From Bayes to MAP, and the Gaussian bridge (20 marks)

A sensor measures a scalar physical quantity $\theta$ (e.g., a voltage). You obtain $n$ i.i.d. noisy readings $x_i = \theta + \varepsilon_i, \qquad \varepsilon_i \sim \mathcal{N}(0,\sigma^2),$ with $\sigma^2$ known. Your prior belief about the true value is $\theta \sim \mathcal{N}(\mu_0, \tau^2)$ .

(a) Derive the maximum likelihood estimator $\hat\theta_{\mathrm{MLE}}$ of $\theta$ from the data $x_1,\dots,x_n$ . Show the log-likelihood and the stationarity condition. (4)

(b) Derive the full posterior $p(\theta \mid x_{1:n})$ . Show it is Gaussian and give closed-form expressions for its posterior mean $\mu_n$ and variance $\sigma_n^2$ . Hence write $\hat\theta_{\mathrm{MAP}}$ . (7)

(c) Express $\mu_n$ as a precision-weighted average of the prior mean and the sample mean. Prove that as $n\to\infty$ , $\hat\theta_{\mathrm{MAP}} \to \hat\theta_{\mathrm{MLE}}$ , and interpret this in terms of the influence of the prior. (4)

(d) Write a short vectorized function posterior(x, sigma2, mu0, tau2) returning $(\mu_n, \sigma_n^2)$ , and state (with justification) how the posterior variance behaves as $n \to \infty$ . (5)

Question 2 — Estimator quality, CLT, and a confidence interval you must build (20 marks)

Let $X_1,\dots,X_n$ be i.i.d. from an Exponential distribution with rate $\lambda>0$ , density $f(x)=\lambda e^{-\lambda x}$ for $x\ge 0$ .

(a) Show $E[X_i]=1/\lambda$ and $\operatorname{Var}(X_i)=1/\lambda^2$ . Derive the MLE $\hat\lambda$ from a sample. (5)

(b) Using the Central Limit Theorem, state the approximate sampling distribution of the sample mean $\bar X_n$ for large $n$ . Then use the delta method to find the approximate distribution of $\hat\lambda = 1/\bar X_n$ , i.e. give its asymptotic variance. (6)

(c) Construct an approximate two-sided $95\%$ confidence interval for $\lambda$ based on part (b). A dataset of $n=100$ readings gives $\bar X_n = 2.0$ . Compute the numeric interval (use $z_{0.975}=1.96$ ). (5)

(d) Explain precisely what the phrase " $95\%$ confidence" means (and does not mean) in the frequentist sense, contrasting it with a Bayesian credible interval. (4)

Question 3 — Entropy, cross-entropy, and why ML minimizes KL (20 marks)

Let $p$ be a true discrete distribution over classes $\{1,\dots,K\}$ and $q$ a model distribution.

(a) Define Shannon entropy $H(p)$ , cross-entropy $H(p,q)$ , and KL divergence $D_{\mathrm{KL}}(p\|q)$ . Prove the identity $H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q).$ (4)

(b) Prove that $D_{\mathrm{KL}}(p\|q)\ge 0$ with equality iff $p=q$ . (Gibbs' inequality — use Jensen's inequality on the concave function $\log$ .) (6)

(c) For a classifier trained on data with empirical label distribution, show that minimizing average cross-entropy loss over the training set is equivalent to maximum likelihood estimation. Make the argument precise with the negative log-likelihood. (6)

(d) Compute a concrete value: for $K=2$ with true $p=(0.5,0.5)$ and model $q=(0.8,0.2)$ , compute $H(p)$ , $H(p,q)$ , and $D_{\mathrm{KL}}(p\|q)$ in bits (log base 2). Verify your identity from (a) numerically. (4)

Answer keyMark scheme & solutions

Question 1

(a) MLE (4) Likelihood: $L(\theta)=\prod_i \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\big(-\frac{(x_i-\theta)^2}{2\sigma^2}\big)$ . (1) Log-likelihood: $\ell(\theta) = C - \frac{1}{2\sigma^2}\sum_i (x_i-\theta)^2$ . (1) Stationarity: $\frac{d\ell}{d\theta} = \frac{1}{\sigma^2}\sum_i (x_i-\theta) = 0$ . (1) $\Rightarrow \hat\theta_{\mathrm{MLE}} = \bar x = \frac1n\sum_i x_i$ (second derivative $<0$ , so maximum). (1)

(b) Posterior (7) $p(\theta\mid x) \propto p(x\mid\theta)\,p(\theta)$ . (1) Exponent $= -\frac{1}{2\sigma^2}\sum_i(x_i-\theta)^2 - \frac{(\theta-\mu_0)^2}{2\tau^2}$ . (1) Collect quadratic in $\theta$ : coefficient of $\theta^2$ is $-\frac12\big(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\big)$ , so (2) $\sigma_n^2 = \left(\frac{n}{\sigma^2}+\frac{1}{\tau^2}\right)^{-1}.$ Linear term gives mean (2) $\mu_n = \sigma_n^2\left(\frac{n\bar x}{\sigma^2} + \frac{\mu_0}{\tau^2}\right).$ Posterior is Gaussian $\mathcal N(\mu_n,\sigma_n^2)$ ; since Gaussian mode = mean, $\hat\theta_{\mathrm{MAP}}=\mu_n$ . (1)

(c) Precision weighting & limit (4) Write prior precision $a=1/\tau^2$ , data precision $b=n/\sigma^2$ . Then (2) $\mu_n = \frac{a\,\mu_0 + b\,\bar x}{a+b},$ a precision-weighted average of prior mean and sample mean. As $n\to\infty$ , $b=n/\sigma^2\to\infty$ dominates, so $\mu_n\to\bar x=\hat\theta_{\mathrm{MLE}}$ . (1) The prior's fixed precision $a$ becomes negligible against growing data precision — data overwhelms the prior. (1)

(d) Code + variance (5)

import numpy as np
def posterior(x, sigma2, mu0, tau2):
    n = x.size
    xbar = x.mean()
    prec = n/sigma2 + 1/tau2          # posterior precision
    sigma_n2 = 1.0/prec               # posterior variance
    mu_n = sigma_n2*(n*xbar/sigma2 + mu0/tau2)
    return mu_n, sigma_n2

(3 for correct formulas/vectorization) As $n\to\infty$ : $\sigma_n^2=(n/\sigma^2+1/\tau^2)^{-1}\to 0$ like $\sigma^2/n$ . (1) Interpretation: infinite data $\Rightarrow$ certainty about $\theta$ ; posterior collapses to a point at $\bar x$ . (1)

Question 2

(a) Moments & MLE (5) $E[X]=\int_0^\infty x\lambda e^{-\lambda x}dx = 1/\lambda$ . (1) $E[X^2]=2/\lambda^2 \Rightarrow \operatorname{Var}=2/\lambda^2-1/\lambda^2=1/\lambda^2$ . (1) Log-likelihood $\ell(\lambda)=n\ln\lambda - \lambda\sum x_i$ . (1) $\ell'(\lambda)=n/\lambda - \sum x_i=0 \Rightarrow \hat\lambda = n/\sum x_i = 1/\bar X_n$ . (2)

(b) CLT + delta method (6) CLT: $\bar X_n \approx \mathcal N\!\big(1/\lambda,\ \frac{1}{n\lambda^2}\big)$ (variance $=\operatorname{Var}(X)/n$ ). (2) Delta method with $g(t)=1/t$ , $g'(t)=-1/t^2$ , evaluated at $t=1/\lambda$ : $g'(1/\lambda)=-\lambda^2$ . (2) Asymptotic variance $= (g')^2\cdot\operatorname{Var}(\bar X_n) = \lambda^4\cdot\frac{1}{n\lambda^2}=\frac{\lambda^2}{n}$ . (1) So $\hat\lambda \approx \mathcal N\!\big(\lambda, \frac{\lambda^2}{n}\big)$ . (1)

(c) Confidence interval (5) Standard error $\approx \hat\lambda/\sqrt n$ (plug-in). (1) CI: $\hat\lambda \pm z_{0.975}\,\hat\lambda/\sqrt n$ . (1) $\hat\lambda = 1/\bar X_n = 1/2.0 = 0.5$ . (1) SE $=0.5/\sqrt{100}=0.05$ . Margin $=1.96\times0.05=0.098$ . (1) CI $=[0.402,\ 0.598]$ . (1)

(d) Interpretation (4) " $95\%$ confidence" is a statement about the procedure: if we repeated the experiment many times, about $95\%$ of the intervals so constructed would contain the fixed true $\lambda$ . (2) It does not mean there is a $95\%$ probability that $\lambda$ lies in this particular computed interval — $\lambda$ is a fixed constant, and the realized interval either does or does not contain it. (1) A Bayesian credible interval, by contrast, does assign probability to $\lambda$ lying in the interval given the data (treating $\lambda$ as random via a prior). (1)

Question 3

(a) Identity (4) $H(p)=-\sum_k p_k\log p_k$ ; $H(p,q)=-\sum_k p_k\log q_k$ ; $D_{\mathrm{KL}}(p\|q)=\sum_k p_k\log\frac{p_k}{q_k}$ . (2) Then $H(p)+D_{\mathrm{KL}}(p\|q)= -\sum p_k\log p_k + \sum p_k(\log p_k-\log q_k)=-\sum p_k\log q_k = H(p,q)$ . (2)

(b) Gibbs (6) $-D_{\mathrm{KL}}(p\|q)=\sum_k p_k\log\frac{q_k}{p_k}$ . (1) By Jensen (log concave, $E[\log Y]\le\log E[Y]$ with $Y=q/p$ under $p$ ): (2) $\sum_k p_k\log\frac{q_k}{p_k} \le \log\sum_k p_k\frac{q_k}{p_k}=\log\sum_k q_k=\log 1=0.$ (2) Hence $D_{\mathrm{KL}}\ge 0$ . Equality iff $q_k/p_k$ constant $=1$ (strict concavity of $\log$ ), i.e. $p=q$ . (1)

(c) Cross-entropy = MLE (6) Given data $\{(x_i,y_i)\}_{i=1}^N$ with model $q_\theta(\cdot\mid x)$ , negative log-likelihood is (2) $\mathrm{NLL}=-\sum_i \log q_\theta(y_i\mid x_i).$ Using one-hot targets $p^{(i)}$ (indicator of true class), the per-sample cross-entropy is $H(p^{(i)},q_\theta)=-\sum_k p^{(i)}_k\log q_\theta(k\mid x_i)=-\log q_\theta(y_i\mid x_i)$ . (2) So average cross-entropy $=\frac1N\sum_i H(p^{(i)},q_\theta) = \frac1N\mathrm{NLL}$ . (1) Minimizing average cross-entropy over $\theta$ is therefore identical (up to the positive constant $1/N$ ) to maximizing the log-likelihood — i.e. MLE. (1)

(d) Numeric (4) $H(p)= -2(0.5\log_2 0.5)=1$ bit. (1) $H(p,q)=-(0.5\log_2 0.8 + 0.5\log_2 0.2)= -0.5(-0.3219)-0.5(-2.3219)=0.16096+1.16096=1.32193$ bits. (1) $D_{\mathrm{KL}}=\sum p_k\log_2(p_k/q_k)=0.5\log_2(0.5/0.8)+0.5\log_2(0.5/0.2)=0.5(-0.6781)+0.5(1.3219)=0.32193$ bits. (1) Check: $H(p)+D_{\mathrm{KL}}=1+0.32193=1.32193=H(p,q)$ . ✓ (1)

[
  {"claim":"Q1 posterior mean is precision-weighted average; with n=4, xbar=3, sigma2=1, mu0=0, tau2=1 gives mu_n=12/5",
   "code":"n=4; sigma2=1; tau2=1; mu0=0; xbar=3; sig_n2=1/(Rational(n,sigma2)+Rational(1,tau2)); mu_n=sig_n2*(Rational(n*xbar,sigma2)+Rational(mu0,tau2)); result=(mu_n==Rational(12,5) and sig_n2==Rational(1,5))"},
  {"claim":"Q2 delta-method asymptotic variance of 1/Xbar is lambda**2/n",
   "code":"lam,n,t=symbols('lam n t',positive=True); g=1/t; gp=diff(g,t).subs(t,1/lam); varXbar=1/(n*lam**2); asympt=simplify(gp**2*varXbar); result=simplify(asympt-lam**2/n)==0"},
  {"claim":"Q2c CI endpoints for xbar=2,n=100,z=1.96 are [0.402,0.598]",
   "code":"lh=Rational(1,2); se=lh/sqrt(100); lo=lh-Rational(196,100)*se; hi=lh+Rational(196,100)*se; result=(lo==Rational(402,1000) and hi==Rational(598,1000))"},
  {"claim":"Q3d identity H(p,q)=H(p)+KL in bits for p=(.5,.5), q=(.8,.2)",
   "code":"p=[Rational(1,2),Rational(1,2)]; q=[Rational(4,5),Rational(1,5)]; Hp=-sum(pi*log(pi,2) for pi in p); Hpq=-sum(pi*log(qi,2) for pi,qi in zip(p,q)); KL=sum(pi*log(pi/qi,2) for pi,qi in zip(p,q)); result=simplify(Hpq-(Hp+KL))==0"}
]