Probability Theory & Statistics

Question 1

(a) [4] $M_X(t)=\mathbb E[e^{tX}]=\int_0^\infty \lambda e^{-(\lambda-t)x}dx = \dfrac{\lambda}{\lambda-t}$ for $t<\lambda$ (integral converges only then). [2] $M'(t)=\lambda(\lambda-t)^{-2}$, so $\mathbb E[X]=M'(0)=1/\lambda$. $M''(t)=2\lambda(\lambda-t)^{-3}$, $\mathbb E[X^2]=2/\lambda^2$, hence $\operatorname{Var}(X)=2/\lambda^2-1/\lambda^2=1/\lambda^2$. [2]

(b) [4] By independence $M_{S_n}(t)=\prod_i M_{X_i}(t)=\left(\frac{\lambda}{\lambda-t}\right)^n$. [2] This is exactly the MGF of $\text{Gamma}(n,\lambda)$ (shape $n$, rate $\lambda$); MGFs uniquely determine distributions, so $S_n\sim\text{Gamma}(n,\lambda)$. [1] Thus $\mathbb E[S_n]=n/\lambda$, $\operatorname{Var}(S_n)=n/\lambda^2$. [1]

(c) [6] CLT statement: $Z_n=\dfrac{\bar X_n-\mu}{\sigma/\sqrt n}=\dfrac{S_n-n\mu}{\sigma\sqrt n}\xrightarrow{d}N(0,1)$ with $\mu=1/\lambda,\ \sigma=1/\lambda$. [2] Sketch: Let $W_i=(X_i-\mu)/\sigma$, mean 0, variance 1. MGF $M_W(t)=1+\tfrac{t^2}{2}+o(t^2)$ (from Taylor expansion, using $\mathbb E W=0,\ \mathbb E W^2=1$). [2] By independence the MGF of $Z_n=\frac1{\sqrt n}\sum W_i$ is $\left(M_W(t/\sqrt n)\right)^n=\left(1+\frac{t^2}{2n}+o(1/n)\right)^n\to e^{t^2/2}$, the $N(0,1)$ MGF; convergence of MGFs on a neighbourhood of 0 $\Rightarrow$ convergence in distribution (Lévy/continuity theorem). The i.i.d. assumption is used to factor the joint MGF into the $n$-th power and to use a common $M_W$. [2]

(d) [6] Unbiased: $\mathbb E[\hat\theta_N]=\frac1N\sum\mathbb E[g(X_i)]=\theta$. [1] Consistency: $\operatorname{Var}(\hat\theta_N)=\sigma_g^2/N\to0$ where $\sigma_g^2=\operatorname{Var}(g(X))<\infty$ (bounded $g$); by WLLN (or Chebyshev) $\hat\theta_N\xrightarrow{P}\theta$. [2] SE $=\sigma_g/\sqrt N=O(N^{-1/2})$. [1] Code: [2]

import numpy as np
def mc_estimate(g, sampler, N, rng):
    x = sampler(N, rng)          # draws X_i
    vals = g(x)                  # vectorized g
    theta_hat = vals.mean()
    se = vals.std(ddof=1)/np.sqrt(N)
    ci = (theta_hat - 1.96*se, theta_hat + 1.96*se)
    return theta_hat, ci

Question 2

(a) [4] $\ell(\mu)=\sum_i(-\mu+Y_i\ln\mu-\ln Y_i!) = -n\mu+\ln\mu\sum Y_i - \text{const}$. [2] $\ell'(\mu)=-n+\frac{\sum Y_i}{\mu}=0\Rightarrow\hat\mu=\bar Y$. Second derivative $-\sum Y_i/\mu^2<0$ so it is a max. [1] MoM: population mean $=\mu$, set equal to sample mean $\Rightarrow \hat\mu=\bar Y$ — same. [1]

(b) [6] $\mathbb E[\hat\mu]=\frac1n\sum\mathbb E[Y_i]=\mu$ ⇒ unbiased. [1] Per-observation info: $-\mathbb E[\partial^2_\mu \ln f]$. $\ln f=-\mu+Y\ln\mu-\ln Y!$, $\partial^2_\mu=-Y/\mu^2$, $-\mathbb E[-Y/\mu^2]=\mu/\mu^2=1/\mu$. So $I_1(\mu)=1/\mu$ and $I_n(\mu)=n/\mu$. [3] CRLB $=1/I_n(\mu)=\mu/n$. [1] $\operatorname{Var}(\hat\mu)=\operatorname{Var}(\bar Y)=\mu/n$ attains the bound ⇒ $\hat\mu$ is efficient (MVUE). [1]

(c) [5] By CLT $\hat\mu\approx N(\mu,\mu/n)$, and $\widehat{\operatorname{Var}}=\hat\mu/n$. [1] CI: $\hat\mu\pm z_{1-\alpha/2}\sqrt{\hat\mu/n}$. [2] Numeric: $\hat\mu=250/100=2.5$, SE $=\sqrt{2.5/100}=\sqrt{0.025}=0.15811$. Margin $=1.96\times0.15811=0.30990$. CI $\approx(2.190,\,2.810)$. [2]

(d) [5] Posterior $\propto$ likelihood $\times$ prior $\propto \mu^{\sum Y_i}e^{-n\mu}\cdot \mu^{a-1}e^{-b\mu}=\mu^{a+\sum Y_i-1}e^{-(b+n)\mu}$. [2] So posterior $=\text{Gamma}(a+\sum Y_i,\ b+n)$. [1] Posterior mean $=\dfrac{a+\sum Y_i}{b+n}=\dfrac{b}{b+n}\cdot\frac{a}{b}+\dfrac{n}{b+n}\cdot\bar Y$ — a convex combination of prior mean $a/b$ and MLE $\bar Y$. [1] As $n\to\infty$ the weight on $\bar Y\to1$, posterior mean $\to\hat\mu$ (data dominates). [1]

Question 3

(a) [4] $\int_0^1\int_0^1 c(x+y)\,dx\,dy = c\int_0^1(\tfrac12+y)dy=c(\tfrac12+\tfrac12)=c=1\Rightarrow c=1$. [2] $f_X(x)=\int_0^1(x+y)dy=x+\tfrac12,\ 0\le x\le1$ (same form for $f_Y$). [1] Since $f(x,y)=x+y\ne (x+\tfrac12)(y+\tfrac12)=f_X f_Y$, they are not independent. [1]

(b) [6] $\mathbb E[X]=\int_0^1 x(x+\tfrac12)dx=\tfrac13+\tfrac14=\tfrac{7}{12}$. [1] $\mathbb E[X^2]=\int_0^1 x^2(x+\tfrac12)dx=\tfrac14+\tfrac16=\tfrac{5}{12}$; $\operatorname{Var}(X)=\tfrac{5}{12}-(\tfrac7{12})^2=\tfrac{60-49}{144}=\tfrac{11}{144}$. [2] $\mathbb E[XY]=\int_0^1\int_0^1 xy(x+y)dxdy=\int\!\int (x^2y+xy^2)=\tfrac13\cdot\tfrac12+\tfrac12\cdot\tfrac13=\tfrac13$. [1] $\operatorname{Cov}=\tfrac13-(\tfrac7{12})^2=\tfrac{48-49}{144}=-\tfrac1{144}$. [1] $\rho=\dfrac{-1/144}{11/144}=-\tfrac{1}{11}$. [1]

(c) [5] $f_U(u)=\int f(x,u-x)dx$ over valid region. For $0\le u\le1$: $x\in[0,u]$, integrand $=u$ ⇒ $f_U(u)=\int_0^u u\,dx=u^2$. [2] For $1\le u\le2$: $x\in[u-1,1]$, length $2-u$, integrand $=u$ ⇒ $f_U(u)=u(2-u)$. [2] Check: $\int_0^1 u^2du+\int_1^2 u(2-u)du=\tfrac13+\big[u^2-\tfrac{u^3}{3}\big]_1^2=\tfrac13+(4-\tfrac83)-(1-\tfrac13)=\tfrac13+\tfrac43-\tfrac23=1.$ ✓ [1]

(d) [5] Minimize $\sum(y_i-\beta_0-\beta_1 x_i)^2$. Normal equations give $\hat\beta_1=\dfrac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}=\dfrac{S_{xy}}{S_{xx}}$, $\hat\beta_0=\bar y-\hat\beta_1\bar x$. [2] Unbiasedness: $\hat\beta_1=\sum w_i y_i$ with $w_i=(x_i-\bar x)/S_{xx}$, $\sum w_i=0,\ \sum w_i x_i=1$. Then $\mathbb E[\hat\beta_1]=\sum w_i(\beta_0+\beta_1x_i)=\beta_0\cdot0+\beta_1\cdot1=\beta_1$. [2] Distribution: $\hat\beta_1\sim N\!\big(\beta_1,\ \sigma^2/S_{xx}\big)$; test $H_0:\beta_1=0$ via $t=\hat\beta_1/(\hat\sigma/\sqrt{S_{xx}})\sim t_{n-2}$. [1]

[
{"claim":"Exponential MGF gives mean 1/lambda, var 1/lambda^2","code":"t,lam,x=symbols('t lam x',positive=True); M=integrate(lam*exp(-(lam-t)*x),(x,0,oo)); M=simplify(M.rewrite(Piecewise)); Mf=lam/(lam-t); mean=diff(Mf,t).subs(t,0); ex2=diff(Mf,t,2).subs(t,0); var=simplify(ex2-mean**2); result=(simplify(mean-1/lam)==0 and simplify(var-1/lam**2)==0)"},
{"claim":"Poisson 95% CI endpoints ~ (2.190,2.810)","code":"mu=Rational(25,10); se=sqrt(mu/100); lo=float(mu-1.96*se); hi=float(mu+1.96*se); result=(abs(lo-2.190)<0.01 and abs(hi-2.810)<0.01)"},
{"claim":"Joint density: Var(X)=11/144, Cov=-1/144, rho=-1/11","code":"x,y=symbols('x y'); fX=x+Rational(1,2); EX=integrate(x*fX,(x,0,1)); EX2=integrate(x**2*fX,(x,0,1)); VarX=EX2-EX**2; EXY=integrate(integrate(x*y*(x+y),(x,0,1)),(y,