Exercises — Statistical estimation — MLE, method of moments

Har item ek yes/no trial hai → Bernoulli($p$), unknown $p$ = defect ki probability. Bernoulli probability ka MLE sample proportion hota hai: $\hat{p}=\bar{x}=\frac{s}{n}=\frac{3}{10}=\mathrm{0.3.}$ Yahan $s=\sum x_i=3$ successes (defects) ki number hai.

Exponential ke liye, MoM aur MLE dono dete hain $\hat{\lambda }=\frac{1}{\bar{x}}=\frac{1}{4}=0.25\text{ per minute}.$ Exponential ka mean $1/\lambda$ hota hai, isliye invariance property se (kisi function ka MLE woh function hai us function ke MLE ka) estimated mean $1/\hat{\lambda }=\bar{x}=4$ minutes hai.

Step 1 — data ko average karo. Kyun? Exponential ka MLE sirf $\bar{x}$ par depend karta hai. $\bar{x}=\frac{2+4+4+6+9}{5}=\frac{25}{5}=5.$ Step 2 — formula apply karo. $\hat{\lambda }=\frac{1}{\bar{x}}=\frac{1}{5}=\mathrm{0.2.}$

Step 1 — mean. Normal ke liye, $\hat{\mu }=\bar{x}$. $\bar{x}=\frac{1+3+5+7+9}{5}=\frac{25}{5}=5.$ Step 2 — variance. ${\sigma }^2$ ka MLE $n$ se divide karta hai ($n-1$ se nahi): ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2.$ $5$ se deviations: $-4,-2,0,2,4$. Squares: $16,4,0,4,16$, sum $=40$. ${\hat{\sigma }}^2=\frac{40}{5}=8.$

Step 1 — likelihood (saare data par pmf multiply karo; independence): $L(\lambda )={\prod }_{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}=\frac{e^{-n\lambda }{\lambda }^{\sum x_i}}{\prod x_i!}.$ Step 2 — log-likelihood. Kyun log? Kyunki $\log$ ugly product/power ko sum mein convert karta hai, aur (strictly increasing hone ke karan, dekho Logarithms) uska same maximizer hota hai: $\ell (\lambda )=-n\lambda +\left(\sum x_i\right)\ln \lambda -\ln \left(\prod x_i!\right).$ Last term mein koi $\lambda$ nahi, isliye maximization ke liye yeh constant hai. Step 3 — differentiate, zero set karo. Kyun? Smooth curve ka peak zero slope par hota hai: ${\ell }^{\prime }(\lambda )=-n+\frac{\sum x_i}{\lambda }=0\Rightarrow \hat{\lambda }=\frac{\sum x_i}{n}=\bar{x}.$ Step 4 — confirm karo ki maximum hai (second derivative negative): ${\ell }^{\prime \prime }(\lambda )=-\frac{\sum x_i}{{\lambda }^2}<0.✓$ Apply: $\bar{x}=\frac{0+1+2+2+5}{5}=\frac{10}{5}=2$, isliye $\hat{\lambda }=2$.

MLE — yahan differentiation kyun fail hoti hai. Likelihood hai $L(\theta )={\prod }_{i=1}^n\frac{1}{\theta }\cdot 1\{0\le x_i\le \theta \}=\frac{1}{{\theta }^n}\text{but only if }\theta \ge {\max }_i{x}_i.$ Indicator $1\{\cdot \}$ ek switch hai: jab condition hold hoti hai toh $1$ hota hai, warna $0$. Agar koi bhi data point $\theta$ se exceed kare, toh woh factor $0$ ho jaata hai, aur $L$ zero ho jaati hai. Toh $L(\theta )=0$ for $\theta <\max x_i$, aur $L(\theta )=1/{\theta }^n$ for $\theta \ge \max x_i$. Figure 1 dekho: red curve $\theta =\max x_i$ par jump karti hai aur phir decrease hoti hai ($1/{\theta }^n$ shrink hota hai jab $\theta$ badhta hai). Sabse allowed $\theta$ ka highest point exactly maximum data value par hai: ${\hat{\theta }}_{\text{MLE}}={\max }_i{x}_i=\mathrm{0.9.}$ ${\ell }^{\prime }=0$ set karne se $-n/\theta =0$ milta hai, jiska koi solution nahi — maximum ek boundary par hai, calculus se nahi balki inspection se milta hai.

MoM. $U(0,\theta )$ ka mean midpoint $\theta /2$ hota hai. $\theta /2=\bar{x}$ set karo: $\bar{x}=\frac{0.2+0.5+0.9+0.7}{4}=\frac{2.3}{4}=0.575,{\hat{\theta }}_{\text{MoM}}=2\bar{x}=\mathrm{1.15.}$ Analysis. Dono legitimate hain, lekin MoM ne $1.15$ diya jabki humne $0.9$ ka value dekha — consistent hai. MLE ($0.9$) kabhi bhi largest observation se chota nahi ho sakta (achha hai), phir bhi yeh biased low hai (yeh true $\theta$ se kabhi exceed nahi kar sakta). MoM yahan unbiased hai lekin unlucky samples mein kisi data point se chota ho sakta hai, jo Uniform ke upper limit ke liye absurd hai. Yeh classic "MoM out-of-range values deta hai" warning hai.

Goal: $E[{\hat{\sigma }}^2]$ compute karo aur ${\sigma }^2$ se compare karo. Ek known identity (woh "sum-of-squared-deviations" fact) deta hai $E\left[{\sum }_{i=1}^n(x_i-\bar{x}{)}^2\right]=(n-1){\sigma }^2.$ $n$ ki jagah $n-1$ kyun? Kyunki estimated mean $\bar{x}$ subtract karna (true mean ki jagah) ek "degree of freedom" remove karta hai — deviations ka sum zero hona forced hai, isliye unme se ek free nahi hai. $n$ se divide karne par: $E[{\hat{\sigma }}^2]=\frac{(n-1){\sigma }^2}{n}=\frac{n-1}{n}{\sigma }^2<{\sigma }^2.$ Toh MLE average mein underestimate karta hai — uska bias hai $-{\sigma }^2/n$. Unbiased fix $n-1$ se divide karta hai: $s^2=\frac{1}{n-1}\sum (x_i-\bar{x}{)}^2,E[s^2]={\sigma }^2.$ Dekho Bias and Variance: MLE likelihood maximize karne ke liye thodi si bias trade karta hai.

(a) MLE. Log-likelihood (product split karne ke liye $\log$ use karke; dekho Logarithms): $\ell (p)={\sum }_{i=1}^n[(x_i-1)\ln (1-p)+\ln p]=(\sum x_i-n)\ln (1-p)+n\ln p.$ Differentiate karo aur zero set karo: ${\ell }^{\prime }(p)=-\frac{\sum x_i-n}{1-p}+\frac{n}{p}=0.$ $p(1-p)$ se multiply karo: $-p(\sum x_i-n)+n(1-p)=0\Rightarrow n=p\sum x_i\Rightarrow \hat{p}=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}.$ Second derivative ${\ell }^{\prime \prime }(p)=-\frac{\sum x_i-n}{(1-p{)}^2}-\frac{n}{p^2}<0$ (dono terms negative kyunki $\sum x_i\ge n$), maximum confirm karta hai. (b) MoM. Mean $=1/p$, $1/p=\bar{x}$ set karo $\Rightarrow \hat{p}=1/\bar{x}$. (c) Woh agree karte hain: $\hat{p}=1/\bar{x}$. Jaise Exponential ke saath, jab mean single parameter ka clean function hota hai, MoM aur MLE coincide karte hain. Apply: $\bar{x}=\frac{1+2+2+3+2}{5}=\frac{10}{5}=2$, isliye $\hat{p}=\frac{1}{2}=0.5$.

Bias: $\bar{x}$ mean ke liye unbiased hai, $E[\bar{x}]=\mu$, isliye bias $=0$. Variance: $\text{Var}(\bar{x})={\sigma }^2/n=9/4=2.25$. MSE: $\text{MSE}=\text{Var}+{\text{bias}}^2=2.25+0^2=\mathrm{2.25.}$ True mean $\mu =10$ ek red herring tha — ek unbiased estimator ke liye MSE sirf variance hota hai, true value kahan bhi ho usse independent.

Likelihood. Har data point ko $x_i\ge \theta$ satisfy karna chahiye, warna uski density $0$ hai: $L(\theta )={\prod }_{i=1}^n{e}^{-(x_i-\theta )}\cdot 1\{x_i\ge \theta \}=e^{-\sum (x_i-\theta )}=e^{-\sum x_i}{e}^{n\theta },\text{valid only if }\theta \le {\min }_i{x}_i.$ Shape ka analysis (Figure 2). Fixed data ke liye, $e^{n\theta }$ $\theta$ mein increase karta hai — bada $\theta$ better hai. Lekin hum capped hain: $\theta$ smallest data point se exceed nahi kar sakta, warna koi indicator $0$ ho jaayega. Toh $\theta$ ko jitna allowed ho utna high push karo: ${\hat{\theta }}_{\text{MLE}}={\min }_i{x}_i=\mathrm{3.1.}$ Phir bhi calculus akela fail karta hai (${\ell }^{\prime }(\theta )=n>0$, koi interior root nahi); maximum boundary $\theta =\min x_i$ par hai.

Invariance kehta hai: agar $\hat{\lambda }$, $\lambda$ ka MLE hai, toh kisi bhi function $g$ ke liye, $g(\lambda )$ ka MLE $g(\hat{\lambda })$ hoga — aap bas hat plug in karo. Yahan $g(\lambda )=e^{-\lambda }$: $\widehat{P(X=0)}=e^{-\hat{\lambda }}=e^{-\bar{x}}=e^{-2}\approx \mathrm{0.1353.}$ Koi nayi maximization nahi chahiye — yahi invariance ka poora point hai. (Ise Bayesian Estimation se contrast karo, jahan aap single best value plug in karne ki jagah $\lambda$ ki posterior distribution par average karte.)

(a) MLE deta hai ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2$, factor $\frac{n-1}{n}$ se biased. Lekin jab $n\to \infty$, $\frac{n-1}{n}\to 1$, toh bias vanish hota hai aur uska variance bhi $0$ tak shrink hota hai; isliye yeh consistent hai (${\hat{\sigma }}^2\stackrel{P}{\to }{\sigma }^2$). Finite $n$ par bias consistency ko prevent nahi karta. (b) Cramér–Rao Lower Bound woh floor hai jo kisi bhi unbiased estimator ka variance achieve kar sakta hai. MLEs asymptotically efficient hote hain — large $n$ ke liye woh us floor ko attain karte hain. Toh MLE small $n$ par perfect nahi hai (biased ho sakta hai), lekin jab data accumulate hota hai toh best-possible unbiased estimator ban jaata hai. Yehi wajah hai hum small-sample bias tolerate karte hain.

Uniform $U(0,\theta )$ ka MLE hai ==${\max }_i{x}_i$, jo inspection/boundary, not differentiation== se milta hai. Shifted-Exponential ke shift $\theta$ ka MLE hai ==${\min }_i{x}_i$==. Invariance property se aap $e^{-\lambda }$ ko ==$e^{-\hat{\lambda }}$== se estimate kar sakte ho bina nayi maximization ke. Unbiased estimator ke liye, $\text{MSE}$ the variance ke barabar hota hai kyunki bias term zero hai.