Exercises — Maximum likelihood estimation (MLE)

HUM KYA KARTE HAIN: har head ek factor $p$ (uski probability) multiply karta hai, har tail $(1-p)$ multiply karta hai. $L(p)=\underset{6}{\underbrace{p\cdot p\cdots p}}\cdot \underset{2}{\underbrace{(1-p)\cdot (1-p)}}=p^6(1-p{)}^2.$ KYU: flips independent hain, isliye joint probability single-flip probabilities ka product hai. Order se product ki value matter nahi karti — sirf counts (6 aur 2) matter karte hain. Log lo, $\log (ab)=\log a+\log b$ aur $\log (a^k)=k\log a$ use karke: $\ell (p)=6\log p+2\log (1-p).$

False. MLE $P(\text{data}\mid \theta )=L(\theta )$ maximise karta hai — parameter diye gaye data ki probability. "Data diye gaye parameters ki probability" wala phrase $P(\theta \mid \text{data})$ hai, posterior, jo Bayesian Estimation / MAP se belong karta hai, MLE se nahi. MLE $\theta$ ko ek fixed unknown constant maanta hai, koi random cheez nahi.

Step — differentiate karo. $\frac{d}{dp}\log p=\frac{1}{p}$ aur $\frac{d}{dp}\log (1-p)=-\frac{1}{1-p}$ use karke: $\frac{d\ell }{dp}=\frac{6}{p}-\frac{2}{1-p}.$ KYU ye tool (derivative)? $\ell$-hill ka peak wo ek flat spot hai; derivative hi slope hai, isliye usse $0$ set karna peak locate karta hai (upar figure dekho). Step — zero set karo aur solve karo. $\frac{6}{p}=\frac{2}{1-p}\Rightarrow 6(1-p)=2p\Rightarrow 6=8p\Rightarrow \boxed{\hat{p}=\frac{6}{8}=\mathrm{0.75.}}$ Sanity check: $\hat{p}=\frac{\text{heads}}{\text{flips}}=\frac{6}{8}$ — exactly observed frequency, jaise intuition demand karta hai.

General Gaussian-mean result (parent note, Example 2) hai ${\hat{\mu }}_{\text{MLE}}=\bar{x}=\frac{1}{n}\sum x_i$. KYU mean: log-likelihood maximise karna matlab $\sum (x_i-\mu {)}^2$ minimise karna hai (baaki $\mu$ mein constant hai); wo point jo numbers ke set se total squared distance minimise karta hai woh unka average hota hai. $\hat{\mu }=\frac{4+7+9+10}{4}=\frac{30}{4}=\boxed{\mathrm{7.5.}}$ Note karo ki ${\sigma }^2$ kabhi enter nahi hua — mean ke liye, known variance cancel ho jaata hai.

${\hat{\sigma }}^2=\frac{2^2+(-1{)}^2+3^2+(-4{)}^2}{4}=\frac{4+1+9+16}{4}=\frac{30}{4}=\boxed{\mathrm{7.5.}}$ ${\sigma }^2$ mein differentiate kyun karte hain, $\sigma$ mein nahi? Log-likelihood mein ${\sigma }^2$ sirf do clean pieces ke through aata hai: $\log ({\sigma }^2)$ aur $\frac{1}{{\sigma }^2}$. Poore block $v={\sigma }^2$ ko ek variable maano: $\frac{d}{dv}\log v=\frac{1}{v}$ aur $\frac{d}{dv}\frac{1}{v}=-\frac{1}{v^2}$ painless hain. $\sigma$ mein differentiate karne se extra chain-rule factors aate hain aur identical answer milta hai zyada mess ke saath. (Invariance of MLE se, $\hat{\sigma }=\sqrt{{\hat{\sigma }}^2}$ regardless.)

Log-likelihood. $\log p(x\mid \lambda )=\log \lambda -\lambda x$, isliye $\ell (\lambda )={\sum }_{i=1}^n(\log \lambda -\lambda x_i)=n\log \lambda -\lambda {\sum }_{i=1}^n{x}_i.$ Differentiate & solve. $\frac{d\ell }{d\lambda }=\frac{n}{\lambda }-\sum x_i=0\Rightarrow \hat{\lambda }=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}.$ Curvature check: ${\ell }^{\prime \prime }=-\frac{n}{{\lambda }^2}<0$ ⇒ maximum. $x=\{2,4,6\}$ ke liye: $\sum x_i=12$, $n=3$, $\hat{\lambda }=\frac{3}{12}=\boxed{\mathrm{0.25.}}$ Intuition: bada total wait ⇒ chhota rate. Rate average wait ka reciprocal hai — dimensionally aur physically sensible.

Do unknowns mein log-likelihood: $\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2.$ $\mu$ mein partial (sirf last term $\mu$ par depend karta hai): $\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu )=0\Rightarrow \hat{\mu }=\bar{x}.$ Positive factor $\frac{1}{{\sigma }^2}$ sum ko kabhi zero nahi bana sakta, isliye $\hat{\mu }=\bar{x}$ ${\sigma }^2$ se independent hai — yahi wajah hai ki hum pehle $\mu$ solve kar sakte hain. ${\sigma }^2$ mein partial, phir $\mu =\bar{x}$ substitute karo: $\frac{\partial \ell }{\partial ({\sigma }^2)}=-\frac{n}{2{\sigma }^2}+\frac{1}{2({\sigma }^2{)}^2}\sum (x_i-\hat{\mu }{)}^2=0\Rightarrow {\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2.$ Ye ek profile likelihood move hai: easy parameter ko uske optimum par freeze karo, doosre ko optimise karo. EM Algorithm is "ek block optimise karo, phir agla" idea ko hidden variables tak generalise karta hai.

$\hat{\mu }=\bar{x}=\frac{2+4+4+4+6}{5}=\frac{20}{5}=4.$ Squared deviations: $(2-4{)}^2=4,0,0,0,(6-4{)}^2=4$, sum $=8$. ${\hat{\sigma }}_{\text{MLE}}^2=\frac{8}{5}=\boxed{1.6},s^2=\frac{8}{4}=\boxed{\mathrm{2.0.}}$ Kyun alag hain: MLE $\bar{x}$ reuse karta hai — usi data se estimated — isliye deviations thodi chhoti hoti hain (data apne mean ke paas hug karta hai). $n$ ki jagah $n-1$ se divide karna isko correct karta hai. MLE variance biased low hai lekin consistent hai (bias $\to 0$ jab $n\to \infty$). Dekho Bias-Variance Tradeoff.

Log-likelihood (valid $\theta$ ke liye, yaani $\theta \ge {\max }_i{x}_i$): $\ell (\theta )={\sum }_{i=1}^n(\log (2x_i)-2\log \theta )=\text{const}-2n\log \theta .$ Differentiate karo: $\frac{d\ell }{d\theta }=-\frac{2n}{\theta }$, jo hamesha negative hai — kabhi zero nahi hota! Isliye koi interior stationary point nahi hai. KYU calculus "fail" hota hai: $\ell$ strictly $\theta$ mein decreasing hai, isliye hum $\theta$ ko jitna ho sake chhota rakhna chahte hain. Lekin ek hard wall hai: har observed $x_i$ ko $x_i\le \theta$ satisfy karna chahiye, isliye $\theta \ge {\max }_i{x}_i$. $\theta$ ko $\max x_i$ se neeche dhakka dene par likelihood $0$ ho jaati hai (wo data impossible hoga). Maximum boundary par baitha hai (figure dekho): $\boxed{{\hat{\theta }}_{\text{MLE}}=\underset{i}{\max }{x}_i.}$ Ye ek support boundary MLE hai — peak ek corner hai, smooth summit nahi. Method of Moments se compare karo, jo instead $\mathbb{E}[X]=\frac{2\theta }{3}=\bar{x}\Rightarrow \theta =\frac{3\bar{x}}{2}$ solve karega — ek alag estimator, jo dikhata hai ki MLE aur MoM ka agree karna zaroori nahi.

Asymptotic normality (parent note property 2) kehta hai ki $\hat{\lambda }$ ka variance approximately $I(\lambda {)}^{-1}=\frac{{\lambda }^2}{n}$ hai, isliye standard error hai $\text{SE}(\hat{\lambda })\approx \sqrt{\frac{{\hat{\lambda }}^2}{n}}=\frac{\hat{\lambda }}{\sqrt{n}}=\frac{0.25}{\sqrt{3}}\approx \boxed{\mathrm{0.1443.}}$ Iska matlab: ye Gaussian bell ki width hai jo $\lambda$ ke baare mein hamari uncertainty describe karti hai. Cramér-Rao Bound ke by, koi bhi unbiased estimator asymptotically is variance ko beat nahi kar sakta — MLE efficient hai. ($n=3$ ke saath ye sirf ek rough approximation hai; guarantee ek large-$n$ statement hai.)

MLE $P(\text{data}\mid \theta )$ maximise karta hai ya $P(\theta \mid \text{data})$? ::: $P(\text{data}\mid \theta )$, the likelihood. 8 flips mein 6 heads ke liye Bernoulli MLE ::: $\hat{p}=0.75$. $\{4,7,9,10\}$ ka Gaussian MLE mean ::: $\hat{\mu }=7.5$. $\{2,4,6\}$ ke liye Exponential MLE ::: $\hat{\lambda }=0.25$. $\{2,4,4,4,6\}$ ka MLE variance ($\frac{1}{n}$) ::: ${\hat{\sigma }}^2=1.6$; unbiased $s^2=2.0$. Triangular-on-$[0,\theta ]$ MLE ::: $\hat{\theta }={\max }_i{x}_i$ (boundary!). Unbiased variance ke liye $n-1$ se kyun divide karte hain? ::: mean estimate karne par ek degree of freedom kharach hoti hai.