Worked examples — Statistical estimation — MLE, method of moments

4.9.17 · D3 · Maths › Probability Theory & Statistics › Statistical estimation — MLE, method of moments

Shuru karne se pehle, char chhoti reminders taaki koi bhi symbol unclear na rahe:

• $x_1,\dots ,x_n$ observed numbers hain (the data). $n$ yeh batata hai ki kitne hain.
• $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$ sample mean hai — bas "sab jodo, kitne hain us se divide karo".
• $\hat{\theta }$ (theta with a hat) matlab hai "true hidden number $\theta$ ka hamara guess". Hat ka matlab hamesha data se estimate kiya gaya hota hai.
• $L(\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$ likelihood hai: aap har data point ki density (ya probability) multiply karte ho, data ko fixed maanke aur $\theta$ ko woh dial maanke jise aap ghuma rahe ho. Yeh jawab deta hai "yeh $\theta$ kitna believable hai, jo maine dekha usse?" — dekho parent note. Kyunki bahut saare chhote numbers ka product differentiate karna mushkil hota hai, hum ise log karte hain:
• $\ell (\theta )=\log L(\theta )={\sum }_{i=1}^n\log f(x_i;\theta )$ log-likelihood hai. Kyunki $\log$ strictly increasing hai, woh $\theta$ jo $\ell$ maximize karta hai wahi $\theta$ $L$ bhi maximize karta hai — lekin sum, product se kaafi aasaan hota hai. Dekho Logarithms.

---

The scenario matrix

Har estimation problem jo aap miloge woh in case classes mein se kisi ek mein aata hai. Table unhe list karta hai aur us worked example ki taraf point karta hai jo har ek ko cover karta hai.

#	Case class	Kya mushkil banata hai	Example jo cover karta hai
A	Standard continuous, 1 parameter	baseline — differentiate & solve	Ex 1 (Exponential rate)
B	Discrete count data	pmf density nahi; factorials	Ex 2 (Poisson)
C	Do parameters ek saath	system of equations	Ex 3 (Normal $\mu ,{\sigma }^2$)
D	Boundary / support parameter	derivative = 0 koi answer nahi deta	Ex 4 (Uniform $U(0,\theta )$)
E	MoM aur MLE disagree karte hain	do recipes, do answers	Ex 5 (Uniform, MoM vs MLE)
F	Degenerate sample (sab $x_i$ equal hain, ya ek zero)	estimator blow up hota hai ya edge pe pin hota hai	Ex 6 (Exponential with $\bar{x}\to 0$)
G	MoM illegal (out-of-range) value deta hai	answer parameter constraints violate karta hai	Ex 7 (Binomial, unknown $p$ with fixed $m$)
H	Real-world word problem	English → likelihood mein translate karo	Ex 8 (call-centre waiting times)
I	Bias check / exam twist	kya "obvious" estimator honest hai?	Ex 9 (variance divisor $n$ vs $n-1$)
J	Invariance twist	$g(\theta )$ manga gaya hai, $\theta$ nahi	Ex 10 (median lifetime from $\hat{\lambda }$)

Ab hum har cell walk karenge.

---

Ex 1 — Case A: Exponential rate (baseline warm-up)

1. Likelihood, phir log-likelihood. $L(\lambda )=\prod \lambda e^{-\lambda x_i}={\lambda }^n{e}^{-\lambda \sum x_i}$, toh $\ell (\lambda )=\log L=n\ln \lambda -\lambda \sum x_i$. Yeh step kyun? Independence ki wajah se $L$ ek product ban jaata hai (upar di gayi definition); logging se woh product ek friendly sum mein badal jaata hai.
2. Differentiate karo, zero pe set karo (the score equation). ${\ell }^{\prime }(\lambda )=\frac{n}{\lambda }-\sum x_i=0$. Yeh step kyun? Ek smooth hill ka peak zero slope pe hota hai; ${\ell }^{\prime }=0$ solve karna woh peak dhundta hai.
3. Solve karo. $\hat{\lambda }=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}$. Yahan $\sum x_i=10$, $n=4$, toh $\hat{\lambda }=\frac{4}{10}=0.4$ per year. Yeh step kyun? Pure algebra — $\lambda$ ko isolate karo.
4. Confirm karo ki yeh maximum hai. ${\ell }^{\prime \prime }(\lambda )=-n/{\lambda }^2<0$ for all $\lambda >0$. Yeh step kyun? Zero-slope point valley bhi ho sakta hai; negative second derivative prove karta hai ki yeh hilltop hai.

---

Ex 2 — Case B: Poisson counts (discrete data)

1. Likelihood (pmf values ka product). $L(\lambda )={\prod }_{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}=\frac{e^{-n\lambda }{\lambda }^{\sum x_i}}{\prod x_i!}$. Yeh step kyun? Discrete data ke liye, hamare likelihood definition mein $f$ bas ek probability hai; hum phir bhi multiply karte hain kyunki trials independent hain.
2. Log-likelihood. $\ell (\lambda )=-n\lambda +(\sum x_i)\ln \lambda -\ln \prod x_i!$. Yeh step kyun? Aakhri term $\lambda$ mein constant hai (andar koi $\lambda$ nahi), toh differentiate karne par yeh gayab ho jaayega — hum ise sirf honesty ke liye rakhte hain. $\ln (ab)=\ln a+\ln b$ ke liye dekho Logarithms.
3. Differentiate & solve. ${\ell }^{\prime }(\lambda )=-n+\frac{\sum x_i}{\lambda }=0\Rightarrow \hat{\lambda }=\frac{\sum x_i}{n}=\bar{x}$. Yahan $\sum x_i=10$, $n=5$, toh $\hat{\lambda }=2$. Yeh step kyun? Peak ka zero slope hai; constant term promise ke mutabiq drop ho gaya.
4. Second-derivative check. ${\ell }^{\prime \prime }(\lambda )=-\frac{\sum x_i}{{\lambda }^2}<0$ (kyunki $\sum x_i>0$ hai). True maximum. ✓

---

Ex 3 — Case C: Normal, do parameters ek saath

1. Log-likelihood. $\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\ln (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}\sum (x_i-\mu {)}^2$. Yeh step kyun? Parent note ki normal density, logged.
2. Pehla score: $\mu$ mein differentiate karo. $\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}\sum (x_i-\mu )=0\Rightarrow \hat{\mu }=\bar{x}=13$. Yeh step kyun? Do unknowns hone par ${\sigma }^2$ ko fixed rakho aur $\mu$ ko top tak slide karo — ek partial derivative. ${\sigma }^2$ multiplier positive hai toh yeh change nahi karta ki zero kahan hai.
3. Doosra score: ${\sigma }^2$ mein differentiate karo. $\frac{\partial \ell }{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum (x_i-\mu {)}^2=0\Rightarrow {\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\hat{\mu }{)}^2$. Yeh step kyun? Ab ${\sigma }^2$ ko slide karo $\mu$ ko $\hat{\mu }$ pe pin karke. $\hat{\mu }=13$ plug karo: deviations hain $(-3,-1,1,3)$, squares $9+1+1+9=20$, toh ${\hat{\sigma }}^2=20/4=5$.
4. Confirm karo ki yeh maximum hai (multivariate check). $(\hat{\mu },{\hat{\sigma }}^2)$ pe Hessian (second derivatives ka matrix) diagonal hai: ${\partial }^2\ell /\partial {\mu }^2=-n/{\sigma }^2<0$ aur ${\partial }^2\ell /\partial ({\sigma }^2{)}^2=-n/(2{\sigma }^4)<0$ (${\hat{\sigma }}^2$ pe evaluate karke, $\sum (x_i-\hat{\mu }{)}^2=n{\hat{\sigma }}^2$ use karke), stationary point pe zero cross-term ke saath. Yeh step kyun? Do dimensions mein zero-gradient point saddle bhi ho sakta hai. Negative-definite Hessian (yahan: dono diagonal entries negative aur cross-term zero, toh dono eigenvalues negative) prove karta hai ki yeh genuine peak hai, na saddle na valley.

---

Ex 4 — Case D: Boundary parameter, jahan differentiation JHOOTH BOLTA HAI

Yahan geometry kaam karta hai, algebra nahi — toh steps padhne se pehle figure ko dhyaan se study karo.

Yeh figure kaise padhein. Horizontal axis $\theta$ hai, upper bound ka candidate value. Vertical axis likelihood $L(\theta )$ hai page ke upar hamare definition se. Teen cheezein drawn hain:

• Baayein coral shaded band ($\theta <5$) illegal region hai: agar $\theta$ sabse bade data value $5$ se chhota hai, toh woh data point $[0,\theta ]$ ke bahar gir jaata hai, uski density $0$ ho jaati hai, aur poora product collapse ho jaata hai $L=0$ tak.
• Lavender curve ($\theta \ge 5$) $L(\theta )={\theta }^{-n}$ hai, jo steadily neeche ki taraf slope karta hai — bada $\theta$ hamesha chhoti likelihood deta hai.
• Red dot peak mark karta hai: yeh exactly legal region ke left edge par baith jaata hai, $\hat{\theta }=5$. Yahan koi smooth "top of a hill" nahi hai; maximum ek corner hai.
• Bottom par mint ticks char data points $(2,3,4,5)$ hain; note karo ki peak sabse bade par land karta hai.

1. Likelihood. $L(\theta )=\prod \frac{1}{\theta }=\frac{1}{{\theta }^n}$ — lekin sirf tab jab har $x_i\le \theta$ ho; warna koi factor $0$ hai aur $L=0$ (figure mein coral band). Yeh step kyun? Support ($0\le x\le \theta$) wahan hai jahan tricky part chhupa hai. Agar $\theta$ sabse bade data point se chhota hai, toh woh point model ke under "impossible" tha, poore product ko khatam kar deta hai.
2. Log-likelihood, likha hua. $\theta \ge \max x_i$ ke liye hamare paas $L(\theta )={\theta }^{-n}$ hai, toh $\ell (\theta )=\ln ({\theta }^{-n})=-n\ln \theta$. $\theta <\max x_i$ ke liye, $L=0$ aur $\ell =-\infty$ (zero ka log). Yeh step kyun? Derivative ke baare mein baat karne se pehle hume $\ell$ explicitly chahiye. $\ln (a^b)=b\ln a$ use karke (dekho Logarithms) ${\theta }^{-n}$ clean $-n\ln \theta$ mein badal jaata hai.
3. Picture dekho, derivative nahi. Legal piece differentiate karne par ${\ell }^{\prime }(\theta )=\frac{d}{d\theta }(-n\ln \theta )=-\frac{n}{\theta }$ milta hai, jo $\theta >0$ ke liye kabhi zero nahi hota. Toh score equation ka koi solution nahi hai. Iske bajaaye lavender curve padho: $\ell (\theta )=-n\ln \theta$ strictly decreasing hai, toh $L$ chahta hai $\theta$ jitna chhota ho sake — lekin usse at least $\max x_i=5$ rehna hi padhega. Yeh step kyun? Differentiation yeh maximum nahi dhundh sakta; peak allowed region ke edge par baith jaata hai, aur sirf graph hi ise reveal karta hai.
4. Maximizer padho. Sabse badi $L$ with $\theta \ge \max x_i$ sabse chhote legal $\theta$ par milti hai: $\hat{\theta }={\max }_i{x}_i=5$ (red dot). Yeh step kyun? Ek monotone-decreasing curve par, $\theta \ge 5$ tak restricted, top left end par hai.

---

Ex 5 — Case E: MoM vs MLE disagree karte hain (wahi uniform)

1. Population mean. $E[X]=\theta /2$ (interval $[0,\theta ]$ ka centre). Yeh step kyun? Ek parameter → ek moment match karo (pehla). Law of Large Numbers dekho ki kyun $\bar{x}\to E[X]$.
2. Sample mean se match karo. $\frac{\hat{\theta }}{2}=\bar{x}\Rightarrow {\hat{\theta }}_{\text{MoM}}=2\bar{x}$. Yahan $\bar{x}=(2+5+3+4)/4=3.5$, toh ${\hat{\theta }}_{\text{MoM}}=7$. Yeh step kyun? Theoretical moment ko measured moment ke equal set karo, solve karo.
3. Compare karo. MLE ne $5$ kaha, MoM ne $7$ kaha. Yeh kyun matter karta hai? Dono "reasonable" hain, lekin MoM yahan zyada variable hai aur doosre samples par $\max x_i$ se neeche bhi ja sakta hai — jo logically impossible hoga (aap $\theta <5$ nahi rakh sakte jab aapne already ek $5$ dekh liya).

---

Ex 6 — Case F: Degenerate sample (estimator blow up karta hai)

1. Directly compute karo. $\bar{x}=0.01$, toh $\hat{\lambda }=1/0.01=100$ per unit time. Yeh step kyun? Bas Ex 1 ka formula apply karo.
2. Limiting behaviour. Jab $\bar{x}\to 0^{+}$, $\hat{\lambda }=1/\bar{x}\to +\infty$. Yeh step kyun? Near-zero lifetimes ka pile matlab hai "cheezein almost instantly fail ho jaati hain" → bahut bada failure rate. Model sensibly respond karta hai, lekin estimate unbounded hai.
3. True degenerate case $\bar{x}=0$ se guard karo. Agar har $x_i=0$ exactly hai, $\hat{\lambda }$ undefined hai ($1/0$). Physically matlab hai "kabhi koi time nahi guzarta" — exponential model toot gaya hai; aapko model reject karna hoga, $\infty$ report nahi karna. Yeh step kyun? Matrix ki genuinely-degenerate cell cover karna: zero-mean sample ek modelling red flag hai, koi number nahi.

---

Ex 7 — Case G: MoM illegal (out-of-range) value produce karta hai

1. Honest MoM (first moment). $E[X]=mp$, toh $\hat{p}=\frac{\bar{x}}{m}$. Yahan $\bar{x}=(6+7+8)/3=7$, $m=8$, deta hai $\hat{p}=7/8=0.875\in [0,1]$. ✓ Yeh step kyun? Ek parameter → pehla moment match karo; mean equation $p\le 1$ respect karta hai kyunki $\bar{x}\le m$ hamesha hota hai.
2. Flawed MoM (variance match karo instead). Sample variance ($n$ se divide karke) $\overline{x^2}-{\bar{x}}^2$ hai. Yahan $\overline{x^2}=(36+49+64)/3=149/3\approx 49.667$ hai, aur ${\bar{x}}^2=49$ hai, toh sample variance hai $149/3-49=2/3\approx 0.667$. Model variance iske equal set karo: $mp(1-p)=2/3$, yani $8p(1-p)=2/3\Rightarrow p(1-p)=1/12\approx 0.0833$. Yeh step kyun? Yeh ek legitimate-looking alternative moment equation hai — lekin variance ignore karta hai ki $\bar{x}$ ceiling $m$ ke kitna paas hai, toh yeh $p$ ko kisi silly jagah bhej sakta hai.
3. Quadratic solve karo. $p-p^2=\frac{1}{12}\Rightarrow p^2-p+\frac{1}{12}=0\Rightarrow p=\frac{1\pm \sqrt{1-\frac{1}{3}}}{2}=\frac{1\pm \sqrt{2/3}}{2}$. Numerically do roots hain $p\approx 0.908$ aur $p\approx 0.092$. Yeh step kyun? Variance equation $p$ mein quadratic hai, toh yeh do candidate answers deta hai — pehle se hi mean equation ke single clean root se bura. Bada root $0.908$ ek mean $mp=8(0.908)\approx 7.27$ predict karta hai, mean-based $\hat{p}$ se disagree karta hua; ab aapko choose karna padega, aur MoM koi principled tarika nahi deta.
4. Genuinely illegal case expose karo. Model variance $mp(1-p)$ kabhi bhi apne maximum $m\cdot \frac{1}{4}=2$ ($p=0.5$ par milta hai) se zyada nahi ho sakta. Toh agar kisi sample ki variance 2 se badi ho, equation $8p(1-p)=v$ mein square root ke andar $1-\frac{v}{2}<0$ hoga — koi real solution nahi, yani $\hat{p}$ complex nikaega, isliye bilkul illegal. Yeh kyun matter karta hai? Yeh woh cell hai jahan MoM apna bura naam kamaata hai: "galat" moment aapko do roots ka ambiguous pair de sakta hai ya, jab sample variance model ceiling se zyada ho, koi legal value hi nahi. MLE, jo support aur full likelihood ko respect karta hai, kabhi out-of-range answer nahi de sakta.

---

Ex 8 — Case H: Real-world word problem

1. English ko model mein translate karo. "Random events ke beech ka time" → Exponential($\lambda$), jahan $\lambda$ = calls per minute, density $f(x;\lambda )=\lambda e^{-\lambda x}$. Yeh step kyun? Distribution choose karna asli skill hai; algebra dobara Ex 1 wala hai. Dekho Probability Distributions.
2. $\bar{x}$ compute karo. $\bar{x}=(0.5+1.5+2.0+1.0+3.0)/5=8/5=1.6$ minutes. Yeh step kyun? Exponential ka MLE data pe sirf sample mean ke through depend karta hai, toh yeh ek hi number hai jo hume chahiye.
3. Log-likelihood, phir solve karo (Ex 1 reuse karo). $\ell (\lambda )=n\ln \lambda -\lambda \sum x_i$; ${\ell }^{\prime }(\lambda )=n/\lambda -\sum x_i=0$ set karne par $\hat{\lambda }=1/\bar{x}$ milta hai. Yeh step kyun? Ex 1 se identical machinery; hum result quote karte hain rather than scratch se re-derive karne ke.
4. (a) Rate. $\hat{\lambda }=1/\bar{x}=1/1.6=0.625$ calls per minute. Yeh step kyun? $\hat{\lambda }=1/\bar{x}$ mein direct substitution.
5. (b) Mean wait. Exponential mean hi $1/\lambda$ hai, toh estimated mean wait hai $\widehat{1/\lambda }=1/\hat{\lambda }=\bar{x}=1.6$ minutes. Yeh step kyun? MLE invariance se, $g(\lambda )$ ka MLE $g(\hat{\lambda })$ hai (parent note) — yahan $g(\lambda )=1/\lambda$.

---

Ex 9 — Case I: Bias check / exam twist

Yeh figure kaise padhein. Imagine karo ki poora experiment hazaaron baar repeat kiya; har baar aap ek variance estimate compute karte ho. Do coloured humps dikhate hain ki woh estimates kaise spread out hote hain. Coral hump MLE hai ($n$ se divide karke): iska centre true ${\sigma }^2$ mark karne wali dashed vertical line ke baayein baith jaata hai — average par yeh under-shoot karta hai. Mint hump unbiased estimator hai ($n-1$ se divide karke): iska centre dashed line par baith jaata hai. Coral hump ka woh leftward shift exactly bias hai.

1. Bias fact. $E[{\hat{\sigma }}_{\text{MLE}}^2]=\frac{n-1}{n}{\sigma }^2$, toh MLE average par underestimate karta hai. Yeh step kyun? Variance + bias${}^2$ mein split karna dikhata hai ki $n$ se divide karna estimate ko shrink karta hai (dekho Bias and Variance). Picture coral cloud ko true ${\sigma }^2$ ke baayein baitha dikhati hai.
2. Divisor fix karo. $\frac{n}{n-1}$ se multiply karo: unbiased estimator hai $s^2=\frac{1}{n-1}\sum (x_i-\bar{x}{)}^2$. Yeh step kyun? $\frac{n}{n-1}\cdot \frac{n-1}{n}{\sigma }^2={\sigma }^2$ — correction exactly shrink cancel kar deta hai.
3. Numbers. Sum of squared deviations $=20$ (Ex 3 se). Unbiased: $s^2=20/(4-1)=20/3\approx 6.667$. Yeh step kyun? Wahi numerator, $n=4$ ki jagah $n-1=3$ se divide karo.

---

Ex 10 — Case J: Invariance twist ($g(\theta )$ manga gaya hai)

1. Invariance shortcut. Agar $\hat{\lambda }$, $\lambda$ ka MLE hai, toh kisi bhi function $g$ ke liye, $g(\hat{\lambda })$, $g(\lambda )$ ka MLE hai (parent note). Yeh step kyun? Yeh hume ek poori nayi likelihood re-differentiate karne se bachata hai — hum bas $\hat{\lambda }$ ko median formula mein plug karte hain.
2. Function. $g(\lambda )=\frac{\ln 2}{\lambda }$ (rate ka function ke roop mein median). $\ln 2\approx 0.6931$ ke liye dekho Logarithms. Yeh step kyun? $g$ identify karo, phir invariance apply karo.
3. Plug in karo. $\hat{m}=\frac{\ln 2}{\hat{\lambda }}=\frac{0.6931}{0.4}\approx 1.733$ years. Yeh step kyun? $\hat{\lambda }=0.4$ ka direct substitution.

---

Recall Har cell ka ek-line summary

Smooth 1-param ke liye differentiate & solve (A), counts ke liye same (B) aur do params ke liye (C); support parameters ke liye boundary inspect karo (D); MoM & MLE disagree kar sakte hain (E); degenerate samples guard karo jahan estimates blow up hote hain (F); MoM out of range ja sakta hai (G); word problems model-choice + Ex 1 hain (H); MLE variance biased hai, $n-1$ se fix karo (I); $g(\theta )$ ke liye invariance se $\hat{\theta }$ reuse karo (J).

Reveal-yourself checks: