Visual walkthrough — Statistical estimation — MLE, method of moments

4.9.17 · D2 · Maths › Probability Theory & Statistics › Statistical estimation — MLE, method of moments

Hum exponential distribution ko apna worked example isliye use karte hain kyunki isme guess karne ke liye exactly ek hi unknown number hai, to hum sab kuch ek flat page par draw kar sakte hain.

Step 0 — Koi symbol aane se pehle ki kahani

Data measured lifetimes hain: $x_1,x_2,\dots ,x_n$

• $x_i$ ::: bulb number $i$ ki lifetime (ek positive number, jaise ghante).
• $n$ ::: humne kitne bulbs measure kiye.

Abhi tak kuch bhi scary nahi hai — bas positive numbers ki ek list.

Step 1 — Ek bulb ki probability kaisi dikhti hai

YEH SHAPE KYUN. Exponential curves "kisi random event ka wait karna" model karne ke liye natural hain — memoryless, hamesha positive, ek knob. Woh ek knob hi exactly woh hai jo hum estimate karna chahte hain.

PICTURE. Agla figure do exponential curves dikhata hai: ek bada-$\lambda$ (fast, steep) wala aur ek chhota-$\lambda$ (slow, flat) wala. Dhyan do ki bada $\lambda$ ooncha start hota hai lekin tezi se girta hai.

Step 2 — Saare bulbs ki probability ek saath

• ${\prod }_{i=1}^n$ ::: "$i=1$ se $n$ tak ke liye inhe multiply karo" — sum ka product cousin.
• $L(\lambda )$ ::: likelihood; data ab frozen hain, aur $\lambda$ woh variable hai jise hum slide karte hain.

PRODUCT KYUN. Independent events: "A aur B" ki probability $P(A)\times P(B)$ hoti hai. Ise saare $n$ bulbs par chain karo.

Ab pieces ikatta karo. $\lambda$ ke $n$ copies hain (har bulb ke liye ek), aur exponents add ho jaate hain: $L(\lambda )={\lambda }^n{e}^{-\lambda {\sum }_{i=1}^n{x}_i}.$

• ${\lambda }^n$ ::: $n$ height-knobs multiply hue.
• ${\sum }_{i=1}^n{x}_i$ ::: saari lifetimes ka total; exponents $-\lambda x_1,-\lambda x_2,\dots$ merge ho gaye kyunki $e$ ki powers ko multiply karne se unke exponents add hote hain.

PICTURE. Neeche, ek fixed toy dataset ke liye $L(\lambda )$ ko $\lambda$ ke against plot kiya gaya hai. Yeh utha, peak par pahuncha, phir gira — ek best $\lambda$ hai. Woh peak hi hamara target hai.

Step 3 — $L$ ko directly maximize kyun nahi karte (log lo)

• $\ell (\lambda )$ ::: log-likelihood ("script-L").
• $\ln$ ::: natural logarithm — $e^{(\cdot )}$ ka inverse (dekho Logarithms). Yeh ${\lambda }^n$ ko $n\ln \lambda$ mein badalta hai aur $e^{-\lambda \sum x_i}$ ko $-\lambda \sum x_i$ mein.

LOG KYUN. Do wajah:

1. $\ln$ strictly increasing hai — yeh order kabhi reverse nahi karta — isliye $\lambda$ jo $L$ maximize karta hai wahi $\ell$ bhi maximize karta hai. Same peak location.
2. Product ${\lambda }^n{e}^{-\lambda \sum x_i}$ ek sum $n\ln \lambda -\lambda \sum x_i$ ban jaata hai, jise differentiate karna kahin aasaan hai. (Aur computer par tiny numbers zero par underflow hone se bhi bachata hai.)

PICTURE. Agla figure $L(\lambda )$ aur $\ell (\lambda )$ dono ko same $\lambda$-axis par overlay karta hai. Dotted vertical line dikhata hai ki unke peaks bilkul same $\lambda$ par hain — logging ne shape badla lekin maximum ki location nahi.

Step 4 — Peak ka slope zero hota hai (score equation)

Term by term, $\ell =n\ln \lambda -\lambda \sum x_i$ ko $\lambda$ ke respect mein differentiate karke:

• $\frac{d}{d\lambda }\left(n\ln \lambda \right)=\frac{n}{\lambda }$ ::: $\ln \lambda$ ka derivative $1/\lambda$ hai, constant $n$ se multiply.
• $\frac{d}{d\lambda }(-\lambda \sum x_i)=-\sum x_i$ ::: $\sum x_i$ sirf ek fixed number hai (data frozen!), isliye yeh term $\lambda$ mein linear hai aur iska slope wahi constant hai.

ZERO KYUN. Derivative $\ell$ ka slope hai. Upar jaate waqt positive hota hai, neeche aate waqt negative; peak par exactly $0$ hota hai. Woh crossing point hi hamara estimate hai.

PICTURE. Figure $\ell (\lambda )$ ko teen jagah tangent line ke saath dikhata hai: utha hua (positive slope), peak par (flat, slope $0$), aur gira hua (negative slope). Flat wala winner hai.

Step 5 — $\hat{\lambda }$ ke liye solve karo

• $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$ ::: sample mean (average lifetime). Isliye $n/\sum x_i$ exactly $1/\bar{x}$ hai.
• $\hat{\lambda }$ mein hat ::: yeh batata hai ki yeh data se bana hua ek estimate hai, na ki true hidden $\lambda$.

YEH SENSIBLE KYUN HAI. Exponential ke liye, true mean lifetime $1/\lambda$ hoti hai (lambi zindagi wale bulbs ↔ chhota rate). Agar bulbs average $\bar{x}$ ghante chale, to sabse natural rate jo blame ki ja sake woh $1/\bar{x}$ hai. Math exactly common sense par aa gaya — aur Law of Large Numbers ke hisaab se, $n$ barhne par $\bar{x}$ true mean ke pass aata jaata hai, isliye $\hat{\lambda }\to \lambda$ (consistent hai).

PICTURE. Neeche, toy dataset ke dots $x$-axis par hain; unka balance point $\bar{x}$ hai; recovered curve $f(x;1/\bar{x})$ unke through draw ki gayi hai. Yeh woh exponential hai jo data ko sabse achha "hug" karta hai.

Step 6 — Confirm karo ki yeh peak hai, valley nahi

• $\frac{d^2\ell }{d{\lambda }^2}$ ::: second derivative — slope khud kitna badal raha hai.
• $n/\lambda$ ko phir differentiate karo: $\frac{d}{d\lambda }(n{\lambda }^{-1})=-n{\lambda }^{-2}=-n/{\lambda }^2$. $-\sum x_i$ term constant hai, isliye woh gayab ho jaata hai.
• $-n/{\lambda }^2<0$ ::: hamesha negative ($n>0$, ${\lambda }^2>0$), matlab $\ell$ har jagah downward curve karta hai — ek single hilltop.

KYUN MATTER KARTA HAI. Negative curvature = concave = genuinely maximum. Accidentally minimum report karne ka koi risk nahi. Kyunki $\ell$ poore $\lambda >0$ range par concave hai, peak unique hai.

PICTURE. Figure peak mark karta hai aur dikhata hai ki curve dono sides par downward bend kar rahi hai ("frown" shape) — maximum ka visual signature.

Step 7 — Degenerate cases (reader ko kabhi stranded mat chhordo)

• Saari lifetimes zero ($\bar{x}=0$). Tab $\hat{\lambda }=1/0$ undefined hai — formula "chahta" hai infinite rate. Picture: har death instant hai, jise exponential sirf $\lambda \to \infty$ se mimic kar sakta hai ($0$ par infinitely tall spike). Real bulbs aisa kabhi nahi karte, lekin tumhe boundary pehchanni chahiye.
• Ek bulb ($n=1$). Phir bhi kaam karta hai: $\hat{\lambda }=1/x_1$. $\ell$ ka peak abhi bhi exist karta hai; bas noisier guess hai (badi variance — dekho Bias and Variance).
• Support-dependent parameter alag hota hai. Uniform $U(0,\theta )$ ke liye, differentiate karne par koi interior solution nahi milta — likelihood $\theta$ ke $\max x_i$ ki taraf shrink hote waqt badhti rehti hai, phir zero ho jaati hai. MLE $\hat{\theta }={\max }_i{x}_i$ hai, jo inspection se milta hai, calculus se nahi. Yahi woh warning hai jo parent note ne flag ki thi.

PICTURE. Figure smooth exponential $\ell$ (achha interior peak) ko uniform ke cliff-edge likelihood se contrast karta hai, jiska maximum sabse bade data point par hota hai.

Ek-picture summary

Upar sab kuch ek single diagram mein compress: data → product → log → derivative $=0$ → solve → check.