Statistical estimation — MLE, method of moments

Method of Moments (the easy one first)

Maximum Likelihood Estimation

Worked example MLE for Exponential — full derivation

$f(x;\lambda)=\lambda e^{-\lambda x}$ .

Step 1 Likelihood: $L=\prod \lambda e^{-\lambda x_i}=\lambda^n e^{-\lambda\sum x_i}$ . Why? Product of densities (independence).
Step 2 Log: $\ell = n\ln\lambda - \lambda\sum x_i$ . Why log? Converts power & exp into sum.
Step 3 Differentiate & set $0$ : $\ell'=\dfrac{n}{\lambda}-\sum x_i = 0$ . Why? Maximum has zero slope.
Step 4 Solve: $\hat\lambda = \dfrac{n}{\sum x_i}=\dfrac1{\bar x}$ .
Step 5 Confirm max: $\ell''=-n/\lambda^2<0$ . ✓ Here MLE = MoM (lucky!).

Figure — Statistical estimation — MLE, method of moments

Properties to judge an estimator

MLE vs MoM — when to use which

Common mistake Steel-manned errors

"Likelihood is a probability of $\theta$ ." Feels right because $L$ looks like a density. Fix: $L$ is a function of $\theta$ for fixed data; it doesn't integrate to 1 over $\theta$ . It's the data's probability, read backwards.
"Maximize $L$ directly." Feels right, it's the actual target. Fix: products underflow and are hard to differentiate; maximize $\ell=\log L$ — same maximizer.
"MLE is always unbiased." Feels right since it's "optimal." Fix: $\hat\sigma^2_{MLE}$ divides by $n$ → biased. MLE optimizes likelihood, not unbiasedness.
"Forgetting the second-derivative / boundary check." A score-equation root can be a minimum or sit at a boundary (e.g. uniform $U(0,\theta)$ : MLE is $\hat\theta=\max x_i$ , found by inspection, not by differentiation).

What does MLE maximize?

The likelihood

L(\theta)=\prod f(x_i;\theta)

, i.e. the probability of the observed data as a function of

\theta

Why take the log of the likelihood?

log is strictly increasing so it preserves the maximizer, and it turns the product into a sum that's easy to differentiate.

Method of moments recipe?

Equate population moments

\mu_k=E[X^k]

to sample moments

\frac1n\sum x_i^k

for

k=1..p

and solve for the

p

parameters.

MLE of exponential rate

\lambda

\hat\lambda=1/\bar x

MLE of Bernoulli

p

\hat p=\bar x=

sample proportion of successes.

Is the MLE of

\sigma^2

for a normal unbiased?

No; it divides by

n

, giving

E[\hat\sigma^2]=\frac{n-1}{n}\sigma^2

. The unbiased version divides by

n-1

MSE decomposition?

\text{MSE}=\text{Var}(\hat\theta)+\text{bias}^2

When does differentiating fail to find the MLE?

At boundary/support-dependent parameters, e.g.

U(0,\theta)

where

\hat\theta=\max x_i

Invariance property of MLE?

\hat\theta

is the MLE of

\theta

, then

g(\hat\theta)

is the MLE of

g(\theta)

Define a consistent estimator.

\hat\theta\xrightarrow{P}\theta

n\to\infty

Recall Feynman: explain to a 12-year-old

Imagine you find footprints in the mud. You don't know which animal made them, but you ask: which animal would most likely leave exactly these prints? That animal is your best guess. That's MLE — pick the cause that best explains what you saw. The method of moments is even simpler: if you know that on average dogs weigh 20kg, and your mystery pile of dogs averages 20kg, you assume they're normal dogs — you match the averages and solve. Both are just clever ways to turn "data we saw" into "best guess of the hidden number."

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Estimation ka matlab hai: humare paas data hai jo kisi distribution se aaya hai, lekin uska parameter (jaise mean ya rate $\lambda$ ) hidden hai. Hum us hidden number ka best guess lagana chahte hain. Do popular tareeke hain — MLE aur Method of Moments.

MLE (Maximum Likelihood) kehta hai: woh $\theta$ choose karo jiske under tumhara actual observed data sabse zyada probable lage. Mathematically likelihood $L(\theta)=\prod f(x_i;\theta)$ banate hain, fir uska log lete hain (kyunki log product ko sum bana deta hai aur maximizer same rehta hai), differentiate karke zero set karte hain. Jaise exponential ke liye answer aata hai $\hat\lambda=1/\bar x$ — bilkul intuitive!

Method of Moments aur bhi simple hai: population ka mean/variance jo formula me $\theta$ ke terms me aata hai, usse sample ke mean/variance ke barabar rakh do aur solve kar do. Law of Large Numbers kehta hai sample moments true moments ke paas chale jaate hain, isliye yeh kaam karta hai.

Important baat: MLE hamesha unbiased nahi hota — normal ka $\hat\sigma^2$ $n$ se divide karta hai, isliye thoda chhota (biased) aata hai, aur unbiased ke liye $n-1$ chahiye. Exam me 80% kaam yahi hai: $\ell$ likho, differentiate karo, solve karo, aur second derivative check kar lo. Boundary wale cases (jaise $U(0,\theta)$ ) me differentiate mat karo — wahan $\hat\theta=\max x_i$ hota hai.

Statistical estimation — MLE, method of moments

WHAT is an estimator?

Method of Moments (the easy one first)

Maximum Likelihood Estimation

Properties to judge an estimator

MLE vs MoM — when to use which

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability Theory & Statistics

Connections