Maximum likelihood estimation (MLE): Question bank

1.3.16 · D5 · AI-ML › Probability & Statistics › Maximum likelihood estimation (MLE)

Shuru karne se pehle, teen words jo hum baar baar use karenge, plain language mein:

"Peak", "curvature", aur "multiple bumps" words ko concrete banane ke liye, questions work karte waqt in do chalkboard pictures ko dimag mein rakho.

True or false — justify karo

MLE hamesha true parameter value return karta hai.

False. Yeh woh value return karta hai jo is dataset ko best explain kare; woh truth ka ek noisy pointer hai. Yeh truth par sirf tab converge karta hai jab

n \to \infty

(consistency), kisi finite sample ke liye nahi.

L (θ)

aur

ℓ (θ) = lo g L (θ)

ko maximize karna same estimate deta hai.

True.

lo g

strictly increasing hai, isliye maximum ki location preserve hoti hai — yeh peak ki height move karta hai, kabhi uski position nahi (pehle figure mein left curve dekho).

Likelihood function

L (θ ∣ X)

ek probability distribution hai

θ

par.

False. Yeh

θ

par integrate karke jo chahe woh de sakta hai, 1 nahi. Yeh

p (X ∣ θ)

hai jo

θ

ke function ki tarah dekha jaata hai — ek distribution data axis par rehti hai, parameter axis par nahi.

Variance ka MLE unbiased hota hai.

False.

\overset{σ}{^}_{M L E}^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}

mein

n

se divide hota hai aur systematically underestimate karta hai; unbiased version

n - 1

se divide karta hai. Dekho Bias-Variance Tradeoff.

Agar ek estimator biased hai toh woh unbiased se worse choice zaroor hoga.

False. MLE thodi si bias ke badle lower variance leta hai; total error (mean-squared error) unbiased rival se chhota ho sakta hai. Yahi exactly Bias-Variance Tradeoff hai.

Log-likelihood hamesha concave hoti hai, isliye iska derivative zero karna hamesha maximum dhund leta hai.

False. Concavity kai achhe models (Bernoulli, Gaussian mean) ke liye hoti hai lekin generally nahi — doosra figure ek do-bump curve dikhata hai jahan "derivative = 0" ke teen solutions hain aur sirf ek global max hai.

MLE aur Maximum A Posteriori (MAP) estimation alag methods hain jo kabhi coincide nahi karte.

False. MLE, MAP ke barabar hota hai jab prior flat (uniform) ho parameter par, kyunki constant prior peak ko move nahi karta — lekin dhyan rakho ki unbounded space par uniform prior improper hota hai (1 tak integrate nahi karta), isliye coincidence bilkul exact sirf bounded region par ya formal limit ke roop mein hota hai. Dekho Bayesian Estimation.

Zyada i.i.d. data add karna log-likelihood peak ko sirf sharpen kar sakta hai (kabhi flatten nahi).

True expectation mein, sample-path by sample-path nahi. Average par har point curvature add karta hai (Fisher Information) aur peak

1/ n

ki tarah tighten hoti hai, lekin ek akela unlucky point momentarily observed curve ko flatten ya shift kar sakta hai — guarantee expected curvature ke baare mein hai, har individual sample ke baare mein nahi.

Cramér-Rao bound guarantee karta hai ki koi estimator MLE ki variance ko kabhi beat nahi kar sakta.

False. Cramér-Rao Bound unbiased estimators ki variance ko lower-bound karta hai; ek biased estimator iske neeche ja sakta hai. MLE bound ko sirf asymptotically meet karta hai, chhote

n

ke liye nahi.

Error dhundo

"Main data diye hue sabse likely parameters chahta hun, isliye main

P (θ ∣ X)

maximize karta hun."

Woh posterior hai, yaani Bayesian Bayesian Estimation / MAP. Plain MLE mein koi prior nahi hota aur woh

P (X ∣ θ)

maximize karta hai; bina prior

P (θ)

ke aap

P (θ ∣ X)

likh bhi nahi sakte.

"1000 points ke saath main seedha saare

p (x_{i} ∣ θ)

multiply kar dunga aur usse maximize karunga."

1000 sub-one probabilities ka product floating point mein

0

tak underflow ho jaata hai (jaise

0. 5^{1000} \approx 1 0^{- 301}

). Optimize karne se pehle product ko numerically safe sum mein badle ke liye log lo.

"Maine

\frac{d ℓ}{d θ} = 0

find kar liya, toh bas ho gaya — yahi MLE hai."

Zero derivative ek critical point mark karta hai, jo minimum, saddle, ya boundary artefact ho sakta hai. Confirm karo ki yeh max hai: check karo

\frac{d ^{2} ℓ}{d θ ^{2}} < 0

, ya

ℓ

ko critical points aur boundaries par compare karo.

"10 flips mein 7 heads wale coin ke liye,

p

ka MLE

0.5

ke aaspaas hai kyunki coins fair hote hain."

MLE aapki prior belief ignore karta hai aur sirf data padhta hai:

\overset{p}{^} = k / n = 0.7

0.5

ki taraf pull ke liye ek prior chahiye — woh Bayesian Estimation hai, MLE nahi.

"Invariance property ka matlab hai ki agar

\hat{θ}

unbiased hai toh

g (\hat{θ})

bhi unbiased hai."

Invariance kehta hai

g (θ) = g (\hat{θ})

— estimate cleanly transform hota hai. Unbiasedness nonlinear

g

se preserve nahi hoti (Jensen's inequality se

E [g (\hat{θ})] \neq = g (E [\hat{θ}])

"Invariance theorem mujhe MLE ko bilkul kisi bhi function

g

ke through push karne deta hai."

Apne sabse clean form mein theorem

g

se maangta hai ki woh ek one-to-one, differentiable reparameterization ho taaki "kaunsa

θ

maximize karta hai" clearly translate ho "kaunsa

g (θ)

maximize karta hai" mein. Many-to-one

g

ke liye claim precise rakhne ke liye aapko ek induced likelihood define karni padegi.

"MLE optimal hai, isliye yeh har sample size par efficient hona chahiye."

Efficiency (Cramér-Rao Bound ko hit karna) ek asymptotic (

n \to \infty

) statement hai. Chhote

n

ke liye doosra estimator chhoti variance ya chhota total error rakh sakta hai.

Why questions

Hum likelihood ka log kyun lete hain?

Teen reasons: yeh products ko sums mein badalta hai (cleaner derivatives), yeh floating-point underflow rokta hai, aur monotonic hone ki wajah se maximum ki location untouched rehti hai.

i.i.d. assumption kyun likelihood ko product ke roop mein likhne deti hai?

Independence ka matlab joint probability factorize hoti hai,

P (x_{1}, \dots, x_{n} ∣ θ) = \prod_{i} p (x_{i} ∣ θ)

; "identically distributed" ka matlab har factor same

p (\cdot ∣ θ)

use karta hai.

Gaussian mean ka MLE sirf sample mean

\overset{x}{ˉ}

kyun hai?

Gaussian log-likelihood hai

ℓ (μ) = const - \frac{1}{2} \sum (x_{i} - μ)^{2}

, isliye isse maximize karne ka matlab hai

\sum (x_{i} - μ)^{2}

minimize karna. Differentiate karo:

\frac{d}{d μ} \sum (x_{i} - μ)^{2} = - 2 \sum (x_{i} - μ) = 0

, jo deta hai

\sum x_{i} = n μ

, yaani

μ = \overset{x}{ˉ}

. Geometrically,

\overset{x}{ˉ}

woh akela point hai jiska saare data points se total squared pull balance out hota hai — "centre of mass" jo koi doosra point beat nahi kar sakta.

Log-likelihood ka second derivative maximum check karne se aage kyun itna important hai?

Iska negative expectation hi Fisher Information

I (θ)

hai, jo MLE ki asymptotic variance

I (θ)^{- 1} / n

set karta hai — sharp curvature ka matlab confident, low-variance estimate hai.

MLE ko "frequentist" kyun kaha jaata hai "Bayesian" ke bajaye?

Yeh

θ

ko ek fixed unknown constant treat karta hai jisme koi probability distribution nahi hoti; sirf data random hai. Bayesian Estimation ki jagah

θ

par ek prior distribution daali jaati hai aur use update kiya jaata hai.

MLE Method of Moments se efficiency mein kyun beat kar sakta hai?

MLE assumed density ki poori shape use karta hai (woh saari information jo model encode karta hai), jabki Method of Moments sirf kuch summary moments match karta hai aur information throw away kar sakta hai.

Edge cases

Agar aap

k = 0

heads observe karo

n

flips mein toh Bernoulli MLE kya hai?

\overset{p}{^} = 0/ n = 0

. Estimate

[0, 1]

ki boundary par baithta hai, isliye "derivative = 0" trick fail hoti hai; maximum edge par hai aur koi bhi future head assigned-probability zero hogi — chhote data ke saath over-confidence ka warning sign.

Symmetric case

k = n

(saare heads) mein Bernoulli MLE kya hai?

\overset{p}{^} = n / n = 1

. Yeh mirror-image boundary problem hai: peak

[0, 1]

ke right edge par baithti hai, interior derivative kabhi zero nahi hiti, aur model kisi bhi future tail ko probability zero assign karega — phir se over-confident kyunki chhota sample rare outcomes rule out nahi kar sakta.

Kya hota hai jab aapke saare data points identical hote hain (zero sample variance)?

Gaussian variance MLE

\overset{σ}{^}^{2} = 0

tak collapse kar jaata hai, likelihood ko ek spike (infinite density) bana deta hai. Yeh ek degenerate maximum hai — red flag ki model ne sample ko overfit kar liya.

Kya MLE unique hota hai?

Zaruri nahi. Agar log-likelihood ek region par flat ho ya multiple equal peaks hon (jaise mixtures mein label-swapping jo EM Algorithm handle karta hai), toh kai

θ

values tie karte hain, isliye "the" MLE ek point nahi balki ek set hai.

n = 1

data point ke saath MLE kya karta hai?

Yeh phir bhi ek estimate return karta hai, lekin bina averaging ke variance bahut bada hota hai aur, Gaussian ke liye,

\overset{σ}{^}^{2}

apni degenerate value

0

par pahunch jaata hai. Consistency ko

n \to \infty

chahiye; ek akele point par essentially koi trust nahi hoti.

Agar true data-generating distribution aapki model family mein bilkul nahi hai toh kya hota hai?

MLE phir bhi converge karta hai — lekin us parameter par jiska model truth ke sabse kareeb hai (KL divergence mein), truth par nahi. "Best wrong model" "no model" se behtar hai, lekin yaad rakho guarantee weak ho gayi.

Likelihood equation ka koi finite solution kyun nahi ho sakta — ek concrete toy model do?

Ek Pareto tail lo unknown index

α

ke saath: agar data light-tailed nikla, toh log-likelihood

α \to \infty

ke saath badhti rehti hai (ya

α

ki boundary ki taraf bhagti hai) bina kisi interior critical point ke. Tab

ℓ^{'} (α) = 0

ka koi finite root nahi hota, aur aapko "solve for zero" ki bajaye limiting behaviour inspect karni padti hai.

Recall Quick self-test

MLE ke saath khud ko embarrass karne ke teen sabse fast tarike ::: posterior maximize karna likelihood ki jagah, log bhool jaana aur underflow karna, aur curvature check kiye bina critical point ko maximum kehna. Ek line mein Bias vs consistency ::: MLE finite $n$ ke liye biased ho sakta hai lekin consistent hai — jaise jaise data badhta hai yeh truth par centre hota jaata hai.