Maximum likelihood estimation (MLE): Worked examples

1.3.16 · D3 · AI-ML › Probability & Statistics › Maximum likelihood estimation (MLE)

Yeh page MLE ki "training ground" hai. Parent note ne aapko machine dikhaayi. Yahan hum use har tarah ke input pe run karte hain jo machine ko mil sakti hai: easy cases, sign-flip cases, zero cases, degenerate cases, ek word problem, aur ek exam twist. Iske baad aapko koi bhi MLE problem nahi milni chahiye jiska shape aapne pehle nahi dekha ho.

Scenario matrix

Neeche har cell ek alag kisam ki situation hai. Uske baad ke examples ko un cell(s) se tag kiya gaya hai jo woh cover karte hain.

Cell	Kya special hai	Covered by
A. Interior maximum, bounded parameter	Parameter $[0, 1]$ mein rehta hai; answer andar hota hai, derivative $= 0$ se milta hai	Ex 1
B. Degenerate data (sab ek jaise)	Data ek extreme pe pahunch jaata hai (sab heads / sab tails)	Ex 2
C. Positive-only parameter	Parameter $> 0$ hona chahiye (ek rate/scale)	Ex 3
D. Boundary maximum	Derivative kabhi zero nahi hoti; max support ke edge pe baitha hota hai	Ex 4
E. Do parameters ek saath	$μ$ aur $σ^{2}$ jointly solve karo	Ex 5
F. Bias check / limiting behaviour	Kya estimator unbiased hai? $n \to \infty$ pe kya hota hai?	Ex 6
G. Real-world word problem	Messy story ko ek model mein translate karo, phir estimate karo	Ex 7
H. Exam twist (reparametrisation / invariance)	Parameter ki function ka MLE	Ex 8

Algebra se pehle, poore idea ki ek picture: log-likelihood ek curved landscape hai parameter values ke upar, aur MLE uski choti tak jaata hai.

Cell A — Interior maximum, bounded parameter

Worked example Example 1 — Normal mix of outcomes ke saath Bernoulli

Statement: Aap ek spam filter ko $n = 12$ emails pe test karte ho; woh sahi se $k = 9$ flag karta hai. Har email ek independent success/failure hai. $p$ , yaani sahi success probability, estimate karo.

Forecast: Answer padhne se pehle guess karo. (Hint: kitna fraction succeed hua?)

Step 1 — Likelihood likho. $L (p) = p^{9} (1 - p)^{3}$ Yeh step kyun? Har success ek factor $p$ multiply karta hai, har failure ek factor $1 - p$ ; independence ki wajah se hum unhe multiply kar sakte hain.

Step 2 — Log lo. $ℓ (p) = 9 lo g p + 3 lo g (1 - p)$ Yeh step kyun? Product ko sum mein badal deta hai taaki derivative ek clean sum ho, aur tiny numbers ko zero pe underflow hone se bachata hai.

Step 3 — Differentiate karo aur zero set karo. $ℓ^{'} (p) = \frac{9}{p} - \frac{3}{1 - p} = 0$ Yeh step kyun? Ek smooth hill ki choti ka slope zero hota hai; $ℓ^{'} = 0$ solve karna woh flat spot dhundta hai.

Step 4 — Solve karo. $9 (1 - p) = 3 p ⟹ 9 = 12 p ⟹ \overset{p}{^} = \frac{9}{12} = 0.75$ Yeh step kyun? Cross-multiply karne se fractions saaf ho jaate hain; answer bas $k / n$ hai.

Verify: $ℓ^{''} (p) = - \frac{9}{p ^{2}} - \frac{3}{( 1 - p ) ^{2}} < 0$ har jagah, toh yeh ek maximum hai, minimum nahi. Sanity: $0.75 \in [0, 1]$ ✓, aur yeh observed fraction $9/12$ ke barabar hai ✓.

Cell B — Degenerate data (sab outcomes ek jaise)

Worked example Example 2 — Har flip heads

Statement: Ek coin $n = 5$ baar flip kiya jaata hai aur saari 5 baar heads aata hai. $\overset{p}{^}$ kya hai?

Forecast: Kya derivative trick yahan kaam bhi karegi?

Step 1 — Likelihood. $k = 5$ heads, $0$ tails ke saath: $L (p) = p^{5} (1 - p)^{0} = p^{5}$ Yeh step kyun? $(1 - p)$ factor ka exponent $0$ hai, toh woh gayab ho jaata hai — koi tails evidence hai hi nahi.

Step 2 — Log-likelihood. $ℓ (p) = 5 lo g p$ Yeh step kyun? Wohi log rule; ab sirf ek term bachta hai.

Step 3 — Slope dekho. $ℓ^{'} (p) = \frac{5}{p} > 0 for all p \in (0, 1]$ Yeh step kyun? Slope kabhi zero nahi hota — derivative-equals-zero recipe ka yahan koi interior solution nahi hai. Yeh boundary answer ka warning sign hai.

Step 4 — Edge tak jao. Kyunki $ℓ$ strictly increase karta hai, maximum sabse bade allowed $p$ pe baitha hai: $\overset{p}{^} = 1$ Yeh step kyun? Jab hill sirf upar ki taraf slope karti ho, toh tum boundary $p = 1$ tak chadh jaate ho.

Verify: $L (1) = 1^{5} = 1$ , jo ek probability ho sakti hai sabse bada; koi bhi $p < 1$ deta hai $L < 1$ . Formula check: $k / n = 5/5 = 1$ ✓. Plain formula $\overset{p}{^} = k / n$ sahi boundary pe land karta hai.

Cell C — Positive-only parameter (ek rate)

Worked example Example 3 — Exponential waiting times

Statement: Server response times (seconds) $x_{1}, \dots, x_{n}$ ko $Exponential (λ)$ se model kiya gaya hai jisme density $p (x ∣ λ) = λ e^{- λ x}$ hai $x > 0$ ke liye. Aap $n = 4$ times observe karte ho: $2, 1, 3, 2$ (mean $= 2$ ). Rate $λ > 0$ estimate karo.

Forecast: Mean waiting time $2$ s hai. Guess karo $\hat{λ}$ kya hona chahiye.

Step 1 — Log-likelihood. $ℓ (λ) = \sum_{i = 1}^{n} (lo g λ - λ x_{i}) = n lo g λ - λ \sum_{i = 1}^{n} x_{i}$ Yeh step kyun? $lo g (λ e^{- λ x}) = lo g λ - λ x$ ; points pe sum karne se total milta hai.

Step 2 — $λ$ mein differentiate karo. $ℓ^{'} (λ) = \frac{n}{λ} - \sum_{i = 1}^{n} x_{i}$ Yeh step kyun? $\frac{d}{d λ} lo g λ = 1/ λ$ , aur linear term $- \sum x_{i}$ differentiate hota hai.

Step 3 — Zero set karo aur solve karo. $\frac{n}{λ} = \sum x_{i} ⟹ \hat{λ} = \frac{n}{\sum x _{i}} = \frac{1}{x ˉ}$ Hamare numbers ke saath: $\sum x_{i} = 8$ , $n = 4$ , toh $\hat{λ} = 4/8 = 0.5$ . Yeh step kyun? MLE rate mean wait ka reciprocal hai — faster rate ⇔ chhote waits.

Verify: $ℓ^{''} (λ) = - n / λ^{2} < 0$ ⇒ maximum ✓. $\hat{λ} = 0.5 > 0$ positivity constraint respect karta hai ✓. Units: mean seconds mein hai, rate per-second hai, $0.5 s^{- 1}$ ✓ (har 2s pe ek response).

Cell D — Boundary maximum (uniform support)

Worked example Example 4 —

$[0, θ]$ pe Uniform Statement: Numbers $[0, θ]$ se uniformly draw kiye jaate hain, density $p (x ∣ θ) = 1/ θ$ for $0 \leq x \leq θ$ (aur baaki jagah $0$ ). Aap $x = {3, 7, 5, 7, 2}$ observe karte ho. $θ$ estimate karo.

Forecast: Yahan derivative aapko mislead karega. Pehle common sense se $\hat{θ}$ guess karo.

Step 1 — Likelihood, carefully. $L (θ) = \prod_{i = 1}^{n} \frac{1}{θ} = \frac{1}{θ ^{n}}, but ONLY if θ \geq max_{i} x_{i} .$ Yeh step kyun? Agar $θ$ kisi observed $x_{i}$ se chhota hai, toh us point ki model ke under probability $0$ thi, toh $L = 0$ . Constraint hi poori baat hai.

Step 2 — Derivative try karo (fail hote dekhne ke liye). $\frac{d}{d θ} θ^{- n} = - n θ^{- n - 1} \neq = 0 for any finite θ .$ Yeh step kyun? Koi zero-slope point exist nahi karta; $L$ sirf decrease karta rehta hai jaise $θ$ badhta hai. Calculus recipe akele kuch nahi deti.

Step 3 — Constraint use karo. $L = 1/ θ^{n}$ tab sabse bada hota hai jab $θ$ jitna chhota allowed ho utna ho. Sabse chhota legal $θ$ sabse bada data point hai: $\hat{θ} = max_{i} x_{i} = 7$ Yeh step kyun? Hum $θ$ chhota karte hain $1/ θ^{n}$ badhane ke liye, lekin $max_{i} x_{i}$ se neeche nahi ja sakte warna likelihood zero ho jaayegi.

Verify: $θ = 7$ pe, $L = 1/ 7^{5}$ ; $θ = 6.9$ pe, ek point ( $7$ ) bahar hai toh $L = 0 < 1/ 7^{5}$ ; $θ = 8$ pe, $L = 1/ 8^{5} < 1/ 7^{5}$ ✓. Toh $θ = 7$ hi peak hai. $1/ 7^{5} = 1/16807$ .

Cell E — Do parameters jointly solve kiye

Worked example Example 5 — Gaussian:

$μ$ aur $σ^{2}$ saath estimate karo Statement: Data $x = {2, 4, 4, 4, 5, 5, 7, 9}$ ( $n = 8$ ) Gaussian $N (μ, σ^{2})$ hai jisme dono unknown hain. Dono estimate karo.

Forecast: $\overset{μ}{^}$ (centre) aur roughly $\overset{σ}{^}^{2}$ (spread) compute karne se pehle guess karo.

Step 1 — Do variables mein log-likelihood. $ℓ (μ, σ^{2}) = - \frac{n}{2} lo g (2 π) - \frac{n}{2} lo g σ^{2} - \frac{1}{2 σ ^{2}} \sum_{i = 1}^{n} (x_{i} - μ)^{2}$ Yeh step kyun? Parent note jaisa hi Gaussian log, lekin ab kuch bhi fixed nahi hai.

Step 2 — $μ$ mein partial derivative. $\frac{\partial ℓ}{\partial μ} = \frac{1}{σ ^{2}} \sum_{i = 1}^{n} (x_{i} - μ) = 0 ⟹ \overset{μ}{^} = \overset{x}{ˉ} = \frac{2 + 4 + 4 + 4 + 5 + 5 + 7 + 9}{8} = 5$ Yeh step kyun? $σ^{2}$ hold fix karne pe, sirf last term $μ$ pe depend karta hai; uska slope zero set karne se sample mean milta hai. $σ^{2}$ divide out ho jaata hai, toh $\overset{μ}{^}$ variance pe depend nahi karta.

Step 3 — $σ^{2}$ mein partial derivative. $\frac{\partial ℓ}{\partial σ ^{2}} = - \frac{n}{2 σ ^{2}} + \frac{1}{2 σ ^{4}} \sum_{i = 1}^{n} (x_{i} - \overset{μ}{^})^{2} = 0 ⟹ \overset{σ}{^}^{2} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{μ}{^})^{2}$ Yeh step kyun? Pehle se mila $\overset{μ}{^}$ plug in karo, phir $σ^{2}$ ke liye solve karo. Yeh "profile" trick hai — ek optimise karo, substitute karo, agla optimise karo.

Step 4 — Numbers. $5$ se deviations: $- 3, - 1, - 1, - 1, 0, 0, 2, 4$ ; squares $9, 1, 1, 1, 0, 0, 4, 16$ ka sum $32$ hai. $\overset{σ}{^}^{2} = \frac{32}{8} = 4$

Verify: $\overset{μ}{^} = 5$ data range $[2, 9]$ ke andar hai ✓. $\overset{σ}{^}^{2} = 4 > 0$ ✓, toh $\overset{σ}{^} = 2$ . Yeh Method of Moments se connect karta hai, jo yahan wohi $\overset{μ}{^}, \overset{σ}{^}^{2}$ deta hai.

Cell F — Bias aur limiting behaviour

Worked example Example 6 — Kya variance estimate unbiased hai?

Statement: Upar wale two-parameter Gaussian ke liye, kya $\overset{σ}{^}_{MLE}^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}$ average pe sahi hai? $n \to \infty$ pe kya hota hai?

Forecast: "Maximum likelihood" optimal lagg raha hai — guess karo kya $E [\overset{σ}{^}^{2}] = σ^{2}$ hai.

Step 1 — Exact expectation. Ek standard result: $E [\overset{σ}{^}_{MLE}^{2}] = \frac{n - 1}{n} σ^{2}$ Yeh step kyun? $\overset{x}{ˉ}$ use karna (jo khud data se estimate hua hai) ek degree of freedom "kha jaata hai", spread ko shrink karta hai.

Step 2 — Bias read off karo. $bias = E [\overset{σ}{^}^{2}] - σ^{2} = - \frac{σ ^{2}}{n} < 0$ Yeh step kyun? Average se truth subtract karna dikhata hai ki MLE variance ko under-estimate karta hai.

Step 3 — Limit check karo. $\frac{n - 1}{n} = 1 - \frac{1}{n} n \to \infty 1$ Yeh step kyun? Jaise data flood karta hai, bias factor $1$ ki taraf jaata hai: MLE asymptotically unbiased (consistent) hai, bhale hi finite $n$ ke liye biased ho.

Verify: $n = 8$ ke liye, factor $7/8 = 0.875$ hai. De-bias karne ke liye, multiply karo: unbiased estimate $= \frac{n}{n - 1} \overset{σ}{^}^{2} = \frac{8}{7} \cdot 4 = \frac{32}{7} \approx 4.571$ , jo exactly $\frac{1}{n - 1} \sum (x_{i} - \overset{x}{ˉ})^{2} = 32/7$ hai ✓. Bias-Variance Tradeoff aur Cramér-Rao Bound se ties hain.

Cell G — Real-world word problem

Worked example Example 7 — Factory line se defect rate estimate karna

Statement: Ek factory $n = 200$ items inspect karta hai aur $k = 6$ defective paata hai. Maano har item independently defective hai unknown probability $p$ ke saath. (a) $p$ estimate karo. (b) Manager poochh ta hai: "Agar agle do items dono acche hone ki estimated probability kya hai?"

Forecast: Defect rate guess karo, phir guess karo kya hum part (b) ke liye bas usi ko reuse kar sakte hain.

Step 1 — Model aur MLE. Yeh phir se Bernoulli hai (Cell A shape): $\overset{p}{^} = \frac{k}{n} = \frac{6}{200} = 0.03$ Yeh step kyun? Bernoulli MLE hamesha observed success (yahan defect) fraction hota hai.

Step 2 — Part (b) translate karo. "Agle dono acche" $=$ probability $(1 - p)^{2}$ independence ke under. Yeh step kyun? Do independent acche items apni good-probabilities $1 - p$ multiply karte hain.

Step 3 — Invariance apply karo. $(1 - p)^{2}$ ka MLE hai $(1 - \overset{p}{^})^{2}$ : $(1 - 0.03)^{2} = 0.9 7^{2} = 0.9409$ Yeh step kyun? MLE invariance: kisi function $g (p)$ ko estimate karne ke liye, bas $\overset{p}{^}$ ko $g$ mein plug in karo. Koi re-derivation nahi chahiye. (Agla example, Cell H, set up kar raha hai.)

Verify: $\overset{p}{^} = 0.03 \in [0, 1]$ ✓, real line ke liye plausible. $0.9 7^{2} = 0.9409$ ✓, aur yeh $0.97$ se neeche hai (do hurdles ek se mushkil hote hain) ✓.

Cell H — Exam twist: reparametrisation / invariance

Worked example Example 8 — Probability ka nahi, odds ka MLE

Statement: Ek exam poochh ta hai: Example 1 se ( $\overset{p}{^} = 0.75$ ), odds $o = \frac{p}{1 - p}$ ka MLE do, aur log-odds $lo g o$ ka bhi. Yeh karo bina koi calculus dobara kiye.

Forecast: Kya aapko likelihood ko $o$ ke terms mein rebuild karna padega?

Step 1 — Invariance invoke karo. Agar $\overset{p}{^}$ likelihood maximise karta hai, toh kisi bhi function $g$ ke liye, $g (\overset{p}{^})$ us likelihood ko maximise karta hai jo $g (p)$ ke terms mein likhi gayi hai. Yeh step kyun? Functional invariance property matlab answer directly transfer hota hai — yahi twist ka poora point hai.

Step 2 — Odds ke liye plug in karo. $\overset{o}{^} = \frac{p ^}{1 - p ^} = \frac{0.75}{0.25} = 3$ Yeh step kyun? $\overset{p}{^} = 0.75$ ko $g (p) = p / (1 - p)$ mein substitute karo. Koi nayi differentiation nahi.

Step 3 — Log-odds ke liye plug in karo. $lo g o = lo g \overset{o}{^} = lo g 3 \approx 1.0986$ Yeh step kyun? Log-odds $\overset{p}{^}$ ki ek aur function hai; invariance phir se $lo g (\overset{o}{^})$ deta hai.

Verify: Odds of $3$ matlab "3 successes har 1 failure ke liye", $p = 0.75$ ( $4$ mein se $3$ ) se match karta hai ✓. $lo g 3 = 1.0986 \dots$ ✓. Yeh exactly wahi quantity hai jo logistic regression estimate karta hai, Loss Functions in ML aur Likelihood Ratio Test se link karta hai.

Recall Kaunse cell ko second-derivative / boundary check ki sabse zyada zaroorat hai?

Cell D (Uniform) aur Cell B (degenerate) — derivative kabhi zero nahi hoti, toh blindly $ℓ^{'} = 0$ solve karna fail hota hai; maximum support ki boundary pe baitha hota hai.

Recall

$\overset{σ}{^}_{MLE}^{2}$ average pe "true" variance se chhota kyun hota hai? Kyunki sample mean $\overset{x}{ˉ}$ use karna (jo khud data pe fit hua hai) ek degree of freedom remove karta hai, $E [\overset{σ}{^}^{2}] = \frac{n - 1}{n} σ^{2}$ deta hai.

Connections

Parent: Maximum likelihood estimation (MLE) (index 1.3.16) · 1.3.16 Maximum likelihood estimation (MLE) (Hinglish)
Jab ek prior add hota hai: Bayesian Estimation
In estimators ka variance/efficiency: Fisher Information · Cramér-Rao Bound
Alternative fitting recipe: Method of Moments
Hidden-variable extension: EM Algorithm