Maximum likelihood estimation (MLE): Visual walkthrough

1.3.16 · D2 · AI-ML › Probability & Statistics › Maximum likelihood estimation (MLE)

Hum coin flip ko apna running example use karenge kyunki isme sirf ek unknown number hai, toh hum sab kuch ek flat sheet of paper pe draw kar sakte hain.

Step 1 — "Parameter" kya hota hai aur "data" kya hota hai?

KYA. Hamare paas ek coin hai. Yeh heads par utarta hai kisi unknown chance ke saath jise hum $p$ kehte hain. Yahan $p$ ek single number hai $0$ aur $1$ ke beech — yeh hamara parameter hai (woh dial jo hum ghuma sakte hain). Hum coin ko $n$ baar flip karte hain aur jo hua use record karte hain; woh recorded outcomes hamare data hain. Hum == $k$ ko un heads ki sankhya ke liye likhte hain== jo humne dekhe (toh $n - k$ tails ki sankhya hai). Hamare running example mein $n = 10$ aur $k = 7$ hai.

YE se shuru KYUN. Kisi bhi maths se pehle, hume crystal clear hona chahiye ki kaunsi cheez fixed hai aur kaunsi cheez hum search kar rahe hain. Data ho chuka hai — woh frozen hai. Parameter $p$ woh mystery knob hai. $n$ aur $k$ letters ko abhi nail down karna matlab hai ki baad ke har formula mein koi surprise nahi aayega.

PICTURE. Neeche: baayein taraf $p$ ke liye ek dial (ghumaane layak), aur daayein taraf frozen data — 7 heads, 3 tails, pehle se gir chuke hain aur badal nahi sakte.

Figure — Maximum likelihood estimation (MLE)

Step 2 — Yeh exact data ek knob setting ke liye kitna likely hai?

KYA. Knob ki ek value choose karo, maano $p = 0.5$ . Poochho: agar heads ki sach mein chance $0.5$ hoti, toh kya probability hai ki exactly 7 heads aur phir 3 tails dikhein? Har head ek factor $p$ ka cost lagata hai, har tail ek factor $(1 - p)$ ka. Inhe multiply karo:

$L (p) = head p \cdot head p \dots tail (1 - p) = p^{7} (1 - p)^{3} = p^{k} (1 - p)^{n - k}$

Yahan har symbol apni jagah earn karta hai: $p^{7} = p^{k}$ matlab "har ek $k$ head ki chance $p$ hai, multiplied," aur $(1 - p)^{3} = (1 - p)^{n - k}$ matlab "har ek $n - k$ tail ki chance $1 - p$ hai." Poora product $L (p)$ likelihood hai — frozen data ki probability, knob $p$ ke function ke roop mein padhi gayi.

MULTIPLY KYUN karte hain. Flips independent hain — ek flip ka result tumhe agla flip ke baare mein kuch nahi batata. Independent events ke liye, "yeh AND woh AND doosra" ki probability individual probabilities ka product hoti hai. Yahi ek reason hai ki product yahan aata hai.

PICTURE. Daayein taraf do dial settings: $p = 0.5$ par data kaafi likely hai; $p = 0.9$ par teeno tails bahut unlikely ho jaate hain, toh product shrink ho jaata hai. Bar ki height = $L (p)$ .

Common mistake "Kya

$(k n)$ combinatorial factor nahi hona chahiye?" Yeh depend karta hai ki tumne kya model kiya. Agar tum exact sequence record karte ho (HHT...), toh $L (p) = p^{k} (1 - p)^{n - k}$ exactly hai — koi combinatorial factor nahi. Agar tumne sirf heads ki count $k$ rakhi (order bhool ke), toh model binomial hai aur $L (p) = (k n) p^{k} (1 - p)^{n - k}$ hai, jahan $(k n)$ un orderings ki sankhya count karta hai jo $k$ heads dete hain. Dono cases mein MLE identical hai. Factor $(k n)$ ek constant hai — isme $p$ nahi hai. Poori curve ko ek fixed positive number se multiply karne se uski height stretch hoti hai lekin peak kabhi sideways nahi move hoti. Kyunki $ar g max$ sirf top ki location ki parwah karta hai, hum $(k n)$ ko shuru se hi drop kar sakte hain. (Log terms mein yeh ek additive constant $lo g (k n)$ ban jaata hai jiska $p$ -derivative $0$ hai.)

Step 3 — Knob slide karo aur likelihood ko rise aur fall hote dekho

KYA. $p$ ko ek value par test karne ki jagah, $p$ ko $0$ se $1$ tak smoothly sweep karo aur height $L (p) = p^{7} (1 - p)^{3}$ plot karo. Tumhe ek single hump milega.

HUMP KYUN. $p = 0$ par tum claim karte ho ki heads impossible hain, phir bhi tumne 7 heads dekhe → likelihood $0$ . $p = 1$ par tum claim karte ho ki tails impossible hain, phir bhi tumne 3 tails dekhe → likelihood $0$ . Kahin beech mein dono pressures balance hote hain aur curve peak karti hai. MLE us peak ki location hai — woh knob setting jo frozen data ko sabse zyada probable banati hai.

PICTURE. Poori likelihood curve. Yeh dono ends par zero par pinned hai, beech mein bulge karti hai, aur iska highest point ek dashed line se $p$ -axis par mark kiya gaya hai.

Step 4 — Hum curve ke log par kyun climb karte hain

KYA. $L (p)$ ko uske logarithm se replace karo, jo log-likelihood hai:

$ℓ (p) = lo g L (p) = 7 heads se 7 lo g p + 3 tails se 3 lo g (1 - p)$

$lo g$ ne product $p^{7} (1 - p)^{3}$ ko sum mein badal diya, kyunki $lo g (ab) = lo g a + lo g b$ aur $lo g (a^{k}) = k lo g a$ . (Agar humne Step 2 ka binomial factor rakha hota, toh woh yahan sirf ek extra constant $lo g (k n)$ ke roop mein appear karta, $p$ mein flat — peak-finder ke liye invisible.)

LOG KYUN — aur yeh safe kyun hai. Do reasons. (1) Same peak. $lo g$ strictly increasing hai: agar curve $A$ kisi point par curve $B$ se zyada tall hai, toh uska log bhi wahan zyada tall hai. Toh $ℓ$ climb karna hume exact same $p$ par land karta hai jaise $L$ climb karna. (2) Practical. Sums, products se kahin zyada aasaan differentiate hote hain, aur large $n$ ke liye raw product computer par $0$ tak underflow kar jaata hai jabki uska log ek sensible number rehta hai.

PICTURE. Left axis: hump $L (p)$ . Right axis: curve $ℓ (p)$ . Dashed vertical line notice karo — dono peaks same $p$ par hain. Log-curve stretched aur shifted hai lekin uska summit move nahi hua.

Step 5 — Slope summit par zero hoti hai

KYA. Ek smooth hill ke top par zameen momentarily flat hoti hai — slope (derivative) zero hoti hai. Toh hum $ℓ$ ki slope compute karte hain aur use zero ke barabar set karte hain:

$\frac{d ℓ}{d p} = heads se slope \frac{7}{p} - tails se slope \frac{3}{1 - p} = 0$

Term by term: $\frac{d}{d p} (7 lo g p) = \frac{7}{p}$ (yeh $p$ ko upar push karta hai, kyunki zyada heads-weight bada $p$ chahta hai), aur $\frac{d}{d p} (3 lo g (1 - p)) = - \frac{3}{1 - p}$ (tails $p$ ko neeche kheenchte hain). Peak par yeh dono forces exactly cancel out ho jaati hain.

DERIVATIVE KYUN. Yeh tool kyun, aur kuch nahi, maano sirf graph dekh ke? Kyunki derivative woh precise machine hai jo answer deti hai "curve kahan flat hai?" — aur ek smooth hump sirf apne summit par flat hoti hai. Use zero ke barabar set karna ek searching problem ko ek equation-solving problem mein convert kar deta hai.

PICTURE. Log-likelihood curve teen tangent lines ke saath: baayein taraf ek uphill (positive slope), daayein taraf ek downhill (negative slope), aur top par ek perfectly horizontal jahan slope $= 0$ hai.

Step 6 — Solve karo, aur answer ka meaning padho

KYA. Flat-slope equation ko rearrange karo:

$\frac{7}{p} = \frac{3}{1 - p} ⟹ 7 (1 - p) = 3 p ⟹ 7 = 10 p ⟹ \overset{p}{^} = \frac{7}{10} = 0.7$

Generally, $n$ flips mein se $k$ heads ke saath wahi algebra deta hai $\overset{p}{^} = k / n$ : MLE sirf observed fraction of heads hai.

YEH sundar KYUN hai. Abstract machinery — products, logs, derivatives — collapse ho jaati hai sabse obvious estimate par jo ek bachcha guess karta. Yeh ek sign hai ki method sane hai.

PICTURE. Likelihood hump phir se, peak $p = 0.7$ par annotated ke saath, aur fraction $7/10$ frozen data ke paas likha hua dikhaane ke liye ki woh same number hain.

Step 7 — Kya yeh sach mein ek maximum tha? Aur degenerate cases

KYA. Zero slope akela ek valley bhi mark kar sakta hai, peak nahi. Hum second derivative check karte hain (woh rate jis par slope khud change hoti hai — negative matlab curve neeche ki taraf bend karti hai, yani ek true summit):

$\frac{d ^{2} ℓ}{d p ^{2}} = - \frac{7}{p ^{2}} - \frac{3}{( 1 - p ) ^{2}} < 0 for every p \in (0, 1) .$

Dono terms valid $p$ ke liye negative hain, toh curve har jagah concave-down hai — hamara critical point beshak maximum hai.

Edge cases — sab cover karo:

Saare heads ( $k = n$ ): $ℓ (p) = n lo g p$ ka koi interior flat point nahi hai; yeh bas $p = 1$ ki taraf rise karta rehta hai. MLE boundary $\overset{p}{^} = 1$ par hota hai. Interpretation: data ne zero evidence diya hai ki tails ho sakta hai.
Saare tails ( $k = 0$ ): mirror image, $\overset{p}{^} = 0$ .
Zero flips ( $n = 0$ ): koi data nahi, koi hump nahi — har $p$ equally (un)supported hai aur MLE undefined hai. Tum ek aisa knob estimate nahi kar sakte jise tumne kabhi test hi nahi kiya.

YEH KYUN dikhate hain. Clean formula $k / n$ in teeno cases mein abhi bhi $1$ , $0$ , aur $0/0$ return karta hai — exactly us boundary/undefined behaviour se match karta hai jo humne abhi reason out kiya. Ek method jis par tum trust karte ho use apne extremes par sensibly behave karna chahiye.

PICTURE. Teen mini-curves side by side: normal interior-peak case, all-heads case (peak right wall $p = 1$ par pinned), aur all-tails case (peak left wall $p = 0$ par pinned).

Ek-picture summary

Yeh final figure poore safar ko compress karta hai: frozen data → likelihood hump → log reshaping → peak par flat slope → answer $k / n$ , margin mein boundary cases ke saath.

Recall Feynman retelling — plain words mein wapas bolo

Hamare paas ek coin thi unknown heads-chance $p$ ke saath, aur humne $k = 7$ heads aur $n - k = 3$ tails table par frozen dekhe ( $n = 10$ flips mein se). $p$ ki kisi bhi guess ke liye, us exact record ko dekhne ki chance $p^{7} (1 - p)^{3}$ hai — badi jab $p$ fit karta hai, tiny jab nahi karta. Agar humne order yaad rakhne ki jagah sirf heads count kiye hote, toh aage ek extra $(k n)$ hota, lekin yeh ek constant hai jo peak kabhi move nahi karta, toh hum ise ignore karte hain. Humne $p$ ko 0 se 1 tak sweep kiya aur ek hump mili jo dono ends par zero hai (yeh guess ki "heads impossible" ya "tails impossible" data ko contradict karta hai). Sabse achha $p$ hump ka top hai. Top dhundne ke liye humne log liya — jo hump ko reshape karta hai lekin summit ko same spot par rakhta hai aur product ko ek easy sum mein badal deta hai. Summit par zameen flat hoti hai, toh humne slope ko zero set kiya, aur heads-force $7/ p$ tails-force $3/ (1 - p)$ ke saath balance hua, $p = 7/10$ deta hai. Second derivative har jagah negative tha, toh yeh sach mein ek peak tha. Aur generally answer sirf heads ka fraction hai, $k / n$ — edges (saare heads → 1, saare tails → 0, koi flips nahi → undefined) ke saath jo exactly common sense ki demand karte hain.

Recall Quick self-test

Likelihood curve $p = 0$ aur $p = 1$ dono par zero kyun touch karti hai? ::: Kyunki $p = 0$ 7 observed heads ko impossible banata hai aur $p = 1$ 3 observed tails ko impossible banata hai, toh product $p^{7} (1 - p)^{3}$ har end par vanish ho jaata hai. Maximize karne se pehle hum $(k n)$ factor kyun drop kar sakte hain? ::: Isme $p$ nahi hai, toh yeh sirf curve ki height ko ek positive constant se scale karta hai — peak ki location ( $ar g max$ ) unchanged rehti hai. Log lena allowed kyun hai bina answer change kiye? ::: $lo g$ strictly increasing hai, toh yeh kabhi ek taller point ko shorter ke neeche nahi laata — peak ki location preserve hoti hai. Kya confirm karta hai ki critical point maximum hai, minimum nahi? ::: Second derivative $- 7/ p^{2} - 3/ (1 - p)^{2}$ valid $p$ ke liye negative hai, toh curve concave down hai (ek peak).

Yeh bhi dekho: Method of Moments (ek alag fitting recipe), Bayesian Estimation ( $θ$ par prior add karta hai), Fisher Information aur Cramér-Rao Bound (peak kitni sharp hai), Gaussian Distribution (agle worked case), Bias-Variance Tradeoff (MLE variance estimate biased kyun hai).