Visual walkthrough — Maximum a posteriori estimation (MAP)

1.3.17 · D2 · AI-ML › Probability & Statistics › Maximum a posteriori estimation (MAP)

#ai-ml/probability-statistics #bayesian-inference #parameter-estimation

Koi bhi math se pehle: ek parameter sirf ek unknown number hai jise hum guess karna chahte hain. Ise $\theta$ (Greek letter "theta") kehte hain. Ek coin ke liye, $\theta$ uske heads aane ki chance hai — $0$ aur $1$ ke beech ka ek number. Bas itna hi $\theta$ ka matlab hai. Neeche sab kuch us unknown number ke liye best single value chunne ke baare mein hai.

---

Step 1 — Jo cheez hum guess kar rahe hain use draw karo

KYA. $\theta$ ko ek horizontal line par rakh do. Us line ka har point coin ki bias ke liye ek possible guess hai: $0$ (kabhi heads nahi), $0.5$ (fair coin), $1$ (hamesha heads).

KYUN. Tum "best guess" ke baare mein tab tak baat nahi kar sakte jab tak tum saare guesses laid out dekh nahi lete. Parameter line ki picture woh stage hai jis par poori kahani hoti hai.

PICTURE. $0.5$ par amber tick ek fair coin mark karta hai. Humne abhi decide nahi kiya ki kaunsa value jitega — line sirf options ka menu hai.

---

Step 2 — Height = "kitna believable hai". Do curves aate hain

KYA. Line ke upar hum heights draw karte hain. Kisi $\theta$ ke upar zyada height ka matlab hai "woh value believable hai"; kam height ka matlab hai "unlikely". Do alag curves yahan rehte hain.

• Likelihood $P(\mathbf{x}\mid \theta )$ — padho "data $\mathbf{x}$ ki probability, agar true value $\theta$ hoti". Yahan $\mathbf{x}$ woh hai jo humne actually dekha (10 flips mein 7 heads). Yeh curve wahan peak karta hai jahan data best explain hota hai.
• Prior $P(\theta )$ — padho "$\theta$ kitna believable tha data dekhne se pehle". Yeh tumhari common sense hai, ek curve ki tarah draw ki gayi.

KYUN. MLE (Maximum Likelihood Estimation (MLE)) sirf likelihood curve ko dekha karta hai. Lekin real belief ke do sources hote hain — data aur jo tum pehle se jaante the. Dono curves stage par chahiye inhe combine karne se pehle.

PICTURE. Cyan = likelihood ($0.7$ ke paas peak, data ki kahani). Amber = prior ($0.6$ ke paas peak, hamari pehle ki hunch). Notice karo ki unke peaks alag jagahon par hain — yahi tension saara point hai.

Dono curves ko explicitly likhte hain, kyunki Step 6 mein unke exponents aate hain aur tumhe deserve hai ki tum jaano har number kahan se aaya.

Likelihood $k=7$ heads $n=10$ flips mein binomial probability hai: $P(\mathbf{x}\mid \theta )=\underset{\text{orderings count karta hai}}{\underbrace{\left(\genfrac{}{}{0ex}{}{10}{7}\right)}}\underset{\text{7 heads}}{\underbrace{{\theta }^7}}\underset{\text{3 tails}}{\underbrace{(1-\theta {)}^3}}$

• $\left(\genfrac{}{}{0ex}{}{10}{7}\right)$ — un alag-alag tareekon ki sankhya jinse 7 heads 10 flips mein aa sakte hain. Ismein koi $\theta$ nahi, toh yeh ek fixed constant $C$ hai.
• ${\theta }^7$ — har head $\theta$ probability se hota hai, aur 7 hain, sab multiply hote hain.
• $(1-\theta {)}^3$ — har tail $1-\theta$ probability se hota hai, aur 3 hain.

Prior hum Beta$(3,2)$ curve choose karte hain (dekho Beta Distribution). Uska functional form hai $P(\theta )=\underset{\text{fixed normaliser}}{\underbrace{\frac{1}{B(3,2)}}}\underset{={\theta }^2}{\underbrace{{\theta }^{3-1}}}\underset{=(1-\theta {)}^1}{\underbrace{(1-\theta {)}^{2-1}}}=\frac{1}{B(3,2)}{\theta }^2(1-\theta )$

• Beta ke do shape numbers $3$ aur $2$ hain; formula mein exponents hamesha ek kam hote hain, jisse ${\theta }^2$ aur $(1-\theta {)}^1$ milte hain. Wahan se woh "2" aur "1" aate hain jo tum Step 6 mein dekhoge.
• $B(3,2)$ — ek normalising constant jisme koi $\theta$ nahi, toh yeh bhi ek fixed number hai jise hum baad mein drop kar sakte hain.

---

Step 3 — Bayes' theorem: arrow ko flip karo

KYA. Hum $P(\mathbf{x}\mid \theta )$ compute kar sakte hain (data ki probability given ek guess), lekin actually hum $P(\theta \mid \mathbf{x})$ chahte hain (ek guess ki probability given data). Bayes' theorem yeh arrow flip karta hai:

$P(\theta \mid \mathbf{x})=\frac{\stackrel{\text{likelihood}}{\overbrace{P(\mathbf{x}\mid \theta )}}\cdot \stackrel{\text{prior}}{\overbrace{P(\theta )}}}{\underset{\text{evidence: har }\theta \text{ ke liye same number}}{\underbrace{P(\mathbf{x})}}}$

KYUN. Left side ki quantity, posterior $P(\theta \mid \mathbf{x})$, exactly "data dekhne ke baad $\theta$ kitna believable hai" hai. Yahi woh curve hai jiska peak hum chahte hain. Bayes ek hi lawful tarika hai Step 2 ke do ingredients ko usmein badalne ka.

PICTURE. Ise ek factory socho: do input curves upar se jaate hain (likelihood $\times$ prior, har $\theta$ par height-by-height multiply hoke), evidence $P(\mathbf{x})$ sab kuch ek fixed number se divide karta hai, aur posterior neeche se nikalti hai.

Har symbol, jahan woh baitha hai:

• $P(\theta \mid \mathbf{x})$ — output, $\theta$ par posterior height.
• $P(\mathbf{x}\mid \theta )\cdot P(\theta )$ — numerator, likelihood height times prior height, pointwise multiply karke.
• $P(\mathbf{x})$ — denominator, saare $\theta$ par data ki total probability; ismein $\theta$ nahi hai, toh yeh ek constant number hai.

---

Step 4 — Denominator hatao (yeh peak ko kabhi move nahi karta)

KYA. Hum tallest point ki location dhundh rahe hain, likha jaata hai $\arg {\max }_{\theta }$ — "$\theta$ jo neeche wali cheez ko jitna bada ho sake karta hai". Har height ko same constant $P(\mathbf{x})$ se divide karna poori curve ko uniformly shrink karta hai lekin uska peak sideways nahi karta.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }\frac{P(\mathbf{x}\mid \theta )P(\theta )}{P(\mathbf{x})}=\arg {\max }_{\theta }P(\mathbf{x}\mid \theta )P(\theta )$

KYUN. Hum chahte hain ki peak kahan hai, kitna tall nahi. Ek positive constant se scale karna photo zoom karne jaisa hai — koi bhi horizontal cheez nahi badlti. Toh evidence, jise ignore karna humhe kuch bhi cost nahi karta, drop kar diya jaata hai. Yahi reasoning $\left(\genfrac{}{}{0ex}{}{10}{7}\right)$ aur $B(3,2)$ ko Step 2 se drop karne ki bhi permission deti hai — dono $\theta$-free constants hain. Jo bachta hai sirf $\theta$-wala part hai: $P(\mathbf{x}\mid \theta )P(\theta )\propto {\theta }^7(1-\theta {)}^3\cdot {\theta }^2(1-\theta )={\theta }^{7+2}(1-\theta {)}^{3+1}={\theta }^9(1-\theta {)}^4$ Same base ke powers multiply karne se unke exponents add hote hain: $7+2=9$ heads-worth, $3+1=4$ tails-worth. Yeh combined curve Beta$(10,5)$ shape hai — posterior.

PICTURE. Tall cyan curve aur short cyan curve same shape hain scaled; amber dashed line dono peaks se same $\theta$ par drop hoti hai. Peak ka address unchanged hai.

---

Step 5 — Log lo: multiply becomes add

KYA. Jo cheez hum maximize kar rahe hain usmein $\log$ (logarithm) apply karo. Key property: $\log (a\cdot b)=\log a+\log b$. Product sum ban jaata hai. Hamare surviving curve ${\theta }^9(1-\theta {)}^4$ par apply karke: $\log [{\theta }^9(1-\theta {)}^4]=9\log \theta +4\log (1-\theta )$ Rule $\log (a^m)=m\log a$ exponents $9$ aur $4$ ko front par le aata hai — wahin se Step 6 mein coefficients aate hain.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }[\underset{\text{log-likelihood}}{\underbrace{\log P(\mathbf{x}\mid \theta )}}+\underset{\text{log-prior}}{\underbrace{\log P(\theta )}}]$

KYUN do reasons.

1. Sums products se easier hote hain — sum ka derivative derivatives ka sum hota hai, toh Step 6 ka calculus clean hai.
2. $\log$ monotonic hai (hamesha increasing): agar curve $A$ kisi point par curve $B$ se taller hai, toh $\log A$ wahan bhi $\log B$ se taller hai. Toh log lene par peak ki location kabhi nahi badlti — exactly wही jo humhe Step 4 ki spirit mein chahiye tha.

PICTURE. Left panel: original product-shaped curve. Right panel: uska log — same peak position (amber line line up karti hai), lekin shape ab do gentle bowls ka sum hai, ek data se, ek prior se.

---

Step 6 — Peak dhundho: slope zero

KYA. Ek smooth curve ka highest point wahan hota hai jahan uski slope zero ho — top par flat. Slope derivative $\frac{d}{d\theta }$ hai. Use zero par set karo aur solve karo.

Step 5 se log-posterior hai $9\log \theta +4\log (1-\theta )$ (woh Beta$(10,5)$ shape jo humne assemble ki). Differentiate karo, $\frac{d}{d\theta }\log \theta =\frac{1}{\theta }$ aur $\frac{d}{d\theta }\log (1-\theta )=\frac{-1}{1-\theta }$ use karke:

$\frac{d}{d\theta }[9\log \theta +4\log (1-\theta )]=\underset{\text{heads se upar pull}}{\underbrace{\frac{9}{\theta }}}-\underset{\text{tails se neeche pull}}{\underbrace{\frac{4}{1-\theta }}}=0$

KYUN. Hilltop par tum momentarily level ground par chal rahe ho — slope $0$ hai. $\text{slope}=0$ solve karna calculus ka standard tarika hai maximum locate karne ka. Do fractions literally do forces hain: ek $\theta$ ko upar push karta hai (heads ka evidence), ek neeche pull karta hai (tails ka evidence). Balance = estimate.

Solve karo: $\frac{9}{\theta }=\frac{4}{1-\theta }\Rightarrow 9(1-\theta )=4\theta \Rightarrow 9=13\theta \Rightarrow \boxed{{\hat{\theta }}_{\text{MAP}}=\frac{9}{13}\approx 0.692}$

PICTURE. Posterior curve jiske top par flat amber tangent line resting hai, $\theta =9/13$ tak drop karti hai. Cyan MLE peak $0.7$ se compare karo: prior ne answer ko leftward nudge kiya, apni $0.6$ ki hunch ki taraf.

Beta$(3,2)$ ka prior mode $\frac{3-1}{3+2-2}=\frac{2}{3}\approx 0.667$ par hai; data akela $0.7$ kehta hai; MAP $0.692$ par unke beech land karta hai — data ke zyada paas kyunki woh zyada hai. (Dekho Conjugate Priors, Posterior Distribution.)

---

Step 7 — Edge case A: flat prior MLE wapas deta hai

KYA. Maano prior flat hai — har $\theta$ equally believable, $P(\theta )=\text{const}$. Tab $\log P(\theta )$ ek constant hai, aur constant ka slope har jagah zero hota hai.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }[\log P(\mathbf{x}\mid \theta )+\underset{\text{koi tilt nahi}}{\underbrace{\log (\text{const})}}]=\arg {\max }_{\theta }\log P(\mathbf{x}\mid \theta )={\hat{\theta }}_{\text{MLE}}$

KYUN. Flat prior ka matlab hai "mujhe pehle se kuch believe nahi karna." Log-posterior mein ek level slab add karna use uniformly lift karta hai lekin kabhi tilt nahi karta, toh peak exactly wahan rehti hai jahan likelihood ne rakhi thi. Isliye MLE, MAP hai ek khali dimag ke saath.

PICTURE. Amber flat prior ek straight horizontal slab hai; posterior peak seedha cyan likelihood peak ke neeche baithti hai — MAP aur MLE coincide karte hain.

---

Step 8 — Edge case B: jaise data flood karta hai, prior fade hota hai

KYA. Ab hum ek Gaussian mean $\mu$ estimate karne ki taraf switch karte hain, aur uska MAP formula derive karte hain quote karne ki jagah, taaki koi bhi symbol unearned na aaye. Hum $n$ numbers $x_1,\dots ,x_n$ observe karte hain jo bell curve $N(\mu ,1)$ (variance known aur equal to 1) se draw hain, aur hamara prior ek bell curve $N(0,4)$ hai — $0$ par centred belief $4$ ke spread ke saath.

Likelihood (i.i.d. points par product, constants dropped) ka log-form hai $\log P(\mathbf{x}\mid \mu )=-\frac{1}{2}{\sum }_{i=1}^n(x_i-\mu {)}^2+\text{const}$ Har term $-\frac{1}{2}(x_i-\mu {)}^2$ bell $\exp (-\frac{1}{2}(x_i-\mu {)}^2)$ ka log hai; sums products ki jagah kyunki logs $\times$ ko $+$ mein turn karte hain.

Prior $N(0,4)$ ka log-form hai $\log P(\mu )=-\frac{{\mu }^2}{2\cdot 4}+\text{const}=-\frac{{\mu }^2}{8}+\text{const}$ "$8$" hai $2{\sigma }_0^2=2\times 4$; $-{\mu }^2$ $\mu$ ko $0$ ki taraf pull karta hai, prior ke centre ki taraf.

Inhe add karo, differentiate karo, zero par set karo ($\frac{d}{d\mu }(x_i-\mu {)}^2=-2(x_i-\mu )$ use karke): $\frac{d}{d\mu }[-\frac{1}{2}{\sum }_i(x_i-\mu {)}^2-\frac{{\mu }^2}{8}]={\sum }_{i=1}^n(x_i-\mu )-\frac{\mu }{4}=0$ $\bar{x}=\frac{1}{n}{\sum }_i{x}_i$ likho (data average), toh ${\sum }_i{x}_i=n\bar{x}$: $n\bar{x}-n\mu -\frac{\mu }{4}=0\Rightarrow n\bar{x}=\mu (n+\frac{1}{4})\Rightarrow \boxed{{\hat{\mu }}_{\text{MAP}}=\frac{n\bar{x}}{n+\frac{1}{4}}=\frac{4n}{4n+1}\bar{x}}$

KYUN yeh shape hai. Prior ek fixed dab ki tarah act karta hai pretend evidence ki jo roughly ek data point ke quarter ki value hai. Jab real data scarce hoti hai, woh dab loud hoti hai; jab data flood karta hai, woh dab dub jaata hai:

• $n=1,\bar{x}=2$: $\hat{\mu }=\frac{4}{5}\cdot 2=\frac{8}{5}=1.6$ — prior mean $0$ ki taraf heavily shrunk.
• $n\to \infty$: $\frac{4n}{4n+1}\to 1$, toh $\hat{\mu }\to \bar{x}$ — prior vanish hota hai aur MAP MLE ban jaata hai.

Yeh shrinking pull exactly wahi hai jo tiny datasets par Overfitting ko rokti hai — same idea Regularization in ML, Ridge Regression aur Lasso Regression ki tarah (Gaussian prior L2 hai, Laplace prior L1 hai). Yeh bhi dekho Fisher Information ke liye ki data likelihood ko kaise "sharpen" karta hai.

PICTURE. $n=1,5,50$ ke liye teen posteriors stacked: jaise $n$ badhta hai, peak $0$ ke paas se $\bar{x}=2$ ki taraf march karti hai, aur curve narrow hoti hai — belief data par tighten hoti hai.

---

Ek picture mein summary

Upar sab kuch, ek single frame mein: do curves (data cyan mein, belief amber mein) ko multiply kiya jaata hai, logs usse addition of two scores mein turn kar dete hain, aur hum flat top ki taraf uphill chalte hain. Peak ek tug-of-war winner hai — data se evidence ki taraf khicha, belief se prior ki taraf, wahan settle karta hai jahan dono forces balance karte hain.

Recall Feynman retelling — ise ek story ki tarah bolo

Main ek unknown number $\theta$ guess karna chahta hoon. Main uske saare possible values ki ek line draw karta hoon. Us line ke upar main do hills sketch karta hoon: ek hai ki har value mera data kitna achha explain karta hai, doosri hai ki main us value mein pehle se kitna believe karta tha. Bayes' theorem mujhe kehta hai ki dono hills ko multiply karo (aur ek constant se divide karo jise main ignore kar sakta hoon, kyunki woh sirf shrink karta hai, shift nahi). Multiply karna annoying hai, toh main logs leta hoon aur woh do scores ko add karna ban jaata hai. Best guess summed hill ka top hai, jahan slope flat ho jaata hai. Agar main kuch nahi believe karta tha (flat prior), toh belief-hill level hai aur main plain likelihood par wapas aa jaata hoon — woh MLE hai. Agar mere paas loads of data hai, data-hill belief-hill ke upar tower karta hai aur phir se basically MLE milta hai. Beech mein — thoda data, real belief — MAP quietly mera guess common sense ki taraf pull karta hai, jo poora reason hai ki yeh overfitting ko beat karta hai.

Recall Quick self-test

MAP estimate dhundhte waqt hum $P(\mathbf{x})$ kyun drop kar sakte hain? ::: Yeh $\theta$ par depend nahi karta; har height ko same constant se divide karna curve ko scale karta hai lekin uska peak kabhi nahi move karta. Log-posterior mein exponents $9$ aur $4$ kahan se aate hain? ::: Likelihood exponents ${\theta }^7(1-\theta {)}^3$ aur Beta$(3,2)$ prior exponents ${\theta }^2(1-\theta {)}^1$ add karne se: $7+2=9$ aur $3+1=4$. Uniform prior ke under MAP kya reduce hota hai, aur kyun? ::: MLE — ek constant log-prior ek level slab add karta hai jo log-posterior ko lift karta hai bina tilt kiye, toh peak likelihood ke peak par hi rehti hai. Coin example mein, ${\hat{\theta }}_{\text{MAP}}=0.692$ prior mode $0.667$ aur MLE $0.7$ ke beech kyun hai? ::: Posterior dono influences ko multiply karta hai; 10 flips ke saath data prior se thoda zyada outweigh karta hai, toh answer $0.7$ ke zyada paas hai. Gaussian case mein $n\to \infty$ hone par prior ke influence ka kya hota hai? ::: Yeh zero tak fade ho jaata hai ($\frac{4n}{4n+1}\to 1$), toh ${\hat{\mu }}_{\text{MAP}}\to \bar{x}$, plain data mean.