Exercises — Maximum a posteriori estimation (MAP)

• MLE poochta hai: "kaun sa $\theta$ observed data ko most probable banata hai?" → $P(\mathbf{x}\mid \theta )$ maximise karta hai.
• MAP poochta hai: "kaun sa $\theta$ data AUR meri prior belief dono ko dete hue most probable hai?" → posterior $P(\theta \mid \mathbf{x})$ maximise karta hai. MAP, MLE ke data term ke upar belief term $\log P(\theta )$ add karta hai.

Jab prior uniform (flat) ho, $P(\theta )=\text{const}$. Tab $\log P(\theta )$ ek aisa constant hai jo $\theta$ par depend nahi karta, isliye woh $\arg \max$ se bahar nikal jaata hai aur sirf data term bachta hai. MLE isliye MAP hai jisme "data se pehle koi opinion nahi hai."

MAP mode return karta hai — posterior ka peak (highest point). Ek symmetric posterior ke liye (jaise Gaussian) peak centre par hoti hai, isliye mode = mean aur koi fark nahi padta. Skewed posteriors (Beta, Gamma) ke liye mode $\ne$ mean, isliye fark padta hai.

Beta$(\alpha ,\beta )$ prior ke saath Binomial likelihood ka posterior Beta$(\alpha +k,\beta +n-k)$ hota hai (yahi wajah hai ki Beta conjugate prior hai). $\alpha +k=2+9=11,\beta +n-k=2+3=5\Rightarrow \text{Beta}(11,5).$ Beta$(a,b)$ ka mode jab $a,b>1$ ho $\frac{a-1}{a+b-2}$ hota hai: ${\hat{\theta }}_{\text{MAP}}=\frac{11-1}{11+5-2}=\frac{10}{14}=\frac{5}{7}\approx \mathrm{0.714.}$ Compare karo ${\hat{\theta }}_{\text{MLE}}=9/12=0.75$ se: prior mean $0.5$ ne estimate ko thoda neeche kheecha.

Log lo (products → sums, aur log monotonic hai isliye peak location nahi badalti): $\log P=(a-1)\log \theta +(b-1)\log (1-\theta )+\text{const}.$ $\theta$ ke respect mein differentiate karo aur zero ke barabar karo (top par flat slope): $\frac{a-1}{\theta }-\frac{b-1}{1-\theta }=0\Rightarrow (a-1)(1-\theta )=(b-1)\theta .$ Expand karo: $(a-1)=\theta (a-1+b-1)=\theta (a+b-2)$, isliye $\theta =\frac{a-1}{a+b-2}.■$ KYUN yeh minimum ya boundary point nahi balki maximum hai ($a,b>1$ ke liye):

1. Interior second-derivative check. $\frac{d^2}{d{\theta }^2}\log P=-\frac{a-1}{{\theta }^2}-\frac{b-1}{(1-\theta {)}^2}$. $a>1,b>1$ ke saath dono terms negative hain, isliye log-posterior $(0,1)$ par har jagah concave hai — single critical point genuinely maximum hai.
2. Boundary check. Jab $\theta \to 0^{+}$, $(a-1)\log \theta \to -\infty$; jab $\theta \to 1^{-}$, $(b-1)\log (1-\theta )\to -\infty$. Isliye density dono ends par zero ho jaati hai aur boundary par koi bada value nahi chupta. Interior peak jeetti hai.

Parent note se, ${\hat{\mu }}_{\text{MAP}}=\frac{4n}{4n+1}\bar{x}$. $n=1,\bar{x}=2$ ke saath: ${\hat{\mu }}_{\text{MAP}}=\frac{4}{5}\cdot 2=\frac{8}{5}=\mathrm{1.6.}$ Prior mean $0$ ne raw data mean $2$ ko shrink karke $1.6$ kar diya.

• $n\to 0$: ${\hat{\mu }}_{\text{MAP}}\to 0$. Koi data nahi to estimate prior mean par collapse ho jaata hai — belief hi sab kuch hai. Yeh red curve ka left end hai, orange dashed line par baitha hua.
• $n\to \infty$: $\frac{4n}{4n+1}\to 1$, isliye ${\hat{\mu }}_{\text{MAP}}\to \bar{x}$, yaani MLE. Data prior ko duba deta hai. Yeh red curve ka right mein blue dashed line ki taraf approach karna hai. Red curve ka $0$ se $1$ tak smooth climb hi shrinkage ki kahani hai: evidence dheere-dheere belief ko overtake karta hai.

Log-posterior (constants drop karo): $\log P(\mu \mid \mathbf{x})=-\frac{1}{2{\sigma }^2}{\sum }_i(x_i-\mu {)}^2-\frac{(\mu -{\mu }_0{)}^2}{2{\sigma }_0^2}.$ Differentiate karo aur zero ke barabar karo: $\frac{1}{{\sigma }^2}{\sum }_i(x_i-\mu )-\frac{\mu -{\mu }_0}{{\sigma }_0^2}=0.$ KYUN hum terms ko is tarah group karte hain: hum ek single equation chahte hain "(coefficient)$\cdot \mu$ = (jaani hui cheez)", isliye hum har piece jo $\mu$ contain karta hai ek side le jaate hain aur har constant doosri side. ${\sum }_i{x}_i=n\bar{x}$ use karke sum expand karo (wahan $n$ identical $-\mu$ terms hain, jo $-n\mu$ dete hain): $\frac{n\bar{x}}{{\sigma }^2}-\frac{n\mu }{{\sigma }^2}-\frac{\mu }{{\sigma }_0^2}+\frac{{\mu }_0}{{\sigma }_0^2}=0.$ Ab do $\mu$-terms collect karo (woh unknown $\mu$ share karte hain) left side par aur do known means right side par push karo: $\mu \left(\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}\right)=\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0.$ $\mu$ isolate karne ke liye bracket se divide karo: ${\hat{\mu }}_{\text{MAP}}=\frac{\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0}{\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}}.$ Yeh ek precision-weighted average hai: har mean ko uski precision (1/variance) se weight kiya jaata hai — hum us par kitna confident hain. Zyada data ($n\uparrow$) ya sharper prior (${\sigma }_0^2\downarrow$) balance accordingly tip karta hai. ${\sigma }^2=1,{\sigma }_0^2=4,{\mu }_0=0$ plug karne par $\frac{4n}{4n+1}\bar{x}$ wapas milta hai.

Posterior Beta$(1+k,1+n-k)$ hai. Mode $=\frac{(1+k)-1}{(1+k)+(1+n-k)-2}=\frac{k}{n}$, exactly MLE. Kyun: Beta$(1,1)$ $[0,1]$ par flat, uniform density hai — completely opinion-free prior. Flat prior log-posterior mein ek constant add karta hai, likelihood peak ko untouched chhodta hai. Yeh L1 principle action mein hai.

MAP negative log-posterior minimise karta hai: $-\log P(\mathbf{x}\mid \theta )-\log P(\theta ).$ Gaussian prior $\theta \sim N(0,{\sigma }^{2}I)$ ke liye (yahan ${\sigma }^2$ weights par prior variance hai), $-\log P(\theta )=\frac{\Vert \theta {\Vert }^2}{2{\sigma }^2}+\text{const}$. Isliye objective ban jaata hai $\underset{\text{data loss}}{\underbrace{-\log P(\mathbf{x}\mid \theta )}}+\underset{\text{L2 penalty}}{\underbrace{\frac{1}{2{\sigma }^2}\Vert \theta {\Vert }^2}}.$ Ise standard ridge form $\text{loss}+\lambda \Vert \theta {\Vert }^2$ se match karne par milta hai $\boxed{\lambda =\frac{1}{2{\sigma }^2}}.$ Tighter prior (chhota ${\sigma }^2$) → bada $\lambda$ → $0$ ki taraf stronger shrinkage. Yahi wajah hai ki Ridge Regression aur L2 regularization dil se Bayesian hain. Convention caveat: exact constant depend karta hai ki tum penalty kaise likhte ho. Agar $\lambda \Vert \theta {\Vert }^2$ use karo to $\lambda =\frac{1}{2{\sigma }^2}$ milta hai; agar instead penalty $\frac{\lambda }{2}\Vert \theta {\Vert }^2$ likho (bahut textbooks $\frac{1}{2}$ ko $\lambda$ mein absorb karte hain) to $\lambda =\frac{1}{{\sigma }^2}$ milta hai. Invariant message yeh hai ki $\lambda \propto 1/{\sigma }^2$ — tighter prior matlab stronger penalty; leading factor sirf bookkeeping hai.

Laplace (double-exponential) prior $P(\theta )\propto \exp (-|\theta |/b)$ deta hai $-\log P(\theta )=\frac{|\theta |}{b}+\text{const}$, yaani L1 penalty $\lambda \Vert \theta {\Vert }_1$ jisme $\lambda =1/b$. Sparsity kyun: Laplace prior ka $0$ par ek sharp spike hota hai (uski density wahan non-differentiable hai). Woh kink — upar figure mein green V — matlab MAP objective ka optimum exactly ${\theta }_j=0$ par baith sakta hai, kai coefficients ko precisely zero drive karta hai.

$\lambda =\frac{1}{2{\sigma }^2}$ se: ${\sigma }^2=\frac{1}{2\lambda }=\frac{1}{2\cdot 0.5}=1.$ Weights par unit-variance Gaussian prior.

Interior-mode formula $\frac{a-1}{a+b-2}$ sirf tab valid hai jab $a>1$ aur $b>1$ ho; warna log-posterior concave nahi rahta aur hume directly endpoints inspect karne padte hain. Recall karo log-density $\log P=(a-1)\log \theta +(b-1)\log (1-\theta )$.

(a) Symmetric $a=b=0.5$, koi data nahi (posterior = prior). Density ${\theta }^{-0.5}(1-\theta {)}^{-0.5}$ dono endpoints par $+\infty$ tak blow up karti hai. Formula deta hai $\frac{-0.5}{-1}=0.5$, lekin woh critical point ek minimum hai, maximum nahi. Curvature check karo: $\frac{d^2}{d{\theta }^2}\log P=-\frac{a-1}{{\theta }^2}-\frac{b-1}{(1-\theta {)}^2}$; $a-1=b-1=-0.5$ ke saath dono terms positive hain, isliye log-posterior convex hai — interior point $0.5$ ek valley bottom hai. Jab $\theta \to 0^{+}$ aur $\theta \to 1^{-}$ density $\to +\infty$, isliye maxima boundaries par hain. Honest MAP: non-unique — maximizers dono $\theta =0$ aur $\theta =1$ par; posterior mode ek single point nahi hai.

(b) One-sided $a\le 1<b$. Term $(a-1)\le 0$ density ko $\theta =0$ ki taraf non-decreasing pressure deta hai (jab $\theta \to 0^{+}$, $(a-1)\log \theta \to +\infty$ agar $a<1$ ho, ya finite rahta hai agar $a=1$ ho), jabki $(b-1)>0$ density ko $\theta =1$ par $0$ ki taraf push karta hai. Single maximum left boundary par hota hai. Honest MAP: ${\hat{\theta }}_{\text{MAP}}=0$.

(c) One-sided $b\le 1<a$. Mirror-image argument se, $(b-1)\le 0$ density ko $\theta =1$ ki taraf drive karta hai jabki $(a-1)>0$ use $\theta =0$ par kill karta hai. Single maximum right boundary par hota hai. Honest MAP: ${\hat{\theta }}_{\text{MAP}}=1$.

Take-away: interior formula par trust karne se pehle hamesha confirm karo ki $a>1$ aur $b>1$ hai; warna maximum boundaries check karke locate karo, aur non-unique mode report karne ke liye taiyaar raho.

Posterior $=$ Beta$(2+4,10+0)=$ Beta$(6,10)$. Dono shapes $>1$ hain, isliye mode valid hai: ${\hat{\theta }}_{\text{MAP}}=\frac{6-1}{6+10-2}=\frac{5}{14}\approx \mathrm{0.357.}$ MLE $4/4=1.0$ hai (insist karta ki coin kabhi tails nahi aati — ek classic small-sample overfit). Strong prior estimate ko $0.357$ tak kheeench leta hai, extreme claim ko refuse karta hai. Yeh scarce data ke saath MAP ka safety-rail value hai.

Exercise 3.2 se, prior weight prior precision $1/{\sigma }_0^2$ hai. Jab ${\sigma }_0^2\to \infty$ to yeh weight $\to 0$: ${\hat{\mu }}_{\text{MAP}}=\frac{\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0}{\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}}⟶\frac{\frac{n}{{\sigma }^2}\bar{x}}{\frac{n}{{\sigma }^2}}=\bar{x}.$ Infinitely vague prior koi information carry nahi karta, isliye MAP → MLE. Data ka weight $n/{\sigma }^2$ exactly $n$ Gaussian samples ka $\mu$ ke baare mein Fisher information hai — data jitna zyada information carry karta hai, woh utna hi ek fixed prior ko dominate karta hai.

Koi data nahi to data term vanish ho jaata hai; log-posterior hi log-prior hai, jiska peak prior mean hai: ${\hat{\mu }}_{\text{MAP}}={\mu }_0=3.$ Prior variance $0.25$ hamari confidence affect karta hai, lekin zero observations ke saath peak location simply prior mean hai.

Posterior $P(\theta \mid \mathbf{x})$ ka mode (peak), uska mean nahi.

Uniform (flat) prior — belief term ek constant ban jaata hai aur $\arg \max$ se bahar nikal jaata hai.

$(a,b)$ ka mode jab $a,b>1$ ho? $\frac{a-1}{a+b-2}$ (sirf tab valid jab log-posterior concave ho, yaani $a>1$ aur $b>1$; warna maximum boundary par hota hai).

$\lambda$ usse kaise related hai? Zero-mean Gaussian prior; $\lambda \Vert \theta {\Vert }^2$ convention ke saath, $\lambda =\frac{1}{2{\sigma }^2}$ (isliye $\lambda \propto 1/{\sigma }^2$).

Laplace prior; uska sharp spike (kink) $0$ par optimum ko exactly $\theta =0$ par baithne deta hai.

Prior mean aur data mean ki precision-weighted average, jisme har ek ko uski precision (1/variance) se weight kiya jaata hai.

Precision kya hai? Precision $=1/\text{variance}$ — high jab tum certain ho (narrow bell), low jab vague ho (wide bell).