Maximum a posteriori estimation (MAP): Question bank

1.3.17 · D5 · AI-ML › Probability & Statistics › Maximum a posteriori estimation (MAP)

Quick vocabulary refresher taaki neeche kuch bhi mystery na lage:

Definition Woh words aur symbols jo tum confuse nahi kar sakte

$x$ = observations ka poora collection, yaani $x \equiv (x_{1}, x_{2}, \dots, x_{n})$ . Bold letter poori data list ka shorthand hai, koi single number nahi.
Posterior = $P (θ ∣ x)$ , data $x$ dekhne ke baad parameter ke baare mein hamara belief. Dekho Posterior Distribution.
Likelihood = $P (x ∣ θ)$ , observed data $x$ kitna probable hai ek given value of $θ$ ke liye. Yahi woh quantity hai jo MLE akele maximize karta hai.
Evidence = $P (x)$ , data ki total probability jo saare parameter values par average ki gayi ho — Bayes Theorem mein normalizing denominator. Yeh $θ$ par depend nahi karta.
Precision = variance ka inverse, $1/ σ^{2}$ . Badi precision = chhota variance = high confidence; chhoti precision = bada variance = vagueness.
Mode = distribution ka peak (sabse ऊँचा point) ka location — sabse zyada probable single value.
Mean = distribution ka balance point (average) — jahan yeh see-saw par baithti.
$n$ = observations ki number (sample size), jaise kitne coin flips ya data points collect kiye.
$σ_{0}^{2}$ = prior variance — data dekhne se pehle hamara prior belief kitna spread out hai. Chhota $σ_{0}^{2}$ = confident prior; bada $σ_{0}^{2}$ = vague prior. Iska inverse $1/ σ_{0}^{2}$ prior precision hai. MAP posterior ka mode report karta hai, uska mean nahi.

Formula Gaussian-mean example — closed form ke peeche poora "kyun"

$n$ i.i.d. points $x_{1}, \dots, x_{n}$ (isliye $x \equiv (x_{1}, \dots, x_{n})$ ) observe karo $N (μ, 1)$ se (known variance $1$ ), aur Gaussian prior $μ \sim N (0, σ_{0}^{2})$ ke saath. Log-posterior yeh hai: $lo g P (μ ∣ x) = - \frac{1}{2} \sum_{i = 1}^{n} (x_{i} - μ)^{2} - \frac{μ ^{2}}{2 σ _{0}^{2}} + const .$ Step 1 — differentiate karo. Ek smooth curve ke peak par slope zero hota hai, isliye hum poochhte hain kahan $\partial / \partial μ = 0$ hai. Pehle term differentiate karo: har $- \frac{1}{2} (x_{i} - μ)^{2}$ ka derivative $+ (x_{i} - μ)$ hai (chain rule, $(x_{i} - μ)$ ka $μ$ w.r.t. inner derivative $- 1$ hai, jo leading minus cancel kar deta hai). $- μ^{2} / (2 σ_{0}^{2})$ differentiate karne par $- μ / σ_{0}^{2}$ milta hai. Toh: $\frac{\partial}{\partial μ} lo g P (μ ∣ x) = \sum_{i = 1}^{n} (x_{i} - μ) - \frac{μ}{σ _{0}^{2}} = 0.$ Step 2 — sum expand karo. Kyunki $\sum_{i = 1}^{n} x_{i} = n \overset{x}{ˉ}$ (mean $\overset{x}{ˉ}$ ki definition) aur $\sum_{i = 1}^{n} μ = n μ$ : $n \overset{x}{ˉ} - n μ - \frac{μ}{σ _{0}^{2}} = 0.$ Step 3 — $μ$ terms collect karo. Data wala part ek side le jao aur baaki se $μ$ factor out karo: $n \overset{x}{ˉ} = n μ + \frac{μ}{σ _{0}^{2}} = μ (n + \frac{1}{σ _{0}^{2}}) .$ Step 4 — divide karo. $μ$ isolate karo: $\overset{μ}{^}_{MAP} = \frac{n x ˉ}{n + 1/ σ _{0}^{2}} .$ Ise ek weighted average ki tarah padho: data mean $\overset{x}{ˉ}$ ka weight $n$ hai (unit variance ke $n$ observations se uski precision), prior mean $0$ ka weight $1/ σ_{0}^{2}$ hai (prior precision), aur hum total weight se divide karte hain. Jab $σ_{0}^{2} \to \infty$ tab prior weight $1/ σ_{0}^{2} \to 0$ aur $\overset{μ}{^}_{MAP} \to \overset{x}{ˉ}$ (MLE).

True or false — justify karo

MAP hamesha MLE se alag answer deta hai.

False. Uniform (flat) prior ke saath log-prior constant hota hai, isliye woh argmax se drop ho jaata hai aur MAP exactly MLE par collapse ho jaata hai.

Jaise sample size

n

(observations ki number) badhta hai, MAP estimate MLE se aur dur drift karta hai.

False — bilkul ulta. Likelihood har data point ke saath badhti hai jabki prior fixed rehta hai, isliye prior ka relative weight shrink hota hai aur MAP

\to

MLE jab

n \to \infty

MAP posterior distribution ka mean return karta hai.

False. MAP mode return karta hai (peak). Mean aur mode sirf symmetric posteriors jaise Gaussian mein agree karte hain; skewed ones jaise Beta ya Gamma mein differ karte hain.

MAP estimate hamesha ek valid probability (0 aur 1 ke beech) hoti hai jab coin bias estimate karte hain.

Parameter ki apni range ke liye True, kyunki prior support

θ \in [0, 1]

force karta hai; lekin yeh guarantee model ke domain se milti hai, MAP se khud nahi.

Prior se multiply karna estimate ko sirf zero ki taraf shrink kar sakta hai.

False. Prior estimate ko prior ke high-density region ki taraf kheenchta hai, jo MLE se upar ya neeche ho sakta hai. Ek Beta(3,2) prior (mean 0.6) ek low MLE ko upar kheench sakta hai.

MAP estimate find karne ke liye evidence term

P (x)

zaroor compute karni chahiye.

False. Evidence

P (x)

θ

mein constant hai, isliye yeh maximum ki location ko kabhi affect nahi karta aur drop kar diya jaata hai. Tumhe isse sirf normalized posterior ke liye chahiye, argmax ke liye nahi.

L2 regularization aur Gaussian prior ke saath MAP ek hi optimization ke do naam hain.

True.

- lo g P (x ∣ θ) + λ ∥ θ ∥^{2}

minimize karna exactly log-posterior maximize karna hai

θ \sim N (0, \frac{1}{2 λ})

ke saath. Dekho Regularization in ML aur Ridge Regression.

Ek stronger prior (chhota prior variance

σ_{0}^{2}

) ka matlab hai data ka zyada influence hai.

False. Chhota prior variance = high prior precision = high confidence, isliye data ka pull kam hota hai aur estimate prior mean ke paas rehti hai.

MAP ke liye prior aur likelihood conjugate hone zaroori hain.

False. Conjugate Priors posterior ko ek clean closed form banate hain, lekin MAP ko sirf log-likelihood + log-prior maximize karna hota hai — yeh kisi bhi prior ke liye numerically kiya ja sakta hai.

Error dhundho

"Kyunki posterior likelihood times prior ke proportional hai, MAP = MLE times prior mode."

Galat. Tum functions

P (x ∣ θ) P (θ)

ko multiply karte ho aur phir product ka argmax find karte ho; tum do alag estimates ko multiply nahi kar sakte.

"Main

lo g P (θ)

drop kar dunga kyunki yeh constant hai, jaise maine

P (x)

drop kiya."

Error. Evidence

P (x)

θ

mein constant hai, lekin

P (θ)

$θ$ par depend karta hai — ise drop karna prior ko delete kar deta hai aur MAP ko wapas MLE bana deta hai.

"Beta(1,1) ek informative prior hai jo low values favor karta hai."

Galat. Beta(1,1)

[0, 1]

par uniform prior hai — yeh flat aur uninformative hai, isliye iske under MAP equals MLE.

"Beta

(α, β)

ka mode

α / (α + β)

hai."

Yeh mean hai. Mode

\frac{α - 1}{α + β - 2}

hai (jab

α, β > 1

); inhe confuse karna classic mode/mean slip hai.

"Log lene se maximum ki location change ho jaati hai, isliye log-posterior MAP ≠ posterior MAP."

Galat.

lo g

monotonically increasing hai, isliye yeh argmax ki location preserve karta hai. Yeh sirf products ko sums mein badle leta hai easier differentiation ke liye.

"Ridge regression mein bada

λ

wider (weaker) Gaussian prior correspond karta hai."

Ulta hai. Gaussian prior

N (0, σ^{2})

se penalty

- lo g P (θ) = \frac{∥ θ ∥ ^{2}}{2 σ ^{2}}

hai, isliye ise

λ ∥ θ ∥^{2}

se match karne par

λ = \frac{1}{2 σ ^{2}}

milta hai; bada

λ

isliye chhota

σ^{2}

matlab hai — ek narrower, stronger prior jo weights ko zyada shrink karta hai.

Why questions

MAP Overfitting kyun rokta hai jab data scarce ho?

Thode data points ke saath likelihood flat aur noisy hoti hai; prior extra structure supply karta hai jo estimate ko plausible values par anchor karta hai sampling noise chase karne ki jagah.

Hum objective mein prior multiply karne ki jagah log-prior add kyun karte hain?

Kyunki humne ek product (likelihood × prior) ka log liya, aur

lo g (ab) = lo g a + lo g b

product ko ek sum mein convert kar deta hai jo differentiate karna bahut easy hai.

L1 / Lasso regularization Gaussian prior ke under nahi balki Laplace prior ke under MAP estimate kyun hai?

Laplace density ki shape

\propto e^{- ∣ θ ∣/ b}

hai (jahan

b > 0

iska scale parameter hai, spread control karta hai), isliye iska negative log

\propto ∣ θ ∣

hai — exactly L1 penalty, jabki Gaussian squared L2 penalty deta hai.

Gaussian-mean example weighted average of data mean aur prior mean kyun deta hai?

Log-posterior do quadratics ka sum hai; iska derivative zero set karne par har term linearly balance hoti hai apni precision (inverse variance) se, jo literally

\overset{x}{ˉ}

(weight = data precision

n

) aur

0

(weight = prior precision

1/ σ_{0}^{2}

) ka weighted average hai.

Hum MAP mode ki jagah hamesha posterior mean report kyun nahi kar sakte?

Mean ke liye poore posterior ko integrate (normalize) karna padta hai, jo aksar intractable hota hai; mode ko sirf ek argmax chahiye, jo optimization cheaply handle karta hai.

Flat prior MAP ko "objective-free" kyun banata hai?

Flat prior har parameter value ko a priori equally plausible maanta hai, isliye koi preference inject nahi karta — estimate purely data se drive hoti hai, yaani MLE.

Edge cases

Kya hoga jab prior ek zero probability region of

θ

par rakhta hai?

MAP wahan kabhi nahi ja sakta — log-prior

- \infty

hai, isliye woh region forbidden hai chahe data kitna bhi strongly use support kare.

Beta-coin MAP mein kya hota hai agar tum zero flips observe karo (

n = 0

Likelihood constant hai, isliye posterior prior ke equal ho jaata hai aur MAP prior mode return karta hai — pure belief, koi evidence nahi. Caveat: agar prior uniform Beta(1,1) hai, toh woh prior flat hai, isliye uska mode unique nahi hai (har

θ

ek peak hai) aur MAP yahan undefined hai.

Beta

(α, β)

posterior ke liye exactly

α < 1

β < 1

mein se ek ke saath mode kahan hai?

Density ek endpoint par blow up karti hai (

α < 1

ho toh 0 par,

β < 1

ho toh 1 par), isliye mode us boundary par hota hai, aur smooth "derivative zero set karo" formula ab apply nahi hota.

Beta

(α, β)

posterior ke liye dono

α < 1

aur

β < 1

ke saath mode kahan hai?

Density dono endpoints par blow up karti hai, do equal boundary modes 0 aur 1 par deta hai — ek genuine tie, aur MAP unke beech choose nahi kar sakta.

Poori real line par improper (unnormalizable) flat prior ke under MAP estimate kya hai?

Yeh tab bhi kaam karta hai jab posterior khud proper ho — constant prior drop ho jaata hai aur MAP equals MLE, chahe

P (θ)

ek real distribution nahi hai.

Agar posterior bimodal hai (do equal peaks), MAP kya report karta hai?

MAP sirf ek peak return karta hai (ya unke beech ambiguous rehta hai), jo misleading hai — ek single point do-humped belief ko badly summarize karta hai.

Jab Gaussian-mean example mein prior variance

σ_{0}^{2} \to \infty

, toh

\overset{μ}{^}_{MAP}

kiske paas jaata hai?

Yeh

\overset{x}{ˉ}

, yaani MLE, ke paas jaata hai, kyunki infinitely wide prior effectively flat hota hai aur optimization mein kuch contribute nahi karta.

Kya hota hai jab likelihood aur prior opposite ends of the range par peak karte hain?

Ek compromise dono ke beech mein milta hai, jis term ki curvature zyada sharp hai (higher curvature / more confident) uski taraf pull hoti hai, sirf midpoint par nahi.

Recall Lock in karne ke liye one-line summary

MAP = argmax of (log-likelihood + log-prior) = posterior ka mode; flat prior ke saath MLE ban jaata hai aur Gaussian/Laplace prior ke saath L2/L1 regularization ban jaata hai.