Maximum a posteriori estimation (MAP) — Hinglish notes

1.3.17 · AI-ML › Probability & Statistics

#ai-ml/probability-statistics #bayesian-inference #parameter-estimation

Definition & Mathematical Foundation

Derivation from Bayes' Theorem

Chaliye ise Bayes' theorem se scratch se build karte hain:

Step 1: Posterior ke liye Bayes' rule se shuru karo $P (θ ∣ x) = \frac{P ( x ∣ θ ) \cdot P ( θ )}{P ( x )}$

Yeh form kyun? Hum $P (x ∣ θ)$ (likelihood) se $P (θ ∣ x)$ (posterior) mein "flip" karna chahte hain. Bayes' theorem is inversion ke liye mathematical machinery hai.

Step 2: Woh $θ$ dhundho jo posterior ko maximize kare $\hat{θ}_{MAP} = ar g max_{θ} P (θ ∣ x) = ar g max_{θ} \frac{P ( x ∣ θ ) \cdot P ( θ )}{P ( x )}$

Yeh step kyun? Hum posterior distribution ka peak dhundh rahe hain — ek sabse zyada likely parameter value.

Step 3: Denominator hata do (woh $θ$ ke saath constant hai) $\hat{θ}_{MAP} = ar g max_{θ} P (x ∣ θ) \cdot P (θ)$

Hum ise kyun hata sakte hain? $P (x)$ marginal likelihood (evidence) hai, jo $θ$ par depend nahi karta. Kyunki hum $θ$ par optimize kar rahe hain, constants argmax location ko affect nahi karte.

Step 4: Computational convenience ke liye logarithm lo $\hat{θ}_{MAP} = ar g max_{θ} [lo g P (x ∣ θ) + lo g P (θ)]$

Log kyun? Products sums ban jaate hain (optimize karna aasaan), aur log monotonic hai isliye argmax preserve hota hai. Yahi standard log-posterior hai.

Relationship to MLE

Maximum Likelihood Estimation (MLE), MAP ka ek special case hai:

$\hat{θ}_{MLE} = ar g max_{θ} P (x ∣ θ) = ar g max_{θ} lo g P (x ∣ θ)$

Derivation: Agar hum uniform prior $P (θ) = const$ use karein, toh: $\hat{θ}_{MAP} = ar g max_{θ} [lo g P (x ∣ θ) + lo g (const)] = ar g max_{θ} lo g P (x ∣ θ) = \hat{θ}_{MLE}$

Kyun? Uniform prior ka matlab hai "data dekhne se pehle sabhi parameter values equally likely hain." Prior optimization mein kuch contribute nahi karta, toh hum pure likelihood maximization par wapas aa jaate hain.

Key insight: MAP, MLE ko non-uniform priors allow karke generalize karta hai. MLE, MAP hai jisme koi prior knowledge nahi hai.

Worked Examples

Worked example Example 1: Coin Flip with Beta Prior

Setup: Hum ek coin 10 baar flip karte hain, 7 heads aate hain. Beta(3, 2) prior ke saath MAP use karke bias $θ$ (heads ki probability) estimate karo.

Beta(3,2) kyun? Yeh prior encode karta hai "mujhe lagta hai coin data dekhne se pehle thoda heads ki taraf biased hai" (mean = 3/(3+2) = 0.6).

Step 1: Likelihood likho $n = 10$ flips mein $k = 7$ heads ke liye: $P (x ∣ θ) = (7 10) θ^{7} (1 - θ)^{3}$

Yeh form kyun? Binomial distribution — hum 10 flips mein 7 heads paane ke tarike count karte hain, har sequence ki probability se weight karke.

Step 2: Prior likho Beta(3, 2) prior: $P (θ) = \frac{θ ^{3 - 1} ( 1 - θ ) ^{2 - 1}}{B ( 3 , 2 )} = \frac{θ ^{2} ( 1 - θ )}{B ( 3 , 2 )}$

Beta kyun? Yeh binomial likelihood ke liye conjugate prior hai — posterior bhi Beta rahega, jisse math tractable ho jaata hai.

Step 3: Posterior form karo (constant tak) $P (θ ∣ x) \propto θ^{7} (1 - θ)^{3} \cdot θ^{2} (1 - θ) = θ^{7 + 2} (1 - θ)^{3 + 1} = θ^{9} (1 - θ)^{4}$

Exponents add kyun karte hain? Hum terms multiply kar rahe hain, isliye exponents add hote hain. Yeh Beta(10, 5) hai.

Step 4: Mode dhundho Log lo aur differentiate karo: $lo g P (θ ∣ x) = 9 lo g θ + 4 lo g (1 - θ) + const$ $\frac{d}{d θ} = \frac{9}{θ} - \frac{4}{1 - θ} = 0$ $9 (1 - θ) = 4 θ ⟹ 9 = 13 θ ⟹ θ = \frac{9}{13} \approx 0.692$

Derivative zero kyun set karte hain? Maximum par, slope zero hota hai. Yeh standard calculus optimization hai.

$\hat{θ}_{MAP} = \frac{9}{13} = 0.692$

MLE se comparison: $\hat{θ}_{MLE} = \frac{7}{10} = 0.7$

Interpretation: Prior ne hamare estimate ko 0.7 se prior mean 0.6 ki taraf thoda "kheencha". Zyada data ke saath, yeh effect kam ho jaata hai.

Worked example Example 2: Gaussian Mean with Gaussian Prior

Setup: Hum $x = {x_{1}, \dots, x_{n}}$ observe karte hain $N (μ, σ^{2})$ se, jaana hua $σ^{2} = 1$ ke saath. Gaussian prior $μ \sim N (μ_{0}, σ_{0}^{2})$ use karo jisme $μ_{0} = 0, σ_{0}^{2} = 4$ hai.

Gaussian prior kyun? Hum believe karte hain ki $μ$ 0 ke aas-paas hai lekin kuch uncertainty ke saath ( $σ_{0}^{2} = 4$ ka matlab hai humare prior belief ka 95% [-4, 4] mein hai).

Step 1: Likelihood $P (x ∣ μ) = \prod_{i = 1}^{n} \frac{1}{2 π} exp (- \frac{( x _{i} - μ ) ^{2}}{2}) \propto exp (- \frac{1}{2} \sum_{i = 1}^{n} (x_{i} - μ)^{2})$ Product kyun? i.i.d. samples assume karke, joint probability factorize hoti hai.

Step 2: Prior $P (μ) = \frac{1}{2 π \cdot 4} exp (- \frac{μ ^{2}}{8}) \propto exp (- \frac{μ ^{2}}{8})$

Step 3: Log-posterior $lo g P (μ ∣ x) = - \frac{1}{2} \sum_{i = 1}^{n} (x_{i} - μ)^{2} - \frac{μ ^{2}}{8} + const$ Step 4: Differentiate karo aur solve karo $\frac{d}{d μ} = \sum_{i = 1}^{n} (x_{i} - μ) - \frac{μ}{4} = 0$ $n \overset{x}{ˉ} - n μ - \frac{μ}{4} = 0$ $n \overset{x}{ˉ} = μ (n + \frac{1}{4})$ $μ = \frac{n x ˉ}{n + 1/4} = \frac{4 n x ˉ}{4 n + 1}$

Yeh form kyun? Hum data mean $\overset{x}{ˉ}$ (jo $n$ se weighted hai) aur prior mean 0 (jo $1/ σ_{0}^{2} = 1/4$ se weighted hai) ke beech balance kar rahe hain.

$\overset{μ}{^}_{MAP} = \frac{4 n}{4 n + 1} \overset{x}{ˉ}$ Special cases:

$n \to \infty$ : $\overset{μ}{^}_{MAP} \to \overset{x}{ˉ}$ (MLE)
$n = 1, \overset{x}{ˉ} = 2$ : $\overset{μ}{^}_{MAP} = \frac{8}{5} = 1.6$ (0 ki taraf shrunk) Interpretation: Data mean aur prior mean ke beech ek weighted average, jisme weights unke relative precisions se determine hote hain.

Common Mistakes

Common mistake Mistake 2: Regularization mein Prior Bhool Jaana

Galat idea: "ML mein Regularization (jaise L2) sirf overfitting rokne ka ek trick hai." Kyun sahi lagta hai: Hum regularization ko loss functions mein add hone wale penalty term ke roop mein sikhte hain, aksar bina Bayesian context ke.

Fix: L2 regularization IS MAP estimation with a Gaussian prior. Regularization term $λ ∥ θ ∥^{2}$ exactly $- lo g P (θ)$ hai jab $θ \sim N (0, 1/ (2 λ))$ . Derivation: $MAP objective: - lo g P (x ∣ θ) - lo g P (θ)$ $If P (θ) = N (0, σ^{2}) : - lo g P (θ) = \frac{∥ θ ∥ ^{2}}{2 σ ^{2}} + const$ $Set λ = \frac{1}{2 σ ^{2}} : MAP = MLE + λ ∥ θ ∥^{2}$

Similarly: L1 regularization = Laplace prior ke saath MAP.

Yeh kyun important hai: Is connection ko samajhna explain karta hai ki regularization kyun kaam karta hai — yeh prior belief encode kar raha hai ki parameters chhote hone chahiye.

MAP vs MLE vs Full Bayes

Method	Kya deta hai	Formula	Kab use karein
MLE	Single point estimate	$ar g max P (x ∣ θ)$	Bahut data ho, koi prior info na ho
MAP	Single point estimate	$ar g max P (θ ∣ x)$	Moderate data ho, useful prior ho
Full Bayes	Poori distribution	$P (θ ∣ x)$	Uncertainty quantification chahiye

MAP kyun?

Chhote data ke saath MLE se zyada robust (prior regularize karta hai)
Full Bayes se computationally simpler (integration nahi chahiye)
Optimization mein regularization se natural connection

Connection to Regularization (Deep Dive)

Negative log-posterior ek regularized loss hai:

$- lo g P (θ ∣ x) = - lo g P (x ∣ θ) - lo g P (θ) + const$

Ise minimize karna (posterior maximize karne ke equivalent) deta hai:

$\hat{θ}_{MAP} = ar g min_{θ} Data loss - lo g P (x ∣ θ) + Regularizer (- lo g P (θ))$

Common correspondences:

Gaussian prior $N (0, σ^{2})$ ⟷ L2 regularization ( $λ = 1/ (2 σ^{2})$ )
Laplace prior $Laplace (0, b)$ ⟷ L1 regularization ( $λ = 1/ b$ )
Uniform prior ⟷ Koi regularization nahi (MLE)

Yeh powerful kyun hai: Har regularization scheme ko parameters ke baare mein ek prior belief ke roop mein interpret kiya ja sakta hai. Regularization choose karna = prior choose karna.

Recall Ise 12 saal ke bacche ko samjhao

Socho tum ek jar mein candies count karne ki koshish kar rahe ho. Tumhare dost ne kaha "maine glass se 47 candies gine" (yeh tumhara data hai). Lekin tum yeh bhi jaante ho ki jar usually full hone par lagbhag 50 candies rakhta hai (yeh tumhari prior knowledge hai).

MLE kehta: "Count par poori tarah bharosa karo — 47 guess karo."

MAP kehta: "Count probably sahi ke aas-paas hai, lekin jar usually 50 ke aas-paas hota hai, toh shayad count thoda off tha. Main 48 ya 49 guess karunga — jo maine dekha aur jo main expect karta tha, uske beech kuch."

MAP jo aap dekhte ho (data) aur jo aap pehle se jaante ho (prior) ko combine karta hai. Yeh do experts se poochne aur unke beech ek smart compromise nikalne jaisa hai. Jitna confident har expert hai, utna aap unhe sunte ho. Agar aapne 1000 candies clearly dekhi hain, toh aap data par zyada bharosa karte ho. Agar sirf ek nazar dekhi hai, toh aap apni prior knowledge par zyada bharosa karte ho.

Active Recall Challenges

Answers (pehle try karo!):

(Upar derivation section dekho)
Uniform prior; kyunki $lo g P (θ) = const$ argmax mein kuch contribute nahi karta
Posterior hai Beta(2+2, 2+1) = Beta(4,3), mode = (4-1)/(4+3-2) = 3/5 = 0.6
L2 regularization $λ ∥ θ ∥^{2}$ = Gaussian prior $N (0, 1/ (2 λ))$ ke saath MAP
Data term $n \to \infty$ ke saath prior term par dominate karta hai → MAP → MLE (prior wash out ho jaata hai)

Connections

Maximum Likelihood Estimation (MLE) — MAP ise priors add karke generalize karta hai
Bayes Theorem — MAP derive karne ki foundation
Conjugate Priors — MAP ko computationally tractable banate hain
Regularization in ML — Specific priors ke saath MAP ke equivalent hai
Beta Distribution — Probabilities ke liye common prior
Posterior Distribution — MAP uska mode dhundhta hai
Overfitting — MAP ise chhote data ke saath MLE se behtar rokta hai
Fisher Information — Reparameterization invariance ke liye Jeffreys prior mein use hoti hai
Ridge Regression — Coefficients par Gaussian prior ke saath MAP
Lasso Regression — Coefficients par Laplace prior ke saath MAP

#flashcards/ai-ml

What is the MAP estimation formula in terms of log-posterior? :: $\hat{θ}_{MAP} = ar g max_{θ} [lo g P (x ∣ θ) + lo g P (θ)]$ — log-likelihood aur log-prior ka sum

How does MAP relate to MLE? :: MLE, MAP hai uniform prior ke saath: jab $P (θ) = const$ , prior term vanish ho jaata hai aur MAP, MLE mein reduce ho jaata hai

In MAP estimation, why can we drop

P (x)

from Bayes' theorem?

Kyunki

P (x)

θ

par depend nahi karta — yeh optimization variable ke w.r.t. ek constant hai, isliye yeh argmax ko affect nahi karta

For coin flips with

k

heads in

n

flips and Beta(

α

β

) prior, what is the posterior?

Beta(

α + k

β + n - k

) — observed heads ko

α

mein add karo, observed tails ko

β

mein add karo

What is the mode of a Beta(

α

β

) distribution?

\frac{α - 1}{α + β - 2}

jab

α, β > 1

— log-PDF ka derivative zero set karke mila

What type of regularization corresponds to a Gaussian prior in MAP?

L2 regularization — Gaussian prior

N (0, σ^{2})

regularization term

\frac{∥ θ ∥ ^{2}}{2 σ ^{2}}

deta hai

What type of regularization corresponds to a Laplace prior in MAP?

L1 regularization — Laplace prior regularization term

λ ∥ θ ∥_{1}

deta hai

Does MAP give the mean or mode of the posterior?

Mode — MAP woh parameter value dhundhta hai jisme highest posterior probability ho, average nahi

What happens to MAP estimate as sample size approaches infinity?

MAP, MLE mein converge hota hai — data term

n

ke saath badhta hai jabki prior term constant rehta hai, isliye prior ka influence khatam ho jaata hai

For Gaussian data with Gaussian prior, is MAP estimate closer to sample mean or prior mean?

Ek weighted average — specifically

\overset{μ}{^}_{MAP} = \frac{n σ _{0}^{2}}{n σ _{0}^{2} + σ ^{2}} \overset{x}{ˉ} + \frac{σ ^{2}}{n σ _{0}^{2} + σ ^{2}} μ_{0}

Why is Beta distribution a conjugate prior for binomial likelihood?

Kyunki Beta prior × Binomial likelihood = Beta posterior — posterior same family mein rehta hai, jisse computation analytical ho jaata hai

What is the relationship between negative log-posterior and regularized loss?

Woh equivalent hain:

- lo g P (θ ∣ x) = data loss + regularization term

If you use MAP with Beta(1,1) prior for coin bias, what estimate do you get?

Exactly MLE — Beta(1,1), [0,1] par uniform hai, koi prior information na hone ke equivalent hai

What is Jeffreys prior and why use it?

P (θ) \propto ∣ I (θ) ∣

jahan

I

Fisher information hai — yeh reparameterization ke under invariant hai, ise truly non-informative prior banata hai

For Gaussian mean estimation, how does prior variance $σ_{0}^{2}$ affect MAP? :: Bada $σ_{0}^{2}$ matlab kam confident prior, toh MAP data par zyada rely karta hai aur MLE ke paas jaata hai; chhota $σ_{0}^{2}$ estimate ko prior mean ki taraf kheenchta hai