Maximum a posteriori estimation (MAP)

#ai-ml/probability-statistics #bayesian-inference #parameter-estimation

Definition & Mathematical Foundation

Derivation from Bayes' Theorem

Let's build this from scratch using Bayes' theorem:

Step 1: Start with Bayes' rule for the posterior $P(\theta \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid \theta) \cdot P(\theta)}{P(\mathbf{x})}$

Why this form? We want to "flip" from $P(\mathbf{x} \mid \theta)$ (the likelihood) to $P(\theta \mid \mathbf{x})$ (the posterior). Bayes' theorem is the mathematical machinery for this inversion.

Step 2: Find the $\theta$ that maximizes the posterior $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} P(\theta \mid \mathbf{x}) = \arg\max_{\theta} \frac{P(\mathbf{x} \mid \theta) \cdot P(\theta)}{P(\mathbf{x})}$

Why this step? We're looking for the peak of the posterior distribution—the single most likely parameter value.

Step 3: Drop the denominator (it's constant w.r.t. $\theta$ ) $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} P(\mathbf{x} \mid \theta) \cdot P(\theta)$

Why can we drop it? $P(\mathbf{x})$ is the marginal likelihood (evidence), which doesn't depend on $\theta$ . Since we're optimizing over $\theta$ , constants don't affect the argmax location.

Step 4: Take logarithm for computational convenience $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \left[ \log P(\mathbf{x} \mid \theta) + \log P(\theta) \right]$

Why log? Products become sums (easier to optimize), and log is monotonic so it preserves argmax. This is the standard log-posterior.

Relationship to MLE

Maximum Likelihood Estimation (MLE) is a special case of MAP:

$\hat{\theta}_{\text{MLE}} = \arg\max_{\theta} P(\mathbf{x} \mid \theta) = \arg\max_{\theta} \log P(\mathbf{x} \mid \theta)$

Derivation: If we use a uniform prior $P(\theta) = \text{const}$ , then: $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \left[ \log P(\mathbf{x} \mid \theta) + \log(\text{const}) \right] = \arg\max_{\theta} \log P(\mathbf{x} \mid \theta) = \hat{\theta}_{\text{MLE}}$

Why? A uniform prior means "all parameter values are equally likely before seeing data." The prior contributes nothing to the optimization, so we're back to pure likelihood maximization.

Key insight: MAP generalizes MLE by allowing non-uniform priors. MLE is MAP with no prior knowledge.

Worked Examples

Worked example Example 1: Coin Flip with Beta Prior

Setup: We flip a coin 10 times, get 7 heads. Estimate the bias $\theta$ (probability of heads) using MAP with a Beta(3, 2) prior.

Why Beta(3,2)? This prior encodes "I think the coin is slightly biased toward heads before seeing data" (mean = 3/(3+2) = 0.6).

Step 1: Write the likelihood For $n=10$ flips with $k=7$ heads: $P(\mathbf{x} \mid \theta) = \binom{10}{7} \theta^7 (1-\theta)^{3}$

Why this form? Binomial distribution—we count the number of ways to get 7 heads in 10 flips, weighted by the probability of each sequence.

Step 2: Write the prior Beta(3, 2) prior: $P(\theta) = \frac{\theta^{3-1}(1-\theta)^{2-1}}{B(3,2)} = \frac{\theta^2(1-\theta)}{B(3,2)}$

Why Beta? It's the conjugate prior for binomial likelihood—posterior will also be Beta, making math tractable.

Step 3: Form the posterior (up to constant) $P(\theta \mid \mathbf{x}) \propto \theta^7(1-\theta)^3 \cdot \theta^2(1-\theta) = \theta^{7+2}(1-\theta)^{3+1} = \theta^9(1-\theta)^4$

Why add exponents? We're multiplying terms, so exponents add. This is Beta(10, 5).

Step 4: Find the mode Take log and differentiate: $\log P(\theta \mid \mathbf{x}) = 9\log\theta + 4\log(1-\theta) + \text{const}$ $\frac{d}{d\theta} = \frac{9}{\theta} - \frac{4}{1-\theta} = 0$ $9(1-\theta) = 4\theta \implies 9 = 13\theta \implies \theta = \frac{9}{13} \approx 0.692$

Why set derivative to zero? At the maximum, the slope is zero. This is standard calculus optimization.

$\boxed{\hat{\theta}_{\text{MAP}} = \frac{9}{13} = 0.692}$

Compare to MLE: $\hat{\theta}_{\text{MLE}} = \frac{7}{10} = 0.7$

Interpretation: The prior "pulled" our estimate slightly down from 0.7 toward the prior mean of 0.6. With more data, this effect diminishes.

Worked example Example 2: Gaussian Mean with Gaussian Prior

Setup: We observe $\mathbf{x} = \{x_1, \ldots, x_n\}$ from $N(\mu, \sigma^2)$ with known $\sigma^2=1$ . Use a Gaussian prior $\mu \sim N(\mu_0, \sigma_0^2)$ with $\mu_0=0, \sigma_0^2=4$ .

Why Gaussian prior? We believe $\mu$ is around 0 but with some uncertainty ( $\sigma_0^2=4$ means95% of our prior belief is in [-4, 4]).

Step 1: Likelihood $P(\mathbf{x} \mid \mu) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(x_i-\mu)^2}{2}\right) \propto \exp\left(-\frac{1}{2}\sum_{i=1}^n(x_i-\mu)^2\right)$ Why product? Assuming i.i.d. samples, joint probability factorizes.

Step 2: Prior $P(\mu) = \frac{1}{\sqrt{2\pi \cdot 4}}\exp\left(-\frac{\mu^2}{8}\right) \propto \exp\left(-\frac{\mu^2}{8}\right)$

Step 3: Log-posterior $\log P(\mu \mid \mathbf{x}) = -\frac{1}{2}\sum_{i=1}^n(x_i-\mu)^2 - \frac{\mu^2}{8} + \text{const}$ Step 4: Differentiate and solve $\frac{d}{d\mu} = \sum_{i=1}^n(x_i - \mu) - \frac{\mu}{4} = 0$ $n\bar{x} - n\mu - \frac{\mu}{4} = 0$ $n\bar{x} = \mu\left(n + \frac{1}{4}\right)$ $\mu = \frac{n\bar{x}}{n + 1/4} = \frac{4n\bar{x}}{4n+1}$

Why this form? We're balancing the data mean $\bar{x}$ (weighted by $n$ ) against the prior mean 0 (weighted by $1/\sigma_0^2 = 1/4$ ).

$\boxed{\hat{\mu}_{\text{MAP}} = \frac{4n}{4n+1}\bar{x}}$ Special cases:

$n \to \infty$ : $\hat{\mu}_{\text{MAP}} \to \bar{x}$ (MLE)
$n = 1, \bar{x}=2$ : $\hat{\mu}_{\text{MAP}} = \frac{8}{5} = 1.6$ (shrunk toward0) Interpretation: A weighted average between the data mean and prior mean, with weights determined by their relative precisions.

Common Mistakes

Common mistake Mistake 2: Forgetting the Prior in Regularization

Wrong idea: "Regularization in ML (like L2) is just a trick to prevent overfitting." Why it feels right: We learn regularization as a penalty term added to loss functions, often without Bayesian context.

The fix: L2 regularization IS MAP estimation with a Gaussian prior. The regularization term $\lambda|\theta\|^2$ is exactly $-\log P(\theta)$ for $\theta \sim N(0, 1/(2\lambda))$ . Derivation: $\text{MAP objective: } -\log P(\mathbf{x} \mid \theta) - \log P(\theta)$ $\text{If } P(\theta) = N(0, \sigma^2): \quad -\log P(\theta) = \frac{\|\theta\|^2}{2\sigma^2} + \text{const}$ $\text{Set } \lambda = \frac{1}{2\sigma^2}: \quad \text{MAP} = \text{MLE} + \lambda\|\theta\|^2$

Similarly: L1 regularization = MAP with Laplace prior.

Why this matters: Understanding this connection explains why regularization works—it's encoding prior belief that parameters should be small.

Common mistake Mistake 3: Using MAP with Uninformative Priors on Transformed Spaces

Wrong idea: "A uniform prior is always non-informative." Why it feels right: Uniform = "all values equally likely" sounds like maximum ignorance.

The fix: Priors transform under reparameterization. A uniform prior on $\theta$ is NOT uniform on $\log\theta$ or $\theta^2$ .

Example: If $\theta \sim \text{Uniform}(0,1)$ and $\phi = \theta^2$ , then $P(\phi) = \frac{1}{2\sqrt{\phi}}$ (not uniform).

The principle: Use Jeffreys prior $P(\theta) \propto \sqrt{|I(\theta)|}$ (where $I$ is Fisher information) for true reparameterization-invariant priors. When it matters: Scale parameters, variance estimation, any problem where the natural parameterization isn't obvious.

MAP vs MLE vs Full Bayes

Method	What it gives	Formula Use when
MLE	Single point estimate	$\arg\max P(\mathbf{x} \mid \theta)$
MAP	Single point estimate	$\arg\max P(\theta \mid \mathbf{x})$
Full Bayes	Entire distribution	$P(\theta \mid \mathbf{x})$

Why MAP?

More robust than MLE with small data (prior regularizes)
Computationally simpler than full Bayes (no integration)
Natural connection to regularization in optimization

Connection to Regularization (Deep Dive)

The negative log-posterior is a regularized loss:

$-\log P(\theta \mid \mathbf{x}) = -\log P(\mathbf{x} \mid \theta) - \log P(\theta) + \text{const}$

Minimizing this (equivalent to maximizing posterior) gives:

$\hat{\theta}_{\text{MAP}} = \arg\min_{\theta} \left[\underbrace{-\log P(\mathbf{x} \mid \theta)}_{\text{Data loss}} + \underbrace{(-\log P(\theta))}_{\text{Regularizer}}\right]$

Common correspondences:

Gaussian prior $N(0, \sigma^2)$ ⟷ L2 regularization ( $\lambda = 1/(2\sigma^2)$ )
Laplace prior $\text{Laplace}(0, b)$ ⟷ L1 regularization ( $\lambda = 1/b$ )
Uniform prior ⟷ No regularization (MLE)

Why this is powerful: Every regularization scheme can be interpreted as a prior belief about parameters. Choosing regularization = choosing a prior.

Recall Explain Like I'm 12

Imagine you're trying to guess how many candies are in a jar. Your friend says "I counted 47 candies through the glass" (this is your data). But you also know that the jar usually holds about 50 candies when it's full (this is your prior knowledge).

MLE would say: "Trust the count completely—guess 47."

MAP would say: "The count is probably close, but the jar is usually around 50, so maybe the count was off by a bit. I'll guess 48 or 49—something between what I saw and what I expected."

MAP combines what you see (the data) with what you already know (the prior). It's like asking two experts and finding a smart compromise between them. The more confident each expert is, the more you listen to them. If you've seen 1000 candies clearly, you trust the data more. If you only glimpsed a few, you trust your prior knowledge more.

Active Recall Challenges

Answers (try first!):

(See derivation section above)
Uniform prior; because $\log P(\theta) = \text{const}$ contributes nothing to argmax
Posterior is Beta(2+2, 2+1) = Beta(4,3), mode = (4-1)/(4+3-2) = 3/5 = 0.6
L2 regularization $\lambda\|\theta\|^2$ = MAP with Gaussian prior $N(0, 1/(2\lambda))$
Data term $n\to\infty$ dominates prior term → MAP → MLE (prior washes out)

Connections

Maximum Likelihood Estimation (MLE) — MAP generalizes this by adding priors
Bayes Theorem — The foundation for deriving MAP
Conjugate Priors — Make MAP computationally tractable
Regularization in ML — Equivalent to MAP with specific priors
Beta Distribution — Common prior for probabilities
Posterior Distribution — MAP finds its mode
Overfitting — MAP prevents this better than MLE with small data
Fisher Information — Used in Jeffreys prior for reparameterization invariance
Ridge Regression — MAP with Gaussian prior on coefficients
Lasso Regression — MAP with Laplace prior on coefficients

#flashcards/ai-ml

What is the MAP estimation formula in terms of log-posterior? :: $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \left[ \log P(\mathbf{x} \mid \theta) + \log P(\theta) \right]$ — sum of log-likelihood and log-prior

How does MAP relate to MLE? :: MLE is MAP with a uniform prior: when $P(\theta) = \text{const}$ , the prior term vanishes and MAP reduces to MLE

In MAP estimation, why can we drop

P(\mathbf{x})

from Bayes' theorem?

Because

P(\mathbf{x})

doesn't depend on

\theta

—it's a constant w.r.t. the optimization variable, so it doesn't affect the argmax

For coin flips with

k

heads in

n

flips and Beta(

\alpha

\beta

) prior, what is the posterior?

Beta(

\alpha + k

\beta + n - k

) — add observed heads to

\alpha

, observed tails to

\beta

What is the mode of a Beta(

\alpha

\beta

) distribution?

\frac{\alpha - 1}{\alpha + \beta - 2}

for

\alpha, \beta > 1

— found by setting derivative of log-PDF to zero

What type of regularization corresponds to a Gaussian prior in MAP?

L2 regularization — Gaussian prior

N(0, \sigma^2)

gives regularization term

\frac{\|\theta\|^2}{2\sigma^2}

What type of regularization corresponds to a Laplace prior in MAP?

L1 regularization — Laplace prior gives regularization term

\lambda\|\theta\|_1

Does MAP give the mean or mode of the posterior?

Mode — MAP finds the parameter value with highest posterior probability, not the average

What happens to MAP estimate as sample size approaches infinity?

MAP converges to MLE — the data term grows with

n

while prior term stays constant, so prior influence vanishes

For Gaussian data with Gaussian prior, is MAP estimate closer to sample mean or prior mean?

A weighted average — specifically

\hat{\mu}_{\text{MAP}} = \frac{n\sigma_0^2}{n\sigma_0^2 + \sigma^2}\bar{x} + \frac{\sigma^2}{n\sigma_0^2 + \sigma^2}\mu_0

Why is Beta distribution a conjugate prior for binomial likelihood?

Because Beta prior × Binomial likelihood = Beta posterior — the posterior stays in the same family, making computation analytical

What is the relationship between negative log-posterior and regularized loss?

They are equivalent:

-\log P(\theta \mid \mathbf{x}) = \text{data loss} + \text{regularization term}

If you use MAP with Beta(1,1) prior for coin bias, what estimate do you get?

Exactly the MLE — Beta(1,1) is uniform on [0,1], equivalent to no prior information

What is Jeffreys prior and why use it?

P(\theta) \propto \sqrt{|I(\theta)|}

where

I

is Fisher information — it's invariant under reparameterization, making it a truly non-informative prior

For Gaussian mean estimation, how does prior variance $\sigma_0^2$ affect MAP? :: Larger $\sigma_0^2$ means less confident prior, so MAP relies more on data and approaches MLE; smaller $\sigma_0^2$ pulls estimate toward prior mean

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

MAP Estimation ka Intuition

Dekho, jab bhi hum kisi parameter ka estimate karna chahte hain (jaise coin ke bias ya kisi distribution ka mean), toh humein do chezein consider karni padti hain: pehli, jo data humein dikha raha hai (likelihood), aur dosri, jo hum pehle se jante the (prior belief). Maximum Likelihood (MLE) sirf data ko dekhta hai, lekin MAP dono ko balance karta hai — ye Bayesian approach hai.

Samjho ek example se: tumhare pas ek coin hai aur tum 10 baar flip karte ho, 7 baar heads ata hai. MLE kehta hai bias = 7/10 = 0.7. Lekin agar tumhe pehle se pata tha ki ye coin thoda biased hai (prior knowledge), toh MAP us information ko bhi use karega. Agar prior kehta hai "I think bias 0.6 ke as-paas hoga", toh MAP final estimate ko thoda 0.6 ki taraf pull karega — result ayega around 0.69. Jaise jaise zyada data milta hai (100 flips

Definition & Mathematical Foundation

Derivation from Bayes' Theorem

Relationship to MLE

Worked Examples

Common Mistakes

MAP vs MLE vs Full Bayes

Connection to Regularization (Deep Dive)

Active Recall Challenges

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections