Exercises — Maximum a posteriori estimation (MAP)

• MLE asks: "which $\theta$ makes the observed data most probable?" → maximises $P(\mathbf{x}\mid \theta )$.
• MAP asks: "which $\theta$ is most probable given the data AND my prior belief?" → maximises the posterior $P(\theta \mid \mathbf{x})$. MAP adds the belief term $\log P(\theta )$ on top of the MLE data term.

When the prior is uniform (flat), $P(\theta )=\text{const}$. Then $\log P(\theta )$ is a constant that does not depend on $\theta$, so it drops out of the $\arg \max$ and only the data term remains. MLE is thus MAP with "no opinion before the data."

MAP returns the mode — the peak (highest point) of the posterior. For a symmetric posterior (e.g. Gaussian) the peak sits at the centre, so mode = mean and it makes no difference. For skewed posteriors (Beta, Gamma) mode $\ne$ mean, so it matters.

The posterior of a Binomial likelihood with a Beta$(\alpha ,\beta )$ prior is Beta$(\alpha +k,\beta +n-k)$ (this is why Beta is the conjugate prior). $\alpha +k=2+9=11,\beta +n-k=2+3=5\Rightarrow \text{Beta}(11,5).$ The mode of Beta$(a,b)$ with $a,b>1$ is $\frac{a-1}{a+b-2}$: ${\hat{\theta }}_{\text{MAP}}=\frac{11-1}{11+5-2}=\frac{10}{14}=\frac{5}{7}\approx \mathrm{0.714.}$ Compare ${\hat{\theta }}_{\text{MLE}}=9/12=0.75$: the prior mean $0.5$ pulled the estimate down slightly.

Take the log (products → sums, and log is monotonic so the peak location is unchanged): $\log P=(a-1)\log \theta +(b-1)\log (1-\theta )+\text{const}.$ Differentiate w.r.t. $\theta$ and set to zero (flat slope at the top): $\frac{a-1}{\theta }-\frac{b-1}{1-\theta }=0\Rightarrow (a-1)(1-\theta )=(b-1)\theta .$ Expand: $(a-1)=\theta (a-1+b-1)=\theta (a+b-2)$, so $\theta =\frac{a-1}{a+b-2}.■$ WHY this is a maximum, not a min or a boundary point (for $a,b>1$):

1. Interior second-derivative check. $\frac{d^2}{d{\theta }^2}\log P=-\frac{a-1}{{\theta }^2}-\frac{b-1}{(1-\theta {)}^2}$. With $a>1,b>1$ both terms are negative, so the log-posterior is concave everywhere on $(0,1)$ — the single critical point is a genuine maximum.
2. Boundary check. As $\theta \to 0^{+}$, $(a-1)\log \theta \to -\infty$; as $\theta \to 1^{-}$, $(b-1)\log (1-\theta )\to -\infty$. So the density vanishes at both ends and no larger value hides at a boundary. The interior peak wins.

From the parent note, ${\hat{\mu }}_{\text{MAP}}=\frac{4n}{4n+1}\bar{x}$. With $n=1,\bar{x}=2$: ${\hat{\mu }}_{\text{MAP}}=\frac{4}{5}\cdot 2=\frac{8}{5}=\mathrm{1.6.}$ The prior mean $0$ shrank the raw data mean $2$ down to $1.6$.

• $n\to 0$: ${\hat{\mu }}_{\text{MAP}}\to 0$. With no data the estimate collapses to the prior mean — belief is all we have. This is the left end of the red curve, sitting on the orange dashed line.
• $n\to \infty$: $\frac{4n}{4n+1}\to 1$, so ${\hat{\mu }}_{\text{MAP}}\to \bar{x}$, the MLE. Data drowns out the prior. This is the red curve approaching the blue dashed line on the right. The smooth climb of the red curve from $0$ up to $1$ is the shrinkage story: evidence gradually overtakes belief.

Log-posterior (drop constants): $\log P(\mu \mid \mathbf{x})=-\frac{1}{2{\sigma }^2}{\sum }_i(x_i-\mu {)}^2-\frac{(\mu -{\mu }_0{)}^2}{2{\sigma }_0^2}.$ Differentiate and set to zero: $\frac{1}{{\sigma }^2}{\sum }_i(x_i-\mu )-\frac{\mu -{\mu }_0}{{\sigma }_0^2}=0.$ WHY we now group the terms this way: we want a single equation of the form "(coefficient)$\cdot \mu$ = (known stuff)", so we sweep every piece containing $\mu$ to one side and every constant to the other. Expand the sum using ${\sum }_i{x}_i=n\bar{x}$ (there are $n$ identical $-\mu$ terms, giving $-n\mu$): $\frac{n\bar{x}}{{\sigma }^2}-\frac{n\mu }{{\sigma }^2}-\frac{\mu }{{\sigma }_0^2}+\frac{{\mu }_0}{{\sigma }_0^2}=0.$ Now collect the two $\mu$-terms (they share the unknown $\mu$) on the left and push the two known means to the right: $\mu \left(\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}\right)=\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0.$ Divide by the bracket to isolate $\mu$: ${\hat{\mu }}_{\text{MAP}}=\frac{\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0}{\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}}.$ This is a precision-weighted average: each mean is weighted by its precision (1/variance) — how confident we are in it. More data ($n\uparrow$) or a sharper prior (${\sigma }_0^2\downarrow$) tips the balance accordingly. Plugging ${\sigma }^2=1,{\sigma }_0^2=4,{\mu }_0=0$ recovers $\frac{4n}{4n+1}\bar{x}$.

Posterior is Beta$(1+k,1+n-k)$. Mode $=\frac{(1+k)-1}{(1+k)+(1+n-k)-2}=\frac{k}{n}$, exactly the MLE. Why: Beta$(1,1)$ is the flat, uniform density on $[0,1]$ — a totally opinion-free prior. A flat prior adds a constant to the log-posterior, leaving the likelihood peak untouched. This is the L1 principle in action.

MAP minimises the negative log-posterior: $-\log P(\mathbf{x}\mid \theta )-\log P(\theta ).$ For a Gaussian prior $\theta \sim N(0,{\sigma }^{2}I)$ (here ${\sigma }^2$ is the prior variance on the weights), $-\log P(\theta )=\frac{\Vert \theta {\Vert }^2}{2{\sigma }^2}+\text{const}$. So the objective becomes $\underset{\text{data loss}}{\underbrace{-\log P(\mathbf{x}\mid \theta )}}+\underset{\text{L2 penalty}}{\underbrace{\frac{1}{2{\sigma }^2}\Vert \theta {\Vert }^2}}.$ Matching this to the standard ridge form $\text{loss}+\lambda \Vert \theta {\Vert }^2$ gives $\boxed{\lambda =\frac{1}{2{\sigma }^2}}.$ A tighter prior (small ${\sigma }^2$) → larger $\lambda$ → stronger shrinkage toward $0$. This is why Ridge Regression and L2 regularization are Bayesian at heart. Convention caveat: the exact constant depends on how you write the penalty. If you use $\lambda \Vert \theta {\Vert }^2$ you get $\lambda =\frac{1}{2{\sigma }^2}$; if instead you write the penalty as $\frac{\lambda }{2}\Vert \theta {\Vert }^2$ (many textbooks absorb the $\frac{1}{2}$ into $\lambda$) you get $\lambda =\frac{1}{{\sigma }^2}$. The proportionality $\lambda \propto 1/{\sigma }^2$ — tighter prior means stronger penalty — is the invariant message; the leading factor is just bookkeeping.

A Laplace (double-exponential) prior $P(\theta )\propto \exp (-|\theta |/b)$ gives $-\log P(\theta )=\frac{|\theta |}{b}+\text{const}$, i.e. the L1 penalty $\lambda \Vert \theta {\Vert }_1$ with $\lambda =1/b$. Why sparsity: The Laplace prior has a sharp spike at $0$ (its density is non-differentiable there). That kink — the green V in the figure above — means the MAP objective can have its optimum sit exactly at ${\theta }_j=0$, driving many coefficients to precisely zero.

From $\lambda =\frac{1}{2{\sigma }^2}$: ${\sigma }^2=\frac{1}{2\lambda }=\frac{1}{2\cdot 0.5}=1.$ A unit-variance Gaussian prior on the weights.

The interior-mode formula $\frac{a-1}{a+b-2}$ is valid only when $a>1$ and $b>1$; otherwise the log-posterior is no longer concave and we must inspect the endpoints directly. Recall the log-density $\log P=(a-1)\log \theta +(b-1)\log (1-\theta )$.

(a) Symmetric $a=b=0.5$, no data (posterior = prior). The density ${\theta }^{-0.5}(1-\theta {)}^{-0.5}$ blows up to $+\infty$ at both endpoints. The formula gives $\frac{-0.5}{-1}=0.5$, but that critical point is a minimum, not a maximum. Check the curvature: $\frac{d^2}{d{\theta }^2}\log P=-\frac{a-1}{{\theta }^2}-\frac{b-1}{(1-\theta {)}^2}$; with $a-1=b-1=-0.5$ both terms are positive, so the log-posterior is convex — the interior point $0.5$ is a valley bottom. As $\theta \to 0^{+}$ and $\theta \to 1^{-}$ the density $\to +\infty$, so the maxima sit at the boundaries. Honest MAP: non-unique — maximizers at both $\theta =0$ and $\theta =1$; the posterior mode is not a single point.

(b) One-sided $a\le 1<b$. The term $(a-1)\le 0$ makes the density non-decreasing pressure toward $\theta =0$ (as $\theta \to 0^{+}$, $(a-1)\log \theta \to +\infty$ if $a<1$, or stays finite if $a=1$), while $(b-1)>0$ pushes the density to $0$ at $\theta =1$. The single maximum sits at the left boundary. Honest MAP: ${\hat{\theta }}_{\text{MAP}}=0$.

(c) One-sided $b\le 1<a$. By the mirror-image argument, $(b-1)\le 0$ drives the density up toward $\theta =1$ while $(a-1)>0$ kills it at $\theta =0$. The single maximum sits at the right boundary. Honest MAP: ${\hat{\theta }}_{\text{MAP}}=1$.

Take-away: always confirm $a>1$ and $b>1$ before trusting the interior formula; otherwise locate the maximum by checking the boundaries, and be ready to report a non-unique mode.

Posterior $=$ Beta$(2+4,10+0)=$ Beta$(6,10)$. Both shapes $>1$, so mode is valid: ${\hat{\theta }}_{\text{MAP}}=\frac{6-1}{6+10-2}=\frac{5}{14}\approx \mathrm{0.357.}$ The MLE is $4/4=1.0$ (would insist the coin never lands tails — a classic small-sample overfit). The strong prior drags the estimate all the way to $0.357$, refusing the extreme claim. This is the safety-rail value of MAP with scarce data.

From Exercise 3.2, prior weight is the prior precision $1/{\sigma }_0^2$. As ${\sigma }_0^2\to \infty$ this weight $\to 0$: ${\hat{\mu }}_{\text{MAP}}=\frac{\frac{n}{{\sigma }^2}\bar{x}+\frac{1}{{\sigma }_0^2}{\mu }_0}{\frac{n}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}}⟶\frac{\frac{n}{{\sigma }^2}\bar{x}}{\frac{n}{{\sigma }^2}}=\bar{x}.$ An infinitely vague prior carries no information, so MAP → MLE. The data's weight $n/{\sigma }^2$ is exactly the Fisher information of $n$ Gaussian samples about $\mu$ — the more information the data carries, the more it dominates a fixed prior.

With no data the data term vanishes; the log-posterior is the log-prior, whose peak is the prior mean: ${\hat{\mu }}_{\text{MAP}}={\mu }_0=3.$ The prior variance $0.25$ affects our confidence, but with zero observations the peak location is simply the prior mean.

The mode (peak) of the posterior $P(\theta \mid \mathbf{x})$, not its mean.

A uniform (flat) prior — the belief term becomes a constant and drops out of the $\arg \max$.

$(a,b)$ when $a,b>1$? $\frac{a-1}{a+b-2}$ (valid only when the log-posterior is concave, i.e. $a>1$ and $b>1$; otherwise the maximum sits on a boundary).

$\lambda$ related to it? A zero-mean Gaussian prior; with the $\lambda \Vert \theta {\Vert }^2$ convention, $\lambda =\frac{1}{2{\sigma }^2}$ (so $\lambda \propto 1/{\sigma }^2$).

A Laplace prior; its sharp spike (kink) at $0$ lets the optimum sit exactly at $\theta =0$.

A precision-weighted average of the prior mean and the data mean, each weighted by its precision (1/variance).

precision? Precision $=1/\text{variance}$ — high when you are certain (narrow bell), low when vague (wide bell).