Visual walkthrough — Maximum a posteriori estimation (MAP)

#ai-ml/probability-statistics #bayesian-inference #parameter-estimation

Before any math: a parameter is just an unknown number we want to guess. Call it $\theta$ (the Greek letter "theta"). For a coin, $\theta$ is the chance it lands heads — a number between $0$ and $1$. That is all $\theta$ means. Everything below is about picking the best single value for that unknown number.

---

Step 1 — Draw the thing we are guessing

WHAT. Put $\theta$ on a horizontal line. Every point on that line is one possible guess for the coin's bias: $0$ (never heads), $0.5$ (fair), $1$ (always heads).

WHY. You cannot talk about "the best guess" until you can see all the guesses laid out. A picture of the parameter line is the stage on which the whole story happens.

PICTURE. The amber tick at $0.5$ marks a fair coin. We have not yet decided which value wins — the line is just the menu of options.

---

Step 2 — Height = "how believable". Two curves appear

WHAT. Above the line we draw heights. A tall height over some $\theta$ means "that value is believable"; a short height means "unlikely". Two different curves live here.

• The likelihood $P(\mathbf{x}\mid \theta )$ — read "probability of the data $\mathbf{x}$, if the true value were $\theta$". Here $\mathbf{x}$ is what we actually saw (7 heads in 10 flips). This curve peaks where the data is best explained.
• The prior $P(\theta )$ — read "how believable $\theta$ was before we saw any data". This is your common sense, drawn as a curve.

WHY. MLE (Maximum Likelihood Estimation (MLE)) only ever looks at the likelihood curve. But real belief has two sources — the data and what you knew before. We need both curves on the stage before we can combine them.

PICTURE. Cyan = likelihood (peaks near $0.7$, the data's story). Amber = prior (peaks near $0.6$, our earlier hunch). Notice their peaks are in different places — that tension is the whole point.

Let us write both curves down explicitly, because in Step 6 their exponents appear and you deserve to know where every number came from.

The likelihood for $k=7$ heads in $n=10$ flips is the binomial probability: $P(\mathbf{x}\mid \theta )=\underset{\text{counts the orderings}}{\underbrace{\left(\genfrac{}{}{0ex}{}{10}{7}\right)}}\underset{\text{7 heads}}{\underbrace{{\theta }^7}}\underset{\text{3 tails}}{\underbrace{(1-\theta {)}^3}}$

• $\left(\genfrac{}{}{0ex}{}{10}{7}\right)$ — the number of different ways 7 heads can land among 10 flips. It contains no $\theta$, so it is a fixed constant $C$.
• ${\theta }^7$ — each head happens with probability $\theta$, and there are 7 of them, multiplied together.
• $(1-\theta {)}^3$ — each tail happens with probability $1-\theta$, and there are 3.

The prior we pick is a Beta$(3,2)$ curve (see Beta Distribution). Its functional form is $P(\theta )=\underset{\text{fixed normaliser}}{\underbrace{\frac{1}{B(3,2)}}}\underset{={\theta }^2}{\underbrace{{\theta }^{3-1}}}\underset{=(1-\theta {)}^1}{\underbrace{(1-\theta {)}^{2-1}}}=\frac{1}{B(3,2)}{\theta }^2(1-\theta )$

• The two shape numbers of Beta are $3$ and $2$; the exponents in the formula are always one less, giving ${\theta }^2$ and $(1-\theta {)}^1$. That is where the "2" and the "1" you will see in Step 6 are born.
• $B(3,2)$ — a normalising constant with no $\theta$ in it, so it too is a fixed number we can later drop.

---

Step 3 — Bayes' theorem: flip the arrow

WHAT. We can compute $P(\mathbf{x}\mid \theta )$ (probability of data given a guess), but we actually want $P(\theta \mid \mathbf{x})$ (probability of a guess given the data). Bayes' theorem flips this arrow:

$P(\theta \mid \mathbf{x})=\frac{\stackrel{\text{likelihood}}{\overbrace{P(\mathbf{x}\mid \theta )}}\cdot \stackrel{\text{prior}}{\overbrace{P(\theta )}}}{\underset{\text{evidence: same number for every }\theta }{\underbrace{P(\mathbf{x})}}}$

WHY. The quantity on the left, the posterior $P(\theta \mid \mathbf{x})$, is exactly "how believable is $\theta$ after seeing the data". That is the curve whose peak we want. Bayes is the only lawful way to turn the two ingredients of Step 2 into it.

PICTURE. Think of it as a factory: two input curves go in the top (likelihood $\times$ prior, multiplied height-by-height at each $\theta$), the evidence $P(\mathbf{x})$ divides everything by one fixed number, and the posterior comes out the bottom.

Each symbol, right where it sits:

• $P(\theta \mid \mathbf{x})$ — output, the posterior height at $\theta$.
• $P(\mathbf{x}\mid \theta )\cdot P(\theta )$ — numerator, likelihood height times prior height, multiplied pointwise.
• $P(\mathbf{x})$ — denominator, the total probability of the data added over all $\theta$; it does not contain $\theta$, so it is one constant number.

---

Step 4 — Throw away the denominator (it never moves the peak)

WHAT. We are hunting the location of the tallest point, written $\arg {\max }_{\theta }$ — "the $\theta$ that makes the following as large as possible". Dividing every height by the same constant $P(\mathbf{x})$ shrinks the whole curve uniformly but does not slide its peak sideways.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }\frac{P(\mathbf{x}\mid \theta )P(\theta )}{P(\mathbf{x})}=\arg {\max }_{\theta }P(\mathbf{x}\mid \theta )P(\theta )$

WHY. We want where the peak is, not how tall. Scaling by a positive constant is like zooming a photo — nothing horizontal changes. So the evidence, which cost us nothing to ignore, is dropped. The same reasoning also lets us drop $\left(\genfrac{}{}{0ex}{}{10}{7}\right)$ and $B(3,2)$ from Step 2 — both are $\theta$-free constants. What survives is only the $\theta$-carrying part: $P(\mathbf{x}\mid \theta )P(\theta )\propto {\theta }^7(1-\theta {)}^3\cdot {\theta }^2(1-\theta )={\theta }^{7+2}(1-\theta {)}^{3+1}={\theta }^9(1-\theta {)}^4$ Multiplying powers of the same base adds their exponents: $7+2=9$ heads-worth, $3+1=4$ tails-worth. This combined curve is a Beta$(10,5)$ shape — the posterior.

PICTURE. The tall cyan curve and the short cyan curve are the same shape scaled; the amber dashed line drops from both peaks to the same $\theta$. The peak's address is unchanged.

---

Step 5 — Take logs: multiply becomes add

WHAT. Apply $\log$ (the logarithm) to the thing we maximize. The key property: $\log (a\cdot b)=\log a+\log b$. A product turns into a sum. Applied to our surviving curve ${\theta }^9(1-\theta {)}^4$: $\log [{\theta }^9(1-\theta {)}^4]=9\log \theta +4\log (1-\theta )$ The rule $\log (a^m)=m\log a$ pulls the exponents $9$ and $4$ out to the front — that is where the coefficients in Step 6 come from.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }[\underset{\text{log-likelihood}}{\underbrace{\log P(\mathbf{x}\mid \theta )}}+\underset{\text{log-prior}}{\underbrace{\log P(\theta )}}]$

WHY two reasons.

1. Sums are easier than products — the derivative of a sum is the sum of derivatives, so Step 6's calculus is clean.
2. $\log$ is monotonic (always increasing): if curve $A$ is taller than curve $B$ at some point, then $\log A$ is still taller than $\log B$ there. So the peak's location never moves when we take logs — exactly what we needed in Step 4's spirit.

PICTURE. Left panel: the original product-shaped curve. Right panel: its log — same peak position (amber line lines up), but the shape is now a sum of two gentle bowls, one from the data, one from the prior.

---

Step 6 — Find the peak: slope zero

WHAT. A smooth curve's highest point is where its slope is zero — flat on top. The slope is the derivative $\frac{d}{d\theta }$. Set it to zero and solve.

From Step 5 the log-posterior is $9\log \theta +4\log (1-\theta )$ (the Beta$(10,5)$ shape we assembled). Differentiate, using $\frac{d}{d\theta }\log \theta =\frac{1}{\theta }$ and $\frac{d}{d\theta }\log (1-\theta )=\frac{-1}{1-\theta }$:

$\frac{d}{d\theta }[9\log \theta +4\log (1-\theta )]=\underset{\text{pull up from heads}}{\underbrace{\frac{9}{\theta }}}-\underset{\text{pull down from tails}}{\underbrace{\frac{4}{1-\theta }}}=0$

WHY. At a hilltop you are momentarily walking on level ground — the slope is $0$. Solving $\text{slope}=0$ is the standard way calculus locates a maximum. The two fractions are literally two forces: one pushes $\theta$ up (evidence for heads), one pulls it down (evidence for tails). Balance = the estimate.

Solve it: $\frac{9}{\theta }=\frac{4}{1-\theta }\Rightarrow 9(1-\theta )=4\theta \Rightarrow 9=13\theta \Rightarrow \boxed{{\hat{\theta }}_{\text{MAP}}=\frac{9}{13}\approx 0.692}$

PICTURE. The posterior curve with a flat amber tangent line resting on its top, dropping to $\theta =9/13$. Compare the cyan MLE peak at $0.7$: the prior nudged the answer leftward, toward its own hunch of $0.6$.

The prior mode of Beta$(3,2)$ is at $\frac{3-1}{3+2-2}=\frac{2}{3}\approx 0.667$; the data alone says $0.7$; MAP lands between them at $0.692$ — closer to the data because there is more of it. (See Conjugate Priors, Posterior Distribution.)

---

Step 7 — Edge case A: a flat prior gives back MLE

WHAT. Suppose the prior is flat — every $\theta$ equally believable, $P(\theta )=\text{const}$. Then $\log P(\theta )$ is a constant, and a constant has slope zero everywhere.

${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }[\log P(\mathbf{x}\mid \theta )+\underset{\text{no tilt}}{\underbrace{\log (\text{const})}}]=\arg {\max }_{\theta }\log P(\mathbf{x}\mid \theta )={\hat{\theta }}_{\text{MLE}}$

WHY. A flat prior means "I believe nothing in advance." Adding a level slab to the log-posterior lifts it uniformly but never tilts it, so the peak stays exactly where the likelihood put it. This is why MLE is MAP with an empty mind.

PICTURE. The amber flat prior is a straight horizontal slab; the posterior peak sits directly under the cyan likelihood peak — MAP and MLE coincide.

---

Step 8 — Edge case B: as data floods in, the prior fades

WHAT. We now switch to estimating a Gaussian mean $\mu$, and derive its MAP formula rather than quoting it, so no symbol appears unearned. We observe $n$ numbers $x_1,\dots ,x_n$ drawn from a bell curve $N(\mu ,1)$ (variance known and equal to 1), and our prior is a bell curve $N(0,4)$ — belief centred at $0$ with spread $4$.

The likelihood (product over i.i.d. points, constants dropped) has log-form $\log P(\mathbf{x}\mid \mu )=-\frac{1}{2}{\sum }_{i=1}^n(x_i-\mu {)}^2+\text{const}$ Each term $-\frac{1}{2}(x_i-\mu {)}^2$ is the log of the bell $\exp (-\frac{1}{2}(x_i-\mu {)}^2)$; sums replace products because logs turn $\times$ into $+$.

The prior $N(0,4)$ has log-form $\log P(\mu )=-\frac{{\mu }^2}{2\cdot 4}+\text{const}=-\frac{{\mu }^2}{8}+\text{const}$ The "$8$" is $2{\sigma }_0^2=2\times 4$; the $-{\mu }^2$ pulls $\mu$ toward $0$, the prior's centre.

Add them, differentiate, set to zero (using $\frac{d}{d\mu }(x_i-\mu {)}^2=-2(x_i-\mu )$): $\frac{d}{d\mu }[-\frac{1}{2}{\sum }_i(x_i-\mu {)}^2-\frac{{\mu }^2}{8}]={\sum }_{i=1}^n(x_i-\mu )-\frac{\mu }{4}=0$ Write $\bar{x}=\frac{1}{n}{\sum }_i{x}_i$ (the data average), so ${\sum }_i{x}_i=n\bar{x}$: $n\bar{x}-n\mu -\frac{\mu }{4}=0\Rightarrow n\bar{x}=\mu (n+\frac{1}{4})\Rightarrow \boxed{{\hat{\mu }}_{\text{MAP}}=\frac{n\bar{x}}{n+\frac{1}{4}}=\frac{4n}{4n+1}\bar{x}}$

WHY this shape. The prior acts like a fixed dab of pretend evidence worth about a quarter of a data point. When real data is scarce, that dab is loud; when data floods in, it is drowned out:

• $n=1,\bar{x}=2$: $\hat{\mu }=\frac{4}{5}\cdot 2=\frac{8}{5}=1.6$ — heavily shrunk toward the prior mean $0$.
• $n\to \infty$: $\frac{4n}{4n+1}\to 1$, so $\hat{\mu }\to \bar{x}$ — the prior vanishes and MAP becomes MLE.

This shrinking pull is exactly what prevents Overfitting on tiny datasets — the same idea as Regularization in ML, Ridge Regression and Lasso Regression (a Gaussian prior is L2, a Laplace prior is L1). See also Fisher Information for how data "sharpens" the likelihood.

PICTURE. Three posteriors stacked for $n=1,5,50$: as $n$ grows, the peak marches from near $0$ out to $\bar{x}=2$, and the curve narrows — belief tightens onto the data.

---

The one-picture summary

Everything above, in a single frame: two curves (data in cyan, belief in amber) get multiplied, logs turn that into an addition of two scores, and we walk uphill to the flat top. The peak is a tug-of-war winner — pulled toward the data by evidence, toward the prior by belief, settling wherever the two forces balance.

Recall Feynman retelling — say it like a story

I want to guess one unknown number $\theta$. I draw a line of all its possible values. Over that line I sketch two hills: one is how well each value explains my data, the other is how much I believed in each value beforehand. Bayes' theorem tells me to multiply the two hills together (and divide by a constant I can ignore, because it only shrinks, never shifts). Multiplying is annoying, so I take logs and it becomes adding two scores. The best guess is the top of the summed hill, found where the slope goes flat. If I believed nothing (flat prior), the belief-hill is level and I'm back to plain likelihood — that's MLE. If I have loads of data, the data-hill towers over the belief-hill and again I basically get MLE. In between — little data, real belief — MAP quietly pulls my guess toward common sense, which is the whole reason it beats overfitting.

Recall Quick self-test

Why can we drop $P(\mathbf{x})$ when finding the MAP estimate? ::: It does not depend on $\theta$; dividing every height by the same constant scales the curve but never moves its peak. Where do the exponents $9$ and $4$ in the log-posterior come from? ::: From adding the likelihood exponents ${\theta }^7(1-\theta {)}^3$ and the Beta$(3,2)$ prior exponents ${\theta }^2(1-\theta {)}^1$: $7+2=9$ and $3+1=4$. What does MAP reduce to under a uniform prior, and why? ::: MLE — a constant log-prior adds a level slab that lifts the log-posterior without tilting it, so the peak stays at the likelihood's peak. In the coin example, why is ${\hat{\theta }}_{\text{MAP}}=0.692$ between the prior mode $0.667$ and the MLE $0.7$? ::: The posterior multiplies both influences; with 10 flips the data outweighs the prior slightly, so the answer sits closer to $0.7$. As $n\to \infty$ in the Gaussian case, what happens to the prior's influence? ::: It fades to zero ($\frac{4n}{4n+1}\to 1$), so ${\hat{\mu }}_{\text{MAP}}\to \bar{x}$, the plain data mean.