Foundations — Maximum a posteriori estimation (MAP)

Before you can read the parent note Maximum a posteriori estimation (MAP), you must own every symbol it uses without flinching. This page builds each one from nothing — plain words, then a picture, then why the topic needs it. Read top to bottom; each block leans on the one above.

1. What is a "parameter" $\theta$?

Picture it. Imagine a coin whose chance of landing heads is not fixed at $0.5$. That hidden chance is $\theta$. If $\theta =0.7$, the coin is heads-biased; if $\theta =0.2$, it is tails-biased. $\theta$ lives on a number line from $0$ to $1$ here.

Why the topic needs it. MAP's (Maximum a Posteriori's) whole job is to estimate $\theta$ — to point at one spot on that line and say "this is my best guess." Everything else is machinery for choosing that spot.

2. The "hat" $\hat{\theta }$ — an estimate of $\theta$

Picture it. The true $\theta$ is a fixed but invisible dot on the number line. Our method plants a flag $\hat{\theta }$ where it thinks the dot is. They may not coincide — the hat is a guess.

Why the topic needs it. MAP produces one particular guess, written ${\hat{\theta }}_{\text{MAP}}$ (read "theta-hat MAP"): the estimate of $\theta$ chosen by the Maximum a Posteriori rule. The subscript just labels which estimation method made the guess — you will also meet ${\hat{\theta }}_{\text{MLE}}$ from a different method.

3. What is "data" $\mathbf{x}$?

Picture it. You flip the coin 10 times and write down the results: $\mathbf{x}=\{H,H,T,H,H,H,T,T,H,H\}$ That list of 10 outcomes is $\mathbf{x}$, and here $n=10$.

Why the topic needs it. $\theta$ is invisible; $\mathbf{x}$ is what we can see. MAP (Maximum a Posteriori) is the bridge that walks from the visible data to the hidden knob.

4. Probability $P(\cdot )$ and the pipe $\mid$

Picture it. $P(\text{rain})$ might be $0.3$. But $P(\text{rain}\mid \text{dark clouds})$ jumps to $0.8$ — knowing about the clouds changed the probability. The pipe is a filter: it throws away every world where $B$ is false, then measures $A$ inside what remains.

Why the topic needs it. MAP is built on two conditional probabilities pointing in opposite directions:

• $P(\mathbf{x}\mid \theta )$ — "if the knob were $\theta$, how likely is this data?"
• $P(\theta \mid \mathbf{x})$ — "given this data, how likely is the knob to be $\theta$?"

The entire subject is about flipping the first into the second.

5. Independence and i.i.d. data

Before we can multiply probabilities together, we must know when that is legal.

Picture it. Ten separate coin flips: the 3rd flip does not care what the 2nd did, and every flip uses the same biased coin. That is exactly i.i.d.

Why the topic needs it. Independence is the permission slip that lets a joint probability factor into a product: $P(x_1\text{ and }{x}_2)=P(x_1)P(x_2)$. Without it, the neat $\prod$ notation in Section 11 would be wrong. Every worked example in the parent silently assumes i.i.d. data.

6. The three named probabilities

Each conditional above has a name. Learn them like faces.

Picture it. Start with a wide, vague "prior" hump over $\theta$. The data reshapes it into a sharper "posterior" hump. The peak of that final hump is the MAP estimate ${\hat{\theta }}_{\text{MAP}}$.

Why "posterior" is the target. It is the only one of the three that answers the actual question: given everything I now know, what is $\theta$? Likelihood ignores prior knowledge; prior ignores data; the posterior fuses both.

7. Bayes' theorem — the flipping machine

Plain words. Posterior equals likelihood times prior, divided by the evidence $P(\mathbf{x})$.

Why we need it. We can compute $P(\mathbf{x}\mid \theta )$ (models tell us how likely data is for a given knob). We want $P(\theta \mid \mathbf{x})$. Bayes' theorem is the exact machinery that turns the one we can compute into the one we want. See Bayes Theorem.

8. The $\arg \max$ operator

Picture it. On a curve of posterior-vs-$\theta$, $\max$ is the height of the summit; $\arg \max$ is the $\theta$-coordinate directly under the summit. MAP wants the location, so it uses $\arg \max$.

Why the topic needs it. "Most probable parameter" literally means "the $\theta$ where the posterior is tallest" — that is exactly ${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }P(\theta \mid \mathbf{x})$.

9. The logarithm $\log$ — why it appears

Why the topic needs it. Likelihoods are products of many tiny probabilities (one per data point). Multiplying hundreds of numbers near zero underflows a computer to $0$. Taking $\log$ converts the product into a friendly sum, and because $\log$ is monotonic, the location of the peak does not move. So: ${\hat{\theta }}_{\text{MAP}}=\arg {\max }_{\theta }[P(\mathbf{x}\mid \theta )P(\theta )]=\arg {\max }_{\theta }[\log P(\mathbf{x}\mid \theta )+\log P(\theta )]$ Same answer, numerically safe, easier calculus.

10. The derivative $\frac{d}{d\theta }$ — finding the peak

Why the topic needs it. At the very top of a smooth hill, the ground is momentarily flat — slope $=0$. So to find the peak of the log-posterior we set its derivative to zero and solve. This is why every worked example ends with "$\frac{d}{d\theta }=0$." The flat-top rule is how we locate $\arg \max$ algebraically.

11. Product $\prod$ and sum $\sum$ notation

Why the topic needs it. Because the data is i.i.d. (Section 5), the joint likelihood factors into the product of each point's likelihood — hence $\prod$. After taking $\log$ (Section 9), that product becomes a $\sum$. This is the exact move in the parent's Gaussian example.

Prerequisite map

Each arrow means "you need the tail before the head makes sense." Notice that the likelihood and prior both feed Bayes' theorem, which produces the posterior, which $\arg \max$ + log + derivative finally squeeze into the MAP estimate ${\hat{\theta }}_{\text{MAP}}$.

Where these lead next

Once these symbols are solid, you can read how MAP (Maximum a Posteriori) connects outward:

• Maximum Likelihood Estimation (MLE) — MAP with a flat prior. A flat (or uniform) prior is one that assigns the same $P(\theta )$ to every allowed $\theta$; being constant, it adds nothing to the $\arg \max$, so MAP collapses back into pure likelihood maximization.
• Conjugate Priors and Beta Distribution — priors that keep the algebra clean.
• Regularization in ML, Ridge Regression, Lasso Regression — MAP wearing a disguise.
• Overfitting — the disease the prior cures.

Equipment checklist

Cover the right side and answer each. If you stall, reread that section.