Foundations — Bayesian statistics — prior, likelihood, posterior (intro)

Before you can update beliefs, you must be able to read the sentence that does the updating. This page builds every piece of notation the parent note silently assumes, in an order where each symbol is earned by the one before it. Never move past a symbol you cannot picture.

1. What is a probability? The symbol $P(\cdot )$

The picture. Draw a big rectangle and call its whole area $1$ — this rectangle is everything that could happen (the sample space). An event $A$ is a blob inside it. $P(A)$ is simply the fraction of the rectangle that the blob covers.

Why the topic needs it. Bayesian statistics is a machine that takes probabilities in and gives a probability out. If "a number that measures how much area a blob covers" is not crystal clear, none of the later symbols mean anything.

2. Two events at once: $\cap$ and the joint region

The picture. Draw two overlapping blobs, $A$ and $B$, inside the rectangle. The lens-shaped overlap is $A\cap B$. Its area is $P(A\cap B)$ — the chance you land in both blobs.

Why the topic needs it. The parent's Step 1 writes "the joint two ways": $P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A).$ That whole derivation is a statement about this one overlapping lens measured from two directions. Without a picture of the overlap, that equation is just symbols.

3. Conditioning: the bar $\mid$ and $P(A\mid B)$

Why this formula and not another? Once we know $B$ happened, the world shrank: only the $B$-blob is still possible. So we throw away the rest of the rectangle and ask what fraction of the new smaller world (area $P(B)$) is also in $A$ (area $P(A\cap B)$)? Dividing rescales the overlap so that the $B$-blob now counts as the full "certain" amount $1$.

Look at the figure. On the left, $A\cap B$ is a small slice of the whole rectangle. On the right we have zoomed so that $B$ fills the frame — the same slice now looks much bigger, because we divided by the shrunken world $P(B)$.

4. Naming the unknowns: $\theta$ and $D$

The picture. Think of $\theta$ as a dial hidden inside a locked box, and $D$ as the readings that leak out through a window. We never touch the dial; we only see readings and reason backwards to the dial's position.

Why the topic needs it. The whole subject is the arrow $D\Rightarrow \theta$: from visible readings back to the hidden dial. Bayes' theorem is literally the machine that reverses the arrow, because nature runs it forwards ($\theta$ causes $D$) and we want it backwards.

5. The four characters as symbols

Now every symbol in the parent's headline formula is a phrase you can read aloud.

Why $P(D)$ is "just a normalizer". It has no $\theta$ in it, so as $\theta$ varies it is a fixed constant. Its only job is to make the posterior's total area equal $1$ (a real probability). That is why the parent's proportional form drops it: $P(\theta \mid D)\propto P(D\mid \theta )P(\theta )$.

6. Summation, integration, and $\propto$

Two symbols appear when we build the evidence $P(D)$.

The picture. Discrete $\theta$: a few bars you add. Continuous $\theta$: a smooth curve whose area underneath you accumulate. The integral is "the area under the curve".

Why the topic needs both. The Law of Total Probability is precisely $P(D)={\sum }_{\theta }P(D\mid \theta )P(\theta )$ (or the integral version) — it is how the denominator is assembled. And $\propto$ is the parent's "80/20": compute the un-normalized shape, then fix the constant at the very end.

7. The two data-generators the parent leans on

Why now, not earlier. These are the specific likelihood and prior shapes in Worked Example 2. You needed $\theta$, $\propto$, and "area under a curve" first before "${\theta }^7(1-\theta {)}^3$ is a Beta" could mean anything. The magic word conjugate (Beta prior + Binomial data → Beta posterior) is just: the shapes multiply and stay in the same family.

8. Prerequisite map

Equipment checklist

Cover the answers and test yourself. If any line stumps you, re-read its section above.