Bayesian statistics — prior, likelihood, posterior (intro)

WHY does this exist?

WHAT problem it solves: You have an unknown quantity (a coin's bias, a disease probability, a model parameter $\theta$ ). You can't observe $\theta$ directly — you only see data $D$ . Bayesian statistics answers: given the data I actually saw, what should I now believe about $\theta$ ?

WHY it's powerful: It treats the unknown $\theta$ itself as a random variable with a probability distribution. This lets you express uncertainty about a parameter, not just about data. Frequentists say " $\theta$ is fixed but unknown"; Bayesians say " $\theta$ is fixed but my knowledge of it is a distribution that I refine."

HOW it works — deriving Bayes' theorem from scratch

Step 1 — write the joint two ways. The joint probability $P(A\cap B)$ doesn't care about order, so: $P(A\cap B) = P(A\mid B)\,P(B) = P(B\mid A)\,P(A)$ Why this step? Both expansions describe the same overlap region — one slices by $B$ first, the other by $A$ first.

Step 2 — equate and divide. Set the two right-hand sides equal and divide by $P(B)$ : $\boxed{\;P(A\mid B) = \dfrac{P(B\mid A)\,P(A)}{P(B)}\;}$ Why this step? We isolate the thing we want ( $A$ given $B$ ) in terms of things we can get ( $B$ given $A$ , and the priors).

Step 3 — rename for statistics. Let $A=\theta$ (hypothesis/parameter) and $B=D$ (data):

The four characters

Figure — Bayesian statistics — prior, likelihood, posterior (intro)

Worked Example 1 — Disease testing (discrete)

A disease affects 1% of people. A test is 99% sensitive ( $P(+\mid \text{sick})=0.99$ ) and 95% specific ( $P(-\mid \text{healthy})=0.95$ , so false-positive rate $0.05$ ). You test positive. What's $P(\text{sick}\mid +)$ ?

Setup. $\theta\in\{\text{sick},\text{healthy}\}$ . Prior: $P(\text{sick})=0.01$ .

Step 1 — likelihoods. $P(+\mid\text{sick})=0.99$ , $P(+\mid\text{healthy})=0.05$ . Why? These are how well each hypothesis explains the data "test = +".

Step 2 — evidence. $P(+)=0.99(0.01)+0.05(0.99)=0.0099+0.0495=0.0594$ Why this step? Total probability of testing positive, summed over both hypotheses (sick and healthy).

Step 3 — posterior. $P(\text{sick}\mid +)=\frac{0.99\times 0.01}{0.0594}=\frac{0.0099}{0.0594}\approx 0.1667$

Punchline: Only ~17%! Even with a "99%" test, the tiny prior dominates. Why? There are far more healthy people, so false positives outnumber true positives.

Worked Example 2 — Coin bias (continuous, Beta–Binomial)

Unknown bias $\theta\in[0,1]$ . You flip and get 7 heads in 10 tosses. Start with a flat prior $P(\theta)=1$ (Uniform on $[0,1]$ ).

Step 1 — likelihood (Binomial). $P(D\mid\theta)=\binom{10}{7}\theta^{7}(1-\theta)^{3}$ Why? Each head contributes $\theta$ , each tail $(1-\theta)$ ; $\binom{10}{7}$ counts orderings (constant in $\theta$ ).

Step 2 — proportional posterior. $P(\theta\mid D)\propto \theta^{7}(1-\theta)^{3}\times 1$ Why this step? posterior $\propto$ likelihood $\times$ prior; flat prior $=1$ drops out, and the binomial constant is absorbed.

Step 3 — recognize the shape. $\theta^{a-1}(1-\theta)^{b-1}$ is a Beta distribution with $a=8,\ b=4$ . So $\theta\mid D \sim \text{Beta}(8,4).$ Why? A Uniform is $\text{Beta}(1,1)$ ; adding $7$ heads and $3$ tails gives $\text{Beta}(1+7,\,1+3)$ — the Beta is conjugate to the Binomial (posterior stays in the same family).

Step 4 — point estimate. Posterior mean $=\dfrac{a}{a+b}=\dfrac{8}{12}=0.667$ . Why this is nice: it sits between the prior mean $0.5$ and the data fraction $0.7$ — the prior shrinks the estimate slightly toward $0.5$ .

Forecast-then-Verify

Common mistakes (Steel-man + fix)

Recall Feynman: explain to a 12-year-old (hidden — try first!)

Imagine you think a cookie jar is probably full (that's your prior guess). Then you shake it and hear only a tiny rattle. A full jar would make a LOUD rattle, an almost-empty jar a tiny one — that "how likely is this sound" is the likelihood. Combining your first guess with what you heard, you now believe the jar is almost empty: that new belief is the posterior. Bayes' rule is just the honest way to mix "what I thought" with "what I saw." And the surprise: if you were super sure the jar was full (strong prior), one tiny rattle won't change your mind much — strong priors are stubborn.

Active-recall flashcards

What does the prior

P(\theta)

represent?

Your belief about the parameter

\theta

before seeing any data.

What is the likelihood

P(D\mid\theta)

, and is it a distribution over

\theta

How probable the observed data is for each value of

\theta

; it is a function of

\theta

but NOT a probability distribution over

\theta

(need not integrate to 1).

State Bayes' theorem for a parameter and data.

P(\theta\mid D)=\dfrac{P(D\mid\theta)P(\theta)}{P(D)}

Write the proportional form of the posterior.

P(\theta\mid D)\propto P(D\mid\theta)\,P(\theta)

— posterior ∝ likelihood × prior.

What is the role of the evidence

P(D)

A normalizing constant (independent of

\theta

) that makes the posterior sum/integrate to 1;

P(D)=\int P(D\mid\theta)P(\theta)d\theta

In disease testing with 1% prevalence and a 99%/95% test, why is

P(\text{sick}\mid+)

only ~17%?

The low prior means healthy people vastly outnumber sick ones, so false positives outnumber true positives.

Why is Beta conjugate to the Binomial?

A Beta prior times a Binomial likelihood gives a Beta posterior; updating

\text{Beta}(a,b)

with

h

heads and

t

tails yields

\text{Beta}(a+h,b+t)

A Uniform prior on

[0,1]

equals which Beta distribution?

\text{Beta}(1,1)

After 7 heads in 10 tosses with a flat prior, what is the posterior and its mean?

\text{Beta}(8,4)

, mean

=8/12\approx0.667

What fallacy is confusing

P(\theta\mid D)

with

P(D\mid\theta)

The prosecutor's fallacy (confusing the conditioning direction).

Connections

Conditional Probability — the foundation Bayes is derived from.
Law of Total Probability — how the evidence $P(D)$ is computed.
Binomial Distribution — the likelihood in the coin example.
Beta Distribution — conjugate prior/posterior for proportions.
Maximum Likelihood Estimation — what you get if you ignore the prior (or use a flat one and take the mode).
Naive Bayes Classifier — a direct ML application of this rule.
Frequentist vs Bayesian Inference — the philosophical contrast.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Bayesian statistics ka core idea bilkul simple hai: yeh "beliefs ko update karne" ka ek honest formula hai. Pehle aapke paas ek prior hota hai — yaani data dekhne se pehle aap kya maante the (jaise "coin shayad fair hai"). Phir data aata hai, aur aap dekhte ho ki har possible $\theta$ us data ko kitni achhi tarah explain karta hai — yeh hai likelihood $P(D\mid\theta)$ . In dono ko multiply karke, normalize karke, aapko milta hai posterior $P(\theta\mid D)$ — yaani updated belief. Mantra yaad rakho: posterior ∝ likelihood × prior.

Bayes ka theorem nikalna easy hai. Joint probability ko do tareeke se likho: $P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)$ . Bas $P(B)$ se divide kar do, mil gaya $P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}$ . Yahi cheez statistics mein $\theta$ aur $D$ ke saath likh do — done!

Sabse important intuition disease-test wala example hai. Test 99% accurate hai, phir bhi positive aane par aap sirf 17% sick ho — kyun? Kyunki disease rare hai (prior chhota), to healthy logon ke false positives, sick logon ke true positives se zyada ho jaate hain. Yahi prosecutor's fallacy hai: $P(+\mid\text{sick})$ aur $P(\text{sick}\mid +)$ ko mat confuse karo — dono bilkul alag hote hain, aur prior hi flip ko control karta hai.

Kyun matter karta hai? Machine learning (Naive Bayes), medical diagnosis, spam filtering, A/B testing — sab jagah hum naye data ke saath beliefs update karte hain. Bayesian soch aapko sikha deti hai ki uncertainty ko number mein kaise express karein, aur strong prior hone par ek chhota sa data evidence aapki rai zyada nahi badalta — yeh deep aur practical dono hai.

Bayesian statistics — prior, likelihood, posterior (intro)

WHY does this exist?

HOW it works — deriving Bayes' theorem from scratch

The four characters

Worked Example 1 — Disease testing (discrete)

Worked Example 2 — Coin bias (continuous, Beta–Binomial)

Forecast-then-Verify

Common mistakes (Steel-man + fix)

Active-recall flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability Theory & Statistics

Connections