Joint, marginal, and conditional distributions

WHY do we need all three?

WHAT we have: two RVs, say $X$ = "is email spam?" and $Y$ = "does it contain the word free?"
WHY we care: a joint $p(X,Y)$ $p (X, Y)$ is the complete description — you can recover any question about $X$ $X$ and $Y$ $Y$ from it. But often we only want part of the story:
- "How often is email spam, ignoring words?" → marginal
- "Given the word free appeared, how likely is spam?" → conditional
HOW they connect: marginal = collapse a dimension; conditional = take a slice and rescale.

Definitions from first principles

Deriving the conditional (don't memorize — build it)

Formula Why conditioning is division

Conditioning means: "restrict attention to the world where $X=x$ ." In that restricted world, the total probability must again be 1. The joint slice $p_{X,Y}(x,\cdot)$ does not sum to 1 — it sums to $p_X(x)$ . So we re-normalize by dividing: $\sum_y \frac{p_{X,Y}(x,y)}{p_X(x)} = \frac{1}{p_X(x)}\sum_y p_{X,Y}(x,y) = \frac{p_X(x)}{p_X(x)} = 1.\ \checkmark$ Rearranging gives the chain rule / product rule: $\boxed{p_{X,Y}(x,y) = p_{Y\mid X}(y\mid x)\, p_X(x) = p_{X\mid Y}(x\mid y)\, p_Y(y)}$

From the two ways of factoring the joint we immediately get Bayes' theorem: $p_{X\mid Y}(x\mid y) = \frac{p_{Y\mid X}(y\mid x)\,p_X(x)}{p_Y(y)}.$

Figure — Joint, marginal, and conditional distributions

Worked Example 1 — a 2×2 joint table (discrete)

Let $X$ = spam? ( $x\in\{0,1\}$ ), $Y$ = contains "free"? ( $y\in\{0,1\}$ ). Joint pmf:

$p(x,y)$	$Y=0$	$Y=1$	Row sum = $p_X$
$X=0$ (ham)	0.60	0.10	0.70
$X=1$ (spam)	0.05	0.25	0.30
Col sum = $p_Y$	0.65	0.35	1.00

Marginals (edge sums — that's literally why they're written in the margins):

$p_X(1) = 0.05+0.25 = 0.30$ . Why this step? Summing over both $Y$ values erases the word info.
$p_Y(1) = 0.10+0.25 = 0.35$ .

Conditional $P(\text{spam}\mid \text{"free"})$ : $p_{X\mid Y}(1\mid 1) = \frac{p_{X,Y}(1,1)}{p_Y(1)} = \frac{0.25}{0.35}\approx 0.714.$ Why this step? We restrict to the "free"-column (total 0.35) and ask what fraction is spam.

Independent? Check $p(1,1)\overset{?}{=}p_X(1)p_Y(1)=0.30\times0.35=0.105 \ne 0.25$ . Not equal ⇒ dependent. Good — the word free really does raise spam probability from 0.30 to 0.714.

Worked Example 2 — continuous joint

Let $f_{X,Y}(x,y) = 4xy$ on the unit square $0\le x\le1,\ 0\le y\le1$ .

Step 1 — valid? Why? A pdf must integrate to 1. $\int_0^1\!\!\int_0^1 4xy\,dx\,dy = 4\Big(\tfrac12\Big)\Big(\tfrac12\Big)=1.\ \checkmark$

Step 2 — marginal of $X$ : integrate out $y$ . $f_X(x)=\int_0^1 4xy\,dy = 4x\cdot\tfrac12 = 2x,\quad 0\le x\le1.$

Step 3 — conditional of $Y$ given $X=x$ : $f_{Y\mid X}(y\mid x)=\frac{4xy}{2x}=2y.$ Why interesting? It doesn't depend on $x$ ⇒ $X$ and $Y$ are independent here. Indeed $f_{X,Y}=2x\cdot2y=f_X f_Y$ . ✔

Common mistakes (Steel-manned)

Recall Feynman: explain to a 12-year-old

Imagine a big grid of boxes counting kids by (favorite fruit) and (favorite sport). The joint is the count in each little box. If you push all the boxes in each row into one pile, you get how many like each fruit no matter the sport — that's the marginal. If you look at only the kids who play football and ask "of these, how many like apples?", you first grab that column then split it into fractions that add to one — that's the conditional. Marginal = squish a direction; conditional = pick a strip and share it out to 100%.

Active recall

How do you get a marginal

p_X(x)

from a joint

p_{X,Y}

Sum (or integrate) the joint over all values of the other variable:

p_X(x)=\sum_y p_{X,Y}(x,y)

Define the conditional pmf

p_{Y\mid X}(y\mid x)

p_{Y\mid X}(y\mid x)=\dfrac{p_{X,Y}(x,y)}{p_X(x)}

for

p_X(x)>0

— the joint slice re-normalized.

Why must we divide by the marginal when conditioning?

Because the joint slice at

X=x

sums to

p_X(x)

, not 1; dividing renormalizes it to a valid distribution.

State the product/chain rule for the joint.

p_{X,Y}(x,y)=p_{Y\mid X}(y\mid x)\,p_X(x)=p_{X\mid Y}(x\mid y)\,p_Y(y)

When are

X

and

Y

independent (in terms of distributions)?

When

p_{X,Y}(x,y)=p_X(x)p_Y(y)

for all

x,y

, equivalently

p_{Y\mid X}=p_Y

Derive Bayes' theorem from the two factorizations of the joint.

Equate

p_{X\mid Y}p_Y=p_{Y\mid X}p_X

⇒

p_{X\mid Y}(x\mid y)=\dfrac{p_{Y\mid X}(y\mid x)p_X(x)}{p_Y(y)}

Given the spam table, compute

P(\text{spam}\mid\text{"free"})

0.25/0.35\approx0.714

True/False: uncorrelated implies independent.

False — independence implies uncorrelated, but not vice versa.

Connections

Bayes' theorem — direct consequence of the two joint factorizations.
Independence and conditional independence — the factorization condition.
Covariance and correlation — weaker summaries of the joint than the full distribution.
Naive Bayes classifier — assumes conditional independence of features given the label.
Marginalization and the law of total probability — $p_Y(y)=\sum_x p_{Y\mid X}(y\mid x)p_X(x)$ .
Probabilistic graphical models — encode which conditionals factor the joint.
Expectation and variance — computed against these distributions.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho tumhare paas ek table hai jisme do cheezein saath-saath likhi hain — jaise "email spam hai ya nahi" aur "email me 'free' word hai ya nahi". Har chhoti box ki probability ko joint distribution kehte hain: yeh poori kahani batata hai, dono variables ke bare me sab kuch isi se nikal sakta hai.

Ab agar tum sirf "spam kitna common hai" jaanna chahte ho, chahe word kuch bhi ho, to tum uss row ko poora add kar dete ho — isko marginal kehte hain (naam isliye kyunki yeh table ke kinare, yaani margin me likha aata hai). Aur agar tum bolo "maan lo 'free' word aa gaya, ab spam ki kya chance hai?", to tum sirf uss column ko dekhte ho, phir usko re-scale karte ho taaki total 1 ho jaye — yeh hai conditional. Formula: joint ko marginal se divide karo.

Sabse important trap: $P(\text{spam}\mid\text{free})$ aur $P(\text{free}\mid\text{spam})$ ek nahi hote — inke denominator alag hote hain, isliye kabhi bhi bar ke dono taraf ko swap mat karo. Isi swap ko theek karne ke liye Bayes theorem use hota hai, jo bas joint ko do tareeke se todne se aa jaata hai.

Yeh cheez ML me har jagah hai: Naive Bayes, generative models, graphical models — sab is ek simple idea (slice karo aur normalize karo, ya squish karke bhulo) par khade hain. Ek baar joint–marginal–conditional ka rishta samajh liya, to aadhi probability apne aap clear ho jaati hai — yahi 80/20 hai.

Joint, marginal, and conditional distributions

WHY do we need all three?

Definitions from first principles

Deriving the conditional (don't memorize — build it)

Worked Example 1 — a 2×2 joint table (discrete)

Worked Example 2 — continuous joint

Common mistakes (Steel-manned)

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections