Foundations — Joint, marginal, and conditional distributions

This page assumes you have seen none of the notation in the parent note. We will earn every symbol — $P$, $X$, $p(x,y)$, $\sum$, $\int$, and the bar $\mid$ — one at a time, each pinned to a picture. When you finish, re-read the parent topic and nothing will look foreign.

0. What is a probability, really?

• Plain words: cut a whole cake into slices, one slice per possible outcome. The size of a slice = its probability.
• The picture: a bar of total length $1$. Chop it into pieces; each piece is one outcome's probability. All pieces together must refill the whole bar — nothing missing, nothing overflowing.
• Why the topic needs it: every entry in a joint table is one of these slices. The rule "all slices sum to $1$" is the single law that forces conditioning to divide (we'll see this in §6).

We write a probability with the letter $P$. The expression $P(\text{something})$ reads "the probability that something happens." The thing in the parentheses is called an event — any statement that is either true or false for a given outcome.

1. The symbol $X$ — a random variable

• Plain words: $X$ is a labelled box; before the experiment happens you don't know the number inside, but you know the list of numbers it could be.
• The picture: a spinner. The spinner itself is $X$; where it lands is a value $x$.
• Why the topic needs it: the whole topic is about two boxes at once — "is this email spam?" ($X$) and "does it contain the word free?" ($Y$). We must be able to name each box and each of its values separately.

In this topic our variables are often binary — they take only two values, coded $0$ and $1$:

$X=1\text{means "spam,"}X=0\text{means "ham (not spam)."}$

The choice of $0$ and $1$ is just convenient labelling; any two labels would do.

2. The symbol $P(X=x)$ and a distribution

• Plain words: $p_X$ is the full price-list — one probability for every value $X$ can take.
• The picture: a bar chart. One bar per value of $X$; the height of each bar is its probability; the heights add to $1$.
• Why "mass"? Probability is spread over the values like weight sitting on points — hence "mass."

The word distribution just means "the complete description of how probability is spread out." A pmf is the distribution for a discrete variable.

3. The symbol $\sum$ — summation

• Plain words: it's a loop: "for each value, take the number, keep a running total."
• The picture: lining up the bars of a bar chart and stacking their heights into one tall bar.
• Why the topic needs it: getting a marginal is a sum — you sweep out one variable by adding across it (§5).

4. Two boxes at once — the joint symbol $p_{X,Y}(x,y)$

• Plain words: instead of one price-list, you now have a grid. Every cell is one combination "$(x,y)$" and holds its own probability.
• The picture: a checkerboard of tiles; each tile's height (or shade) is how much probability lives at that $(x,y)$ corner. All tiles together still sum to $1$.
• Why the topic needs it: the joint is the complete description — every question about $X$ and $Y$ can be recovered from it. Marginals and conditionals are just two readings of this one grid.

The comma in $p_{X,Y}$ means "and both together." Do not read it as a fraction or a list — it is one number attached to one cell.

5. Reading the grid one way — the marginal (a sum)

$p_X(x)={\sum }_y{p}_{X,Y}(x,y)$

• Plain words: hold $x$ fixed, sweep $y$ across every value, add. You "forget" $Y$ by absorbing it into the total.
• The picture: push each row of tiles to the left edge — the stacked heights written in the margin of the table. That is literally why it is called the marginal.
• Why it must be the whole sum: grabbing just one cell would answer "$X=1$ and $Y=$ that one value," not "$X=1$ regardless of $Y$." You need every cell in the row.

6. Reading the grid the other way — the conditional and the bar $\mid$

• Plain words: zoom into a single row (the world where $X=x$ is settled), then ask how probability splits within that row.
• The picture: highlight one row of the grid and blow it up so it fills the whole bar of length $1$ again.
• The catch (this is the heart of the topic): the tiles in that one row do not add to $1$ — they add to $p_X(x)$, the marginal. To turn a strip into a valid distribution we must rescale it so it sums to $1$ again. Rescaling = dividing by that total:

• Why $p_X(x)>0$: you cannot zoom into a row that has zero probability — there's nothing there to rescale, and dividing by $0$ is undefined. This is the degenerate case: conditioning on an impossible event is meaningless.

7. The symbol $\int$ — for smooth (continuous) variables

• Plain words: $\int$ is "$\sum$ for continuous things" — add up infinitely many infinitely-thin slices.
• The picture: area under a curve (1-D) or volume under a surface (2-D). Probability = area/volume, not tile-height.
• The exact parallels — every discrete rule has a continuous twin:

• Why the topic needs it: ML features are often continuous (pixel intensities, prices). Same three operations — sum becomes integral, height becomes density. Nothing conceptually new.

8. The last piece — independence and the product rule

Rearranging the conditional formula (multiply both sides by $p_X(x)$) gives the product rule, and reading it symmetrically gives independence:

• Plain words: independence means "knowing $X$ tells you nothing new about $Y$" — the conditional collapses back to the plain marginal, $p_{Y\mid X}=p_Y$.
• The picture: an independent grid looks like an outer-product — each cell is just (row total) × (column total), a smooth "multiplication table" with no surprises.

Prerequisite map

This map feeds directly into Bayes' theorem, Marginalization and the law of total probability, the Naive Bayes classifier, and Probabilistic graphical models — all of which are just repeated slicing and squishing of the joint you now understand.

Equipment checklist

Test yourself — cover the right side and answer before revealing.