Foundations — Probability mass and density functions

Before you can read the parent note, you must own every symbol it throws at you. We go one symbol at a time. Nothing is used before it is drawn — so on this page the symbols $\sum$, $\int$, $p_X$, $f_X$, and $F_X$ appear only after their own section builds them, never before.

0. The starting picture: an experiment

Picture a bag. Every marble in the bag is one outcome $\omega$. The whole bag is $\Omega$.

Why do we need this? Because "probability" is meaningless until we agree on the full list of things that could happen. $\Omega$ is that list.

• Coin flip: $\Omega =\{\text{Heads},\text{Tails}\}$ — two marbles.
• One die: $\Omega =\{1,2,3,4,5,6\}$ — six marbles.
• A height in cm: $\Omega =$ every real number in some range — infinitely many marbles, packed with no gaps. Hold that thought; it is the whole PMF-vs-PDF split.

1. Probability $P(\cdot )$ — the "how likely" score

The dot in $P(\cdot )$ is a placeholder — you drop an outcome or an event into the parentheses.

Picture a slider running from $0$ (impossible, far left) to $1$ (guaranteed, far right). Every event's probability is a mark on that slider.

Rule 2 is the seed of two facts you will meet later: once we have the "add-them-all-up" symbol (Section 6) it becomes the PMF's total-equals-one law, and once we have the "area" symbol (Section 7) it becomes the PDF's total-area-equals-one law. We are not allowed to write those two equations yet — their symbols are not built. We only plant the idea here: everything together weighs 1.

2. Event, and "disjoint" — why probabilities add

Picture two circles drawn inside the bag that do not overlap.

Why does the parent note keep combining probabilities of separate outcomes? Because separate outcomes are automatically disjoint, so their scores just add up. This single fact powers the dice example ($6\times \frac{1}{36}$), and its infinite version is what makes the running total over all values behave.

3. Random variable $X$ — turning outcomes into numbers

Read $X(\omega )=x$ as: "outcome $\omega$ is labelled with the number $x$."

Why bother? Because math works on numbers, not on the word "Heads". $X$ is the translator: "Heads" $\to 1$, "Tails" $\to 0$. Notice in the figure that two different outcomes can land on the same number — that is exactly why probabilities get combined onto a value.

See 1.3.01-Random-variables-and-distributions for the full story of $X$.

4. The subscript and the two functions: $p_X$ and $f_X$

Now that these names exist, we are allowed to use them from here on. Mnemonic below to keep the letters straight.

5. Discrete vs continuous — the picture that splits the topic

This is the heart of the whole parent topic. Discrete adds up bar areas; continuous measures areas under a curve. The next two sections build the exact symbols for each.

6. Summation $\sum$ — "add these all up"

Now we can finally state Rule 2 of Section 1 in discrete costume: all the masses add to one, ${\sum }_x{p}_X(x)=1.$ Picture stacking the unit-width bars from the discrete panel; when their areas all pile up they total exactly $1$. (This is also where countable additivity from Section 2 earns its keep: the list of values may be infinite, and the infinite sum still lands on $1$.)

7. Integral $\int$ — "add up infinitely many slivers" (area)

We need $\int$ because continuous outcomes cannot be listed one-by-one; there are too many, packed with no gaps. $\sum$ can't run over them. So we ask a different but analogous question: what is the area?

Look at the figure: we chop the interval $[a,b]$ into thin rectangles of width $\Delta x$. Each rectangle's area $\approx f_X(x)\cdot \Delta x$ is a little chunk of probability. As $\Delta x$ shrinks to zero the jagged tops smooth into the curve, and the sum of rectangle areas becomes the exact area ${\int }_a^b{f}_X(x)dx=P(a\le X\le b)$.

Now Rule 2 in continuous costume finally has all its symbols: the total area is one, ${\int }_{-\infty }^{\infty }{f}_X(x)dx=1.$

8. The limit $\lim$ and $\Delta x\to 0$

Picture the rectangles in the last figure getting thinner and thinner until their staircase top hugs the curve perfectly. The limit is that perfect-hug value. This is why the parent's discrete-to-continuous derivation says $P(a\le X\le b)={\lim }_{\Delta x\to 0}\sum f_X(x_i)\Delta x={\int }_a^b{f}_X(x)dx$.

9. The CDF $F_X$ and the derivative $\frac{d}{dx}$

The parent uses $f_X(x)=\frac{d}{dx}{F}_X(x)$. That needs two new pieces: the function $F_X$ and the derivative.

Picture walking along a hill: the derivative is how steep the ground is under your feet.

10. $e$ and $\exp$ — the decay/bell-curve engine

Picture a quantity that keeps shrinking by the same fraction every step — hot coffee cooling, a battery draining. The parent's exponential PDF $\lambda e^{-\lambda x}$ and the Gaussian's $\exp (-(x-\mu {)}^2/2{\sigma }^2)$ both ride on this shape: common values are near the peak, extremes fade away exponentially fast.

The Greek letters that ride along: $\lambda$ ("lambda") = a rate (arrivals per second); $\mu$ ("mu") = the centre/mean; $\sigma$ ("sigma") = the spread; ${\sigma }^2$ = variance. Their full meaning lives in 1.3.08-Expected-value-and-variance and 1.3.09-Common-probability-distributions.

Prerequisite map

Each arrow means "you must own the left box before the right box makes sense." Follow them top to bottom and you arrive at the parent topic fully equipped. Downstream, these feed 1.4.02-Maximum-likelihood-estimation, 2.1.05-Softmax-activation, and 3.2.04-Gaussian-processes.

Equipment checklist

Cover the right side; can you answer before revealing?