Probability mass and density functions

Intuition The Bridge Between Outcomes and Probabilities

Imagine you're designing an AI system that needs to predict house prices. Some predictions will be exactly $300k, $301k, etc. (discrete), but in reality, prices are continuous—$300,245.73 How do we assign probabilities to outcomes?

PMF (Probability Mass Function) is for discrete random variables—it tells you the exact probability of each outcome (like a vending machine: button A gives you chips with probability 0.3).

PDF (Probability Density Function) is for continuous random variables—it doesn't give direct probabilities (a single point has probability 0!), but describes the relative likelihood across intervals (like measuring the concentration of sugar in water at different depths).

WHY THIS MATTERS IN ML: Every prediction model outputs a distribution. Classification → PMF over classes. Regression → PDF over possible values. Understanding these functions is how you go from "my model outputs numbers" to "my model quantifies uncertainty."

Probability Mass Function (PMF)

Derivation from First Principles

START: What is a random variable? A function that maps outcomes to numbers.

STEP 1: For discrete outcomes $\omega_1, \omega_2, \ldots$ in sample space $\Omega$ , each has probability $P(\omega_i)$ .

STEP 2: Random variable $X$ maps $\omega_i \to x_i$ . Multiple outcomes can map to the same $x$ .

STEP 3: The probability that $X$ takes value $x$ is the sum of probabilities of all outcomes that map to $x$ : $p_X(x) = P(X = x) = \sum_{\omega: X(\omega) = x} P(\omega)$

WHY this form? By the axiom of countable additivity: if events are disjoint (different outcomes), their probabilities add.

Property 1 proof: Since $0 \leq P(\omega) \leq 1$ for each outcome, and we're summing non-negative terms, $0 \leq p_X(x) \leq 1$ .

Property 2 proof: $\sum_x p_X(x) = \sum_x \sum_{\omega: X(\omega)=x} P(\omega) = \sum_{\omega \in \Omega} P(\omega) = 1$ We're just regrouping the same outcome probabilities.

Worked example Example 1: Rolling Two Dice, Sum = X

Setup: Roll two fair dice. Let $X$ = sum of faces. Find: $p_X(7)$

Solution:

Sample space: 36 equally likely outcomes $(i,j)$ where $i,j \in \{1,2,3,4,5,6\}$
Outcomes giving $X=7$ : $(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)$ — that's 6 outcomes
Each outcome has probability $1/36$ $p_X(7) = P(X=7) = 6 \times \frac{1}{36} = \frac{1}{6}$

Why this step? We identified which elementary outcomes map to $x=7$ via $X$ , then sumed their probabilities (each $1/36$ ).

Full PMF: For $x \in \{2,3,\ldots,12\}$ : $p_X(x) = \frac{\text{number of ways to get sum } x}{36}$

Check: $\sum_{x=2}^{12} p_X(x) = \frac{1+2+3+4+5+6+5+4+3+2+1}{36} = \frac{36}{36} = 1$ ✓

Worked example Example 2: Bernoulli Random Variable

Setup: Single coin flip. $X=1$ if heads (success), $X=0$ if tails (failure). Coin is biased: $P(\text{heads}) = p$ .

Derive the PMF:

$p_X(1) = P(X=1) = p$
$p_X(0) = P(X=0) = 1-p$
$p_X(x) = 0$ for $x \notin \{0,1\}$

Compact form: $p_X(x) = p^x(1-p)^{1-x} \quad \text{for } x \in \{0,1\}$

Why this form? When $x=1$ : $p^1(1-p)^0 = p$ . When $x=0$ : $p^0(1-p)^1 = 1-p$ . It's a clever way to write both cases.

ML Connection: This is the foundation of logistic regression—predicting binary outcomes.

Figure — Probability mass and density functions

Probability Density Function (PDF)

Why Points Have Zero Probability

The intuition: Imagine measuring someone's height. What's the probability their height is EXACTLY 170.000... cm (infinite precision)? Zero. But the probability their height is between 169.5 and 170.5 cm? Non-zero.

Mathematical reason: $P(X = c) = \int_c^c f_X(x) \, dx = 0$ An integral from $c$ to $c$ has zero width.

Consequence: For continuous variables, $P(a \leq X \leq b) = P(a < X < b)$ —boundary points don't matter.

Derivation from Discrete to Continuous

START: Consider a discrete variable with outcomes at $x_i = i\Delta x$ for small $\Delta x$ .

STEP 1: PMF gives $p_X(xi) = P(X = x_i)$ .

STEP 2: As $\Delta x \to 0$ , we pack more and more outcomes into any interval $[a,b]$ .

STEP 3: The probability of interval $[x, x+\Delta x]$ approaches $p_X(x) \approx f_X(x) \cdot \Delta x$ .

WHY? Think of $f_X(x)$ as "probability per unit length." Multiply by length $\Delta x$ to get probability.

STEP 4: Sum over the interval $[a,b]$ : $P(a \leq X \leq b) = \lim_{\Delta x \to 0} \sum_{x_i \in [a,b]} f_X(x_i) \Delta x = \int_a^b f_X(x) \, dx$

This is the Riemann integral definition.

Worked example Example 3: Uniform Distribution on [0, 1]

Setup: Random variable $X$ equally likely anywhere in $[0,1]$ .

Intuition: "Flat" density—no part of the interval is favored.

Derive the PDF:

Total probability = 1 must be spread over length1
Density must be constant, say $f_X(x) = c$ for $x \in [0,1]$
Normalization: $\int_0^1 c \, dx = c \cdot 1 = 1 \implies c = 1$

$f_X(x) = \begin{cases} 1 & \text{if } 0 \leq x \leq 1 \\ 0 & \text{otherwise} \end{cases}$

Check probability of an interval: $P(0.3 \leq X \leq 0.7) = \int_{0.3}^{0.7} 1 \, dx = 0.4$ ✓

Why this step? The integral of a constant over an interval is constant × length.

Worked example Example 4: Exponential Distribution (Waiting Times)

Setup: Time $X$ until next customer arrives at store. Memoryless property: past doesn't affect future. Heuristic derivation:

In a small time $\Delta t$ , probability of arrival is $\lambda \Delta t$ (constant rate $\lambda$ )
Probability of NO arrival in $[0, x]$ involves exponential decay
This leads to CDF: $F_X(x) = 1 - e^{-\lambda x}$ for $x \geq 0$ PDF by differentiation: $f_X(x) = \frac{d}{dx}(1 - e^{-\lambda x}) = \lambda e^{-\lambda x} \quad \text{for } x \geq 0$ Why this form?
$\lambda$ controls the rate (higher $\lambda$ = shorter waits, steper decay)
Exponential decay: most waits are short, long waits are rare Verify normalization:

\int_0^\infty \lambda e^{-\lambda x} dx = \lambda \left[-\frac{1}{\lambda}e^{-\lambda x}\right]_0^\infty = \lambda \cdot \frac{1}{\lambda} = 1

$f_X(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

Parameters:

$\mu$ = mean (center of the bell curve)
$\sigma^2$ = variance (spread; larger $\sigma$ = wider curve)

Why this specific form?

Symmetry: $(x-\mu)^2$ treats deviations equally in both directions
Exponential decay: Extreme values become exponentially unlikely
Normalization constant $\frac{1}{\sigma\sqrt{2\pi}}$ : ensures $\int_{-\infty}^\infty f_X(x) dx = 1$ Derivation of constant (sketch):

Need $\int_{-\infty}^\infty e^{-x^2/(2\sigma^2)} dx = \sigma\sqrt{2\pi}$
Use polar coordinates, convert to 2D integral of $e^{-r^2}$ , get $\sqrt{\pi}$ factors ML uses: Neural network initialization, noise models, Gaussian processes, variational inference.

PMF vs PDF: The Critical Differences

| Property | PMF (Discrete) | PDF (Continuous) | |----------|------------------| | Values | $p_X(x) = P(X=x)$ | $f_X(x)$ is NOT a probability | | Range | $0 \leq p_X(x) \leq 1$ | $f_X(x) \geq 0$ , can exceed 1! | | Total | $\sum_x p_X(x) = 1$ | $\int_{-\infty}^\infty f_X(x) dx = 1$ | | Point prob | $p_X(x)$ is the answer | $P(X=x) = 0$ | | Interval prob | $\sum_{x \in [a,b]} p_X(x)$ | $\int_a^b f_X(x) dx$ |

Common mistake Common Mistake 1: "PDF values are probabilities"

The wrong idea: "If $f_X(1.5) = 0.8$ , then $P(X=1.5) = 0.8$ ."

Why it feels right: PMF values ARE probabilities, so we assume the same for PDF.

The fix: PDF is probability per unit. A density of 0.8 means if you take a tiny interval of width 0.01around 1.5, the probability is approximately $0.8 \times 0.01 = 0.008$ .

Visual analogy: Think of population density (people per km²). "1000 people/km²" doesn't mean there are 1000 people at single point—it means in a1 km² area, there are 1000 people.

Consequence: PDFs can be greater than 1! The uniform distribution on $[0, 0.5]$ has $f_X(x) = 2$ (so $\int_0^{0.5} 2 \, dx = 1$ ).

Common mistake Common Mistake 3: "If

$f_X(x) = 0$ at some point, $X$ never takes that value" The wrong idea: "The PDF is zero at $x=3$ , so $X$ can't equal 3."

Why it feels right: In the discrete case, $p_X(x) = 0$ means impossible outcome.

The fix: For continuous variables, $P(X=3) = 0$ anyway (single points have zero measure). The PDF being zero means the density is zero there—the variable is unlikely to be near that value, but the set $\{X=3\}$ already has probability zero.

Example: Mix of uniform on $[0,1]$ and $[2,3]$ : $f_X(x) = \begin{cases} 0.5 & \text{if } x \in [0,1] \cup [2,3] \\ 0 & \text{otherwise} \end{cases}$ The PDF is zero on $(1,2)$ , and indeed $P(1< X < 2) = 0$ (impossible region). But $P(X=1.5) = 0$ too, even though we might say " $X$ avoids that region."

Recall Feynman Explanation (Explain to a 12-year-old)

Imagine you have a big jar of marbles, each with a number written on it. If you can COUNT the marbles (1, 2, 3, ..), that's discrete. I can tell you exactly: "The probability of picking marble #7 is 1 out of 20." That's a PMF—it gives the exact chance for each marble.

Now imagine instead the jar is full of sand, and you're measuring how much gold dust is mixed in. You can't pick out a single grain and ask its probability—there are infinite grains! But I can tell you: "In this REGION of the jar, there's a lot of gold; in that region, very little." That's what a PDF does—it tells you where the "gold" (probability) is concentrated.

If I scoop a tiny spoonful from the high-density region, I'll probably get some gold. If I scoop from the low-density region, probably not. But a single grain? Zero gold (zero probability). The density tells you: look in THIS neighborhood if you want to find gold.

In machine learning, every time your model says "I think the answer is around here," it's giving you a density function!

Connections

1.3.01-Random-variables-and-distributions — PMF/PDF formalize the distribution of a random variable
1.3.07-Cumulative-distribution-functions — CDF unifies discrete and continuous cases; $F_X' = f_X$
1.3.08-Expected-value-and-variance — Computed as $E[X] = \sum x \cdot p_X(x)$ or $\int x \cdot f_X(x) dx$
1.3.09-Common-probability-distributions — Specific PMFs (Binomial, Poisson) and PDFs (Gaussian, Exponential)
1.4.02-Maximum-likelihood-estimation — Finding parameters that maximize $\prod p_X(x_i)$ or $\prod f_X(x_i)$
2.1.05-Softmax-activation — Softmax converts logits to a PMF over classes
3.2.04-Gaussian-processes — Model functions as draws from a distribution with PDF over function space

#flashcards/ai-ml

What is the key difference between a PMF and a PDF? :: PMF gives exact probabilities for discrete outcomes: $p_X(x) = P(X=x)$ . PDF gives probability density for continuous variables—probability is the integral (area), not the value itself: $P(a \leq X \leq b) = \int_a^b f_X(x) dx$ .

What are the two properties that any PMF must satisfy?

(1)

0 \leq p_X(x) \leq 1

for all

x

, and (2)

\sum_{\text{all } x} p_X(x) = 1

What are the two properties that any PDF must satisfy?

(1)

f_X(x) \geq 0

for all

x

, and (2)

\int_{-\infty}^{\infty} f_X(x) dx = 1

Why is

P(X = c) = 0

for any specific value

c

of a continuous random variable?

Because the probability is the area under the PDF curve, and a single point has zero width, hence zero area:

P(X=c) = \int_c^c f_X(x) dx = 0

What is the relationship between PDF and CDF?

The PDF is the derivative of the CDF:

f_X(x) = \frac{d}{dx} F_X(x)

. The CDF is the integral of the PDF:

F_X(x) = \int_{-\infty}^x f_X(t) dt

Can a PDF value

f_X(x)

be greater than 1?

Yes! PDF is a density (probability per unit), not a probability. Example: uniform on

[0, 0.5]

has

f_X(x) = 2

What is the PMF of a Bernoulli random variable with success probability

p

p_X(1) = p

and

p_X(0) = 1-p

, or compactly:

p_X(x) = p^x(1-p)^{1-x}

for

x \in \{0,1\}

What is the PDF of a uniform distribution on

[a, b]

f_X(x) = \frac{1}{b-a}

for

x \in [a,b]

, and

0

otherwise. The constant density ensures total probability is 1.

What is the PDF of an exponential distribution with rate

\lambda

f_X(x) = \lambda e^{-\lambda x}

for

x \geq 0

, and

0

otherwise. It models waiting times in memoryless processes.

What is the PDF of a Gaussian distribution with mean

\mu

and variance

\sigma^2

f_X(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

for all

x \in \mathbb{R}

How do you compute

P(a \leq X \leq b)

for a discrete random variable?

P(a \leq X \leq b) = \sum_{x=a}^{b} p_X(x)

(sum the PMF over the interval).

How do you compute

P(a \leq X \leq b)

for a continuous random variable?

P(a \leq X \leq b) = \int_a^b f_X(x) dx

(integrate the PDF over the interval).

Why do we use PMF for classification and PDF for regression in ML?

Classification outputs discrete class labels (finite outcomes) → PMF over classes. Regression outputs continuous values (infinite possibilities) → PDF over the real line.

If two dice are rolled and

X

is their sum, what is

p_X(7)

There are 6 outcomes giving sum 7:

(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)

out of 36 total equally likely outcomes. So

p_X(7) = 6/36 = 1/6

What does "memoryless property" of the exponential distribution mean?

P(X > s+t \mid X > s) = P(X > t)

: the probability of waiting an additional

t

time doesn't depend on how long you've already waited. The past doesn't matter.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

PMF aur PDF: Probability ko samajhne ka tarika

Socho tum ek AI model bana rahe ho jo predict karta hai kikal barish hogi ya nahi. Yeh ek discrete problem hai—do outcomes hain: haan ya na. Har outcome ko ek exact probability milti hai, jaise 0.7 (barish) aur 0.3 (no barish). Yeh PMF (Probability Mass Function) ka kaam hai. Yeh tumhe bolta hai ki kisi specific outcome ki exact probability kya hai. Jaise dice roll karo, toh sum = 7 ane ki probability PMF se milti hai.

Ab socho tum predict kar rahe ho ki kal temperature kya hoga—23.5°C, 23.51°C, 23.512°C... infinite possibilities hain! Yahanek single value ki probability zero hai (kyunki infinite values mein se koiek exact match ho—bahut rare!). Toh hum PDF (Probability Density Function) use karte hain. PDF tumhe bata hai ki probability ka "concentration" kahan zyada hai. Agar tum ek chhota interval lo, jaise 23°C se 24°C, toh PDF ko integrate karke (area nikalke) probability milti hai. PDF is like density of gold dust in sand—jahan density zyada, wahan probability zyada.

ML mein kyun zaroori hai? Jab bhi tumara model kuch predict karta hai—classification mein class probabilities (PMF), regression mein continuous values

Probability Mass Function (PMF)

Derivation from First Principles

Probability Density Function (PDF)

Why Points Have Zero Probability

Derivation from Discrete to Continuous

PMF vs PDF: The Critical Differences

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections