Random variables (discrete and continuous)

A random variable is a function that maps outcomes of a random experiment to real numbers. It's the bridge between the mesy real world (coin flips, pixel intensities, user clicks) and the clean mathematical world where we can calculate probabilities and expectations.

Why Random Variables Matter in ML

In machine learning, almost everything is a random variable:

• Features: pixel values, word counts, sensor readings
• Labels: class assignments, regression targets
• Model parameters: weights sampled during training
• Predictions: outputs with uncertainty

Understanding RVs lets you:

1. Quantify uncertainty in data and predictions
2. Design loss functions that match your data distribution
3. Sample from complex distributions (GANs, VAEs)
4. Reason about model behavior probabilistically

The Core Intuition

Example: Roll a die. The outcome space is {⚀, ⚁, ⚂, ⚃, ⚄, ⚅}. Define $X$ = "number of dots". Then $X$ maps⚂ →3, ⚅ → 6, etc. $X$ is the random variable, not the outcome "3" itself.

Discrete Random Variables

Probability Mass Function (PMF)

The probability mass function $P(X = x)$ gives the probability that $X$ takes the exact value $x$.

Requirements (these ensure it's a valid probability distribution):

1. Non-negativity: $P(X = x) \geq 0$ for all $x$
2. Normalization: $\sum_{x} P(X = x) = 1$ (all probabilities sum to 1)

Why these rules? Non-negativity: probabilities can't be negative. Normalization: something must happen, so total probability = 1.

From first principles:

• Imagine repeating the experiment $N$ times, getting values $x_1, x_2, ..., x_N$
• The sample average is $\frac{1}{N}\sum_{i=1}^{N} x_i$
• Group identical values: if $x_j$ appears $n_j$ times, average = $\frac{1}{N}\sum_{j} n_j x_j$
• As $N \to \infty$, $\frac{n_j}{N} \to P(X = x_j)$ (frequency converges to probability)
• Therefore: $E[X] = \sum_{x} x \cdot P(X = x)$

Why does each term get weighted by $P(X=x)$? More probable values appear more often in the long run, so they contribute more to the average.

PMF: $P(X = 1) = p, \quad P(X = 0) = 1 - p$

Compact form: $P(X = x) = p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$

Why this form? When $x=1$: $p^1 (1-p)^0 = p$. When $x=0$: $p^0 (1-p)^1 = 1-p$. It's a clever way to write both cases in one expression.

Expected value: $E[X] = 0 \cdot P(X=0) + 1 \cdot P(X=1) = 0 \cdot (1-p) + 1 \cdot p = p$

Why $E[X] = p$? If you flip 100 times with $p=0.3$, you expect about 30 heads. Average = 30/100 = 0.3 = $p$.

ML Application: Binary classification labels (spam/not spam). Logistic regression outputs $p$, which is literally $E[X]$ for the Bernoulli RV "is this example class 1?".

Derivation of PMF:

• Probability of any specific sequence with $k$ heads: $p^k (1-p)^{n-k}$ (heads occur $k$ times at prob $p$, tails $n-k$ times at prob $1-p$)
• Number of such sequences (orderings of $k$ heads in $n$ flips): $\binom{n}{k} = \frac{n!}{k!(n-k)!}$
• Why multiplication? Each specific sequence has prob $p^k(1-p)^{n-k}$, and there are $\binom{n}{k}$ mutually exclusive sequences

$P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k = 0, 1, ..., n$

Expected value: $E[X] = np$

Why? Each flip contributes expected value $p$. With $n$ independent flips: $E[X_1 + ... + X_n] = E[X_1] + ... + E[X_n] = p + ... + p = np$.

ML Application: Dropout in neural networks. Each neuron independently "survives" with probability $p$. Number of active neurons in a layer ~ Binomial.

Continuous Random Variables

Key difference from discrete: $P(X = x) = 0$ for any specific $x$. Why? Infinitely many points, probability spread infinitely thin. We must ask about intervals: $P(a \leq X \leq b)$.

Probability Density Function (PDF)

The probability density function $f(x)$ is NOT a probability—it's a density. Think of it as "probability per unit length".

Requirements:

1. Non-negativity: $f(x) \geq 0$ for all $x$
2. Normalization: $\int_{-\infty}^{\infty} f(x) \, dx = 1$

Why integration? You're summing up infinitesimal probabilities $f(x) \, dx$ over the interval. The "$dx$" converts density to probability: $f(x)$ has units "probability per unit", $dx$ has units "unit", product has units "probability".

Geometric intuition: Probability = area under the curve $f(x)$ between $a$ and $b$.

$E[X] = \int_{-\infty}^{\infty} x \cdot f(x) \, dx$

Why $x \cdot f(x)$? Same logic as discrete: weight each value $x$ by its "probability" (here, density × $dx$).

PDF: $f(x) = \begin{cases} \frac{1}{b-a} & \text{if } a \leq x \leq b \\ 0 & \text{otherwise} \end{cases}$

Why this form? Constant density in $[a, b]$ (equally likely). Must integrate to 1: $\int_{a}^{b} \frac{1}{b-a} \, dx = \frac{1}{b-a} \cdot (b - a) = 1 \quad \checkmark$

Expected value: $E[X] = \int_{a}^{b} x \cdot \frac{1}{b-a} \, dx = \frac{1}{b-a} \int_{a}^{b} x \, dx = \frac{1}{b-a} \cdot \frac{x^2}{2}\Big|_a^b = \frac{1}{b-a} \cdot \frac{b^2 - a^2}{2}$

Simplify: $b^2 - a^2 = (b-a)(b+a)$, so: $E[X] = \frac{(b-a)(b+a)}{2(b-a)} = \frac{b+a}{2}$

Why is it the midpoint? Symmetry. Equally likely on both sides, average is the center.

ML Application: Weight initialization in neural networks often uses uniform distributions (e.g., Xavier/Glorot initialization samples from $U[-\sqrt{6/(n_{in}+n_{out})}, \sqrt{6/(n_{in}+n_{out})}]$).

PDF: $f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$

Parameters: $\mu$ (mean, location), $\sigma^2$ (variance, spread)

Why this functional form? (Brief derivation from maximum entropy under mean/variance constraints—full proof is complex, but the intuition):

• We want the "most random" distribution given only knowledge of mean and variance
• Maximizing entropy subject to $E[X] = \mu$, $E[(X-\mu)^2] = \sigma^2$ yields this exponential-quadratic form
• The $1/\sqrt{2\pi\sigma^2}$ ensures $\int f(x) dx = 1$

Why $\exp(-x^2)$ shape? It concentrates probability near the mean ($\mu$), with tails dying off quickly. The quadratic in the exponent ensures smoothness and the "bell" shape.

Expected value: By symmetry around $\mu$, $E[X] = \mu$.

ML Applications:

1. Gaussian noise in data: $y = f(x) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma^2)$
2. Weight initialization (e.g., He initialization: $\mathcal{N}(0, 2/n_{in})$)
3. Gaussian processes, variational autoencoders (VAE latent space)
4. Linear regression assumes $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$

Cumulative Distribution Function (CDF)

Both discrete and continuous RVs have CDF $F(x) = P(X \leq x)$.

For discrete RV: $F(x) = \sum_{x_i \leq x} P(X = x_i)$ (sum up PMF values)

For continuous RV: $F(x) = \int_{-\infty}^{x} f(t) \, dt$ (integrate PDF)

Why useful?

• CDFs always exist (PMFs/PDFs don't for mixed distributions)
• Easy to compute probabilities: $P(a< X \leq b) = F(b) - F(a)$
• Relationship to PDF: $f(x) = \frac{dF(x)}{dx}$ (PDF is the derivative of CDF)

ML Application: Percentile-based metrics, quantile regression, inverse transform sampling (generate samples from any distribution by applying $F^{-1}$ to uniform random numbers).

Common Mistakes

Why it feels right: We're used to $P(X=x)$ from discrete RVs.

The fix: $f(x)$ is density, not probability. Units: $f(x)$ is "probability per unit". $f(x) = 2$ means "locally, probability accumulates at 2 units per unit length". Actual probability: $P(X \in [x, x+dx]) = f(x) \cdot dx$ (which is small for small $dx$).

Check: For $X \sim U[0, 0.3]$, $f(x) = 1/0.3 \approx 3.33 > 1$. But $P(0 \leq X \leq 0.3) = 3.33 \times 0.3 = 1$ ✓

Why it feels right: Both use weighted averages.

The fix:

• Discrete: E_X = sum(x * P(x) for x in values)
• Continuous: E_X = integrate(x * f(x), x, -inf, inf) or E_X = np.mean(samples) for samples

In NumPy, if you have samples from a continuous RV, use np.mean(samples), not sum(x * count(x)) (which assumes discrete).

Why it feels right: We measure "20°C" on thermometers.

The fix: Real measurements have finite precision. Your thermometer reads "20°C" = "between 19.5and 20.5°C" (an interval). For continuous RVs, always ask about intervals: $P(19.5 \leq X \leq 20.5)$.

Active Recall

A random variable is like a nickname for that number. We call it $X$. If we roll a die, $X$ might be 1, 2, 3, 4, 5, or 6 (only these numbers, nothing in between). That's discrete—like counting candies.

But if we spin a wheel that can land anywhere from 0 to 100, $X$ could be 23.7145... or 99.001... or any number in between. That's continuous—like measuring how much water is in a cup.

For discrete, we can say "the probability of getting exactly 3 is..." (like $P(X=3)=1/6$ for a fair die). For continuous, we can't say "exactly 23.7145" because there are infinite possibilities—it's like asking for one specific grain of sand on a beach. Instead, we ask "What's the probability $X$ is between 20 and 30?" and add up probabilities in that range (integration).

The cool part: in AI, almost everything is a random variable—what digit is in an image, what word comes next, whether an email is spam. Understanding RVs means understanding how AI handles uncertainty.

Memory Aids

• Discrete = Distinct, Discrete math (integers)
• PMF = Probability Mass (mass sits at points)
• PDF = Probability Density (spread like a fluid)

Alphabet trick:

• PMF uses Sum ($\Sigma$)
• PDF uses Integral ($\int$)
• "S comes before I" → use sum for discrete, integral for continuous

Connections

• Probability Space – random variables are measurable functions on sample spaces
• Expectation and Variance – moments of random variables, higher-order statistics
• Joint and Conditional Distributions – multivariate RVs, dependence structure
• Central Limit Theorem – why sums converge to Gaussian
• Maximum Likelihood Estimation – fitting distributions to data (finding $p$, $\mu$, $\sigma$)
• Bernoulli and Categorical Distributions – classification labels as discrete RVs
• Gaussian Distribution – continuous RV for regression noise, VAE latents
• Sampling Methods – generating RV samples (inverse transform, rejection sampling)
• Information Theory – entropy of RV distributions

#flashcards/ai-ml

What is a random variable? :: A function that maps outcomes of a random experiment to real numbers. It's the bridge between real-world randomness and mathematical probability.

Expected value formula for continuous RV? :: $E[X] = \int_{-\infty}^{\infty} x \cdot f(x) \, dx$. Weighted integral over the density.

Concept Map

Hinglish (regional understanding)