Gaussian - Normal distribution properties

Core Definition

WHY this formula? Let's derive it from first principles.

Derivation from Maximum Entropy

Goal: Find the "most random" distribution with fixed mean $\mu$ and variance $\sigma^2$.

Step 1: Set up entropy maximization We want to maximize entropy $H = -\int f(x) \ln f(x) dx$ subject to:

• Normalization: $\int f(x)dx = 1$
• Mean constraint: $\int xf(x)dx = \mu$
• Variance constraint: $\int (x-\mu)^2f(x)dx = \sigma^2$

Why entropy? Maximizing entropy means assuming minimum information beyond our constraints—the distribution that makes fewest additional assumptions.

Step 2: Use Lagrange multipliers $\mathcal{L} = -\int f \ln f dx + \lambda_0\left(\int f dx - 1\right) + \lambda_1\left(\int xf dx - \mu\right) + \lambda_2\left(\int (x-\mu)^2 f dx - \sigma^2\right)$

Step 3: Take functional derivative with respect to $f$ $\frac{\delta \mathcal{L}}{\delta f} = -\ln f - 1 + \lambda_0 + \lambda_1 x + \lambda_2(x-\mu)^2 = 0$

Why this step? We're finding where the "slope" of entropy equals the "slope" of constraints—the balance point.

Step 4: Solve for $f(x)$ $\ln f = -1 + \lambda_0 + \lambda_1 x + \lambda_2(x-\mu)^2$ $f(x) = A \exp(\lambda_1 x + \lambda_2(x-\mu)^2)$

Step 5: Apply constraints The mean constraint forces $\lambda_1 = 0$ (by symmetry around $\mu$). The variance constraint gives $\lambda_2 = -\frac{1}{2\sigma^2}$. Normalization gives $A = \frac{1}{\sigma\sqrt{2\pi}}$.

Result: This is exactly our Gaussian formula.

Key Properties

WHY these numbers? They come from integrating the Gaussian PDF: $P(\mu - k\sigma \leq X \leq \mu + k\sigma) = \int_{\mu-k\sigma}^{\mu+k\sigma} \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) dx$

Let $z = \frac{x-\mu}{\sigma}$ (standardize), then: $= \int_{-k}^{k} \frac{1}{\sqrt{2\pi}} e^{-z^2/2} dz = \text{erf}\left(\frac{k}{\sqrt{2}}\right)$

For $k=1$: erf$(1/\sqrt{2}) \approx 0.683$

Property 1: Symmetry

The Gaussian is ==symmetric== around $\mu$: $f(\mu + a) = f(\mu - a)$ for all $a$.

WHY? The exponent $(x-\mu)^2$ only depends on distance from $\mu$, not direction.

ML implication: Mean = Median = Mode. This makes $\mu$ both the center and most likely value.

Property 2: Moments

Derivation: $M_X(t) = E[e^{tX}] = \int_{-\infty}^{\infty} e^{tx} \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) dx$

Step 1: Combine exponents $= \frac{1}{\sigma\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp\left(tx - \frac{(x-\mu)^2}{2\sigma^2}\right) dx$

Step 2: Complete the square $tx - \frac{(x-\mu)^2}{2\sigma^2} = -\frac{1}{2\sigma^2}[x^2 - 2x(\mu + \sigma^2 t) + \mu^2]$ $= -\frac{1}{2\sigma^2}[(x - \mu - \sigma^2 t)^2 - \sigma^4 t^2 - 2\mu\sigma^2 t]$ $= -\frac{(x - \mu - \sigma^2 t)^2}{2\sigma^2} + \mu t + \frac{\sigma^2 t^2}{2}$

Why complete the square? To get another Gaussian that integrates to 1.

Step 3: Factor out constants $= e^{\mu t + \sigma^2 t^2/2} \cdot \frac{1}{\sigma\sqrt{2\pi}} \int_{-\infty}^{\infty} \exp\left(-\frac{(x - \mu - \sigma^2 t)^2}{2\sigma^2}\right) dx$

The integral is1 (Gaussian integral), giving our result.

From MGF, derive moments:

• Mean: $E[X] = M'(0) = \mu$
• Variance: $E[X^2] - E[X]^2 = M''(0) - M'(0)^2 = \sigma^2$
• Skewness: 0 (symmetric)
• Excess kurtosis: 0(defines "normal" tail behavior)

Property 3: Linear Transformations

WHY? Use MGF: $M_Y(t) = E[e^{t(aX+b)}] = e^{bt} E[e^{(at)X}] = e^{bt} M_X(at)$ $= e^{bt} \exp(\mu(at) + \frac{\sigma^2(at)^2}{2}) = \exp((a\mu+b)t + \frac{(a\sigma)^2 t^2}{2})$

This is the MGF of $\mathcal{N}(a\mu + b, a^2\sigma^2)$.

ML application: If features are Gaussian, standardizing $Z = \frac{X-\mu}{\sigma}$ gives $Z \sim \mathcal{N}(0,1)$.

Property 4: Sum of Independent Gaussians

Proof via MGF: $M_{X+Y}(t) = M_X(t) M_Y(t) = \exp(\mu_1 t + \frac{\sigma_1^2 t^2}{2}) \cdot \exp(\mu_2 t + \frac{\sigma_2^2 t^2}{2})$ $= \exp(\mu_1+\mu_2)t + \frac{(\sigma_1^2+\sigma_2^2)t^2}{2})$

Why independence matters? Without it, we'd have covariance terms: $\text{Var}(X+Y) = \sigma_1^2 + \sigma_2^2 + 2\text{Cov}(X,Y)$.

Central Limit Theorem connection: Sum of $n$ i.i.d. variables $\sim \mathcal{N}(n\mu, n\sigma^2)$. As $n \to \infty$, the standardized sum converges to $\mathcal{N}(0,1)$ even if original variables aren't Gaussian.

Property 5: Maximum Likelihood Estimation

Derivation:

Step 1: Write likelihood $L(\mu, \sigma^2) = \prod_{i=1}^n \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right)$

Step 2: Take log-likelihood (easier to maximize) $\ell(\mu, \sigma^2) = -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2$

Why log? Turns products into sums, and maximizing log is equivalent to maximizing the original (log is monotonic).

Step 3: Optimize for $\mu$ $\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^n(x_i-\mu) = 0$ $\Rightarrow \sum_{i=1}^n x_i = n\mu \Rightarrow \hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i$

Step 4: Optimize for $\sigma^2$ $\frac{\partial \ell}{\partial \sigma^2} = -\frac{n}{2\sigma^2} + \frac{1}{2\sigma^4}\sum_{i=1}^n(x_i-\mu)^2 = 0$ $\Rightarrow \hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n(x_i-\hat{\mu})^2$

Note: This is biased (divides by $n$ not $n-1$). The unbiased estimator uses $n-1$ (Bessel's correction).

Worked Examples

Common Mistakes

Connections

This concept connects to:

• Central Limit Theorem — why Gaussians appear everywhere
• Maximum Likelihood Estimation — deriving parameters from data
• Bayesian Inference — Gaussian priors and posteriors
• Linear Regression — assumes Gaussian errors
• Principal Component Analysis — assumes multivariate Gaussian data
• Kalman Filters — state estimation with Gaussian noise
• Naive Bayes Classifier — assumes Gaussian features in continuous case
• Q-Q Plots — testing if data is Gaussian
• Box-Muller Transform — generating Gaussian random numbers
• Multivariate Gaussian — extension to multiple dimensions

#flashcards/ai-ml

What is the PDF of a Gaussian distribution and what do its parameters represent? :: $f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$. Parameter $\mu$ is the mean (center location), and $\sigma^2$ is the variance (spread). Standard deviation $\sigma$ controls width.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)