Gaussian - Normal distribution properties

1.3.11 · AI-ML › Probability & Statistics

Core Definition

YEH FORMULA KYUN? Chaliye ise first principles se derive karte hain.

Derivation from Maximum Entropy

Goal: Fixed mean $\mu$ aur variance ${\sigma }^2$ ke saath "sabse zyada random" distribution dhundho.

Step 1: Entropy maximization set up karo Hum entropy $H=-\int f(x)\ln f(x)dx$ maximize karna chahte hain in constraints ke saath:

• Normalization: $\int f(x)dx=1$
• Mean constraint: $\int xf(x)dx=\mu$
• Variance constraint: $\int (x-\mu {)}^{2}f(x)dx={\sigma }^2$

Entropy kyun? Entropy maximize karna matlab apne constraints se aage minimum information assume karna—woh distribution jo apni constraints ke alawa sabse kam extra assumptions karti hai.

Step 2: Lagrange multipliers use karo $\mathcal{L}=-\int f\ln fdx+{\lambda }_0\left(\int fdx-1\right)+{\lambda }_1\left(\int xfdx-\mu \right)+{\lambda }_2\left(\int (x-\mu {)}^{2}fdx-{\sigma }^2\right)$

Step 3: $f$ ke respect mein functional derivative lo $\frac{\delta \mathcal{L}}{\delta f}=-\ln f-1+{\lambda }_0+{\lambda }_{1}x+{\lambda }_2(x-\mu {)}^2=0$

Yeh step kyun? Hum woh point dhundh rahe hain jahan entropy ka "slope" constraints ke "slope" ke barabar ho—balance point.

Step 4: $f(x)$ solve karo $\ln f=-1+{\lambda }_0+{\lambda }_{1}x+{\lambda }_2(x-\mu {)}^2$ $f(x)=A\exp ({\lambda }_{1}x+{\lambda }_2(x-\mu {)}^2)$

Step 5: Constraints apply karo Mean constraint force karta hai ${\lambda }_1=0$ ($\mu$ ke around symmetry se). Variance constraint deta hai ${\lambda }_2=-\frac{1}{2{\sigma }^2}$. Normalization deta hai $A=\frac{1}{\sigma \sqrt{2\pi }}$.

Result: Yeh exactly hamara Gaussian formula hai.

Key Properties

YEH NUMBERS KYUN? Yeh Gaussian PDF ko integrate karne se aate hain: $P(\mu -k\sigma \le X\le \mu +k\sigma )={\int }_{\mu -k\sigma }^{\mu +k\sigma }\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)dx$

$z=\frac{x-\mu }{\sigma }$ let karo (standardize karo), phir: $={\int }_{-k}^k\frac{1}{\sqrt{2\pi }}{e}^{-z^2/2}dz=\text{erf}\left(\frac{k}{\sqrt{2}}\right)$

$k=1$ ke liye: erf$(1/\sqrt{2})\approx 0.683$

Property 1: Symmetry

Gaussian ==symmetric== hai $\mu$ ke around: $f(\mu +a)=f(\mu -a)$ sabhi $a$ ke liye.

KYUN? Exponent $(x-\mu {)}^2$ sirf $\mu$ se distance par depend karta hai, direction par nahi.

ML implication: Mean = Median = Mode. Isse $\mu$ center bhi hai aur sabse zyada likely value bhi.

Property 2: Moments

Derivation: $M_X(t)=E[e^{tX}]={\int }_{-\infty }^{\infty }{e}^{tx}\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)dx$

Step 1: Exponents combine karo $=\frac{1}{\sigma \sqrt{2\pi }}{\int }_{-\infty }^{\infty }\exp \left(tx-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)dx$

Step 2: Square complete karo $tx-\frac{(x-\mu {)}^2}{2{\sigma }^2}=-\frac{1}{2{\sigma }^2}[x^2-2x(\mu +{\sigma }^{2}t)+{\mu }^2]$ $=-\frac{1}{2{\sigma }^2}[(x-\mu -{\sigma }^{2}t{)}^2-{\sigma }^4{t}^2-2\mu {\sigma }^{2}t]$ $=-\frac{(x-\mu -{\sigma }^{2}t{)}^2}{2{\sigma }^2}+\mu t+\frac{{\sigma }^2{t}^2}{2}$

Square complete kyun karo? Taaki ek aur Gaussian mile jo 1 mein integrate ho jaaye.

Step 3: Constants factor out karo $=e^{\mu t+{\sigma }^2{t}^2/2}\cdot \frac{1}{\sigma \sqrt{2\pi }}{\int }_{-\infty }^{\infty }\exp \left(-\frac{(x-\mu -{\sigma }^{2}t{)}^2}{2{\sigma }^2}\right)dx$

Integral 1 hai (Gaussian integral), jo hamara result deta hai.

MGF se, moments derive karo:

• Mean: $E[X]=M^{\prime }(0)=\mu$
• Variance: $E[X^2]-E[X{]}^2=M^{\prime \prime }(0)-M^{\prime }(0{)}^2={\sigma }^2$
• Skewness: 0 (symmetric)
• Excess kurtosis: 0 ("normal" tail behavior define karta hai)

Property 3: Linear Transformations

KYUN? MGF use karo: $M_Y(t)=E[e^{t(aX+b)}]=e^{bt}E[e^{(at)X}]=e^{bt}{M}_X(at)$ $=e^{bt}\exp (\mu (at)+\frac{{\sigma }^2(at{)}^2}{2})=\exp ((a\mu +b)t+\frac{(a\sigma {)}^2{t}^2}{2})$

Yeh $\mathcal{N}(a\mu +b,a^2{\sigma }^2)$ ka MGF hai.

ML application: Agar features Gaussian hain, to standardize karna $Z=\frac{X-\mu }{\sigma }$ deta hai $Z\sim \mathcal{N}(0,1)$.

Property 4: Sum of Independent Gaussians

MGF se proof: $M_{X+Y}(t)=M_X(t)M_Y(t)=\exp ({\mu }_{1}t+\frac{{\sigma }_1^2{t}^2}{2})\cdot \exp ({\mu }_{2}t+\frac{{\sigma }_2^2{t}^2}{2})$ $=\exp ({\mu }_1+{\mu }_2)t+\frac{({\sigma }_1^2+{\sigma }_2^2)t^2}{2})$

Independence kyun matter karta hai? Iske bina, hamen covariance terms milenge: $\text{Var}(X+Y)={\sigma }_1^2+{\sigma }_2^2+2\text{Cov}(X,Y)$.

Central Limit Theorem connection: $n$ i.i.d. variables ka sum $\sim \mathcal{N}(n\mu ,n{\sigma }^2)$. Jaise $n\to \infty$, standardized sum $\mathcal{N}(0,1)$ mein converge karta hai, chahe original variables Gaussian na hon.

Property 5: Maximum Likelihood Estimation

Derivation:

Step 1: Likelihood likho $L(\mu ,{\sigma }^2)={\prod }_{i=1}^n\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)$

Step 2: Log-likelihood lo (maximize karna aasaan hai) $\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\ln (2\pi )-\frac{n}{2}\ln ({\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2$

Log kyun? Products ko sums mein convert karta hai, aur log maximize karna original maximize karne ke equivalent hai (log monotonic hai).

Step 3: $\mu$ ke liye optimize karo $\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu )=0$ $\Rightarrow {\sum }_{i=1}^n{x}_i=n\mu \Rightarrow \hat{\mu }=\frac{1}{n}{\sum }_{i=1}^n{x}_i$

Step 4: ${\sigma }^2$ ke liye optimize karo $\frac{\partial \ell }{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}{\sum }_{i=1}^n(x_i-\mu {)}^2=0$ $\Rightarrow {\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(x_i-\hat{\mu }{)}^2$

Note: Yeh biased hai ($n$ se divide karta hai, $n-1$ se nahi). Unbiased estimator $n-1$ use karta hai (Bessel's correction).

Worked Examples

Common Mistakes

Connections

Yeh concept in se connect hota hai:

• Central Limit Theorem — kyun Gaussians har jagah appear hote hain
• Maximum Likelihood Estimation — data se parameters derive karna
• Bayesian Inference — Gaussian priors aur posteriors
• Linear Regression — Gaussian errors assume karta hai
• Principal Component Analysis — multivariate Gaussian data assume karta hai
• Kalman Filters — Gaussian noise ke saath state estimation
• Naive Bayes Classifier — continuous case mein Gaussian features assume karta hai
• Q-Q Plots — test karna ki data Gaussian hai ya nahi
• Box-Muller Transform — Gaussian random numbers generate karna
• Multivariate Gaussian — multiple dimensions mein extension

#flashcards/ai-ml

Gaussian distribution ka PDF kya hai aur uske parameters kya represent karte hain? :: $f(x)=\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$. Parameter $\mu$ mean hai (center location), aur ${\sigma }^2$ variance hai (spread). Standard deviation $\sigma$ width control karta hai.

Concept Map