Maximum likelihood estimation (MLE)

Overview

Maximum Likelihood Estimation is a method of estimating the parameters of a probability distribution by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable.

WHY it matters: In ML, we constantly need to fit models (like Gaussian distributions for features, or logistic regression coefficients). MLE gives us a principled, mathematically rigorous way to find the "best" parameters from data.

Figure — Maximum likelihood estimation (MLE)

Core Intuition

The Mathematical Framework

Derivation from First Principles

WHY does the likelihood measure "fit"?

Start with what we actually want: Given data $X$ , find $\theta$ that makes $X$ most probable.

Step 1: The probability of observing data $X$ given parameters $\theta$ is $P(X|\theta)$ .

Step 2: For independent samples (i.i.d. assumption): $P(X|\theta) = P(x_1, x_2, \ldots, x_n | \theta) = \prod_{i=1}^{n} P(x_i | \theta)$

WHY? Independence means joint probability = product of individual probabilities.

Step 3: We treat this as a function of $\theta$ (data is fixed), calling it the likelihood $L(\theta|X)$ .

Step 4: Take logarithm (since $\log$ is monotonic, $\arg\max$ doesn't change): $\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \log p(x_i | \theta)$

WHY this step? Mathematically cleaner (sums vs products), numerically stable (prevents underflow), easier derivatives.

Step 5: Find maximum by calculus: $\frac{\partial \ell}{\partial \theta} = 0$

Solve for $\theta$ → this is $\hat{\theta}_{MLE}$ .

Worked Examples

Worked example Example 1: MLE for Bernoulli Distribution

Setup: You flip a coin $n=10$ times, get $k=7$ heads. What's the MLE for $p$ (probability of heads)? Solution: Step 1: Write the likelihood

Model: $X_i \sim \text{Bernoulli}(p)$ , so $P(X_i = 1) = p$ , $P(X_i = 0) = 1-p$
For our data (7 heads, 3 tails): $L(p) = p^7 (1-p)^3$

WHY? Each head contributes factor $p$ , each tail contributes $(1-p)$ .

Step 2: Take log-likelihood $\ell(p) = 7\log p + 3\log(1-p)$ WHY? $\log(ab) = \log a + \log b$ and $\log(a^k) = k\log a$ .

Step 3: Differentiate and set to zero $\frac{d\ell}{dp} = \frac{7}{p} - \frac{3}{1-p} = 0$

WHY this step? Chain rule: $\frac{d}{dp}\log p = \frac{1}{p}$ , $\frac{d}{dp}\log(1-p) = -\frac{1}{1-p}$ .

Step 4: Solve $\frac{7}{p} = \frac{3}{1-p} \implies 7(1-p) = 3p \implies 7 = 10p \implies p = 0.7$

Answer: $\hat{p}_{MLE} = \frac{7}{10} = \frac{k}{n}$ (intuitive!)

Worked example Example 2: MLE for Gaussian Mean (known variance)

Setup: Data $X = \{x_1, \ldots, x_n\}$ from $\mathcal{N}(\mu, \sigma^2)$ with known $\sigma^2=1$ . Find $\hat{\mu}_{MLE}$ .

Solution:

Step 1: Write likelihood $L(\mu) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}} e^{-\frac{(x_i - \mu)^2}{2}}$

WHY? This is the Gaussian PDF for each point.

Step 2: Log-likelihood $\ell(\mu) = \sum_{i=1}^{n} \left[-\frac{1}{2}\log(2\pi) - \frac{(x_i-\mu)^2}{2}\right] = -\frac{n}{2}\log(2\pi) - \frac{1}{2}\sum_{i=1}^{n}(x_i - \mu)^2$ WHY? $\log(e^a) = a$ and distribute the sum.

Step 3: Differentiate $\frac{d\ell}{d\mu} = -\frac{1}{2}\sum_{i=1}^{n} 2(x_i - \mu)(-1) = \sum_{i=1}^{n}(x_i - \mu)$

WHY? Chain rule: $\frac{d}{d\mu}(x_i - \mu)^2 = 2(x_i - \mu)(-1)$ .

Step 4: Set to zero $\sum_{i=1}^{n}(x_i - \mu) = 0 \implies \sum_{i=1}^{n}x_i = n\mu \implies \mu = \frac{1}{n}\sum_{i=1}^{n}x_i$ Answer: $\hat{\mu}_{MLE} = \bar{x}$ (the sample mean!)

Worked example Example 3: MLE for Gaussian Variance (known mean)

Setup: Same Gaussian data, but now $\mu=0$ is known, estimate $\sigma^2$ . Solution: Step 1: Log-likelihood $\ell(\sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}x_i^2$ Step 2: Differentiate with respect to $\sigma^2$ $\frac{d\ell}{d(\sigma^2)} = -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^{n}x_i^2$

WHY? $\frac{d}{dv}\log v = \frac{1}{v}$ and $\frac{d}{dv}\frac{1}{v} = -\frac{1}{v^2}$ . Step 3: Set to zero and solve $\frac{n}{2\sigma^2} = \frac{1}{2(\sigma^2)^2}\sum_{i=1}^{n}x_i^2 \implies nsigma^2 = \sum_{i=1}^{n}x_i^2$

Answer: $\hat{\sigma}^2_{MLE} = \frac{1}{n}\sum_{i=1}^{n}x_i^2$

Common Mistakes & Steel-manning

Properties of MLE

Formula Key Properties

Under regularity conditions:

Consistency: $\hat{\theta}_{MLE} \xrightarrow{P} \theta_0$ as $n \to \infty$

Meaning: With enough data, MLE converges to the true parameter

Asymptotic Normality: $\sqrt{n}(\hat{\theta}_{MLE} - \theta_0) \xrightarrow{d} \mathcal{N}(0, I(\theta_0)^{-1})$

Where $I(\theta)$ is the Fisher Information
Meaning: MLE is approximately Gaussian for large $n$

Efficiency: MLE achieves the Cramér-Rao lower bound asymptotically

Meaning: No other consistent estimator has lower asymptotic variance

Invariance: If $\hat{\theta}_{MLE}$ estimates $\theta$ , then $g(\hat{\theta}_{MLE})$ estimates $g(\theta)$

Example: If $\hat{\sigma}^2$ is MLE for variance, then $\sqrt{\hat{\sigma}^2}$ is MLE for standard deviation

Connection to Machine Learning

Linear Regression as MLE: If errors are Gaussian, minimizing squared loss = maximizing likelihood!

Given $y = \mathbf{w}^T\mathbf{x} + \epsilon$ where $\epsilon \sim \mathcal{N}(0, \sigma^2)$ : $p(y|\mathbf{x}, \mathbf{w}) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y - \mathbf{w}^T\mathbf{x})^2}{2\sigma^2}\right)$

Maximize log-likelihood: $\ell(\mathbf{w}) = -\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i - \mathbf{w}^T\mathbf{x}_i)^2 + \text{const}$

Maximizing this = minimizing $\sum(y_i - \mathbf{w}^T\mathbf{x}_i)^2$ = least squares!

Logistic Regression: Also derived via MLE with Bernoulli likelihood.

Neural Networks: Cross-entropy loss = negative log-likelihood for classification.

Mnemonic & Recall

Recall Explain to a 12-year-old

Imagine you have a bag of colored marbles, but you can't see inside. You pull out 10 marbles: 7 red, 3 blue.

Now the question is: "What percentage of the marbles in the bag are red?" You could guess anything—maybe 50% red, maybe 80% red. But which guess makes the most sense given what you pulled out?

MLE says: "Pick the percentage that would make it most likely to pull out exactly 7 red and 3 blue."

If the bag were70% red marbles, pulling out 7 red from 10 tries makes a lot of sense. If it were 10% red, pulling out 7 red would be super weird (very unlikely).

So MLE picks 70% red because that's the percentage that best explains what you actually saw. It's like working backwards from the result to figure out what the bag probably looks like inside.

Connections

Bayesian Estimation - MLE vs MAP (prior matters)
Fisher Information - Measures how much data tells us about parameters
Cramér-Rao Bound - Lower bound on estimator variance
Method of Moments - Alternative estimation technique
Loss Functions in ML - Cross-entropy, MSE as negative log-likelihoods
Gaussian Distribution - Most common MLE application
Likelihood Ratio Test - Hypothesis testing using MLE
EM Algorithm - MLE for latent variable models
Bias-Variance Tradeoff - MLE variance vs bias properties

#flashcards/ai-ml

What is Maximum Likelihood Estimation (MLE)? :: A method of estimating parameters of a probability distribution by finding the parameter values that maximize the likelihood function, making the observed data most probable under the assumed model.

What is the likelihood function for i.i.d. data? :: $L(\theta|X) = \prod_{i=1}^{n} p(x_i|\theta)$ , the product of individual probabilities for each data point.

Why do we use log-likelihood instead of likelihood?

(1) Products become sums (easier calculus), (2) Monotonic so argmax is preserved, (3) Numerically stable (prevents underflow with tiny probabilities).

What is the MLE for the parameter

p

of a Bernoulli distribution with

k

successes in

n

trials?

\hat{p}_{MLE} = \frac{k}{n}

, the sample proportion of successes.

What is the MLE for the mean

\mu

of a Gaussian distribution (known variance)?

\hat{\mu}_{MLE} = \bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_i

, the sample mean.

What are the three steps to find MLE analytically?

(1) Write log-likelihood

\ell(\theta)

, (2) Take derivative

\frac{d\ell}{d\theta}

, (3) Set to zero and solve for

\theta

What is the relationship between least squares regression and MLE?

Under Gaussian noise assumption, minimizing squared loss is equivalent to maximizing the likelihood function.

Is MLE always unbiased? :: No! MLE is asymptotically unbiased (consistent) but can be biased for finite samples. Example: MLE for Gaussian variance uses $\frac{1}{n}$ instead of $\frac{1}{n-1}$ .

What is the invariance property of MLE?

\hat{\theta}_{MLE}

is the MLE for

\theta

, then

g(\hat{\theta}_{MLE})

is the MLE for

g(\theta)

for any function

g

What does "asymptotic efficiency" of MLE mean?

MLE achieves the Cramér-Rao lower bound asymptotically, meaning no other consistent estimator has lower variance for large sample sizes.

Why do we assume i.i.d. (independent and identically distributed) samples in MLE?

So the joint probability factorizes as a product:

P(X|\theta) = \prod P(x_i|\theta)

, allowing us to write the likelihood function.

How is cross-entropy loss related to MLE?

Cross-entropy loss is the negative log-likelihood for classification problems. Minimizing cross-entropy = maximizing likelihood under a categorical/Bernoulli model.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Maximum Likelihood Estimation (MLE) ka matlab hai ki ap data dekh kar wo parameters dhoondte ho jouss data ko sabse zyada probable banate hain. Socho ki tumhare pas kuch observations hain—jaise 10 coin flips mein 7 heads aye. Ab sawaal hai: coin kitna biased hai? Matlab, heads ka probability kya hai?

MLE kehta hai: "Wo probability value choose karo jo is result (7 heads out of 10) ko sabse likely banaye." Agar coin 70% heads deta hai, toh 7 heads milna makes sense. Agar 10% heads deta hai, toh 7 heads milna bahut rare event hoga. Toh MLE bolega p=0.7 best fit hai.

Machine learning mein yeh concept har jagah use hota hai. Jab tum linear regression fit karte ho (squared loss minimize karte ho), actually tum Gaussian likelihood maximize kar rahe ho—wahi MLE hai! Jab neural network train karta ho cross-entropy loss se, woh bhi MLE hai Bernoulli/Categorical distribution ke liye. Matlab, ML ka pora foundation MLE pe based hai. Understanding this makes you understand WHY we use certain loss functions, WHY least squares works, aur WHY probabilistic models itne powerful hain.

Formula simple hai: log-likelihood likho (product ko sum banao), derivative lo, zero pe set karo, solve karo. Bass! Har distribution ke liye yeh process repeat karo—Bernoulli ho, Gaussian ho, ya koi aur—aur tumhe optimal parameters mil jayenge data se. Yeh AI-ML ke sabse fundamental tools mein se ek hai.

Overview

Core Intuition

The Mathematical Framework

Derivation from First Principles

WHY does the likelihood measure "fit"?

Worked Examples

Common Mistakes & Steel-manning

Properties of MLE

Connection to Machine Learning

Mnemonic & Recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections