Maximum likelihood estimation (MLE)

1.3.16 · AI-ML › Probability & Statistics

Overview

Maximum Likelihood Estimation ek method hai probability distribution ke parameters estimate karne ki, jisme hum ek likelihood function ko maximize karte hain, taaki assumed statistical model ke under observed data sabse zyada probable ho.

YEH KYUN ZARURI HAI: ML mein hume constantly models fit karne padte hain (jaise features ke liye Gaussian distributions, ya logistic regression coefficients). MLE humein ek principled, mathematically rigorous tarika deta hai data se "best" parameters dhundhne ka.

Core Intuition

The Mathematical Framework

Derivation from First Principles

Likelihood "fit" kyun measure karta hai?

Shuru karo jo hum actually chahte hain se: Data $X$ diya gaya, wo $\theta$ dhundho jo $X$ ko sabse zyada probable banaye.

Step 1: Parameters $\theta$ diye gaye, data $X$ observe karne ki probability hai $P(X|\theta )$.

Step 2: Independent samples ke liye (i.i.d. assumption): $P(X|\theta )=P(x_1,x_2,\dots ,x_n|\theta )={\prod }_{i=1}^{n}P(x_i|\theta )$

KYUN? Independence ka matlab hai joint probability = individual probabilities ka product.

Step 3: Hum ise $\theta$ ka function treat karte hain (data fixed hai), aur ise likelihood $L(\theta |X)$ kehte hain.

Step 4: Logarithm lo (kyunki $\log$ monotonic hai, $\arg \max$ nahi badalta): $\ell (\theta )=\log L(\theta )={\sum }_{i=1}^n\log p(x_i|\theta )$

YEH STEP KYUN? Mathematically cleaner (sums vs products), numerically stable (underflow se bachata hai), derivatives aasaan hoti hain.

Step 5: Calculus se maximum dhundho: $\frac{\partial \ell }{\partial \theta }=0$

$\theta$ ke liye solve karo → yahi ${\hat{\theta }}_{MLE}$ hai.

Worked Examples

Common Mistakes & Steel-manning

Properties of MLE

Connection to Machine Learning

Linear Regression as MLE: Agar errors Gaussian hain, to squared loss minimize karna = likelihood maximize karna!

Diya gaya $y={\mathbf{w}}^T\mathbf{x}+ϵ$ jahan $ϵ\sim \mathcal{N}(0,{\sigma }^2)$: $p(y|\mathbf{x},\mathbf{w})=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(y-{\mathbf{w}}^T\mathbf{x}{)}^2}{2{\sigma }^2}\right)$

Log-likelihood maximize karo: $\ell (\mathbf{w})=-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(y_i-{\mathbf{w}}^T{\mathbf{x}}_i{)}^2+\text{const}$

Ise maximize karna = $\sum (y_i-{\mathbf{w}}^T{\mathbf{x}}_i{)}^2$ minimize karna = least squares!

Logistic Regression: Yeh bhi Bernoulli likelihood ke saath MLE se derive hota hai.

Neural Networks: Cross-entropy loss = classification ke liye negative log-likelihood.

Mnemonic & Recall

Connections

• Bayesian Estimation - MLE vs MAP (prior matters)
• Fisher Information - Measure karta hai ki data parameters ke baare mein kitna bata sakta hai
• Cramér-Rao Bound - Estimator variance ki lower bound
• Method of Moments - Alternative estimation technique
• Loss Functions in ML - Cross-entropy, MSE as negative log-likelihoods
• Gaussian Distribution - Sabse common MLE application
• Likelihood Ratio Test - MLE use karke hypothesis testing
• EM Algorithm - Latent variable models ke liye MLE
• Bias-Variance Tradeoff - MLE variance vs bias properties

#flashcards/ai-ml

Maximum Likelihood Estimation (MLE) kya hai? :: Ek method jo probability distribution ke parameters estimate karta hai, un parameter values dhundh ke jo likelihood function maximize karein, taaki observed data assumed model ke under sabse zyada probable ho.

i.i.d. data ke liye likelihood function kya hai? :: $L(\theta |X)={\prod }_{i=1}^{n}p(x_i|\theta )$, har data point ki individual probabilities ka product.

Kya MLE hamesha unbiased hota hai? :: Nahi! MLE asymptotically unbiased (consistent) hai lekin finite samples ke liye biased ho sakta hai. Example: Gaussian variance ke liye MLE $\frac{1}{n}$ use karta hai $\frac{1}{n-1}$ ki jagah.

Concept Map