Exercises — Maximum likelihood estimation (MLE)

WHAT we do: every head multiplies in a factor $p$ (its probability), every tail multiplies in $(1-p)$. $L(p)=\underset{6}{\underbrace{p\cdot p\cdots p}}\cdot \underset{2}{\underbrace{(1-p)\cdot (1-p)}}=p^6(1-p{)}^2.$ WHY: the flips are independent, so the joint probability is the product of the single-flip probabilities. Order does not matter for the value of the product — only the counts (6 and 2) do. Take logs, using $\log (ab)=\log a+\log b$ and $\log (a^k)=k\log a$: $\ell (p)=6\log p+2\log (1-p).$

False. MLE maximises $P(\text{data}\mid \theta )=L(\theta )$ — the probability of the data given the parameter. The phrase "probability of the parameters given the data" is $P(\theta \mid \text{data})$, the posterior, which belongs to Bayesian Estimation / MAP, not MLE. MLE treats $\theta$ as a fixed unknown constant, not a random thing.

Step — differentiate. Using $\frac{d}{dp}\log p=\frac{1}{p}$ and $\frac{d}{dp}\log (1-p)=-\frac{1}{1-p}$: $\frac{d\ell }{dp}=\frac{6}{p}-\frac{2}{1-p}.$ WHY this tool (the derivative)? The peak of the $\ell$-hill is the one flat spot; the derivative is the slope, so setting it to $0$ locates the peak (see the figure above). Step — set to zero and solve. $\frac{6}{p}=\frac{2}{1-p}\Rightarrow 6(1-p)=2p\Rightarrow 6=8p\Rightarrow \boxed{\hat{p}=\frac{6}{8}=\mathrm{0.75.}}$ Sanity check: $\hat{p}=\frac{\text{heads}}{\text{flips}}=\frac{6}{8}$ — exactly the observed frequency, as intuition demands.

The general Gaussian-mean result (parent note, Example 2) is ${\hat{\mu }}_{\text{MLE}}=\bar{x}=\frac{1}{n}\sum x_i$. WHY the mean: maximising the log-likelihood means minimising $\sum (x_i-\mu {)}^2$ (the rest is constant in $\mu$); the point that minimises total squared distance to a set of numbers is their average. $\hat{\mu }=\frac{4+7+9+10}{4}=\frac{30}{4}=\boxed{\mathrm{7.5.}}$ Note ${\sigma }^2$ never entered — for the mean, the known variance cancels out.

${\hat{\sigma }}^2=\frac{2^2+(-1{)}^2+3^2+(-4{)}^2}{4}=\frac{4+1+9+16}{4}=\frac{30}{4}=\boxed{\mathrm{7.5.}}$ Why differentiate in ${\sigma }^2$, not $\sigma$? The log-likelihood contains ${\sigma }^2$ only through the two clean pieces $\log ({\sigma }^2)$ and $\frac{1}{{\sigma }^2}$. Treat the whole block $v={\sigma }^2$ as one variable: $\frac{d}{dv}\log v=\frac{1}{v}$ and $\frac{d}{dv}\frac{1}{v}=-\frac{1}{v^2}$ are painless. Differentiating in $\sigma$ forces extra chain-rule factors and gives the identical answer with more mess. (By invariance of the MLE, $\hat{\sigma }=\sqrt{{\hat{\sigma }}^2}$ regardless.)

Log-likelihood. $\log p(x\mid \lambda )=\log \lambda -\lambda x$, so $\ell (\lambda )={\sum }_{i=1}^n(\log \lambda -\lambda x_i)=n\log \lambda -\lambda {\sum }_{i=1}^n{x}_i.$ Differentiate & solve. $\frac{d\ell }{d\lambda }=\frac{n}{\lambda }-\sum x_i=0\Rightarrow \hat{\lambda }=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}.$ Curvature check: ${\ell }^{\prime \prime }=-\frac{n}{{\lambda }^2}<0$ ⇒ maximum. For $x=\{2,4,6\}$: $\sum x_i=12$, $n=3$, $\hat{\lambda }=\frac{3}{12}=\boxed{\mathrm{0.25.}}$ Intuition: a large total wait ⇒ small rate. The rate is the reciprocal of the average wait — dimensionally and physically sensible.

Log-likelihood in two unknowns: $\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log ({\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2.$ Partial in $\mu$ (only the last term depends on $\mu$): $\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu )=0\Rightarrow \hat{\mu }=\bar{x}.$ The positive factor $\frac{1}{{\sigma }^2}$ can never make the sum zero, so $\hat{\mu }=\bar{x}$ regardless of ${\sigma }^2$ — that is why we can solve $\mu$ first. Partial in ${\sigma }^2$, then substitute $\mu =\bar{x}$: $\frac{\partial \ell }{\partial ({\sigma }^2)}=-\frac{n}{2{\sigma }^2}+\frac{1}{2({\sigma }^2{)}^2}\sum (x_i-\hat{\mu }{)}^2=0\Rightarrow {\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2.$ This is a profile likelihood move: freeze the easy parameter at its optimum, optimise the other. The EM Algorithm generalises this "optimise one block, then the next" idea to hidden variables.

$\hat{\mu }=\bar{x}=\frac{2+4+4+4+6}{5}=\frac{20}{5}=4.$ Squared deviations: $(2-4{)}^2=4,0,0,0,(6-4{)}^2=4$, sum $=8$. ${\hat{\sigma }}_{\text{MLE}}^2=\frac{8}{5}=\boxed{1.6},s^2=\frac{8}{4}=\boxed{\mathrm{2.0.}}$ Why they differ: the MLE reuses $\bar{x}$ — estimated from the same data — so the deviations are slightly too small (the data hugs its own mean). Dividing by $n-1$ instead of $n$ corrects this. MLE variance is biased low but consistent (bias $\to 0$ as $n\to \infty$). See Bias-Variance Tradeoff.

Log-likelihood (for a valid $\theta$, i.e. $\theta \ge {\max }_i{x}_i$): $\ell (\theta )={\sum }_{i=1}^n(\log (2x_i)-2\log \theta )=\text{const}-2n\log \theta .$ Differentiate: $\frac{d\ell }{d\theta }=-\frac{2n}{\theta }$, which is always negative — it never equals zero! So there is no interior stationary point. WHY the calculus "fails": $\ell$ is strictly decreasing in $\theta$, so we want $\theta$ as small as possible. But there is a hard wall: every observed $x_i$ must satisfy $x_i\le \theta$, so $\theta \ge {\max }_i{x}_i$. Pushing $\theta$ below $\max x_i$ makes the likelihood $0$ (that data would be impossible). The maximum sits on the boundary (see figure): $\boxed{{\hat{\theta }}_{\text{MLE}}=\underset{i}{\max }{x}_i.}$ This is a support boundary MLE — the peak is a corner, not a smooth summit. Compare with Method of Moments, which would instead solve $\mathbb{E}[X]=\frac{2\theta }{3}=\bar{x}\Rightarrow \theta =\frac{3\bar{x}}{2}$ — a different estimator, showing MLE and MoM need not agree.

Asymptotic normality (parent note property 2) says the variance of $\hat{\lambda }$ is approximately $I(\lambda {)}^{-1}=\frac{{\lambda }^2}{n}$, so the standard error is $\text{SE}(\hat{\lambda })\approx \sqrt{\frac{{\hat{\lambda }}^2}{n}}=\frac{\hat{\lambda }}{\sqrt{n}}=\frac{0.25}{\sqrt{3}}\approx \boxed{\mathrm{0.1443.}}$ What it means: this is the width of the Gaussian bell describing our uncertainty about $\lambda$. By the Cramér-Rao Bound, no unbiased estimator can beat this variance asymptotically — MLE is efficient. (With $n=3$ this is only a rough approximation; the guarantee is a large-$n$ statement.)

MLE maximises $P(\text{data}\mid \theta )$ or $P(\theta \mid \text{data})$? ::: $P(\text{data}\mid \theta )$, the likelihood. Bernoulli MLE for 6 heads in 8 flips ::: $\hat{p}=0.75$. Gaussian MLE mean of $\{4,7,9,10\}$ ::: $\hat{\mu }=7.5$. Exponential MLE for $\{2,4,6\}$ ::: $\hat{\lambda }=0.25$. MLE variance of $\{2,4,4,4,6\}$ ($\frac{1}{n}$) ::: ${\hat{\sigma }}^2=1.6$; unbiased $s^2=2.0$. Triangular-on-$[0,\theta ]$ MLE ::: $\hat{\theta }={\max }_i{x}_i$ (boundary!). Why divide by $n-1$ for unbiased variance? ::: estimating the mean costs one degree of freedom.