Exercises — Statistical estimation — MLE, method of moments

Each item is a yes/no trial → Bernoulli($p$), unknown $p$ = probability of defect. The MLE of a Bernoulli probability is the sample proportion: $\hat{p}=\bar{x}=\frac{s}{n}=\frac{3}{10}=\mathrm{0.3.}$ Here $s=\sum x_i=3$ is the number of successes (defects).

For the Exponential, both MoM and MLE give $\hat{\lambda }=\frac{1}{\bar{x}}=\frac{1}{4}=0.25\text{ per minute}.$ The mean of an Exponential is $1/\lambda$, so by the invariance property (the MLE of a function is that function of the MLE) the estimated mean is $1/\hat{\lambda }=\bar{x}=4$ minutes.

Step 1 — average the data. Why? The MLE of an Exponential depends only on $\bar{x}$. $\bar{x}=\frac{2+4+4+6+9}{5}=\frac{25}{5}=5.$ Step 2 — apply the formula. $\hat{\lambda }=\frac{1}{\bar{x}}=\frac{1}{5}=\mathrm{0.2.}$

Step 1 — the mean. For a Normal, $\hat{\mu }=\bar{x}$. $\bar{x}=\frac{1+3+5+7+9}{5}=\frac{25}{5}=5.$ Step 2 — the variance. The MLE of ${\sigma }^2$ divides by $n$ (not $n-1$): ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2.$ Deviations from $5$: $-4,-2,0,2,4$. Squares: $16,4,0,4,16$, sum $=40$. ${\hat{\sigma }}^2=\frac{40}{5}=8.$

Step 1 — likelihood (multiply the pmf over all data; independence): $L(\lambda )={\prod }_{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}=\frac{e^{-n\lambda }{\lambda }^{\sum x_i}}{\prod x_i!}.$ Step 2 — log-likelihood. Why log? Because $\log$ turns the ugly product/power into a sum, and (being strictly increasing, see Logarithms) it has the same maximizer: $\ell (\lambda )=-n\lambda +\left(\sum x_i\right)\ln \lambda -\ln \left(\prod x_i!\right).$ The last term has no $\lambda$, so it's a constant for the maximization. Step 3 — differentiate, set to zero. Why? The peak of a smooth curve has zero slope: ${\ell }^{\prime }(\lambda )=-n+\frac{\sum x_i}{\lambda }=0\Rightarrow \hat{\lambda }=\frac{\sum x_i}{n}=\bar{x}.$ Step 4 — confirm it's a maximum (second derivative negative): ${\ell }^{\prime \prime }(\lambda )=-\frac{\sum x_i}{{\lambda }^2}<0.✓$ Apply: $\bar{x}=\frac{0+1+2+2+5}{5}=\frac{10}{5}=2$, so $\hat{\lambda }=2$.

MLE — why differentiation fails here. The likelihood is $L(\theta )={\prod }_{i=1}^n\frac{1}{\theta }\cdot 1\{0\le x_i\le \theta \}=\frac{1}{{\theta }^n}\text{but only if }\theta \ge {\max }_i{x}_i.$ The indicator $1\{\cdot \}$ is a switch: it is $1$ when the condition holds, $0$ otherwise. If any data point exceeds $\theta$, that factor is $0$, killing $L$. So $L(\theta )=0$ for $\theta <\max x_i$, and $L(\theta )=1/{\theta }^n$ for $\theta \ge \max x_i$. Look at Figure 1: the red curve jumps up at $\theta =\max x_i$ and then decreases ($1/{\theta }^n$ shrinks as $\theta$ grows). The highest allowed point is the leftmost allowed $\theta$, exactly at the maximum data value: ${\hat{\theta }}_{\text{MLE}}={\max }_i{x}_i=\mathrm{0.9.}$ Setting ${\ell }^{\prime }=0$ gives $-n/\theta =0$, which has no solution — the maximum sits at a boundary, found by inspection, not calculus.

MoM. The mean of $U(0,\theta )$ is the midpoint $\theta /2$. Set $\theta /2=\bar{x}$: $\bar{x}=\frac{0.2+0.5+0.9+0.7}{4}=\frac{2.3}{4}=0.575,{\hat{\theta }}_{\text{MoM}}=2\bar{x}=\mathrm{1.15.}$ Analysis. Both are legitimate, but MoM gave $1.15$ while we saw a value $0.9$ — consistent. The MLE ($0.9$) can never be smaller than the largest observation (good), yet it is biased low (it can never exceed the true $\theta$). MoM here is unbiased but can be smaller than a data point in unlucky samples, which is absurd for a Uniform's upper limit. This is the classic "MoM gives out-of-range values" warning.

Goal: compute $E[{\hat{\sigma }}^2]$ and compare to ${\sigma }^2$. A known identity (the "sum-of-squared-deviations" fact) gives $E\left[{\sum }_{i=1}^n(x_i-\bar{x}{)}^2\right]=(n-1){\sigma }^2.$ Why $n-1$ and not $n$? Because subtracting the estimated mean $\bar{x}$ (rather than the true mean) removes one "degree of freedom" — the deviations are forced to sum to zero, so one of them is not free. Dividing by $n$: $E[{\hat{\sigma }}^2]=\frac{(n-1){\sigma }^2}{n}=\frac{n-1}{n}{\sigma }^2<{\sigma }^2.$ So the MLE underestimates on average — its bias is $-{\sigma }^2/n$. The unbiased fix divides by $n-1$: $s^2=\frac{1}{n-1}\sum (x_i-\bar{x}{)}^2,E[s^2]={\sigma }^2.$ See Bias and Variance: the MLE trades a little bias for maximizing likelihood.

(a) MLE. Log-likelihood (using $\log$ to split the product; see Logarithms): $\ell (p)={\sum }_{i=1}^n[(x_i-1)\ln (1-p)+\ln p]=(\sum x_i-n)\ln (1-p)+n\ln p.$ Differentiate and set to zero: ${\ell }^{\prime }(p)=-\frac{\sum x_i-n}{1-p}+\frac{n}{p}=0.$ Multiply through by $p(1-p)$: $-p(\sum x_i-n)+n(1-p)=0\Rightarrow n=p\sum x_i\Rightarrow \hat{p}=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}.$ Second derivative ${\ell }^{\prime \prime }(p)=-\frac{\sum x_i-n}{(1-p{)}^2}-\frac{n}{p^2}<0$ (both terms negative since $\sum x_i\ge n$), confirming a maximum. (b) MoM. Mean $=1/p$, set $1/p=\bar{x}\Rightarrow \hat{p}=1/\bar{x}$. (c) They agree: $\hat{p}=1/\bar{x}$. As with the Exponential, when the mean is a clean function of the single parameter, MoM and MLE coincide. Apply: $\bar{x}=\frac{1+2+2+3+2}{5}=\frac{10}{5}=2$, so $\hat{p}=\frac{1}{2}=0.5$.

Bias: $\bar{x}$ is unbiased for the mean, $E[\bar{x}]=\mu$, so bias $=0$. Variance: $\text{Var}(\bar{x})={\sigma }^2/n=9/4=2.25$. MSE: $\text{MSE}=\text{Var}+{\text{bias}}^2=2.25+0^2=\mathrm{2.25.}$ The true mean $\mu =10$ was a red herring — for an unbiased estimator the MSE is just the variance, independent of where the true value sits.

Likelihood. Every data point must satisfy $x_i\ge \theta$, otherwise its density is $0$: $L(\theta )={\prod }_{i=1}^n{e}^{-(x_i-\theta )}\cdot 1\{x_i\ge \theta \}=e^{-\sum (x_i-\theta )}=e^{-\sum x_i}{e}^{n\theta },\text{valid only if }\theta \le {\min }_i{x}_i.$ Analysis of the shape (Figure 2). For fixed data, $e^{n\theta }$ increases in $\theta$ — bigger $\theta$ is better. But we're capped: $\theta$ cannot exceed the smallest data point, or some indicator flips to $0$. So push $\theta$ as high as allowed: ${\hat{\theta }}_{\text{MLE}}={\min }_i{x}_i=\mathrm{3.1.}$ Again calculus alone fails (${\ell }^{\prime }(\theta )=n>0$, no interior root); the maximum is at the boundary $\theta =\min x_i$.

Invariance says: if $\hat{\lambda }$ is the MLE of $\lambda$, then for any function $g$, the MLE of $g(\lambda )$ is $g(\hat{\lambda })$ — you just plug the hat in. Here $g(\lambda )=e^{-\lambda }$: $\widehat{P(X=0)}=e^{-\hat{\lambda }}=e^{-\bar{x}}=e^{-2}\approx \mathrm{0.1353.}$ No new maximization needed — that is the whole point of invariance. (Contrast this with Bayesian Estimation, where you'd instead average over a posterior distribution of $\lambda$ rather than plug in a single best value.)

(a) MLE gives ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2$, biased by factor $\frac{n-1}{n}$. But as $n\to \infty$, $\frac{n-1}{n}\to 1$, so the bias vanishes and its variance shrinks to $0$; therefore it is consistent (${\hat{\sigma }}^2\stackrel{P}{\to }{\sigma }^2$). Bias at finite $n$ does not prevent consistency in the limit. (b) The Cramér–Rao Lower Bound is a floor on the variance any unbiased estimator can achieve. MLEs are asymptotically efficient — for large $n$ they attain that floor. So MLE isn't perfect at small $n$ (it can be biased), but it becomes the best-possible unbiased estimator as data accumulates. That is precisely why we tolerate the small-sample bias.

The MLE of a Uniform $U(0,\theta )$ is ==${\max }_i{x}_i$, found by inspection/boundary, not differentiation==. The MLE of a shifted-Exponential's shift $\theta$ is ==${\min }_i{x}_i$==. The invariance property lets you estimate $e^{-\lambda }$ by ==$e^{-\hat{\lambda }}$== with no new maximization. For an unbiased estimator, $\text{MSE}$ equals the variance because the bias term is zero.