Worked examples — Statistical estimation — MLE, method of moments

Before we start, four tiny reminders so no symbol is unearned:

• $x_1,\dots ,x_n$ are the observed numbers (the data). $n$ is how many there are.
• $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$ is the sample mean — just "add them up, divide by how many".
• $\hat{\theta }$ (theta with a hat) means "our guess of the true hidden number $\theta$". The hat always means estimated from data.
• $L(\theta )={\prod }_{i=1}^{n}f(x_i;\theta )$ is the likelihood: you multiply the density (or probability) of every data point, treating the data as fixed and $\theta$ as the dial you turn. It answers "how believable is this $\theta$, given what I saw?" — see the parent note. Because a product of many small numbers is nasty to differentiate, we log it:
• $\ell (\theta )=\log L(\theta )={\sum }_{i=1}^n\log f(x_i;\theta )$ is the log-likelihood. Since $\log$ is strictly increasing, the $\theta$ that maximizes $\ell$ is the same $\theta$ that maximizes $L$ — but the sum is far easier to handle than the product. See Logarithms.

---

The scenario matrix

Every estimation problem you'll meet falls into one of these case classes. The table lists them and points at the worked example that nails each one.

#	Case class	What makes it tricky	Example that covers it
A	Standard continuous, 1 parameter	baseline — differentiate & solve	Ex 1 (Exponential rate)
B	Discrete count data	pmf not density; factorials	Ex 2 (Poisson)
C	Two parameters at once	system of equations	Ex 3 (Normal $\mu ,{\sigma }^2$)
D	Boundary / support parameter	derivative = 0 gives NO answer	Ex 4 (Uniform $U(0,\theta )$)
E	MoM disagrees with MLE	two recipes, two answers	Ex 5 (Uniform, MoM vs MLE)
F	Degenerate sample (all $x_i$ equal, or a zero)	estimator blows up or pins to edge	Ex 6 (Exponential with $\bar{x}\to 0$)
G	MoM gives illegal (out-of-range) value	answer violates parameter constraints	Ex 7 (Binomial, unknown $p$ with fixed $m$)
H	Real-world word problem	translate English → likelihood	Ex 8 (call-centre waiting times)
I	Bias check / exam twist	is the "obvious" estimator honest?	Ex 9 (variance divisor $n$ vs $n-1$)
J	Invariance twist	asked for $g(\theta )$, not $\theta$	Ex 10 (median lifetime from $\hat{\lambda }$)

We now walk every cell.

---

Ex 1 — Case A: Exponential rate (baseline warm-up)

1. Likelihood, then log-likelihood. $L(\lambda )=\prod \lambda e^{-\lambda x_i}={\lambda }^n{e}^{-\lambda \sum x_i}$, so $\ell (\lambda )=\log L=n\ln \lambda -\lambda \sum x_i$. Why this step? Independence makes $L$ a product (our definition above); logging turns that product into a friendly sum.
2. Differentiate, set to zero (the score equation). ${\ell }^{\prime }(\lambda )=\frac{n}{\lambda }-\sum x_i=0$. Why this step? The peak of a smooth hill has zero slope; solving ${\ell }^{\prime }=0$ locates that peak.
3. Solve. $\hat{\lambda }=\frac{n}{\sum x_i}=\frac{1}{\bar{x}}$. Here $\sum x_i=10$, $n=4$, so $\hat{\lambda }=\frac{4}{10}=0.4$ per year. Why this step? Pure algebra — isolate $\lambda$.
4. Confirm it's a maximum. ${\ell }^{\prime \prime }(\lambda )=-n/{\lambda }^2<0$ for all $\lambda >0$. Why this step? A zero-slope point could be a valley; the negative second derivative proves it's a hilltop.

---

Ex 2 — Case B: Poisson counts (discrete data)

1. Likelihood (a product of pmf values). $L(\lambda )={\prod }_{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}=\frac{e^{-n\lambda }{\lambda }^{\sum x_i}}{\prod x_i!}$. Why this step? For discrete data, $f$ in our likelihood definition is just a probability; we still multiply because trials are independent.
2. Log-likelihood. $\ell (\lambda )=-n\lambda +(\sum x_i)\ln \lambda -\ln \prod x_i!$. Why this step? The last term is a constant in $\lambda$ (no $\lambda$ inside), so it will vanish when we differentiate — we keep it only for honesty. See Logarithms for $\ln (ab)=\ln a+\ln b$.
3. Differentiate & solve. ${\ell }^{\prime }(\lambda )=-n+\frac{\sum x_i}{\lambda }=0\Rightarrow \hat{\lambda }=\frac{\sum x_i}{n}=\bar{x}$. Here $\sum x_i=10$, $n=5$, so $\hat{\lambda }=2$. Why this step? Peak has zero slope; constant term dropped out as promised.
4. Second-derivative check. ${\ell }^{\prime \prime }(\lambda )=-\frac{\sum x_i}{{\lambda }^2}<0$ (since $\sum x_i>0$). True maximum. ✓

---

Ex 3 — Case C: Normal, two parameters at once

1. Log-likelihood. $\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\ln (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}\sum (x_i-\mu {)}^2$. Why this step? Parent note's normal density, logged.
2. First score: differentiate in $\mu$. $\frac{\partial \ell }{\partial \mu }=\frac{1}{{\sigma }^2}\sum (x_i-\mu )=0\Rightarrow \hat{\mu }=\bar{x}=13$. Why this step? With two unknowns we hold ${\sigma }^2$ fixed and slide $\mu$ to the top — a partial derivative. The ${\sigma }^2$ multiplier is positive so it doesn't change where the zero is.
3. Second score: differentiate in ${\sigma }^2$. $\frac{\partial \ell }{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum (x_i-\mu {)}^2=0\Rightarrow {\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\hat{\mu }{)}^2$. Why this step? Now slide ${\sigma }^2$ with $\mu$ pinned at $\hat{\mu }$. Plug $\hat{\mu }=13$: deviations are $(-3,-1,1,3)$, squares $9+1+1+9=20$, so ${\hat{\sigma }}^2=20/4=5$.
4. Confirm it's a maximum (the multivariate check). The Hessian (matrix of second derivatives) at $(\hat{\mu },{\hat{\sigma }}^2)$ is diagonal: ${\partial }^2\ell /\partial {\mu }^2=-n/{\sigma }^2<0$ and ${\partial }^2\ell /\partial ({\sigma }^2{)}^2=-n/(2{\sigma }^4)<0$ (evaluated at ${\hat{\sigma }}^2$, using $\sum (x_i-\hat{\mu }{)}^2=n{\hat{\sigma }}^2$), with zero cross-term at the stationary point. Why this step? In two dimensions a zero-gradient point could be a saddle. A negative-definite Hessian (here: both diagonal entries negative and cross-term zero, so both eigenvalues negative) proves it's a genuine peak, not a saddle or valley.

---

Ex 4 — Case D: Boundary parameter, where differentiation LIES

Here the geometry, not the algebra, does the work — so study the figure carefully before reading the steps.

How to read this figure. The horizontal axis is $\theta$, the candidate value of the upper bound. The vertical axis is the likelihood $L(\theta )$ from our definition at the top of the page. Three things are drawn:

• The coral shaded band on the left ($\theta <5$) is the illegal region: if $\theta$ is smaller than the biggest data value $5$, then that data point falls outside $[0,\theta ]$, its density is $0$, and the whole product collapses to $L=0$.
• The lavender curve ($\theta \ge 5$) is $L(\theta )={\theta }^{-n}$, which slopes steadily downward — bigger $\theta$ always means smaller likelihood.
• The red dot marks the peak: it sits exactly at the left edge of the legal region, $\hat{\theta }=5$. There is no smooth "top of a hill" here; the maximum is a corner.
• The mint ticks along the bottom are the four data points $(2,3,4,5)$; notice the peak lands on the largest one.

1. Likelihood. $L(\theta )=\prod \frac{1}{\theta }=\frac{1}{{\theta }^n}$ — but only if every $x_i\le \theta$; otherwise some factor is $0$ and $L=0$ (the coral band in the figure). Why this step? The support ($0\le x\le \theta$) is where the tricky part hides. If $\theta$ is smaller than the biggest data point, that point was "impossible" under the model, killing the whole product.
2. Log-likelihood, written out. For $\theta \ge \max x_i$ we have $L(\theta )={\theta }^{-n}$, so $\ell (\theta )=\ln ({\theta }^{-n})=-n\ln \theta$. For $\theta <\max x_i$, $L=0$ and $\ell =-\infty$ (the log of zero). Why this step? We need $\ell$ explicitly before we can talk about its derivative. Using $\ln (a^b)=b\ln a$ (see Logarithms) turns ${\theta }^{-n}$ into the clean $-n\ln \theta$.
3. Look at the picture, not the derivative. Differentiating the legal piece gives ${\ell }^{\prime }(\theta )=\frac{d}{d\theta }(-n\ln \theta )=-\frac{n}{\theta }$, which is never zero for $\theta >0$. So the score equation has no solution. Instead read the lavender curve: $\ell (\theta )=-n\ln \theta$ is strictly decreasing, so $L$ wants $\theta$ as small as possible — but it's forced to be at least $\max x_i=5$. Why this step? Differentiation cannot find this maximum; the peak sits at the edge of the allowed region, and only the graph reveals it.
4. Read off the maximizer. The largest $L$ with $\theta \ge \max x_i$ occurs at the smallest legal $\theta$: $\hat{\theta }={\max }_i{x}_i=5$ (the red dot). Why this step? On a monotone-decreasing curve, restricted to $\theta \ge 5$, the top is at the left end.

---

Ex 5 — Case E: MoM vs MLE disagree (same uniform)

1. Population mean. $E[X]=\theta /2$ (centre of the interval $[0,\theta ]$). Why this step? One parameter → match one moment (the first). See Law of Large Numbers for why $\bar{x}\to E[X]$.
2. Match to sample mean. $\frac{\hat{\theta }}{2}=\bar{x}\Rightarrow {\hat{\theta }}_{\text{MoM}}=2\bar{x}$. Here $\bar{x}=(2+5+3+4)/4=3.5$, so ${\hat{\theta }}_{\text{MoM}}=7$. Why this step? Set theoretical moment equal to measured moment, solve.
3. Compare. MLE said $5$, MoM says $7$. Why this matters? Both are "reasonable", but MoM here is more variable and can even land below $\max x_i$ on other samples — which would be logically impossible (you can't have $\theta <5$ when you already saw a $5$).

---

Ex 6 — Case F: Degenerate sample (estimator blows up)

1. Compute directly. $\bar{x}=0.01$, so $\hat{\lambda }=1/0.01=100$ per unit time. Why this step? Just apply Ex 1's formula.
2. Limiting behaviour. As $\bar{x}\to 0^{+}$, $\hat{\lambda }=1/\bar{x}\to +\infty$. Why this step? A pile of near-zero lifetimes means "things fail almost instantly" → an enormous failure rate. The model responds sensibly, but the estimate is unbounded.
3. Guard against the true degenerate case $\bar{x}=0$. If every $x_i=0$ exactly, $\hat{\lambda }$ is undefined ($1/0$). Physically that means "no time ever elapses" — the exponential model has broken down; you must reject the model, not report $\infty$. Why this step? Covering the genuinely-degenerate cell of the matrix: a zero-mean sample is a modelling red flag, not a number.

---

Ex 7 — Case G: MoM produces an illegal (out-of-range) value

1. Honest MoM (first moment). $E[X]=mp$, so $\hat{p}=\frac{\bar{x}}{m}$. Here $\bar{x}=(6+7+8)/3=7$, $m=8$, giving $\hat{p}=7/8=0.875\in [0,1]$. ✓ Why this step? One parameter → match the first moment; the mean equation respects $p\le 1$ because $\bar{x}\le m$ always.
2. Flawed MoM (match the variance instead). The sample variance (dividing by $n$) is $\overline{x^2}-{\bar{x}}^2$. Here $\overline{x^2}=(36+49+64)/3=149/3\approx 49.667$, and ${\bar{x}}^2=49$, so the sample variance is $149/3-49=2/3\approx 0.667$. Set the model variance equal to it: $mp(1-p)=2/3$, i.e. $8p(1-p)=2/3\Rightarrow p(1-p)=1/12\approx 0.0833$. Why this step? This is a legitimate-looking alternative moment equation — but variance ignores how close $\bar{x}$ sits to the ceiling $m$, so it can send $p$ somewhere silly.
3. Solve the quadratic. $p-p^2=\frac{1}{12}\Rightarrow p^2-p+\frac{1}{12}=0\Rightarrow p=\frac{1\pm \sqrt{1-\frac{1}{3}}}{2}=\frac{1\pm \sqrt{2/3}}{2}$. Numerically the two roots are $p\approx 0.908$ and $p\approx 0.092$. Why this step? A variance equation is quadratic in $p$, so it hands back two candidate answers — already worse than the mean equation's single clean root. The larger root $0.908$ predicts a mean $mp=8(0.908)\approx 7.27$, disagreeing with the mean-based $\hat{p}$; you now have to choose, and MoM gives no principled way to.
4. Expose the genuinely illegal case. The model variance $mp(1-p)$ can never exceed its maximum $m\cdot \frac{1}{4}=2$ (attained at $p=0.5$). So if a sample had variance larger than $2$, the equation $8p(1-p)=v$ would have $1-\frac{v}{2}<0$ under the square root — no real solution at all, i.e. $\hat{p}$ would come out complex, hence utterly illegal. Why this matters? This is the cell where MoM earns its bad name: the "wrong" moment can hand you an ambiguous pair of roots or, when the sample variance overshoots the model ceiling, no legal value whatsoever. MLE, which respects the support and the full likelihood, can never return an out-of-range answer.

---

Ex 8 — Case H: Real-world word problem

1. Translate English to a model. "Time between random events" → Exponential($\lambda$), where $\lambda$ = calls per minute, density $f(x;\lambda )=\lambda e^{-\lambda x}$. Why this step? Choosing the distribution is the real skill; the algebra is Ex 1 again. See Probability Distributions.
2. Compute $\bar{x}$. $\bar{x}=(0.5+1.5+2.0+1.0+3.0)/5=8/5=1.6$ minutes. Why this step? The MLE for an exponential depends on the data only through the sample mean, so this is the one number we need.
3. Log-likelihood, then solve (reusing Ex 1). $\ell (\lambda )=n\ln \lambda -\lambda \sum x_i$; setting ${\ell }^{\prime }(\lambda )=n/\lambda -\sum x_i=0$ gives $\hat{\lambda }=1/\bar{x}$. Why this step? Identical machinery to Ex 1; we quote the result rather than re-derive it from scratch.
4. (a) Rate. $\hat{\lambda }=1/\bar{x}=1/1.6=0.625$ calls per minute. Why this step? Direct substitution into $\hat{\lambda }=1/\bar{x}$.
5. (b) Mean wait. The exponential mean is $1/\lambda$, so the estimated mean wait is $\widehat{1/\lambda }=1/\hat{\lambda }=\bar{x}=1.6$ minutes. Why this step? By MLE invariance, the MLE of $g(\lambda )$ is $g(\hat{\lambda })$ (parent note) — here $g(\lambda )=1/\lambda$.

---

Ex 9 — Case I: Bias check / exam twist

How to read this figure. Imagine repeating the whole experiment thousands of times; each time you compute a variance estimate. The two coloured humps show how those estimates spread out. The coral hump is the MLE (dividing by $n$): its centre sits left of the dashed vertical line marking the true ${\sigma }^2$ — on average it under-shoots. The mint hump is the unbiased estimator (dividing by $n-1$): its centre sits on the dashed line. That leftward shift of the coral hump is exactly the bias.

1. The bias fact. $E[{\hat{\sigma }}_{\text{MLE}}^2]=\frac{n-1}{n}{\sigma }^2$, so the MLE underestimates on average. Why this step? Splitting into variance + bias${}^2$ shows dividing by $n$ shrinks the estimate (see Bias and Variance). The picture shows the coral cloud sitting left of the true ${\sigma }^2$.
2. Fix the divisor. Multiply by $\frac{n}{n-1}$: the unbiased estimator is $s^2=\frac{1}{n-1}\sum (x_i-\bar{x}{)}^2$. Why this step? $\frac{n}{n-1}\cdot \frac{n-1}{n}{\sigma }^2={\sigma }^2$ — the correction exactly cancels the shrink.
3. Numbers. Sum of squared deviations $=20$ (from Ex 3). Unbiased: $s^2=20/(4-1)=20/3\approx 6.667$. Why this step? Same numerator, divide by $n-1=3$ instead of $n=4$.

---

Ex 10 — Case J: Invariance twist (asked for $g(\theta )$)

1. Invariance shortcut. If $\hat{\lambda }$ is the MLE of $\lambda$, then for any function $g$, $g(\hat{\lambda })$ is the MLE of $g(\lambda )$ (parent note). Why this step? This saves us re-differentiating a whole new likelihood — we just plug $\hat{\lambda }$ into the median formula.
2. The function. $g(\lambda )=\frac{\ln 2}{\lambda }$ (median as a function of rate). See Logarithms for $\ln 2\approx 0.6931$. Why this step? Identify $g$, then apply invariance.
3. Plug in. $\hat{m}=\frac{\ln 2}{\hat{\lambda }}=\frac{0.6931}{0.4}\approx 1.733$ years. Why this step? Direct substitution of $\hat{\lambda }=0.4$.

---

Recall One-line summary of every cell

Differentiate & solve for smooth 1-param (A), same for counts (B) and two params (C); inspect the boundary for support parameters (D); MoM & MLE can disagree (E); guard degenerate samples where estimates blow up (F); MoM can go out of range (G); word problems are model-choice + Ex 1 (H); the MLE variance is biased, fix with $n-1$ (I); reuse $\hat{\theta }$ via invariance for $g(\theta )$ (J).

Reveal-yourself checks: