Maximum likelihood estimation (MLE): Worked examples

This page is the "training ground" for MLE. The parent note showed you the machine. Here we run it on every kind of input the machine can meet: nice cases, sign-flip cases, zero cases, degenerate cases, a word problem, and an exam twist. By the end you should never meet an MLE problem whose shape you haven't seen.

The scenario matrix

Every cell below is a distinct kind of situation. The examples after it are each tagged with the cell(s) they cover.

Cell	What makes it special	Covered by
A. Interior maximum, bounded parameter	Parameter lives in $[0, 1]$ ; answer is inside, found by derivative $= 0$	Ex 1
B. Degenerate data (all same)	Data hits an extreme (all heads / all tails)	Ex 2
C. Positive-only parameter	Parameter must be $> 0$ (a rate/scale)	Ex 3
D. Boundary maximum	Derivative is never zero; max sits at edge of the support	Ex 4
E. Two parameters at once	Solve $μ$ and $σ^{2}$ jointly	Ex 5
F. Bias check / limiting behaviour	Is the estimator unbiased? What happens as $n \to \infty$ ?	Ex 6
G. Real-world word problem	Translate messy story into a model, then estimate	Ex 7
H. Exam twist (reparametrisation / invariance)	MLE of a function of the parameter	Ex 8

Before the algebra, one picture of the whole idea: the log-likelihood is a curved landscape over parameter values, and MLE walks to its top.

Cell A — Interior maximum, bounded parameter

Worked example Example 1 — Bernoulli with a normal mix of outcomes

Statement: You test a spam filter on $n = 12$ emails; it correctly flags $k = 9$ . Each email is an independent success/failure. Estimate $p$ , the true success probability.

Forecast: Guess the answer before reading. (Hint: what fraction succeeded?)

Step 1 — Write the likelihood. $L (p) = p^{9} (1 - p)^{3}$ Why this step? Each success multiplies in a factor $p$ , each failure a factor $1 - p$ ; independence lets us multiply them.

Step 2 — Take the log. $ℓ (p) = 9 lo g p + 3 lo g (1 - p)$ Why this step? Turns the product into a sum so the derivative is a clean sum, and avoids tiny numbers underflowing to zero.

Step 3 — Differentiate and set to zero. $ℓ^{'} (p) = \frac{9}{p} - \frac{3}{1 - p} = 0$ Why this step? The top of a smooth hill has zero slope; solving $ℓ^{'} = 0$ finds that flat spot.

Step 4 — Solve. $9 (1 - p) = 3 p ⟹ 9 = 12 p ⟹ \overset{p}{^} = \frac{9}{12} = 0.75$ Why this step? Cross-multiplying clears the fractions; the answer is just $k / n$ .

Verify: $ℓ^{''} (p) = - \frac{9}{p ^{2}} - \frac{3}{( 1 - p ) ^{2}} < 0$ everywhere, so it is a maximum, not a minimum. Sanity: $0.75 \in [0, 1]$ ✓, and it equals the observed fraction $9/12$ ✓.

Cell B — Degenerate data (all outcomes identical)

Worked example Example 2 — Every flip a head

Statement: A coin is flipped $n = 5$ times and lands heads all 5 times. What is $\overset{p}{^}$ ?

Forecast: Will the derivative trick even work here?

Step 1 — Likelihood. With $k = 5$ heads, $0$ tails: $L (p) = p^{5} (1 - p)^{0} = p^{5}$ Why this step? The $(1 - p)$ factor has exponent $0$ , so it vanishes — there is no tail evidence at all.

Step 2 — Log-likelihood. $ℓ (p) = 5 lo g p$ Why this step? Same log rule; now only one term survives.

Step 3 — Look at the slope. $ℓ^{'} (p) = \frac{5}{p} > 0 for all p \in (0, 1]$ Why this step? The slope is never zero — the derivative-equals-zero recipe has no interior solution here. This is the warning sign of a boundary answer.

Step 4 — Push to the edge. Since $ℓ$ strictly increases, the maximum sits at the largest allowed $p$ : $\overset{p}{^} = 1$ Why this step? When the hill only ever slopes upward, you climb to the boundary $p = 1$ .

Verify: $L (1) = 1^{5} = 1$ , the largest a probability can be; any $p < 1$ gives $L < 1$ . Formula check: $k / n = 5/5 = 1$ ✓. The plain formula $\overset{p}{^} = k / n$ still lands on the boundary correctly.

Cell C — Positive-only parameter (a rate)

Worked example Example 3 — Exponential waiting times

Statement: Server response times (seconds) $x_{1}, \dots, x_{n}$ are modelled as $Exponential (λ)$ with density $p (x ∣ λ) = λ e^{- λ x}$ for $x > 0$ . You observe $n = 4$ times: $2, 1, 3, 2$ (mean $= 2$ ). Estimate the rate $λ > 0$ .

Forecast: The mean waiting time is $2$ s. Guess what $\hat{λ}$ should be.

Step 1 — Log-likelihood. $ℓ (λ) = \sum_{i = 1}^{n} (lo g λ - λ x_{i}) = n lo g λ - λ \sum_{i = 1}^{n} x_{i}$ Why this step? $lo g (λ e^{- λ x}) = lo g λ - λ x$ ; summing over points gives the total.

Step 2 — Differentiate in $λ$ . $ℓ^{'} (λ) = \frac{n}{λ} - \sum_{i = 1}^{n} x_{i}$ Why this step? $\frac{d}{d λ} lo g λ = 1/ λ$ , and the linear term differentiates to $- \sum x_{i}$ .

Step 3 — Set to zero and solve. $\frac{n}{λ} = \sum x_{i} ⟹ \hat{λ} = \frac{n}{\sum x _{i}} = \frac{1}{x ˉ}$ With our numbers: $\sum x_{i} = 8$ , $n = 4$ , so $\hat{λ} = 4/8 = 0.5$ . Why this step? The MLE rate is the reciprocal of the mean wait — faster rate ⇔ shorter waits.

Verify: $ℓ^{''} (λ) = - n / λ^{2} < 0$ ⇒ maximum ✓. $\hat{λ} = 0.5 > 0$ respects the positivity constraint ✓. Units: mean is in seconds, rate is per-second, $0.5 s^{- 1}$ ✓ (one response every 2s).

Cell D — Boundary maximum (uniform support)

Worked example Example 4 — Uniform on

$[0, θ]$ Statement: Numbers are drawn uniformly from $[0, θ]$ , density $p (x ∣ θ) = 1/ θ$ for $0 \leq x \leq θ$ (and $0$ otherwise). You observe $x = {3, 7, 5, 7, 2}$ . Estimate $θ$ .

Forecast: The derivative will mislead you here. Guess $\hat{θ}$ from common sense first.

Step 1 — Likelihood, carefully. $L (θ) = \prod_{i = 1}^{n} \frac{1}{θ} = \frac{1}{θ ^{n}}, but ONLY if θ \geq max_{i} x_{i} .$ Why this step? If $θ$ is smaller than some observed $x_{i}$ , that point had probability $0$ under the model, so $L = 0$ . The constraint is the whole story.

Step 2 — Try the derivative (to see it fail). $\frac{d}{d θ} θ^{- n} = - n θ^{- n - 1} \neq = 0 for any finite θ .$ Why this step? No zero-slope point exists; $L$ just keeps decreasing as $θ$ grows. The calculus recipe alone gives nothing.

Step 3 — Use the constraint. $L = 1/ θ^{n}$ is largest when $θ$ is as small as allowed. The smallest legal $θ$ is the largest data point: $\hat{θ} = max_{i} x_{i} = 7$ Why this step? We shrink $θ$ to raise $1/ θ^{n}$ , but cannot go below $max_{i} x_{i}$ without killing the likelihood to zero.

Verify: At $θ = 7$ , $L = 1/ 7^{5}$ ; at $θ = 6.9$ , one point ( $7$ ) is outside so $L = 0 < 1/ 7^{5}$ ; at $θ = 8$ , $L = 1/ 8^{5} < 1/ 7^{5}$ ✓. So $θ = 7$ is the peak. $1/ 7^{5} = 1/16807$ .

Cell E — Two parameters solved jointly

Worked example Example 5 — Gaussian: estimate

$μ$ and $σ^{2}$ together Statement: Data $x = {2, 4, 4, 4, 5, 5, 7, 9}$ (that's $n = 8$ ) is Gaussian $N (μ, σ^{2})$ with both unknown. Estimate both.

Forecast: Guess $\overset{μ}{^}$ (centre) and roughly $\overset{σ}{^}^{2}$ (spread) before computing.

Step 1 — Log-likelihood in two variables. $ℓ (μ, σ^{2}) = - \frac{n}{2} lo g (2 π) - \frac{n}{2} lo g σ^{2} - \frac{1}{2 σ ^{2}} \sum_{i = 1}^{n} (x_{i} - μ)^{2}$ Why this step? Same Gaussian log as the parent note, but now nothing is held fixed.

Step 2 — Partial derivative in $μ$ . $\frac{\partial ℓ}{\partial μ} = \frac{1}{σ ^{2}} \sum_{i = 1}^{n} (x_{i} - μ) = 0 ⟹ \overset{μ}{^} = \overset{x}{ˉ} = \frac{2 + 4 + 4 + 4 + 5 + 5 + 7 + 9}{8} = 5$ Why this step? Holding $σ^{2}$ fixed, only the last term depends on $μ$ ; setting its slope to zero gives the sample mean. The $σ^{2}$ divides out, so $\overset{μ}{^}$ doesn't depend on the variance.

Step 3 — Partial derivative in $σ^{2}$ . $\frac{\partial ℓ}{\partial σ ^{2}} = - \frac{n}{2 σ ^{2}} + \frac{1}{2 σ ^{4}} \sum_{i = 1}^{n} (x_{i} - \overset{μ}{^})^{2} = 0 ⟹ \overset{σ}{^}^{2} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{μ}{^})^{2}$ Why this step? Plug the already-found $\overset{μ}{^}$ in, then solve for $σ^{2}$ . This is the "profile" trick — optimise one, substitute, optimise the next.

Step 4 — Numbers. Deviations from $5$ : $- 3, - 1, - 1, - 1, 0, 0, 2, 4$ ; squares $9, 1, 1, 1, 0, 0, 4, 16$ sum to $32$ . $\overset{σ}{^}^{2} = \frac{32}{8} = 4$

Verify: $\overset{μ}{^} = 5$ is inside the data range $[2, 9]$ ✓. $\overset{σ}{^}^{2} = 4 > 0$ ✓, so $\overset{σ}{^} = 2$ . This connects to Method of Moments, which gives the same $\overset{μ}{^}, \overset{σ}{^}^{2}$ here.

Cell F — Bias and limiting behaviour

Worked example Example 6 — Is the variance estimate unbiased?

Statement: For the two-parameter Gaussian above, is $\overset{σ}{^}_{MLE}^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}$ correct on average? What happens as $n \to \infty$ ?

Forecast: "Maximum likelihood" sounds optimal — guess whether $E [\overset{σ}{^}^{2}] = σ^{2}$ .

Step 1 — The exact expectation. A standard result: $E [\overset{σ}{^}_{MLE}^{2}] = \frac{n - 1}{n} σ^{2}$ Why this step? Using $\overset{x}{ˉ}$ (itself estimated from the data) "eats up" one degree of freedom, shrinking the spread.

Step 2 — Read off the bias. $bias = E [\overset{σ}{^}^{2}] - σ^{2} = - \frac{σ ^{2}}{n} < 0$ Why this step? Subtracting the truth from the average shows MLE under-estimates variance.

Step 3 — Check the limit. $\frac{n - 1}{n} = 1 - \frac{1}{n} n \to \infty 1$ Why this step? As data floods in, the bias factor tends to $1$ : MLE is asymptotically unbiased (consistent), even though it is biased for finite $n$ .

Verify: For $n = 8$ , the factor is $7/8 = 0.875$ . To de-bias, multiply: unbiased estimate $= \frac{n}{n - 1} \overset{σ}{^}^{2} = \frac{8}{7} \cdot 4 = \frac{32}{7} \approx 4.571$ , which is exactly $\frac{1}{n - 1} \sum (x_{i} - \overset{x}{ˉ})^{2} = 32/7$ ✓. Ties into Bias-Variance Tradeoff and Cramér-Rao Bound.

Cell G — Real-world word problem

Worked example Example 7 — Estimating a defect rate from a factory line

Statement: A factory inspects $n = 200$ items and finds $k = 6$ defective. Assume each item independently defective with unknown probability $p$ . (a) Estimate $p$ . (b) The manager asks: "What is the estimated probability the next two items are both good?"

Forecast: Guess the defect rate, then guess whether we can just reuse it for part (b).

Step 1 — Model and MLE. This is Bernoulli again (Cell A shape): $\overset{p}{^} = \frac{k}{n} = \frac{6}{200} = 0.03$ Why this step? The Bernoulli MLE is always the observed success (here defect) fraction.

Step 2 — Translate part (b). "Both of the next two good" $=$ probability $(1 - p)^{2}$ under independence. Why this step? Two independent good items multiply their good-probabilities $1 - p$ .

Step 3 — Apply invariance. The MLE of $(1 - p)^{2}$ is $(1 - \overset{p}{^})^{2}$ : $(1 - 0.03)^{2} = 0.9 7^{2} = 0.9409$ Why this step? MLE invariance: to estimate a function $g (p)$ , just plug $\overset{p}{^}$ into $g$ . No re-derivation needed. (Setting up next example, Cell H.)

Verify: $\overset{p}{^} = 0.03 \in [0, 1]$ ✓, plausible for a real line. $0.9 7^{2} = 0.9409$ ✓, and it's below $0.97$ (two hurdles are harder than one) ✓.

Cell H — Exam twist: reparametrisation / invariance

Worked example Example 8 — MLE of the odds, not the probability

Statement: An exam asks: from Example 1 ( $\overset{p}{^} = 0.75$ ), give the MLE of the odds $o = \frac{p}{1 - p}$ , and also of the log-odds $lo g o$ . Do it without redoing any calculus.

Forecast: Will you have to rebuild the likelihood in terms of $o$ ?

Step 1 — Invoke invariance. If $\overset{p}{^}$ maximises the likelihood, then for any function $g$ , $g (\overset{p}{^})$ maximises the likelihood written in terms of $g (p)$ . Why this step? The functional invariance property means the answer transfers directly — this is the whole point of the twist.

Step 2 — Plug in for the odds. $\overset{o}{^} = \frac{p ^}{1 - p ^} = \frac{0.75}{0.25} = 3$ Why this step? Substitute $\overset{p}{^} = 0.75$ into $g (p) = p / (1 - p)$ . No new differentiation.

Step 3 — Plug in for the log-odds. $lo g o = lo g \overset{o}{^} = lo g 3 \approx 1.0986$ Why this step? Log-odds is another function of $\overset{p}{^}$ ; invariance again gives $lo g (\overset{o}{^})$ .

Verify: Odds of $3$ means "3 successes for every 1 failure", matching $p = 0.75$ ( $3$ out of $4$ ) ✓. $lo g 3 = 1.0986 \dots$ ✓. This is exactly the quantity logistic regression estimates, linking to Loss Functions in ML and Likelihood Ratio Test.

Recall Which cell needs the second-derivative / boundary check most?

Cell D (Uniform) and Cell B (degenerate) — the derivative is never zero, so blindly solving $ℓ^{'} = 0$ fails; the maximum sits on the boundary of the support.

Recall Why is

$\overset{σ}{^}_{MLE}^{2}$ smaller than the "true" variance on average? Because using the sample mean $\overset{x}{ˉ}$ (itself fit to the data) removes one degree of freedom, giving $E [\overset{σ}{^}^{2}] = \frac{n - 1}{n} σ^{2}$ .

Connections

Parent: Maximum likelihood estimation (MLE) (index 1.3.16) · 1.3.16 Maximum likelihood estimation (MLE) (Hinglish)
When a prior is added: Bayesian Estimation
Variance/efficiency of these estimators: Fisher Information · Cramér-Rao Bound
Alternative fitting recipe: Method of Moments
Hidden-variable extension: EM Algorithm