Maximum likelihood estimation (MLE): Question bank

Before you start, three words we'll reuse constantly, in plain language:

To make the words "peak", "curvature", and "multiple bumps" concrete, hold these two chalkboard pictures in mind as you work through the questions.

True or false — justify

MLE always returns the true parameter value.

False. It returns the value that best explains this dataset; that is a noisy pointer at truth. It only converges to truth as

n \to \infty

(consistency), not for any finite sample.

Maximizing

L (θ)

and maximizing

ℓ (θ) = lo g L (θ)

give the same estimate.

True.

lo g

is strictly increasing, so it preserves the location of the maximum — it moves the height of the peak, never its position (see the left curve in the first figure).

The likelihood function

L (θ ∣ X)

is a probability distribution over

θ

False. It integrates to whatever it wants over

θ

, not to 1. It is

p (X ∣ θ)

viewed as a function of

θ

— a distribution lives over the data axis, not the parameter axis.

The MLE of a variance is unbiased.

False.

\overset{σ}{^}_{M L E}^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2}

divides by

n

and systematically underestimates; the unbiased version divides by

n - 1

. See Bias-Variance Tradeoff.

If an estimator is biased it must be a worse choice than an unbiased one.

False. The MLE trades a little bias for lower variance; total error (mean-squared error) can be smaller than an unbiased rival. This is exactly the Bias-Variance Tradeoff.

The log-likelihood is always concave, so setting its derivative to zero always finds the maximum.

False. Concavity holds for many nice models (Bernoulli, Gaussian mean) but not in general — the second figure shows a two-bump curve where "derivative = 0" has three solutions and only one is the global max.

MLE and Maximum A Posteriori (MAP) estimation are different methods that never coincide.

False. MLE equals MAP when the prior is flat (uniform) over the parameter, since a constant prior does not move the peak — but note that a uniform prior over an unbounded space is improper (does not integrate to 1), so the coincidence is exact only on a bounded region or as a formal limit. See Bayesian Estimation.

Adding more i.i.d. data can only sharpen (never flatten) the log-likelihood peak.

True in expectation, not sample-path by sample-path. On average each point adds curvature (the Fisher Information) and the peak tightens like

1/ n

, but a single unlucky point can momentarily flatten or shift the observed curve — the guarantee is about the expected curvature, not every individual sample.

The Cramér-Rao bound guarantees no estimator can ever beat the MLE's variance.

False. The Cramér-Rao Bound lower-bounds the variance of unbiased estimators; a biased estimator can dip below it. The MLE only meets the bound asymptotically, not for small

n

Spot the error

"I want the most likely parameters given the data, so I maximize

P (θ ∣ X)

That is the posterior, i.e. Bayesian Bayesian Estimation / MAP. Plain MLE has no prior and maximizes

P (X ∣ θ)

; without a prior

P (θ)

you cannot even write

P (θ ∣ X)

"With 1000 points I'll just multiply all the

p (x_{i} ∣ θ)

directly and maximize that."

A product of 1000 sub-one probabilities underflows to

0

in floating point (e.g.

0. 5^{1000} \approx 1 0^{- 301}

). Take the log to turn the product into a numerically safe sum before optimizing.

"I found

\frac{d ℓ}{d θ} = 0

, so I'm done — that's the MLE."

A zero derivative marks a critical point, which could be a minimum, saddle, or boundary artefact. Confirm it is a max: check

\frac{d ^{2} ℓ}{d θ ^{2}} < 0

, or compare

ℓ

across critical points and boundaries.

"For the coin with 7 heads in 10 flips, the MLE of

p

is somewhere near

0.5

because coins are fair."

The MLE ignores your prior belief and reads only the data:

\overset{p}{^} = k / n = 0.7

. A pull toward

0.5

would require a prior — that's Bayesian Estimation, not MLE.

"The invariance property means if

\hat{θ}

is unbiased then

g (\hat{θ})

is unbiased too."

Invariance says

g (θ) = g (\hat{θ})

— the estimate transforms cleanly. Unbiasedness is not preserved by nonlinear

g

(by Jensen's inequality

E [g (\hat{θ})] \neq = g (E [\hat{θ}])

"The invariance theorem lets me push the MLE through absolutely any function

g

In its cleanest form the theorem asks

g

to be a one-to-one, differentiable reparameterization so that "which

θ

maximizes" translates unambiguously to "which

g (θ)

maximizes". For many-to-one

g

you must define an induced likelihood to keep the claim precise.

"MLE is optimal, so it must be efficient at every sample size."

Efficiency (hitting the Cramér-Rao Bound) is an asymptotic (

n \to \infty

) statement. For small

n

another estimator can have smaller variance or smaller total error.

Why questions

Why do we take the log of the likelihood at all?

Three reasons: it turns products into sums (cleaner derivatives), it prevents floating-point underflow, and being monotonic it leaves the location of the maximum untouched.

Why does the i.i.d. assumption let us write the likelihood as a product?

Independence means the joint probability factorizes,

P (x_{1}, \dots, x_{n} ∣ θ) = \prod_{i} p (x_{i} ∣ θ)

; "identically distributed" means every factor uses the same

p (\cdot ∣ θ)

Why is the MLE of the Gaussian mean just the sample mean

\overset{x}{ˉ}

The Gaussian log-likelihood is

ℓ (μ) = const - \frac{1}{2} \sum (x_{i} - μ)^{2}

, so maximizing it means minimizing

\sum (x_{i} - μ)^{2}

. Differentiate:

\frac{d}{d μ} \sum (x_{i} - μ)^{2} = - 2 \sum (x_{i} - μ) = 0

, giving

\sum x_{i} = n μ

, i.e.

μ = \overset{x}{ˉ}

. Geometrically,

\overset{x}{ˉ}

is the single point whose total squared pull from all data points balances out — the "centre of mass" that no other point beats.

Why is the second derivative of the log-likelihood so important beyond checking for a maximum?

Its negative expectation is the Fisher Information

I (θ)

, which sets the MLE's asymptotic variance

I (θ)^{- 1} / n

— sharp curvature means a confident, low-variance estimate.

Why is MLE called "frequentist" rather than "Bayesian"?

It treats

θ

as a fixed unknown constant with no probability distribution; only the data is random. Bayesian Estimation instead puts a prior distribution on

θ

and updates it.

Why can MLE beat the Method of Moments in efficiency?

MLE uses the full shape of the assumed density (all the information the model encodes), while Method of Moments only matches a few summary moments and can throw away information.

Edge cases

What is the Bernoulli MLE if you observe

k = 0

heads in

n

flips?

\overset{p}{^} = 0/ n = 0

. The estimate sits on the boundary of

[0, 1]

, so the "derivative = 0" trick fails; the maximum is at the edge and any future head would have assigned-probability zero — a warning sign of over-confidence with tiny data.

What is the Bernoulli MLE in the symmetric case

k = n

(all heads)?

\overset{p}{^} = n / n = 1

. This is the mirror-image boundary problem: the peak sits at the right edge of

[0, 1]

, the interior derivative never hits zero, and the model would assign probability zero to any future tail — again over-confident because a small sample cannot rule out rare outcomes.

What happens to the MLE when all your data points are identical (zero sample variance)?

The Gaussian variance MLE collapses to

\overset{σ}{^}^{2} = 0

, making the likelihood a spike (infinite density). This is a degenerate maximum — a red flag that the model has overfit the sample.

Is the MLE unique?

Not necessarily. If the log-likelihood is flat over a region or has multiple equal peaks (e.g. label-swapping in mixtures handled by the EM Algorithm), several

θ

values tie, so "the" MLE is a set, not a point.

What does MLE do with

n = 1

data point?

It still returns an estimate, but with no averaging the variance is huge and, for the Gaussian,

\overset{σ}{^}^{2}

hits its degenerate value

0

. Consistency needs

n \to \infty

; a single point earns essentially no trust.

What if the true data-generating distribution is not in your model family at all?

MLE still converges — but to the parameter whose model is closest (in KL divergence) to the truth, not to the truth itself. "Best wrong model" beats "no model", but be aware the guarantee weakened.

Why can the likelihood equation have no finite solution — give a concrete toy model?

Take a Pareto tail with unknown index

α

: if the data happen to be light-tailed, the log-likelihood keeps rising as

α \to \infty

(or runs off toward a boundary of

α

) with no interior critical point. Then

ℓ^{'} (α) = 0

has no finite root, and you must inspect the limiting behaviour rather than "solve for zero".

Recall Quick self-test

The three fastest ways to embarrass yourself with MLE ::: maximizing the posterior instead of the likelihood, forgetting the log and underflowing, and calling a critical point a maximum without checking curvature. Bias vs consistency in one line ::: MLE can be biased for finite $n$ but is consistent — it centres on truth as data grows without bound.