Question bank — Statistical estimation — MLE, method of moments

False. $L$ is a function of $\theta$ for fixed data; it does not integrate to $1$ over $\theta$ and has no obligation to. It is the data's probability read backwards, not a belief about $\theta$ — that backwards-to-forwards flip is what Bayesian Estimation adds with a prior.

False. log is strictly increasing, so $\ell (\theta )=\log L(\theta )$ rises and falls in lockstep with $L$; they peak at the same $\theta$. Log only changes the algebra, never the answer.

False. What we care about is total error, $\text{MSE}=\text{Var}(\hat{\theta })+{\text{bias}}^2$. A slightly biased estimator with much smaller variance can beat an unbiased one — see Bias and Variance.

False. MLE optimizes likelihood, not unbiasedness — different goals. The normal MLE ${\hat{\sigma }}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2$ divides by $n$ and satisfies $E[{\hat{\sigma }}^2]=\frac{n-1}{n}{\sigma }^2<{\sigma }^2$, so it is biased low.

False. They coincide for the exponential rate and the normal $\hat{\mu }$, but that is luck of the algebra. For $U(0,\theta )$, MoM gives $\hat{\theta }=2\bar{x}$ while MLE gives $\hat{\theta }=\max x_i$ — very different recipes.

True. This is the invariance property: the MLE of any function $g(\theta )$ is just $g(\hat{\theta })$. It works because relabelling the axis by $g$ doesn't move the peak of the likelihood.

False. MoM solves equations blindly and can spit out impossible values (a negative variance, a probability above $1$). It has no built-in respect for the parameter's support.

False. MLE is consistent — it converges in probability, $\hat{\theta }\stackrel{P}{\to }\theta$ — meaning the chance of being far off shrinks to zero. For any finite $n$ it still wobbles; "exactly" is never guaranteed.

False. A constant (in $\theta$) shifts the whole curve up or down without moving its peak; $\frac{d}{d\theta }(\ell +c)={\ell }^{\prime }$, so the score equation is untouched. This is why we can drop $-\frac{n}{2}\ln (2\pi )$ freely.

True. With $p$ unknowns you match the first $p$ moments — $p$ equations for $p$ unknowns. Fewer leaves the system under-determined; the count is forced.

The error is avoidable pain, not wrongness: products of many small densities underflow numerically and the product rule explodes. Take the log first — the product becomes a sum and differentiation is one line.

The error is $E[1/\bar{X}]\ne 1/E[\bar{X}]$ — you cannot pull expectation through a nonlinear function (Jensen's inequality). In fact $\hat{\lambda }$ is biased upward for finite $n$.

Differentiation fails here. The support $[0,\theta ]$ depends on $\theta$; the likelihood is ${\theta }^{-n}$ where valid and $0$ once $\theta <\max x_i$, so it increases as $\theta$ shrinks and jumps to zero — no interior critical point. The max is at the boundary $\hat{\theta }=\max x_i$, found by inspection.

A root of ${\ell }^{\prime }$ could be a minimum or a saddle. You must confirm ${\ell }^{\prime \prime }(\hat{\theta })<0$ (curving down = a peak), and also check that no boundary of the parameter space beats it.

Both equal $\frac{1}{n}\sum (x_i-\bar{x}{)}^2$ — division by $n$, which is biased. The unbiased sample variance divides by $n-1$; neither MLE nor MoM produces it here.

A likelihood value is not a probability about $\theta$ being correct. It is (proportional to) the probability of the data under that $\theta$; comparing $\theta$'s means comparing these values to each other, not reading them as confidence in $\hat{\theta }$.

They are approximately equal for large $n$ (that's the Law of Large Numbers: $m_k\stackrel{P}{\to }{\mu }_k$). For finite data they differ, so MoM is an estimate, not an exact identity.

Independent observations mean their joint probability multiplies: $P(\text{all data})=\prod f(x_i)$. Maximizing that joint probability is literally "which $\theta$ makes exactly this data least surprising."

Because it is monotonic — strictly increasing — so it preserves order: the biggest $L$ maps to the biggest $\ell$. We change the landscape's height but never move its summit.

Population moment ${\mu }_k=E[X^k]$ is a known formula in $\theta$; sample moment $m_k$ is measurable. Since $m_k\to {\mu }_k$ by the Law of Large Numbers, setting them equal and solving lands us near the true $\theta$.

Because $\bar{x}$ is fit to the same data, the deviations $x_i-\bar{x}$ are slightly too small (the mean is pulled toward the data). One degree of freedom is spent estimating $\mu$, so we should divide by $n-1$; MLE ignores this because its target is likelihood, not unbiasedness.

MLE is asymptotically efficient — for large $n$ it achieves the smallest possible variance any estimator can (the Cramér-Rao Lower Bound) — plus it is consistent and invariant. MoM is easy but often noisier and can leave the legal range.

It needs only moment formulas, no likelihood maximization, so it works when the density is unknown or the likelihood is intractable — a fast, robust starting guess.

Because "optimal" is defined relative to a goal. MLE optimizes likelihood; the unbiased estimator optimizes zero-bias; a minimum-MSE estimator trades the two off. Different objectives, different winners — as Bias and Variance makes explicit.

$\hat{\theta }={\max }_i{x}_i$. The likelihood ${\theta }^{-n}$ (on the valid region) shrinks as $\theta$ grows but must stay $\ge$ every observation, so the smallest legal $\theta$ — the largest data point — wins. It's a boundary maximum, invisible to ${\ell }^{\prime }=0$.

$\hat{p}=1$, a boundary of $[0,1]$. The score equation $\frac{s}{p}-\frac{n-s}{1-p}=0$ has no interior root (the second term vanishes); the likelihood $p^n$ is maximized by pushing $p$ to its edge. Perfectly valid, just not an interior critical point.

$\bar{x}\to 0$ forces $\hat{\lambda }\to \infty$ — a degenerate estimate saying "the rate is enormous." It flags too little data or an ill-fitting model rather than a usable number.

Yes in principle if $\overline{x^2}<{\bar{x}}^2$ — but that can't happen for real data since $\frac{1}{n}\sum (x_i-\bar{x}{)}^2\ge 0$ always. For other parameters, however, MoM genuinely can leave the legal range, which is its notorious weakness.

The likelihood is flat/bimodal near the top, so $\hat{\theta }$ is poorly determined — small data changes swing it a lot (high variance). A sharp, peaked likelihood means a confident estimate; a flat one warns you the data barely constrains $\theta$.

No — with one point $\bar{x}=x_1$, so ${\hat{\sigma }}^2=0$. You cannot estimate spread from a single value; the estimator degenerates and the $n-1$ divisor would even divide by zero, honestly signalling "not enough data."