Visual walkthrough — Statistical estimation — MLE, method of moments

We use the exponential distribution as our worked example because it has exactly one unknown number to guess, so we can draw everything on a flat page.

Step 0 — The story before any symbols

The data are the measured lifetimes: $x_1,x_2,\dots ,x_n$

• $x_i$ ::: the lifetime (a positive number, e.g. hours) of bulb number $i$.
• $n$ ::: how many bulbs we measured.

Nothing here is scary yet — just a list of positive numbers.

Step 1 — What one bulb's probability looks like

WHY this shape. Exponential curves are the natural model for "waiting until a random event" — memoryless, always positive, one knob. That single knob is exactly what we want to estimate.

PICTURE. The next figure shows two exponential curves: a large-$\lambda$ (fast, steep) one and a small-$\lambda$ (slow, flat) one. Notice how a bigger $\lambda$ starts higher but dives faster.

Step 2 — Probability of ALL the bulbs at once

• ${\prod }_{i=1}^n$ ::: "multiply these together for $i=1$ up to $n$" — the product cousin of a sum.
• $L(\lambda )$ ::: the likelihood; the data are now frozen, and $\lambda$ is the variable we slide.

WHY a product. Independent events: probability of "A and B" is $P(A)\times P(B)$. Chain that over all $n$ bulbs.

Now collect the pieces. There are $n$ copies of $\lambda$ (one per bulb), and the exponents add: $L(\lambda )={\lambda }^n{e}^{-\lambda {\sum }_{i=1}^n{x}_i}.$

• ${\lambda }^n$ ::: the $n$ height-knobs multiplied.
• ${\sum }_{i=1}^n{x}_i$ ::: the total of all lifetimes; the exponents $-\lambda x_1,-\lambda x_2,\dots$ merged because multiplying powers of $e$ adds their exponents.

PICTURE. Below, $L(\lambda )$ is plotted against $\lambda$ for a fixed toy dataset. It rises, peaks, then falls — there is one best $\lambda$. That peak is our target.

Step 3 — Why we don't maximize $L$ directly (take logs)

• $\ell (\lambda )$ ::: log-likelihood ("script-L").
• $\ln$ ::: natural logarithm — the inverse of $e^{(\cdot )}$ (see Logarithms). It turns ${\lambda }^n$ into $n\ln \lambda$ and turns $e^{-\lambda \sum x_i}$ into $-\lambda \sum x_i$.

WHY logs. Two reasons:

1. $\ln$ is strictly increasing — it never reverses order — so the $\lambda$ that maximizes $L$ also maximizes $\ell$. Same peak location.
2. The product ${\lambda }^n{e}^{-\lambda \sum x_i}$ becomes a sum $n\ln \lambda -\lambda \sum x_i$, which is far easier to differentiate. (And it avoids tiny numbers underflowing to zero on a computer.)

PICTURE. The next figure overlays $L(\lambda )$ and $\ell (\lambda )$ on the same $\lambda$-axis. The dotted vertical line shows their peaks land at exactly the same $\lambda$ — logging changed the shape but not the location of the maximum.

Step 4 — The peak has zero slope (score equation)

Term by term, differentiating $\ell =n\ln \lambda -\lambda \sum x_i$ with respect to $\lambda$:

• $\frac{d}{d\lambda }\left(n\ln \lambda \right)=\frac{n}{\lambda }$ ::: derivative of $\ln \lambda$ is $1/\lambda$, times the constant $n$.
• $\frac{d}{d\lambda }(-\lambda \sum x_i)=-\sum x_i$ ::: $\sum x_i$ is just a fixed number (data frozen!), so this term is linear in $\lambda$ and its slope is that constant.

WHY zero. The derivative is the slope of $\ell$. On the way up it is positive, on the way down negative; at the peak it is exactly $0$. That crossing point is our estimate.

PICTURE. The figure shows $\ell (\lambda )$ with its tangent line drawn at three places: rising (positive slope), at the peak (flat, slope $0$), and falling (negative slope). The flat one marks the winner.

Step 5 — Solve for $\hat{\lambda }$

• $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$ ::: the sample mean (average lifetime). So $n/\sum x_i$ is exactly $1/\bar{x}$.
• The hat in $\hat{\lambda }$ ::: marks it as an estimate built from data, not the true hidden $\lambda$.

WHY this makes sense. For an exponential, the true mean lifetime is $1/\lambda$ (long-lived bulbs ↔ small rate). If the bulbs averaged $\bar{x}$ hours, the most natural rate to blame is $1/\bar{x}$. The maths landed exactly on common sense — and by the Law of Large Numbers, $\bar{x}$ closes in on the true mean as $n$ grows, so $\hat{\lambda }\to \lambda$ (it is consistent).

PICTURE. Below, the toy dataset's dots sit on the $x$-axis; their balance point is $\bar{x}$; the recovered curve $f(x;1/\bar{x})$ is drawn through them. It's the exponential that "hugs" the data best.

Step 6 — Confirm it's a peak, not a valley

• $\frac{d^2\ell }{d{\lambda }^2}$ ::: the second derivative — how the slope itself is changing.
• Differentiate $n/\lambda$ again: $\frac{d}{d\lambda }(n{\lambda }^{-1})=-n{\lambda }^{-2}=-n/{\lambda }^2$. The $-\sum x_i$ term is constant, so it vanishes.
• $-n/{\lambda }^2<0$ ::: always negative ($n>0$, ${\lambda }^2>0$), meaning $\ell$ curves downward everywhere — one single hilltop.

WHY it matters. Negative curvature = concave = a genuine maximum. No risk of accidentally reporting a minimum. Because $\ell$ is concave over the whole $\lambda >0$ range, the peak is unique.

PICTURE. The figure marks the peak and shows the curve bending downward on both sides (the "frown" shape) — the visual signature of a maximum.

Step 7 — The degenerate cases (never leave the reader stranded)

• All lifetimes zero ($\bar{x}=0$). Then $\hat{\lambda }=1/0$ is undefined — the formula "wants" infinite rate. Picture: every death is instant, which an exponential can only mimic with $\lambda \to \infty$ (an infinitely tall spike at $0$). Real bulbs never do this, but you must recognise the boundary.
• One bulb ($n=1$). Still works: $\hat{\lambda }=1/x_1$. The peak of $\ell$ still exists; it's just a noisier guess (large variance — see Bias and Variance).
• A support-dependent parameter is different. For the uniform $U(0,\theta )$, differentiating gives no interior solution — the likelihood keeps rising as $\theta$ shrinks toward $\max x_i$, then drops to zero. The MLE is $\hat{\theta }={\max }_i{x}_i$, found by inspection, not calculus. This is the warning the parent note flagged.

PICTURE. The figure contrasts the smooth exponential $\ell$ (nice interior peak) with the uniform's cliff-edge likelihood, whose maximum sits right at the biggest data point.

The one-picture summary

Everything above compressed into a single diagram: data → product → log → derivative $=0$ → solve → check.