Maximum likelihood estimation (MLE): Visual walkthrough

We will use the coin flip as our running example because it needs only one unknown number, so we can draw everything on a flat sheet of paper.

Step 1 — What is a "parameter" and what is "data"?

WHAT. We have a coin. It lands heads with some unknown chance we call $p$ . Here $p$ is a single number between $0$ and $1$ — that is our parameter (the dial we get to turn). We flip the coin $n$ times and record what happened; those recorded outcomes are our data. We write == $k$ for the number of heads== we saw (so $n - k$ is the number of tails). In our running example $n = 10$ and $k = 7$ .

WHY start here. Before any maths, we must be crystal clear on which thing is fixed and which thing we are searching over. The data already happened — it is frozen. The parameter $p$ is the mystery knob. Nailing down the letters $n$ and $k$ now means every later formula reads without surprise.

PICTURE. Below: a dial for $p$ on the left (turnable), and the frozen data on the right — 7 heads, 3 tails, already fallen and unchangeable.

Figure — Maximum likelihood estimation (MLE)

Step 2 — How likely is this exact data for one setting of the knob?

WHAT. Pick one value of the knob, say $p = 0.5$ . Ask: if the true chance of heads were $0.5$ , how probable is it to see exactly 7 heads then 3 tails? Each head costs a factor $p$ , each tail costs a factor $(1 - p)$ . Multiply them:

$L (p) = head p \cdot head p \dots tail (1 - p) = p^{7} (1 - p)^{3} = p^{k} (1 - p)^{n - k}$

Here each symbol earns its place: $p^{7} = p^{k}$ is "each of the $k$ heads has chance $p$ , multiplied," and $(1 - p)^{3} = (1 - p)^{n - k}$ is "each of the $n - k$ tails has chance $1 - p$ ." The whole product $L (p)$ is the likelihood — the probability of the frozen data, read as a function of the knob $p$ .

WHY multiply. The flips are independent — one flip's result tells you nothing about the next. For independent events, the probability of "this AND that AND the other" is the product of the individual probabilities. That is the only reason the product appears.

PICTURE. Two dial settings side by side: at $p = 0.5$ the data is fairly likely; at $p = 0.9$ the three tails become very unlikely, so the product shrinks. The bar height = $L (p)$ .

Step 3 — Slide the knob and watch the likelihood rise and fall

WHAT. Instead of testing $p$ one value at a time, sweep $p$ smoothly from $0$ to $1$ and plot the height $L (p) = p^{7} (1 - p)^{3}$ . You get a single hump.

WHY a hump. At $p = 0$ you claim heads are impossible, yet you saw 7 heads → likelihood $0$ . At $p = 1$ you claim tails are impossible, yet you saw 3 tails → likelihood $0$ . Somewhere in between the two pressures balance and the curve peaks. The MLE is the location of that peak — the knob setting that makes the frozen data most probable.

PICTURE. The full likelihood curve. It is pinned to zero at both ends, bulges in the middle, and its highest point is marked with a dashed line dropping to the $p$ -axis.

Step 4 — Why we climb the log of the curve instead

WHAT. Replace $L (p)$ with its logarithm, the log-likelihood:

$ℓ (p) = lo g L (p) = from the 7 heads 7 lo g p + from the 3 tails 3 lo g (1 - p)$

The $lo g$ turned the product $p^{7} (1 - p)^{3}$ into a sum, because $lo g (ab) = lo g a + lo g b$ and $lo g (a^{k}) = k lo g a$ . (If we had kept the binomial factor from Step 2, it would appear here only as an extra constant $lo g (k n)$ , flat in $p$ — invisible to the peak-finder.)

WHY the log — and why it is safe. Two reasons. (1) Same peak. $lo g$ is strictly increasing: if curve $A$ is taller than curve $B$ at some point, its log is also taller there. So climbing $ℓ$ lands us at the exact same $p$ as climbing $L$ . (2) Practical. Sums are far easier to differentiate than products, and for large $n$ the raw product underflows to $0$ on a computer while its log stays a sensible number.

PICTURE. Left axis: the hump $L (p)$ . Right axis: the curve $ℓ (p)$ . Notice the dashed vertical line — both peaks sit at the same $p$ . The log-curve is stretched and shifted but its summit has not moved.

Step 5 — The slope is zero at the summit

WHAT. At the top of a smooth hill the ground is momentarily flat — the slope (derivative) is zero. So we compute the slope of $ℓ$ and set it to zero:

$\frac{d ℓ}{d p} = slope from heads \frac{7}{p} - slope from tails \frac{3}{1 - p} = 0$

Term by term: $\frac{d}{d p} (7 lo g p) = \frac{7}{p}$ (this pushes $p$ up, since more heads-weight wants larger $p$ ), and $\frac{d}{d p} (3 lo g (1 - p)) = - \frac{3}{1 - p}$ (the tails pull $p$ down). At the peak these two forces exactly cancel.

WHY the derivative. Why this tool and not, say, just eyeballing the graph? Because the derivative is the precise machine that answers "where is the curve flat?" — and a smooth hump is flat only at its summit. Setting it to zero converts a searching problem into an equation-solving problem.

PICTURE. The log-likelihood curve with three tangent lines: one uphill (positive slope) on the left, one downhill (negative slope) on the right, and one perfectly horizontal at the top where slope $= 0$ .

Step 6 — Solve, and read the meaning of the answer

WHAT. Rearrange the flat-slope equation:

$\frac{7}{p} = \frac{3}{1 - p} ⟹ 7 (1 - p) = 3 p ⟹ 7 = 10 p ⟹ \overset{p}{^} = \frac{7}{10} = 0.7$

In general, with $k$ heads out of $n$ flips the same algebra gives $\overset{p}{^} = k / n$ : the MLE is simply the observed fraction of heads.

WHY it is beautiful. The abstract machinery — products, logs, derivatives — collapses to the most obvious estimate a child would guess. That is a sign the method is sane.

PICTURE. The likelihood hump again, with the peak annotated at $p = 0.7$ , and the fraction $7/10$ written beside the frozen data to show they are the same number.

Step 7 — Was it really a maximum? And the degenerate cases

WHAT. A zero slope alone could mark a valley, not a peak. We check the second derivative (the rate at which the slope itself changes — negative means the curve bends downward, a true summit):

$\frac{d ^{2} ℓ}{d p ^{2}} = - \frac{7}{p ^{2}} - \frac{3}{( 1 - p ) ^{2}} < 0 for every p \in (0, 1) .$

Both terms are negative for all valid $p$ , so the curve is concave-down everywhere — our critical point is unquestionably the maximum.

Edge cases — cover them all:

All heads ( $k = n$ ): $ℓ (p) = n lo g p$ has no interior flat point; it just keeps rising toward $p = 1$ . The MLE sits at the boundary $\overset{p}{^} = 1$ . Interpretation: the data has given zero evidence that tails can happen.
All tails ( $k = 0$ ): mirror image, $\overset{p}{^} = 0$ .
Zero flips ( $n = 0$ ): no data, no hump — every $p$ is equally (un)supported and the MLE is undefined. You cannot estimate a knob you never tested.

WHY show these. The clean formula $k / n$ still returns $1$ , $0$ , and $0/0$ in these three cases — matching exactly the boundary/undefined behaviour we just reasoned out. A method you trust must behave sensibly at its extremes.

PICTURE. Three mini-curves side by side: the interior-peak normal case, the all-heads case (peak pinned to the right wall $p = 1$ ), and the all-tails case (peak pinned to the left wall $p = 0$ ).

The one-picture summary

This final figure compresses the whole journey: frozen data → likelihood hump → log reshaping → flat slope at the peak → answer $k / n$ , with the boundary cases in the margin.

Recall Feynman retelling — say it back in plain words

We had a coin with an unknown heads-chance $p$ , and we saw $k = 7$ heads and $n - k = 3$ tails frozen on the table (out of $n = 10$ flips). For any guess of $p$ , the chance of seeing that exact record is $p^{7} (1 - p)^{3}$ — big when $p$ fits, tiny when it doesn't. If we had only counted heads instead of remembering the order, there'd be an extra $(k n)$ out front, but it's a constant that never moves the peak, so we ignore it. We swept $p$ from 0 to 1 and got a hump that is zero at both ends (a guess of "heads impossible" or "tails impossible" contradicts the data). The best $p$ is the top of the hump. To find the top we took the log — which reshapes the hump but keeps its summit in the same spot and turns the product into an easy sum. At a summit the ground is flat, so we set the slope to zero, and the heads-force $7/ p$ balanced the tails-force $3/ (1 - p)$ , giving $p = 7/10$ . The second derivative was negative everywhere, so it truly was a peak. And in general the answer is just the fraction of heads, $k / n$ — with the edges (all heads → 1, all tails → 0, no flips → undefined) all behaving exactly as common sense demands.

Recall Quick self-test

Why does the likelihood curve touch zero at both $p = 0$ and $p = 1$ ? ::: Because $p = 0$ makes the 7 observed heads impossible and $p = 1$ makes the 3 observed tails impossible, so the product $p^{7} (1 - p)^{3}$ vanishes at each end. Why can we drop the $(k n)$ factor before maximizing? ::: It contains no $p$ , so it only scales the curve's height by a positive constant — the location of the peak (the $ar g max$ ) is unchanged. Why is taking the log allowed without changing the answer? ::: $lo g$ is strictly increasing, so it never moves a taller point below a shorter one — the location of the peak is preserved. What confirms the critical point is a maximum, not a minimum? ::: The second derivative $- 7/ p^{2} - 3/ (1 - p)^{2}$ is negative for all valid $p$ , so the curve is concave down (a peak).

See also: Method of Moments (a different fitting recipe), Bayesian Estimation (adds a prior over $θ$ ), Fisher Information and Cramér-Rao Bound (how sharp the peak is), Gaussian Distribution (the next worked case), Bias-Variance Tradeoff (why the MLE variance estimate is biased).