Foundations — Properties of estimators — unbiasedness, consistency, efficiency

This page is the toolbox. Before you can read the parent note Properties of estimators — unbiasedness, consistency, efficiency, every squiggle it uses must mean something to you. We build each symbol from nothing, anchor it to a picture, and say why the topic needs it. Order matters: each item leans on the one before.

0. The stage: a hidden number and a random draw

Picture a giant barrel holding millions of numbers — the population. Somewhere inside, the barrel has a true average height, a true spread. These true facts are fixed but hidden from you. You are only allowed to scoop out a handful.

Everything below is machinery for turning that handful (the scoop) into an honest guess about the barrel.

1. $\theta$ — the parameter (the hidden number)

Picture: the exact fill-line printed on the inside of the barrel that you can never directly read.

Why the topic needs it: the entire subject exists to guess $\theta$. Without a target there is nothing to estimate. Every property (unbiased, consistent, efficient) is measured relative to this $\theta$.

2. $X_i$ — one observation, and "i.i.d."

Capital $X$ (not $x$) signals it is random — before you scoop, you don't know what it will be.

Picture: $n$ separate hands reaching into the same barrel, each hand blindfolded and unaware of the others.

Why the topic needs it: i.i.d. is the assumption that lets us add up expectations cleanly (Section 6) and lets more data behave predictably (the Law of Large Numbers). Break it and unbiasedness/consistency proofs collapse.

3. $n$ — the sample size

Picture: the number of dots in your handful.

Why the topic needs it: $n$ is the dial you turn in consistency ("what happens as $n\to \infty$?"). Small $n$ = few dots = jittery guess; large $n$ = many dots = steady guess. Watch how $n$ appears in the denominators later (${\sigma }^2/n$, $n-1$) — those are the heart of the topic.

4. Random variable & its distribution

Why the topic needs it: here is the plot twist that makes the whole subject interesting — since $\hat{\theta }$ is computed from random data, $\hat{\theta }$ is itself a random variable. It has its own cloud of possible values, called a sampling distribution. Unbiased / consistent / efficient are all statements about that cloud's shape.

5. $\mathbb{E}[\cdot ]$ — expectation (the long-run average)

Picture: if the distribution were a physical cardboard cut-out, $\mathbb{E}[X]$ is the point where you could balance it on a fingertip.

Why the topic needs it: unbiasedness is literally the equation $\mathbb{E}[\hat{\theta }]=\theta$ ("the balance point of my guess-cloud sits on the truth"). Linearity is the single tool that makes $\mathbb{E}[\bar{X}]=\mu$ fall out in two lines.

6. $\sum$ and $\bar{X}$ — summation and the sample mean

Picture: the centre of gravity of your handful of dots.

Why the topic needs it: $\bar{X}$ is the starring estimator — the running example for all three properties. Notice $\bar{X}$ is Latin-with-a-bar: it's built from data, and it estimates the Greek $\mu$.

7. $\hat{\theta }$ — the estimator (the hat)

Formally $\hat{\theta }=T(X_1,\dots ,X_n)$ — a function of the data. Feed in random data, out comes a (random) number.

Why the topic needs it: the hat is the whole subject's protagonist. "Is $\hat{\theta }$ good?" is the only question we ever ask.

8. $\mathrm{Var}$ and ${\sigma }^2$ — variance (the wobble)

The population variance is ${\sigma }^2$ (sigma-squared, Greek → truth). Its square root $\sigma$ is the standard deviation, in the same units as the data.

Why the topic needs it: efficiency is a contest of smallest variance. Consistency rides on variance $\to 0$. Variance is the second of the three lenses (the "wobble").

9. $\mathrm{Bias}$ — the systematic offset

Picture: a dartboard. Bias = distance from your average dart to the bullseye. Variance = how scattered the darts are around that average. These are two different things — you can have one without the other.

Why the topic needs it: unbiasedness is exactly "bias $=0$," and MSE (next) is built from bias and variance together.

10. $\mathrm{MSE}$ — the honest scorecard

Why the topic needs it: MSE is the single number that lets you compare any two estimators — even a biased one against an unbiased one (see the parent's Uniform example). It also gives the easy consistency test: if $\mathrm{MSE}\to 0$, the estimator is consistent. This links straight to the Bias–Variance Tradeoff.

11. Convergence in probability — $\stackrel{P}{\to }$

Symbols unpacked: $|\cdot |$ = distance (absolute value, always $\ge 0$); $\epsilon$ (epsilon) = a small positive "how close is close enough"; $P(\cdot )$ = probability of an event; ${\lim }_{n\to \infty }$ = "the value we settle on as $n$ grows without bound."

Picture: the guess-cloud getting narrower and narrower, sliding onto the target, until essentially all its weight sits within any hair's-width band around $\theta$.

Why the topic needs it: this is the exact definition of consistency. It is powered by the Law of Large Numbers, which guarantees $\bar{X}\stackrel{P}{\to }\mu$.

12. The likelihood pieces — $f(X;\theta )$, $\ln$, $\partial /\partial \theta$, $I(\theta )$

These appear only in the efficiency section, but you should recognise them.

Why the topic needs it: the Fisher Information sets the Cramér–Rao lower bound $\mathrm{Var}(\hat{\theta })\ge 1/I(\theta )$ — the hard floor no unbiased estimator can beat. That floor is the very definition of efficient. It ties to Maximum Likelihood Estimation and, via the Central Limit Theorem, to why MLEs are asymptotically efficient.

How the foundations feed the topic

Equipment checklist

Cover the right side and answer each before opening the parent note.