Exercises — Properties of estimators — unbiasedness, consistency, efficiency

The estimator is the rule/random variable $\bar{X}=\frac{1}{n}\sum X_i$ — it exists before you see data and has a whole distribution. The estimate is the single number $12.4$ — the output after plugging real data in. Analogy: the estimator is the recipe; the estimate is the cake you baked today.

Bias $=\mathbb{E}[\hat{\theta }]-\theta =\frac{3}{n}$. Since this is not zero for finite $n$, the estimator is biased. But $\frac{3}{n}\to 0$ as $n\to \infty$, so it is asymptotically unbiased — the systematic error shrinks with more data.

(a) → efficiency (least wobbly among unbiased). (b) → unbiasedness ($\mathbb{E}[\hat{\theta }]=\theta$). (c) → consistency (${\hat{\theta }}_n\stackrel{P}{\to }\theta$).

By linearity of expectation (it passes through sums and constants): $\mathbb{E}[\hat{\mu }]=0.3\mathbb{E}[X_1]+0.7\mathbb{E}[X_2]=0.3\mu +0.7\mu =(0.3+0.7)\mu =\mu .$ Unbiased ✓ — because the weights sum to $1$. That is the whole rule for a weighted average of same-mean variables.

$\mathrm{Var}(\bar{X})=\frac{{\sigma }^2}{n}=\frac{100}{25}=4,\text{standard error}=\sqrt{4}=2.$ Notice: quadruple $n$ and the standard error only halves — the $\sqrt{n}$ law of diminishing returns. (See Law of Large Numbers for why $\mathrm{Var}(\bar{X})\to 0$.)

Mean $\bar{X}=\frac{2+4+6}{3}=4$. Squared deviations: $(2-4{)}^2=4$, $(4-4{)}^2=0$, $(6-4{)}^2=4$; sum $=8$. $s^2=\frac{8}{n-1}=\frac{8}{2}=4,{\hat{\sigma }}^2=\frac{8}{n}=\frac{8}{3}\approx \mathrm{2.667.}$ The biased one is smaller — exactly the "shrink" the $n-1$ correction undoes.

For independent variables, $\mathrm{Var}(\sum w_i{X}_i)=\sum w_i^2{\sigma }^2$. $\mathrm{Var}({\hat{\mu }}_A)=(4\cdot \frac{1}{16}){\sigma }^2=\frac{1}{4}{\sigma }^2=0.25{\sigma }^2.$ $\mathrm{Var}({\hat{\mu }}_B)=(\frac{1}{4}+3\cdot \frac{1}{36}){\sigma }^2=(\frac{1}{4}+\frac{1}{12}){\sigma }^2=\frac{1}{3}{\sigma }^2\approx 0.333{\sigma }^2.$ Since $0.25<0.333$, ${\hat{\mu }}_A$ is more efficient. Relative efficiency of $A$ to $B$: $\frac{\mathrm{Var}({\hat{\mu }}_B)}{\mathrm{Var}({\hat{\mu }}_A)}=\frac{1/3}{1/4}=\frac{4}{3}\approx \mathrm{1.333.}$ Lesson: for equal-variance data, equal weights ($1/n$ each) minimise variance — that's why $\bar{X}$ is the winner.

Using $\mathrm{MSE}=\mathrm{Var}+{\mathrm{Bias}}^2$: ${\mathrm{MSE}}_1=5+2^2=9,{\mathrm{MSE}}_2=8+0=8.$ The unbiased rival wins here ($8<9$) — but note the biased one could have won if its variance were low enough. This is the Bias–Variance Tradeoff in one line: a little bias is worth it only if it buys enough variance reduction.

$\mathrm{MSE}({\hat{\theta }}_n)=\frac{9}{n}+(\frac{4}{n}{)}^2=\frac{9}{n}+\frac{16}{n^2}\underset{n\to \infty }{\to }0.$ Since $\mathrm{MSE}\to 0$, by the sufficient condition (Chebyshev) ${\hat{\theta }}_n\stackrel{P}{\to }\theta$: consistent. Both the wobble and the off-centre part vanish.

Unbiased: $\mathbb{E}[X_1]=\mu$. ✓ Consistency check via variance: $\mathrm{Var}(\hat{\mu })=\mathrm{Var}(X_1)={\sigma }^2$ for every $n$ — it never shrinks. The estimate wobbles by ${\sigma }^2$ no matter how much data you collect, because you throw all the extra data away. So $\mathrm{MSE}={\sigma }^2\nrightarrow 0$, and $P(|X_1-\mu |>\epsilon )$ stays fixed. Not consistent. This proves unbiased $\nRightarrow$ consistent: they measure different things.

(a) $\mathrm{Bias}(M)=\mathbb{E}[M]-\theta =\frac{n}{n+1}\theta -\theta =-\frac{1}{n+1}\theta$. It always underestimates — the sample max can never exceed the true ceiling $\theta$. (b) Multiply by the reciprocal of the shrink factor: ${\hat{\theta }}_{\text{unb}}=\frac{n+1}{n}M$. Then $\mathbb{E}\left[\frac{n+1}{n}M\right]=\frac{n+1}{n}\cdot \frac{n}{n+1}\theta =\theta$. ✓ (c) For $n=4$: factor $=\frac{n+1}{n}=\frac{5}{4}=1.25$.

$\frac{\mathrm{Var}(\text{median})}{\mathrm{Var}(\bar{X})}=\frac{\frac{\pi }{2}{\sigma }^2/n}{{\sigma }^2/n}=\frac{\pi }{2}\approx \mathrm{1.571.}$ The mean is more efficient by a factor of $\pi /2\approx 1.571$: the median wastes about $36\%$ of the information (it only reaches $\approx 64\%$ efficiency, since $2/\pi \approx 0.637$). For Normal data the mean is the efficient choice; the median only pays off when tails are heavy.

Minimise $\mathrm{Var}(\hat{\mu })={\sigma }^2\sum w_i^2$ subject to $\sum w_i=1$. Why squares again? Independence ⇒ variance adds the $w_i^2$. Minimising $\sum w_i^2$ under a fixed sum is a classic: Write $w_i=\frac{1}{n}+d_i$ with $\sum d_i=0$ (deviations from equal weights). Then $\sum w_i^2=\sum (\frac{1}{n}+d_i{)}^2=\sum \frac{1}{n^2}+\frac{2}{n}\underset{=0}{\underbrace{\sum d_i}}+\sum d_i^2=\frac{1}{n}+\sum d_i^2.$ Since $\sum d_i^2\ge 0$ with equality iff every $d_i=0$, the minimum is at $w_i=\frac{1}{n}$. ∎ Minimum variance $={\sigma }^2/n$ — the value $\bar{X}$ achieves. Any deviation from equal weights strictly increases variance.

(a) For $n$ i.i.d. points the information adds: $I_n(\lambda )=n{I}_1(\lambda )=\frac{n}{\lambda }$. So $\mathrm{Var}(\hat{\lambda })\ge \frac{1}{I_n(\lambda )}=\frac{\lambda }{n}.$ (b) For Poisson, $\mathrm{Var}(X_i)=\lambda$, hence $\mathrm{Var}(\bar{X})=\frac{\lambda }{n}$. This equals the bound, and $\bar{X}$ is unbiased ($\mathbb{E}[\bar{X}]=\lambda$). Therefore $\bar{X}$ is the efficient (variance-attaining) estimator of $\lambda$. ∎ Sanity of $I_1=1/\lambda$: $\ln f=-\lambda +x\ln \lambda -\ln x!$, so ${\partial }_{\lambda }\ln f=-1+x/\lambda$; its variance is $\mathrm{Var}(x)/{\lambda }^2=\lambda /{\lambda }^2=1/\lambda$. ✓

$\mathbb{E}[{\hat{\theta }}_c]=c\theta$, so $\mathrm{Bias}=c\theta -\theta =(c-1)\theta$, and $\mathrm{Var}({\hat{\theta }}_c)=c^{2}v$. $\mathrm{MSE}(c)=c^{2}v+(c-1{)}^2{\theta }^2.$ Differentiate and set to $0$: $\frac{d}{dc}=2cv+2(c-1){\theta }^2=0\Rightarrow c(v+{\theta }^2)={\theta }^2$, so $\boxed{c^{\star }=\frac{{\theta }^2}{{\theta }^2+v}}<1.$ Why less than 1? A touch of shrinkage trades a little bias for a bigger cut in variance — the Bias–Variance Tradeoff made quantitative. Example: ${\theta }^2=4$, $v=1$ gives $c^{\star }=4/5=0.8$, and $\mathrm{MSE}(0.8)=0.8^2\cdot 1+(0.8-1{)}^2\cdot 4=0.64+0.16=0.8<v=1$. The unbiased choice $c=1$ gives MSE $=1$, so shrinkage strictly wins.

Recognition of terms ::: L1 — estimator vs estimate, spotting bias. Plugging into formulas ::: L2 — $\mathrm{Var}(\bar{X})={\sigma }^2/n$, $s^2$ with $n-1$. Comparing and decomposing ::: L3 — relative efficiency, MSE $=$ Var $+$ Bias${}^2$. Combining ideas ::: L4 — unbiased $\ne$ consistent, unbiasing the Uniform max. Proving and designing ::: L5 — optimal weights, Cramér–Rao attainment, MSE-optimal shrinkage.