Intuition The big pictureWe have an unknown number θ \theta θ parameter ) describing a population — say its true mean.
We can't see θ \theta θ estimator θ ^ \hat\theta θ ^ rule (a function of the data) that produces a guess.
Because the data is random, θ ^ \hat\theta θ ^ random variable — it has a distribution.
The whole game is: how good is this guessing rule? We judge it by three lenses:
Unbiasedness — is it right on average ? (centered on θ \theta θ Consistency — does it get better as we collect more data ? (converges to θ \theta θ Efficiency — among the good rules, is it the least wobbly ? (smallest variance)
Definition Estimator vs estimateGiven data X 1 , … , X n X_1,\dots,X_n X 1 , … , X n estimator is a statistic θ ^ = T ( X 1 , … , X n ) \hat\theta = T(X_1,\dots,X_n) θ ^ = T ( X 1 , … , X n ) estimate is the number you get after plugging in actual observed data.
Example: X ˉ = 1 n ∑ X i \bar X = \frac1n\sum X_i X ˉ = n 1 ∑ X i μ \mu μ 4.7 4.7 4.7
WHY a random variable? Because before you sample, you don't know which values you'll draw. Different samples → different θ ^ \hat\theta θ ^
Definition Unbiased estimatorθ ^ \hat\theta θ ^ unbiased for θ \theta θ E [ θ ^ ] = θ for every possible θ . \mathbb{E}[\hat\theta] = \theta \quad\text{for every possible } \theta. E [ θ ^ ] = θ for every possible θ . bias is Bias ( θ ^ ) = E [ θ ^ ] − θ \operatorname{Bias}(\hat\theta) = \mathbb{E}[\hat\theta] - \theta Bias ( θ ^ ) = E [ θ ^ ] − θ = 0 =0 = 0
Intuition WHY this matters"On average over infinitely many samples, my rule hits the bullseye." It does not mean any single estimate is correct — it means the errors cancel in the long run, with no systematic over- or under-shoot.
Let X 1 , … , X n X_1,\dots,X_n X 1 , … , X n E [ X i ] = μ \mathbb E[X_i]=\mu E [ X i ] = μ E [ X ˉ ] = E [ 1 n ∑ i = 1 n X i ] = 1 n ∑ i = 1 n E [ X i ] = 1 n ⋅ n μ = μ . \mathbb E[\bar X]=\mathbb E\!\left[\frac1n\sum_{i=1}^n X_i\right]=\frac1n\sum_{i=1}^n\mathbb E[X_i]=\frac1n\cdot n\mu=\mu. E [ X ˉ ] = E [ n 1 ∑ i = 1 n X i ] = n 1 ∑ i = 1 n E [ X i ] = n 1 ⋅ n μ = μ . Why this step? Expectation is linear , so it passes through the sum and the constant 1 / n 1/n 1/ n
We want to estimate σ 2 = Var ( X ) \sigma^2=\operatorname{Var}(X) σ 2 = Var ( X ) 1 n ∑ ( X i − X ˉ ) 2 \frac1n\sum(X_i-\bar X)^2 n 1 ∑ ( X i − X ˉ ) 2 biased low . Let's see exactly why.
Start from the identity (subtract and add μ \mu μ ∑ i ( X i − X ˉ ) 2 = ∑ i ( X i − μ ) 2 − n ( X ˉ − μ ) 2 . \sum_i (X_i-\bar X)^2 = \sum_i (X_i-\mu)^2 - n(\bar X - \mu)^2. ∑ i ( X i − X ˉ ) 2 = ∑ i ( X i − μ ) 2 − n ( X ˉ − μ ) 2 . Why this step? Expand ∑ ( X i − μ ) 2 \sum(X_i-\mu)^2 ∑ ( X i − μ ) 2 X ˉ \bar X X ˉ ∑ ( X i − X ˉ ) = 0 \sum(X_i-\bar X)=0 ∑ ( X i − X ˉ ) = 0
Take expectations:
E [ ∑ i ( X i − X ˉ ) 2 ] = ∑ i E [ ( X i − μ ) 2 ] ⏟ σ 2 − n E [ ( X ˉ − μ ) 2 ] ⏟ Var ( X ˉ ) = σ 2 / n = n σ 2 − σ 2 = ( n − 1 ) σ 2 . \mathbb E\!\Big[\sum_i (X_i-\bar X)^2\Big] = \sum_i \underbrace{\mathbb E[(X_i-\mu)^2]}_{\sigma^2} - n\underbrace{\mathbb E[(\bar X-\mu)^2]}_{\operatorname{Var}(\bar X)=\sigma^2/n}=n\sigma^2-\sigma^2=(n-1)\sigma^2. E [ ∑ i ( X i − X ˉ ) 2 ] = ∑ i σ 2 E [( X i − μ ) 2 ] − n Var ( X ˉ ) = σ 2 / n E [( X ˉ − μ ) 2 ] = n σ 2 − σ 2 = ( n − 1 ) σ 2 . Why this step? Each E [ ( X i − μ ) 2 ] = σ 2 \mathbb E[(X_i-\mu)^2]=\sigma^2 E [( X i − μ ) 2 ] = σ 2 Var ( X ˉ ) = σ 2 / n \operatorname{Var}(\bar X)=\sigma^2/n Var ( X ˉ ) = σ 2 / n
So to make it unbiased we divide by n − 1 n-1 n − 1 n n n
Definition Consistent estimatorθ ^ n \hat\theta_n θ ^ n n n n consistent for θ \theta θ θ ^ n → P θ , i.e. ∀ ε > 0 : lim n → ∞ P ( ∣ θ ^ n − θ ∣ > ε ) = 0. \hat\theta_n \xrightarrow{P}\theta,\quad\text{i.e.}\quad \forall\varepsilon>0:\ \lim_{n\to\infty}P\big(|\hat\theta_n-\theta|>\varepsilon\big)=0. θ ^ n P θ , i.e. ∀ ε > 0 : lim n → ∞ P ( ∣ θ ^ n − θ ∣ > ε ) = 0.
Intuition WHY / what it capturesUnbiasedness is about any fixed n n n ; consistency is about what happens as n n n . A consistent estimator homes in on the truth as you gather more data. More data → almost surely closer.
Let m = E [ θ ^ ] m=\mathbb E[\hat\theta] m = E [ θ ^ ] ± m \pm m ± m E [ ( θ ^ − θ ) 2 ] = E [ ( θ ^ − m + m − θ ) 2 ] . \mathbb E[(\hat\theta-\theta)^2]=\mathbb E[(\hat\theta-m+m-\theta)^2]. E [( θ ^ − θ ) 2 ] = E [( θ ^ − m + m − θ ) 2 ] . Why this step? Adding zero ( − m + m ) (-m+m) ( − m + m ) = E [ ( θ ^ − m ) 2 ] ⏟ Var + 2 ( m − θ ) E [ θ ^ − m ] ⏟ = 0 + ( m − θ ) 2 ⏟ Bias 2 . =\underbrace{\mathbb E[(\hat\theta-m)^2]}_{\operatorname{Var}}+2(m-\theta)\underbrace{\mathbb E[\hat\theta-m]}_{=0}+\underbrace{(m-\theta)^2}_{\operatorname{Bias}^2}. = Var E [( θ ^ − m ) 2 ] + 2 ( m − θ ) = 0 E [ θ ^ − m ] + Bias 2 ( m − θ ) 2 . Why the middle vanishes? E [ θ ^ − m ] = m − m = 0 \mathbb E[\hat\theta-m]=m-m=0 E [ θ ^ − m ] = m − m = 0 MSE = Var + Bias 2 \boxed{\text{MSE}=\text{Var}+\text{Bias}^2} MSE = Var + Bias 2
Var ( X ˉ ) = σ 2 / n → 0 \operatorname{Var}(\bar X)=\sigma^2/n\to 0 Var ( X ˉ ) = σ 2 / n → 0 = 0 =0 = 0 MSE = σ 2 / n → 0 \operatorname{MSE}=\sigma^2/n\to0 MSE = σ 2 / n → 0
Among unbiased estimators, the more efficient one has smaller variance . The most efficient (smallest possible variance) is the MVUE (Minimum-Variance Unbiased Estimator).
Relative efficiency of θ ^ 1 \hat\theta_1 θ ^ 1 θ ^ 2 \hat\theta_2 θ ^ 2 Var ( θ ^ 2 ) Var ( θ ^ 1 ) \dfrac{\operatorname{Var}(\hat\theta_2)}{\operatorname{Var}(\hat\theta_1)} Var ( θ ^ 1 ) Var ( θ ^ 2 )
Intuition WHY a lower bound existsData carry only finite information about θ \theta θ I ( θ ) I(\theta) I ( θ ) sharpness of the likelihood's peak: a sharp peak means data pin down θ \theta θ 1 / ( n I 1 ( θ ) ) 1/(nI_1(\theta)) 1/ ( n I 1 ( θ )) n n n
Worked example (A) Comparing two unbiased estimators ofμ \mu μ X 1 , X 2 , X 3 X_1,X_2,X_3 X 1 , X 2 , X 3 σ 2 \sigma^2 σ 2 μ ^ A = X ˉ = X 1 + X 2 + X 3 3 \hat\mu_A=\bar X=\frac{X_1+X_2+X_3}{3} μ ^ A = X ˉ = 3 X 1 + X 2 + X 3 μ ^ B = X 1 + X 3 2 \hat\mu_B=\frac{X_1+X_3}{2} μ ^ B = 2 X 1 + X 3 X 2 X_2 X 2 Unbiased? E [ μ ^ A ] = E [ μ ^ B ] = μ \mathbb E[\hat\mu_A]=\mathbb E[\hat\mu_B]=\mu E [ μ ^ A ] = E [ μ ^ B ] = μ Why? Linearity, weights sum to 1.
Efficiency: Var ( μ ^ A ) = σ 2 / 3 \operatorname{Var}(\hat\mu_A)=\sigma^2/3 Var ( μ ^ A ) = σ 2 /3 Var ( μ ^ B ) = 2 ⋅ ( σ 2 / 4 ) = σ 2 / 2 \operatorname{Var}(\hat\mu_B)=2\cdot(\sigma^2/4)=\sigma^2/2 Var ( μ ^ B ) = 2 ⋅ ( σ 2 /4 ) = σ 2 /2 σ 2 / 3 < σ 2 / 2 \sigma^2/3<\sigma^2/2 σ 2 /3 < σ 2 /2 μ ^ A \hat\mu_A μ ^ A more efficient . Relative efficiency = σ 2 / 2 σ 2 / 3 = 1.5 =\frac{\sigma^2/2}{\sigma^2/3}=1.5 = σ 2 /3 σ 2 /2 = 1.5 Lesson: using all data and equal weights (for equal variances) minimizes variance.
Worked example (B) MSE can beat unbiasednessFor a Uniform( 0 , θ ) (0,\theta) ( 0 , θ ) θ ^ = max i X i \hat\theta=\max_i X_i θ ^ = max i X i biased (E [ max ] = n n + 1 θ < θ \mathbb E[\max]=\frac{n}{n+1}\theta<\theta E [ max ] = n + 1 n θ < θ n + 1 n max i X i \frac{n+1}{n}\max_i X_i n n + 1 max i X i biased shrinkage can win.
Why this step matters: "Unbiased" is not the same as "best." MSE = Var + Bias² is the honest scorecard.
Worked example (C) Consistency ofs 2 s^2 s 2 s 2 = 1 n − 1 ∑ ( X i − X ˉ ) 2 s^2=\frac1{n-1}\sum(X_i-\bar X)^2 s 2 = n − 1 1 ∑ ( X i − X ˉ ) 2 Var ( s 2 ) → 0 \operatorname{Var}(s^2)\to0 Var ( s 2 ) → 0 n → ∞ n\to\infty n → ∞ MSE → 0 \operatorname{MSE}\to0 MSE → 0 consistent . Note the biased version (divide by n n n also consistent — its bias ∝ 1 / n → 0 \propto 1/n\to0 ∝ 1/ n → 0 Why? Consistency only cares about the limit; the small finite-n n n
Common mistake "Unbiased ⇒ consistent" (and vice-versa)Why it feels right: both sound like "the estimator is good/correct."
Counterexample: θ ^ = X 1 \hat\theta=X_1 θ ^ = X 1 μ \mu μ not consistent — its variance stays σ 2 \sigma^2 σ 2 Fix: Unbiasedness = correct on average at fixed n n n . Consistency = correct in the limit n → ∞ n\to\infty n → ∞ . Different axes entirely.
Common mistake "Divide byn n n Why it feels right: variance is an average of squared deviations, so dividing by n n n Fix: You centered on X ˉ \bar X X ˉ μ \mu μ n − 1 n-1 n − 1
Common mistake "More efficient just means smaller variance, even if biased."Why it feels right: small variance = stable.
Fix: Classical efficiency compares variances only among unbiased estimators. To compare biased vs unbiased fairly, use MSE , not variance alone.
Recall Feynman: explain to a 12-year-oldImagine you're guessing how tall the average kid in your school is by measuring a few friends.
Unbiased = your guessing trick doesn't lean too tall or too short — if you repeated it tons of times, the average of your guesses lands exactly right.Consistent = the more friends you measure, the closer your guess sneaks up to the real answer.Efficient = among all fair tricks, yours wobbles the least from sample to sample. You want a trick that's fair, gets better with more data, and is steady. That's a great estimator!
"U-C-E = Center, Closer, Calm"
U nbiased → guess is C entered on truth.
C onsistent → gets C loser as n n n E fficient → stays C alm (lowest wobble/variance).
Define an unbiased estimator. E [ θ ^ ] = θ \mathbb E[\hat\theta]=\theta E [ θ ^ ] = θ for all
θ \theta θ ; i.e. bias
= E [ θ ^ ] − θ = 0 =\mathbb E[\hat\theta]-\theta=0 = E [ θ ^ ] − θ = 0 .
Why divide sample variance by n − 1 n-1 n − 1 E [ ∑ ( X i − X ˉ ) 2 ] = ( n − 1 ) σ 2 \mathbb E[\sum(X_i-\bar X)^2]=(n-1)\sigma^2 E [ ∑ ( X i − X ˉ ) 2 ] = ( n − 1 ) σ 2 ; one degree of freedom is used estimating
μ \mu μ by
X ˉ \bar X X ˉ , so
/ ( n − 1 ) /(n-1) / ( n − 1 ) removes the bias.
State the bias–variance decomposition of MSE. MSE ( θ ^ ) = Var ( θ ^ ) + Bias ( θ ^ ) 2 \operatorname{MSE}(\hat\theta)=\operatorname{Var}(\hat\theta)+\operatorname{Bias}(\hat\theta)^2 MSE ( θ ^ ) = Var ( θ ^ ) + Bias ( θ ^ ) 2 .
Definition of consistency. θ ^ n → P θ \hat\theta_n\xrightarrow{P}\theta θ ^ n P θ :
∀ ε > 0 , P ( ∣ θ ^ n − θ ∣ > ε ) → 0 \forall\varepsilon>0,\ P(|\hat\theta_n-\theta|>\varepsilon)\to0 ∀ ε > 0 , P ( ∣ θ ^ n − θ ∣ > ε ) → 0 as
n → ∞ n\to\infty n → ∞ .
Easy sufficient condition for consistency. MSE ( θ ^ n ) → 0 \operatorname{MSE}(\hat\theta_n)\to0 MSE ( θ ^ n ) → 0 (Var→0 and Bias→0), via Chebyshev.
Give an unbiased but inconsistent estimator of μ \mu μ μ ^ = X 1 \hat\mu=X_1 μ ^ = X 1 :
E [ X 1 ] = μ \mathbb E[X_1]=\mu E [ X 1 ] = μ but
Var = σ 2 \operatorname{Var}=\sigma^2 Var = σ 2 never shrinks.
What does efficiency compare and how? Among unbiased estimators, smaller variance = more efficient; relative efficiency
= Var ( θ ^ 2 ) / Var ( θ ^ 1 ) =\operatorname{Var}(\hat\theta_2)/\operatorname{Var}(\hat\theta_1) = Var ( θ ^ 2 ) / Var ( θ ^ 1 ) .
State the Cramér–Rao Lower Bound. Var ( θ ^ ) ≥ 1 / I ( θ ) \operatorname{Var}(\hat\theta)\ge 1/I(\theta) Var ( θ ^ ) ≥ 1/ I ( θ ) , with Fisher info
I ( θ ) = E [ ( ∂ θ ln f ) 2 ] I(\theta)=\mathbb E[(\partial_\theta\ln f)^2] I ( θ ) = E [( ∂ θ ln f ) 2 ] .
Why is X ˉ \bar X X ˉ μ \mu μ Unbiased with
Var = σ 2 / n → 0 \operatorname{Var}=\sigma^2/n\to0 Var = σ 2 / n → 0 , so MSE
→ 0 \to0 → 0 (Weak LLN).
Is unbiasedness sufficient for "best"? No; a biased estimator can have lower MSE. Use MSE = Var + Bias² as the true scorecard.
Random variable with sampling distribution
E of theta-hat equals theta
Divide by n-1 for variance s^2
Converges to theta as n grows
Smallest variance among good rules
Intuition Hinglish mein samjhoDekho, hum ek unknown number θ \theta θ θ ^ \hat\theta θ ^ θ ^ \hat\theta θ ^
Teen tareeke se judge karte hain. Unbiased matlab average mein bilkul sahi — agar infinite baar repeat karo toh guesses ka average exactly θ \theta θ n − 1 n-1 n − 1 n n n μ \mu μ X ˉ \bar X X ˉ
Consistent matlab jaise-jaise data badhta hai (n → ∞ n\to\infty n → ∞ = = = + + + 2 ^2 2 θ ^ = X 1 \hat\theta=X_1 θ ^ = X 1
Efficient matlab jitne bhi fair (unbiased) rules hain, unmein sabse kam wobble (variance) wala. Cramér–Rao bound batata hai ki variance ek floor 1 / I ( θ ) 1/I(\theta) 1/ I ( θ ) Center, Closer, Calm — sahi center par, data badhne par closer, aur sabse calm (steady). Yeh teen properties estimator ki quality ka pura scorecard hain.
