Exercises — Properties of estimators — unbiasedness, consistency, efficiency

Estimator woh rule/random variable hai $\bar{X}=\frac{1}{n}\sum X_i$ — yeh data dekhne se pehle exist karta hai aur iska poora distribution hota hai. Estimate woh single number hai $12.4$ — real data plug in karne ke baad ka output. Analogy: estimator recipe hai; estimate aaj aapne jo cake banayi woh hai.

Bias $=\mathbb{E}[\hat{\theta }]-\theta =\frac{3}{n}$. Kyunki finite $n$ ke liye yeh zero nahi hai, estimator biased hai. Lekin $\frac{3}{n}\to 0$ jab $n\to \infty$, toh yeh asymptotically unbiased hai — systematic error zyada data ke saath shrink hoti hai.

(a) → efficiency (unbiased mein sabse kam wobble waala). (b) → unbiasedness ($\mathbb{E}[\hat{\theta }]=\theta$). (c) → consistency (${\hat{\theta }}_n\stackrel{P}{\to }\theta$).

Linearity of expectation se (yeh sums aur constants ke through pass hoti hai): $\mathbb{E}[\hat{\mu }]=0.3\mathbb{E}[X_1]+0.7\mathbb{E}[X_2]=0.3\mu +0.7\mu =(0.3+0.7)\mu =\mu .$ Unbiased ✓ — kyunki weights sum to $1$ hain. Same-mean variables ke weighted average ke liye yahi poora rule hai.

$\mathrm{Var}(\bar{X})=\frac{{\sigma }^2}{n}=\frac{100}{25}=4,\text{standard error}=\sqrt{4}=2.$ Dhyaan do: $n$ ko chaar guna karo aur standard error sirf half hoti hai — diminishing returns ka $\sqrt{n}$ law. (Dekho Law of Large Numbers kyun $\mathrm{Var}(\bar{X})\to 0$ hota hai.)

Mean $\bar{X}=\frac{2+4+6}{3}=4$. Squared deviations: $(2-4{)}^2=4$, $(4-4{)}^2=0$, $(6-4{)}^2=4$; sum $=8$. $s^2=\frac{8}{n-1}=\frac{8}{2}=4,{\hat{\sigma }}^2=\frac{8}{n}=\frac{8}{3}\approx \mathrm{2.667.}$ Biased wala chhota hai — exactly woh "shrink" jo $n-1$ correction undo karta hai.

Independent variables ke liye, $\mathrm{Var}(\sum w_i{X}_i)=\sum w_i^2{\sigma }^2$. $\mathrm{Var}({\hat{\mu }}_A)=(4\cdot \frac{1}{16}){\sigma }^2=\frac{1}{4}{\sigma }^2=0.25{\sigma }^2.$ $\mathrm{Var}({\hat{\mu }}_B)=(\frac{1}{4}+3\cdot \frac{1}{36}){\sigma }^2=(\frac{1}{4}+\frac{1}{12}){\sigma }^2=\frac{1}{3}{\sigma }^2\approx 0.333{\sigma }^2.$ Kyunki $0.25<0.333$, ${\hat{\mu }}_A$ zyada efficient hai. $A$ ki $B$ ke relative mein relative efficiency: $\frac{\mathrm{Var}({\hat{\mu }}_B)}{\mathrm{Var}({\hat{\mu }}_A)}=\frac{1/3}{1/4}=\frac{4}{3}\approx \mathrm{1.333.}$ Lesson: equal-variance data ke liye, equal weights ($1/n$ each) variance minimise karte hain — isliye $\bar{X}$ winner hai.

$\mathrm{MSE}=\mathrm{Var}+{\mathrm{Bias}}^2$ use karke: ${\mathrm{MSE}}_1=5+2^2=9,{\mathrm{MSE}}_2=8+0=8.$ Unbiased rival yahan jeetta hai ($8<9$) — lekin dhyaan do biased wala jeet sakta tha agar uski variance kaafi low hoti. Yahi ek line mein Bias–Variance Tradeoff hai: thoda bias tabhi worth it hai jab yeh enough variance reduction khareed le.

$\mathrm{MSE}({\hat{\theta }}_n)=\frac{9}{n}+(\frac{4}{n}{)}^2=\frac{9}{n}+\frac{16}{n^2}\underset{n\to \infty }{\to }0.$ Kyunki $\mathrm{MSE}\to 0$, sufficient condition (Chebyshev) se ${\hat{\theta }}_n\stackrel{P}{\to }\theta$: consistent hai. Wobble aur off-centre dono parts vanish ho jaate hain.

Unbiased: $\mathbb{E}[X_1]=\mu$. ✓ Consistency check variance se: $\mathrm{Var}(\hat{\mu })=\mathrm{Var}(X_1)={\sigma }^2$ har $n$ ke liye — yeh kabhi shrink nahi karta. Estimate ${\sigma }^2$ se wobble karta hai chahe aap kitna bhi data collect karo, kyunki aap saara extra data throw away kar dete ho. Toh $\mathrm{MSE}={\sigma }^2\nrightarrow 0$, aur $P(|X_1-\mu |>\epsilon )$ fixed rehti hai. Consistent nahi. Yeh prove karta hai unbiased $\nRightarrow$ consistent: yeh dono alag cheezein measure karte hain.

(a) $\mathrm{Bias}(M)=\mathbb{E}[M]-\theta =\frac{n}{n+1}\theta -\theta =-\frac{1}{n+1}\theta$. Yeh hamesha underestimate karta hai — sample max true ceiling $\theta$ se kabhi exceed nahi kar sakta. (b) Shrink factor ke reciprocal se multiply karo: ${\hat{\theta }}_{\text{unb}}=\frac{n+1}{n}M$. Tab $\mathbb{E}\left[\frac{n+1}{n}M\right]=\frac{n+1}{n}\cdot \frac{n}{n+1}\theta =\theta$. ✓ (c) $n=4$ ke liye: factor $=\frac{n+1}{n}=\frac{5}{4}=1.25$.

$\frac{\mathrm{Var}(\text{median})}{\mathrm{Var}(\bar{X})}=\frac{\frac{\pi }{2}{\sigma }^2/n}{{\sigma }^2/n}=\frac{\pi }{2}\approx \mathrm{1.571.}$ Mean $\pi /2\approx 1.571$ ke factor se zyada efficient hai: median lagbhag $36\%$ information waste karta hai (yeh sirf $\approx 64\%$ efficiency tak pahunchta hai, kyunki $2/\pi \approx 0.637$). Normal data ke liye mean efficient choice hai; median tabhi payoff karta hai jab tails heavy hon.

$\mathrm{Var}(\hat{\mu })={\sigma }^2\sum w_i^2$ minimise karo subject to $\sum w_i=1$. Squares kyun phir? Independence ⇒ variance $w_i^2$ add karta hai. $\sum w_i^2$ ko fixed sum ke under minimise karna ek classic hai: Likho $w_i=\frac{1}{n}+d_i$ jahan $\sum d_i=0$ (equal weights se deviations). Tab $\sum w_i^2=\sum (\frac{1}{n}+d_i{)}^2=\sum \frac{1}{n^2}+\frac{2}{n}\underset{=0}{\underbrace{\sum d_i}}+\sum d_i^2=\frac{1}{n}+\sum d_i^2.$ Kyunki $\sum d_i^2\ge 0$ equality ke saath iff har $d_i=0$, minimum $w_i=\frac{1}{n}$ pe hai. ∎ Minimum variance $={\sigma }^2/n$ — woh value jo $\bar{X}$ achieve karta hai. Equal weights se koi bhi deviation variance strictly badhata hai.

(a) $n$ i.i.d. points ke liye information add hoti hai: $I_n(\lambda )=n{I}_1(\lambda )=\frac{n}{\lambda }$. Toh $\mathrm{Var}(\hat{\lambda })\ge \frac{1}{I_n(\lambda )}=\frac{\lambda }{n}.$ (b) Poisson ke liye, $\mathrm{Var}(X_i)=\lambda$, isliye $\mathrm{Var}(\bar{X})=\frac{\lambda }{n}$. Yeh bound ke barabar hai, aur $\bar{X}$ unbiased hai ($\mathbb{E}[\bar{X}]=\lambda$). Isliye $\bar{X}$, $\lambda$ ka efficient (variance-attaining) estimator hai. ∎ $I_1=1/\lambda$ ki sanity: $\ln f=-\lambda +x\ln \lambda -\ln x!$, toh ${\partial }_{\lambda }\ln f=-1+x/\lambda$; iska variance hai $\mathrm{Var}(x)/{\lambda }^2=\lambda /{\lambda }^2=1/\lambda$. ✓

$\mathbb{E}[{\hat{\theta }}_c]=c\theta$, toh $\mathrm{Bias}=c\theta -\theta =(c-1)\theta$, aur $\mathrm{Var}({\hat{\theta }}_c)=c^{2}v$. $\mathrm{MSE}(c)=c^{2}v+(c-1{)}^2{\theta }^2.$ Differentiate karo aur $0$ set karo: $\frac{d}{dc}=2cv+2(c-1){\theta }^2=0\Rightarrow c(v+{\theta }^2)={\theta }^2$, toh $\boxed{c^{\star }=\frac{{\theta }^2}{{\theta }^2+v}}<1.$ 1 se chhota kyun? Thodi si shrinkage ek chhota bias trade karta hai variance mein badi cut ke liye — Bias–Variance Tradeoff quantitatively. Example: ${\theta }^2=4$, $v=1$ deta hai $c^{\star }=4/5=0.8$, aur $\mathrm{MSE}(0.8)=0.8^2\cdot 1+(0.8-1{)}^2\cdot 4=0.64+0.16=0.8<v=1$. Unbiased choice $c=1$ MSE $=1$ deta hai, toh shrinkage strictly jeetta hai.

Terms ki recognition ::: L1 — estimator vs estimate, bias spot karna. Formulas mein plug in karna ::: L2 — $\mathrm{Var}(\bar{X})={\sigma }^2/n$, $s^2$ with $n-1$. Compare aur decompose karna ::: L3 — relative efficiency, MSE $=$ Var $+$ Bias${}^2$. Ideas combine karna ::: L4 — unbiased $\ne$ consistent, Uniform max ko unbiasing karna. Prove aur design karna ::: L5 — optimal weights, Cramér–Rao attainment, MSE-optimal shrinkage.