Noesis
4.9.22 · HinglishProbability Theory & Statistics

Linear regression — least squares, inference on coefficients

1,763 words8 min read↑ Read in English

4.9.22 · Maths › Probability Theory & Statistics

HUM kya model kar rahe hain?

Squared errors kyun? Squaring (1) bade misses ko zyada punish karta hai, (2) smooth/differentiable hai isliye calculus kaam karta hai, aur (3) Gaussian noise ke under yeh maximum likelihood fit ke barabar hai. Absolute errors bhi "misses ko shrink" karte lekin unka clean closed form nahi hota.

KAISE: least squares ko scratch se derive karna

Hum sum of squared residuals minimise karte hain:

Step 1 — partial derivatives ko zero set karo (yeh ek paraboloid hai, toh minimum exist karta hai).

\;\Rightarrow\; \sum y_i = n\beta_0 + \beta_1\sum x_i.$$ *Yeh step kyun?* Ek smooth convex function ka minimum wahin hota hai jahan gradient zero ho. $$\frac{\partial S}{\partial \beta_1}=-2\sum x_i(y_i-\beta_0-\beta_1 x_i)=0 \;\Rightarrow\; \sum x_i y_i = \beta_0\sum x_i + \beta_1\sum x_i^2.$$ Yeh dono **normal equations** hain. **Step 2 — pehli equation ko $n$ se divide karo.** $\bar x=\frac1n\sum x_i$, $\bar y=\frac1n\sum y_i$ ke saath: $$\bar y=\hat\beta_0+\hat\beta_1\bar x\;\Rightarrow\; \boxed{\hat\beta_0=\bar y-\hat\beta_1\bar x.}$$ *Kyun?* Fitted line **hamesha $(\bar x,\bar y)$ se guzarti hai** — data ka centre of mass. **Step 3 — wapas substitute karo taaki $\hat\beta_1$ isolate ho.** Centred sums define karo: $$S_{xx}=\sum (x_i-\bar x)^2,\qquad S_{xy}=\sum (x_i-\bar x)(y_i-\bar y).$$ $\hat\beta_0$ ko doosri normal equation mein daalo aur simplify karo: > [!formula] Least-squares estimates > $$\boxed{\;\hat\beta_1=\frac{S_{xy}}{S_{xx}}=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}, > \qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\;}$$ > $\hat\beta_1$ ko aise padho: "kitna $y$ aur $x$ saath chalte hain, $x$ ki spread se scale karke." ![[4.9.22-Linear-regression-—-least-squares,-inference-on-coefficients.png]] --- ## Yeh estimate achha kyun hai (Gauss–Markov, briefly) Upar ke assumptions ke under, $\hat\beta_1$ aur $\hat\beta_0$ **BLUE** hain — ==Best (minimum variance) Linear Unbiased Estimators==. Unbiased: $E[\hat\beta_1]=\beta_1$. "Best": saare linear unbiased estimators mein se kisi ka bhi variance chhota nahi. Gaussian ki zarurat nahi iske liye. --- ## Inference: slope par kitna trust karein Slope, $y_i$ ka ek weighted sum hai, isliye yeh samples ke across **random** hai. Likhte hain: $$\hat\beta_1=\sum_i c_i y_i,\qquad c_i=\frac{x_i-\bar x}{S_{xx}}.$$ **Variance derivation.** Kyunki $\text{Var}(y_i)=\sigma^2$ aur yeh independent hain: $$\text{Var}(\hat\beta_1)=\sum_i c_i^2\,\sigma^2=\frac{\sigma^2}{S_{xx}^2}\sum(x_i-\bar x)^2 =\boxed{\frac{\sigma^2}{S_{xx}}.}$$ *Yeh intuitive kyun hai:* $x$ jitne zyada **spread out** honge ($S_{xx}$ bada) aur **noise** jitni kam ($\sigma^2$ chhota), slope utna precisely pin down hoga — lamba lever aim karna aasaan hota hai. Isi tarah $\;\text{Var}(\hat\beta_0)=\sigma^2\left(\dfrac1n+\dfrac{\bar x^2}{S_{xx}}\right).$ **Hum $\sigma^2$ nahi jaante**, toh residuals se estimate karo: > [!formula] Residual variance (Mean Squared Error) > $$\hat\sigma^2=s^2=\frac{\sum e_i^2}{n-2}=\frac{\text{SSE}}{n-2}.$$ > **$n-2$ kyun?** Humne $\hat\beta_0,\hat\beta_1$ estimate karte waqt 2 degrees of freedom "use" kar liye, toh $n-2$ > se divide karne par (naki $n$) $s^2$ **unbiased** rehta hai $\sigma^2$ ke liye. > [!formula] Standard error & t-statistic > $$\text{SE}(\hat\beta_1)=\frac{s}{\sqrt{S_{xx}}},\qquad > t=\frac{\hat\beta_1-\beta_1^{(0)}}{\text{SE}(\hat\beta_1)}\sim t_{n-2}.$$ > Gaussian errors ke under. "Koi relation hai kya?" test karne ke liye $\beta_1^{(0)}=0$ use karo. > **CI:** $\hat\beta_1 \pm t_{n-2,\,1-\alpha/2}\cdot \text{SE}(\hat\beta_1).$ **Pehle predict, phir verify:** compute karne se pehle socho — agar $x$ ki spread *double* kar do aur noise fixed rakho, toh $\text{SE}(\hat\beta_1)$ ko *$1/2$ shrink* hona chahiye (kyunki $S_{xx}$ quadruple hota hai, $\sqrt{S_{xx}}$ double hota hai). Formula mein check karo — haan. --- ## Goodness of fit: $R^2$ $$\underbrace{\sum(y_i-\bar y)^2}_{\text{SST}}=\underbrace{\sum(\hat y_i-\bar y)^2}_{\text{SSR (explained)}} +\underbrace{\sum(y_i-\hat y_i)^2}_{\text{SSE (residual)}},\qquad R^2=\frac{\text{SSR}}{\text{SST}}=1-\frac{\text{SSE}}{\text{SST}}.$$ $R^2$ = $y$ mein variance ka woh fraction jo line explain karti hai. Simple regression mein $R^2=r^2$, squared correlation. **Yeh split kyun kaam karta hai:** residuals, fitted values ke orthogonal hote hain ($n$-dimensional space mein Pythagoras). --- ## Worked Examples > [!example] Example 1 — haath se line fit karo > Data: $x=(1,2,3,4,5)$, $y=(2,4,5,4,5)$. > - $\bar x=3,\ \bar y=4$. *Pehle kyun?* Sab kuch means ke aas-paas centre hota hai. > - $S_{xx}=\sum(x-3)^2=4+1+0+1+4=10$. > - $S_{xy}=(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)=4+0+0+0+2=6$. > - $\hat\beta_1=6/10=0.6$. *Kyun?* slope = co-movement / x-spread. > - $\hat\beta_0=4-0.6(3)=2.2$. Line: $\hat y=2.2+0.6x$. > - Check karo ki $(3,4)$ se guzarti hai: $2.2+1.8=4$ ✓. > [!example] Example 2 — slope par inference > Example 1 continue karo. Fitted values: $2.8,3.4,4.0,4.6,5.2$. > Residuals $e=(-0.8,0.6,1.0,-0.6,-0.2)$. $\sum e_i^2=0.64+0.36+1.0+0.36+0.04=2.4$. > - $s^2=\dfrac{2.4}{5-2}=0.8$, $s=0.894$. *$n-2=3$ kyun?* do coefficients estimate kiye. > - $\text{SE}(\hat\beta_1)=\dfrac{0.894}{\sqrt{10}}=0.283$. > - $t=\dfrac{0.6}{0.283}=2.12$ on $3$ df. Critical $t_{3,0.975}=3.18$. > - Kyunki $2.12<3.18$, hum 5% par $\beta_1=0$ ko **reject fail karte hain**: slope significant nahi (chhota sample!). > - 95% CI: $0.6\pm 3.18(0.283)=[-0.30,\,1.50]$ — 0 ko straddle karta hai, consistent. > [!example] Example 3 — $R^2$ kya batata hai > $\text{SST}=\sum(y-4)^2=4+0+1+0+1=6$. $\text{SSE}=2.4$. > $R^2=1-2.4/6=0.6$. Line, $y$ mein variation ka 60% explain karti hai. *Kyun care karein?* Yahan "non-significant" slope hone ke bawajood, point estimate ek accha hissa explain karta hai — chhota $n$ bas hume uncertain banata hai. --- ## Common Mistakes (Steel-manned) > [!mistake] "Zyada $R^2$ matlab model correct / causal hai." > *Kyun sahi lagta hai:* tight fit sach jaisa dikhta hai. **Fix:** $R^2$ sirf variance explain karta hai > *ek line se*; causation, omitted variables, ya yeh ki line sahi shape hai ya nahi, iske baare mein kuch nahi kehta. Hamesha residual plot dekho. > [!mistake] "$\sigma^2$ estimate karne ke liye SSE ko $n$ se divide karo." > *Kyun sahi lagta hai:* variance "average squared deviation" hai, aur hum baaki jagah $n$ se average karte hain. > **Fix:** yahan residuals 2 constraints satisfy karne par majboor hain ($\sum e_i=0$, $\sum x_i e_i=0$), toh > sirf $n-2$ information pieces bachte hain. $n$ se divide karna $\sigma^2$ ko **underestimate** karta hai. > [!mistake] "Chhota SE matlab bada/important effect." > *Kyun sahi lagta hai:* chhota SE → significant → "badi baat." **Fix:** SE *precision* measure karta hai, > *size* nahi. Ek tiny lekin ultra-precisely-estimated slope significant ho sakta hai par practically meaningless. Dono $\hat\beta_1$ aur uska CI report karo. > [!mistake] "Slope ka Confidence interval = interval jo 95% data contain karta hai." > *Kyun sahi lagta hai:* dono "95% intervals" hain. **Fix:** CI *parameter* $\beta_1$ ke baare mein hai, > individual data points ke nahi. Individual $y$ ke liye tumhe ek (wider) **prediction interval** chahiye. --- > [!recall]- Feynman: 12-saal ke bachche ko explain karo > Tum kaagaz par dots scatter karte ho jo *roughly* upar jaate hain. Tum woh ek straight line banana chahte ho jo sabse zyada dots ke paas aaye. "Paas" ka matlab: measure karo ki har dot tumhari line ke *upar ya neeche* kitna hai, woh distances square karo (taaki bade misses sach mein count hon), aur woh line chuno jahan total sabse chhota ho. > Yahi least squares hai. Ab: agar tumhare paas sirf kuch dots hote, ya woh sab ek saath left-right bunched hote, > toh tum slope ke baare mein *sure* nahi ho sakte — ek dot hilao aur line bahut tilt ho jaati hai. "Standard error" bas *kitna tumhari line wobble karti* agar tum experiment dobara karte. Agar wobble slope se badi hai, tum claim nahi kar sakte ki line sach mein upar jaati hai. > [!mnemonic] Slope aur uska trust yaad karo > **"Slope = SxY over SxX; trust badhta hai SPREAD se."** > $\hat\beta_1=\dfrac{S_{xy}}{S_{xx}}$, aur $\text{SE}\propto \dfrac{1}{\sqrt{S_{xx}}}$ — > zyada $x$-**spread** ⇒ kam wobble. --- ## Connections - [[Correlation coefficient]] — $\hat\beta_1 = r\,(s_y/s_x)$, aur $R^2=r^2$. - [[Covariance]] — $S_{xy}/n$ sample covariance hai. - [[Maximum Likelihood Estimation]] — Gaussian errors ke under LS = MLE. - [[t-distribution]] — coefficient test ke liye reference law deta hai. - [[Hypothesis Testing]] — $\beta_1$ par $t$-test. - [[Multiple Regression]] — matrix form $\hat\beta=(X^TX)^{-1}X^Ty$. - [[Gauss–Markov Theorem]] — LS BLUE kyun hai. --- ## #flashcards/maths Least squares kya minimise karta hai? ::: *Vertical* residuals ka sum of squares $\sum (y_i-\hat\beta_0-\hat\beta_1 x_i)^2$. Do normal equations kya zero set karne se aati hain? ::: $S$ ke $\beta_0$ aur $\beta_1$ ke saath partial derivatives. Slope estimate ka formula? ::: $\hat\beta_1=S_{xy}/S_{xx}=\sum(x_i-\bar x)(y_i-\bar y)/\sum(x_i-\bar x)^2$. Intercept estimate ka formula? ::: $\hat\beta_0=\bar y-\hat\beta_1\bar x$. Fitted line hamesha kis point se guzarti hai? ::: Centroid $(\bar x,\bar y)$ se. $\text{Var}(\hat\beta_1)=?$ aur yeh kya batata hai? ::: $\sigma^2/S_{xx}$; precision badhti hai zyada x-spread aur kam noise se. $s^2$ ke liye SSE ko $n-2$ se kyun divide karte hain? ::: $\hat\beta_0,\hat\beta_1$ estimate karne mein do degrees of freedom lost; unbiased $\hat\sigma^2$ deta hai. Slope ka standard error? ::: $\text{SE}(\hat\beta_1)=s/\sqrt{S_{xx}}$ jahan $s=\sqrt{\text{SSE}/(n-2)}$. $H_0:\beta_1=0$ ke liye test statistic aur uski distribution? ::: $t=\hat\beta_1/\text{SE}(\hat\beta_1)\sim t_{n-2}$. $R^2$ kya measure karta hai? ::: Regression se $y$ mein explain hue variance ka fraction: $1-\text{SSE}/\text{SST}$. SST decomposition? ::: $\text{SST}=\text{SSR}+\text{SSE}$ (total = explained + residual). LS estimates ko "BLUE" banane wali property kya hai? ::: Best (min variance) Linear Unbiased Estimators, Gauss–Markov se. $\varepsilon_i$ aur $e_i$ mein kya fark hai? ::: $\varepsilon_i$ true unobserved error hai; $e_i=y_i-\hat y_i$ observed residual hai. ## 🖼️ Concept Map ```mermaid flowchart TD M[Linear model yi eq b0 plus b1 xi plus err] -->|fit via| LS[Least squares min SSR] LS -->|set gradient to zero| NE[Normal equations] NE -->|solve| B1[slope b1 eq Sxy over Sxx] NE -->|solve| B0[intercept b0 eq ybar minus b1 xbar] B0 -->|line passes through| CM[Centre of mass xbar ybar] M -->|noise assumptions| ASM[Errors mean 0 var sigma2 uncorrelated] ASM -->|guarantees| GM[Gauss-Markov BLUE] B1 -->|is| GM LS -->|squared errors give| MLE[Max likelihood under Gaussian noise] M -->|observed gap| RES[Residual ei eq yi minus yhat] GM -->|enables| INF[Inference on coefficients] ``` ## 🔬 Deep Dive > [!intuition] Aur deep jao — visual, zero se > Is topic ke step-by-step 3Blue1Brown-style breakdowns. - [[4.9.22 D1 Foundations|D1 · Foundations — har symbol zero se]] - [[4.9.22 D2 Visual Walkthrough|D2 · Visual walkthrough — derivation pictures mein]] - [[4.9.22 D3 Worked Examples|D3 · Worked examples — har scenario]] - [[4.9.22 D4 Exercises|D4 · Exercises — graded, full solutions]] - [[4.9.22 D5 Question Bank|D5 · Question bank — concept traps]]