4.9.22Probability Theory & Statistics
Linear regression — least squares, inference on coefficients
1,960 words9 min readdifficulty · medium2 backlinks
WHAT are we modelling?
WHY squared errors? Squaring (1) punishes big misses more, (2) is smooth/differentiable so calculus works, and (3) under Gaussian noise it equals the maximum likelihood fit. Absolute errors would also "shrink misses" but give no clean closed form.
HOW: deriving least squares from scratch
We minimise the sum of squared residuals:
Step 1 — set partial derivatives to zero (it's a paraboloid, so a min exists).
\;\Rightarrow\; \sum y_i = n\beta_0 + \beta_1\sum x_i.$$ *Why this step?* The minimum of a smooth convex function is where the gradient vanishes. $$\frac{\partial S}{\partial \beta_1}=-2\sum x_i(y_i-\beta_0-\beta_1 x_i)=0 \;\Rightarrow\; \sum x_i y_i = \beta_0\sum x_i + \beta_1\sum x_i^2.$$ These two are the **normal equations**. **Step 2 — divide the first by $n$.** With $\bar x=\frac1n\sum x_i$, $\bar y=\frac1n\sum y_i$: $$\bar y=\hat\beta_0+\hat\beta_1\bar x\;\Rightarrow\; \boxed{\hat\beta_0=\bar y-\hat\beta_1\bar x.}$$ *Why?* The fitted line **always passes through $(\bar x,\bar y)$** — the centre of mass of the data. **Step 3 — substitute back to isolate $\hat\beta_1$.** Define centred sums: $$S_{xx}=\sum (x_i-\bar x)^2,\qquad S_{xy}=\sum (x_i-\bar x)(y_i-\bar y).$$ Plugging $\hat\beta_0$ into the second normal equation and simplifying gives > [!formula] Least-squares estimates > $$\boxed{\;\hat\beta_1=\frac{S_{xy}}{S_{xx}}=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}, > \qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\;}$$ > Read $\hat\beta_1$ as "how $y$ co-moves with $x$, scaled by how spread out $x$ is." ![[4.9.22-Linear-regression-—-least-squares,-inference-on-coefficients.png]] --- ## WHY this estimate is good (Gauss–Markov, briefly) Under the assumptions above, $\hat\beta_1$ and $\hat\beta_0$ are **BLUE** — the ==Best (minimum variance) Linear Unbiased Estimators==. Unbiased: $E[\hat\beta_1]=\beta_1$. "Best": among all linear unbiased estimators none has smaller variance. No Gaussian needed for this. --- ## Inference: how much to trust the slope The slope is a weighted sum of the $y_i$, so it's **random** across samples. Write $$\hat\beta_1=\sum_i c_i y_i,\qquad c_i=\frac{x_i-\bar x}{S_{xx}}.$$ **Variance derivation.** Since $\text{Var}(y_i)=\sigma^2$ and they're independent: $$\text{Var}(\hat\beta_1)=\sum_i c_i^2\,\sigma^2=\frac{\sigma^2}{S_{xx}^2}\sum(x_i-\bar x)^2 =\boxed{\frac{\sigma^2}{S_{xx}}.}$$ *Why this is intuitive:* the **more spread out** the $x$'s ($S_{xx}$ large) and the **less noise** ($\sigma^2$ small), the more precisely the slope is pinned down — a long lever is easier to aim. Similarly $\;\text{Var}(\hat\beta_0)=\sigma^2\left(\dfrac1n+\dfrac{\bar x^2}{S_{xx}}\right).$ **We don't know $\sigma^2$**, so estimate it from residuals: > [!formula] Residual variance (Mean Squared Error) > $$\hat\sigma^2=s^2=\frac{\sum e_i^2}{n-2}=\frac{\text{SSE}}{n-2}.$$ > **Why $n-2$?** We "used up" 2 degrees of freedom estimating $\hat\beta_0,\hat\beta_1$, so dividing > by $n-2$ (not $n$) makes $s^2$ **unbiased** for $\sigma^2$. > [!formula] Standard error & t-statistic > $$\text{SE}(\hat\beta_1)=\frac{s}{\sqrt{S_{xx}}},\qquad > t=\frac{\hat\beta_1-\beta_1^{(0)}}{\text{SE}(\hat\beta_1)}\sim t_{n-2}.$$ > Under Gaussian errors. To test "is there any relation?" use $\beta_1^{(0)}=0$. > **CI:** $\hat\beta_1 \pm t_{n-2,\,1-\alpha/2}\cdot \text{SE}(\hat\beta_1).$ **Forecast-then-verify:** before computing, predict — if you *doubled* the spread of $x$ while keeping noise fixed, $\text{SE}(\hat\beta_1)$ should *shrink by $1/2$* (since $S_{xx}$ quadruples, $\sqrt{S_{xx}}$ doubles). Check it on the formula — yes. --- ## Goodness of fit: $R^2$ $$\underbrace{\sum(y_i-\bar y)^2}_{\text{SST}}=\underbrace{\sum(\hat y_i-\bar y)^2}_{\text{SSR (explained)}} +\underbrace{\sum(y_i-\hat y_i)^2}_{\text{SSE (residual)}},\qquad R^2=\frac{\text{SSR}}{\text{SST}}=1-\frac{\text{SSE}}{\text{SST}}.$$ $R^2$ = fraction of variance in $y$ explained by the line. For simple regression $R^2=r^2$, the squared correlation. **WHY the split works:** residuals are orthogonal to fitted values (a Pythagoras in $n$-dimensional space). --- ## Worked Examples > [!example] Example 1 — fit a line by hand > Data: $x=(1,2,3,4,5)$, $y=(2,4,5,4,5)$. > - $\bar x=3,\ \bar y=4$. *Why first?* Everything centres on the means. > - $S_{xx}=\sum(x-3)^2=4+1+0+1+4=10$. > - $S_{xy}=(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)=4+0+0+0+2=6$. > - $\hat\beta_1=6/10=0.6$. *Why?* slope = co-movement / x-spread. > - $\hat\beta_0=4-0.6(3)=2.2$. Line: $\hat y=2.2+0.6x$. > - Check it passes through $(3,4)$: $2.2+1.8=4$ ✓. > [!example] Example 2 — inference on the slope > Continue Example 1. Fitted values: $2.8,3.4,4.0,4.6,5.2$. > Residuals $e=(-0.8,0.6,1.0,-0.6,-0.2)$. $\sum e_i^2=0.64+0.36+1.0+0.36+0.04=2.4$. > - $s^2=\dfrac{2.4}{5-2}=0.8$, $s=0.894$. *Why $n-2=3$?* two coefficients estimated. > - $\text{SE}(\hat\beta_1)=\dfrac{0.894}{\sqrt{10}}=0.283$. > - $t=\dfrac{0.6}{0.283}=2.12$ on $3$ df. Critical $t_{3,0.975}=3.18$. > - Since $2.12<3.18$, we **fail to reject** $\beta_1=0$ at 5%: slope not significant (small sample!). > - 95% CI: $0.6\pm 3.18(0.283)=[-0.30,\,1.50]$ — straddles 0, consistent. > [!example] Example 3 — what $R^2$ tells you > $\text{SST}=\sum(y-4)^2=4+0+1+0+1=6$. $\text{SSE}=2.4$. > $R^2=1-2.4/6=0.6$. The line explains 60% of the variation in $y$. *Why care?* Even with a > "non-significant" slope here, the point estimate explains a fair chunk — small $n$ just makes us > uncertain. --- ## Common Mistakes (Steel-manned) > [!mistake] "High $R^2$ means the model is correct / causal." > *Why it feels right:* a tight fit looks like truth. **Fix:** $R^2$ only measures variance explained > *by a line*; it says nothing about causation, omitted variables, or whether a line is even the right > shape. Always look at a residual plot. > [!mistake] "Divide SSE by $n$ to estimate $\sigma^2$." > *Why it feels right:* variance is "average squared deviation," and we average by $n$ everywhere else. > **Fix:** here the residuals are forced to satisfy 2 constraints ($\sum e_i=0$, $\sum x_i e_i=0$), so > only $n-2$ pieces of information remain. Dividing by $n$ **underestimates** $\sigma^2$. > [!mistake] "A small SE means a big/important effect." > *Why it feels right:* small SE → significant → "big deal." **Fix:** SE measures *precision*, not > *size*. A tiny but ultra-precisely-estimated slope can be significant yet practically meaningless. > Report both $\hat\beta_1$ and its CI. > [!mistake] "Confidence interval for the slope = interval that contains 95% of the data." > *Why it feels right:* both are "95% intervals." **Fix:** the CI is about the *parameter* $\beta_1$, > not individual data points. For individual $y$ you need a (wider) **prediction interval**. --- > [!recall]- Feynman: explain to a 12-year-old > You scatter dots on paper that *roughly* go up. You want to draw the one straight line that comes > closest to all the dots. "Closest" means: measure how far each dot is *above or below* your line, > square those distances (so big misses really count), and pick the line where the total is smallest. > That's least squares. Now: if you only had a few dots, or they were all bunched together > left-right, you couldn't be *sure* of the slope — wiggle one dot and the line tilts a lot. The > "standard error" is just *how much your line would wobble* if you redid the experiment. If the > wobble is bigger than the slope itself, you can't claim the line really goes up. > [!mnemonic] Remember the slope and its trust > **"Slope = SxY over SxX; trust grows with SPREAD."** > $\hat\beta_1=\dfrac{S_{xy}}{S_{xx}}$, and $\text{SE}\propto \dfrac{1}{\sqrt{S_{xx}}}$ — > more $x$-**spread** ⇒ less wobble. --- ## Connections - [[Correlation coefficient]] — $\hat\beta_1 = r\,(s_y/s_x)$, and $R^2=r^2$. - [[Covariance]] — $S_{xy}/n$ is the sample covariance. - [[Maximum Likelihood Estimation]] — LS = MLE under Gaussian errors. - [[t-distribution]] — gives the reference law for the coefficient test. - [[Hypothesis Testing]] — the $t$-test on $\beta_1$. - [[Multiple Regression]] — matrix form $\hat\beta=(X^TX)^{-1}X^Ty$. - [[Gauss–Markov Theorem]] — why LS is BLUE. --- ## #flashcards/maths What does least squares minimise? ::: The sum of squared *vertical* residuals $\sum (y_i-\hat\beta_0-\hat\beta_1 x_i)^2$. The two normal equations come from setting what to zero? ::: The partial derivatives of $S$ w.r.t. $\beta_0$ and $\beta_1$. Formula for the slope estimate? ::: $\hat\beta_1=S_{xy}/S_{xx}=\sum(x_i-\bar x)(y_i-\bar y)/\sum(x_i-\bar x)^2$. Formula for the intercept estimate? ::: $\hat\beta_0=\bar y-\hat\beta_1\bar x$. Which point does the fitted line always pass through? ::: The centroid $(\bar x,\bar y)$. $\text{Var}(\hat\beta_1)=?$ and what does it tell you? ::: $\sigma^2/S_{xx}$; precision improves with more x-spread and less noise. Why divide SSE by $n-2$ for $s^2$? ::: Two degrees of freedom lost estimating $\hat\beta_0,\hat\beta_1$; gives an unbiased $\hat\sigma^2$. Standard error of the slope? ::: $\text{SE}(\hat\beta_1)=s/\sqrt{S_{xx}}$ where $s=\sqrt{\text{SSE}/(n-2)}$. Test statistic for $H_0:\beta_1=0$ and its distribution? ::: $t=\hat\beta_1/\text{SE}(\hat\beta_1)\sim t_{n-2}$. What does $R^2$ measure? ::: Fraction of variance in $y$ explained by the regression: $1-\text{SSE}/\text{SST}$. SST decomposition? ::: $\text{SST}=\text{SSR}+\text{SSE}$ (total = explained + residual). What property makes LS estimates "BLUE"? ::: Best (min variance) Linear Unbiased Estimators, by Gauss–Markov. Difference between $\varepsilon_i$ and $e_i$? ::: $\varepsilon_i$ is the true unobserved error; $e_i=y_i-\hat y_i$ is the observed residual. ## 🖼️ Concept Map ```mermaid flowchart TD M[Linear model yi eq b0 plus b1 xi plus err] -->|fit via| LS[Least squares min SSR] LS -->|set gradient to zero| NE[Normal equations] NE -->|solve| B1[slope b1 eq Sxy over Sxx] NE -->|solve| B0[intercept b0 eq ybar minus b1 xbar] B0 -->|line passes through| CM[Centre of mass xbar ybar] M -->|noise assumptions| ASM[Errors mean 0 var sigma2 uncorrelated] ASM -->|guarantees| GM[Gauss-Markov BLUE] B1 -->|is| GM LS -->|squared errors give| MLE[Max likelihood under Gaussian noise] M -->|observed gap| RES[Residual ei eq yi minus yhat] GM -->|enables| INF[Inference on coefficients] ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Socho aapke paas kuch points hain graph par jo roughly ek straight line follow karte hain. Hum > ek aisi line draw karna chahte hain jo sabse "close" ho. Close ka matlab — har point se line tak > ka *vertical* gap nikalo, usko square karo (taaki bade miss zyada count hon), aur woh line chuno > jahan total square gap minimum ho. Yahi **least squares** hai. Calculus se derivative zero karke > do "normal equations" milte hain, jisse seedha slope $\hat\beta_1=S_{xy}/S_{xx}$ aur intercept > $\hat\beta_0=\bar y-\hat\beta_1\bar x$ nikalta hai. Ek mast fact: fitted line hamesha centroid > $(\bar x,\bar y)$ se guzarti hai. > > Ab **inference** ka funda. Agar aap experiment dobara karo, noise alag aayega, toh line thodi tilt > ho jaayegi. Yani slope bhi ek random quantity hai. Uska variance $\sigma^2/S_{xx}$ hai — matlab > agar aapke $x$ values zyada *spread* hain (lamba lever), toh slope zyada precisely pin hota hai, > SE chhota. Noise $\sigma^2$ ko hum residuals se estimate karte hain: $s^2=\text{SSE}/(n-2)$. > Yahan $n-2$ isliye, kyunki do parameters estimate karne me 2 degrees of freedom kharch ho gaye — > $n$ se divide karoge toh underestimate ho jaayega (yeh common galti hai!). > > Yeh kyun important hai? Kyunki sirf line fit kar dena kaafi nahi — humein batana padta hai ki > slope **sach me zero se alag** hai ya bas noise hai. Iske liye $t=\hat\beta_1/\text{SE}$ banao, > $t_{n-2}$ table se compare karo. Agar confidence interval me 0 aa raha hai (jaise hamare chhote > example me), matlab data itna strong nahi ki relation confirm ho. Aur $R^2$ batata hai line ne > $y$ ki kitni variation explain ki — par yaad rakho, high $R^2$ ka matlab causation *nahi* hota. ![[audio/4.9.22-Linear-regression-—-least-squares,-inference-on-coefficients.mp3]]