Foundations — Assumptions of linear regression

Before you can judge whether an assumption holds, you must be fluent in the alphabet the topic speaks. We go symbol by symbol. Each entry: plain words → the picture → why the topic needs it. Nothing is used before it is built.

1. The data: points on a grid

The picture: dots scattered on a 2-D grid. Horizontal axis = $x$, vertical axis = $y$. Observation $i$ is one dot at position $(x_i,y_i)$.

Why the topic needs it: every assumption is a statement about these dots and the line we run through them. No dots, nothing to assume.

2. The true line vs the fitted line: $\beta$ vs $\hat{\beta }$

There are two straight lines in this whole subject, and telling them apart is the single most important habit.

Two symbols to picture, both just numbers we compute from the data:

• ${\hat{\beta }}_0$ (say "beta-nought-hat") = the estimated intercept. Where our line crosses the vertical axis, i.e. the prediction when $x=0$. In the picture, the height of the line's left anchor.
• ${\hat{\beta }}_1$ (say "beta-one-hat") = the estimated slope. How many units $\hat{y}$ climbs each time $x$ goes up by one — the steepness, rise over run.

Why the topic needs it: the estimated intercept and slope are the thing regression computes. When the parent note works with ${\beta }_0+{\beta }_1{X}_1+\cdots$, those bare $\beta$'s are the true parameters the model assumes exist; the fitting procedure hands us the hatted estimates ${\hat{\beta }}_j$ — see §7.

Fitting this line by minimizing squared gaps is exactly Ordinary Least Squares (OLS).

3. The error and the residual: ${ϵ}_i$ and $e_i$

This is the heart of every assumption. There are two gap-symbols and beginners mix them up — and they line up exactly with the two lines of §2.

The picture: for each dot, draw a short vertical stick from the dot down (or up) to the fitted line. The length of that stick is the residual $e_i$. A dot above the line has $e_i>0$; a dot below has $e_i<0$; a dot exactly on the line has $e_i=0$.

Why the topic needs it: all four LINE assumptions are rules about ${ϵ}_i$. We check them using $e_i$.

4. Expectation and the zero-mean rule: $E[\cdot ]$

The picture: imagine the vertical spread of dots at one fixed $x$. They bounce around, but they have a centre of mass. That centre is $E[Y\mid X=x]$ — "the average $y$, given that $x$ is this value."

Why the topic needs it: the Linearity assumption is exactly $E[Y\mid X]={\beta }_0+{\beta }_{1}X,$ i.e. "the centre of the $y$-cloud, for each $x$, lies on nature's straight line." Individual dots scatter; only their average is required to be linear.

5. Variance and ${\sigma }^2$: how wide the scatter is

The picture: two clouds of dots. A tight cloud has small ${\sigma }^2$; a fat cloud has large ${\sigma }^2$. Variance is the "thickness" of the band.

Why the topic needs it: constant ${\sigma }^2$ is what lets the parent's variance formula collapse to the clean ${\sigma }^2(X^{T}X{)}^{-1}$.

6. Covariance and the uncorrelated-errors rule: $\text{Cov}({ϵ}_i,{ϵ}_j)$

The picture: plot ${ϵ}_i$ against the next error ${ϵ}_{i+1}$. If the points form a tilted sausage, they're correlated (bad). If they form a shapeless round blob, covariance $\approx 0$ (good) — though a blob shaped like a ring would also have zero covariance while clearly not being independent.

Why the topic needs it: the Independence assumption of OLS is, strictly, the uncorrelated-errors condition $\text{Cov}({ϵ}_i,{ϵ}_j)=0$ for $i\ne j$ (this is what Gauss–Markov actually requires). Time-series and repeated-measures data break it, which is why the parent points you to generalized least squares (GLS) — an OLS cousin that lets errors have known correlations — or mixed models.

7. Going multi-dimensional: vectors, matrices, $X$, $X^{T}X$

The parent jumps to $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}y$. Let's earn every piece.

• $X^T$ (transpose) = flip the table on its side: rows become columns. Picture rotating the grid 90°.
• $X^{T}X$ = a small square matrix summarising how the predictor columns relate to each other — a "correlation-flavoured" summary.
• $(X^{T}X{)}^{-1}$ (inverse) = the matrix version of "divide". Just as $1/a$ blows up when $a\to 0$, the inverse blows up when $X^{T}X$ is nearly flat (tiny determinant).

Why the topic needs it: the No-Multicollinearity assumption and the ridge fix (Ridge and Lasso Regression) are both about keeping $\det (X^{T}X)$ safely away from zero.

8. The normal curve $N(0,{\sigma }^2)$

The picture: a smooth symmetric hill over the residual sticks — most cluster near zero, tapering both ways. A Q-Q plot compares your actual residuals to this ideal bell; if they match, points lie on the 45° line.

Why the topic needs it: normality is what licenses the $t$-tests and $F$-tests of Hypothesis Testing in Regression. For prediction alone it's optional; the Bias-Variance Tradeoff and Gauss–Markov guarantee unbiasedness without it. It is also the bridge (from §6) that upgrades uncorrelated errors into independent ones.

Prerequisite map

Everything above pours into the parent: the five assumptions. Related destinations: Generalized Linear Models (GLMs) generalises the bell-curve assumption to other distributions.

Equipment checklist

Self-test: can you say each answer before revealing it?