Visual walkthrough — Multiple regression

We only assume you can read a graph and multiply numbers. Everything else — vectors, "length", "perpendicular", matrices — we earn as we go.

Step 1 — What are we even looking at? Points and a guess

WHAT. We have a table of observations. Each row is one thing we measured: some inputs (call them $x$'s) and one output $y$ we want to predict. Below, imagine just one input so we can draw it flat on paper — the whole story generalises later.

WHY start with one dot at a time. Before any algebra, the object we minimise is a gap: the vertical distance between a real data point and wherever our guessing-line sits underneath it. If you cannot picture that gap, no formula will mean anything.

PICTURE. The amber dots are the data. The cyan line is one guess at the trend. The dashed white segments are the vertical gaps — how wrong the guess is at each point.

Step 2 — Why we square the gaps

WHAT. We add up all the gaps to score a line. But we square each gap first: our score is $S=\underset{\text{gap 1, squared}}{\underbrace{e_1^2}}+\underset{\text{gap 2, squared}}{\underbrace{e_2^2}}+\cdots +\underset{\text{gap }n}{\underbrace{e_n^2}}={\sum }_{i=1}^n{e}_i^2.$ Here $\sum$ is just "add them all up", and $n$ is how many data points we have.

WHY square, not just add? Two problems with raw gaps. (1) A point $+4$ above and one $4$ below would cancel to $0$ — a terrible line would score "perfect". Squaring kills the sign since a square is never negative. (2) Squaring bends the score into a smooth bowl, so calculus can find its bottom exactly (Step 6). Absolute value $|e_i|$ also kills signs but has a sharp corner — no clean formula.

PICTURE. Each dashed gap becomes a literal square (amber). The total shaded area is the score $S$. Our whole job: slide and tilt the line until the total amber area is as small as possible.

Step 3 — Stacking the data into arrows (vectors)

WHAT. Instead of $n$ separate numbers, put all the $y_i$ into one tall list and treat that list as a single arrow in a high-dimensional space: $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right].$ The bold $\mathbf{y}$ is now one point living in $n$-dimensional space (one axis per data point).

WHY do this bizarre thing? Because "sum of squared gaps" secretly is a squared length. If two arrows $\mathbf{u},\mathbf{v}$ have a gap $\mathbf{u}-\mathbf{v}$, the squared length of that gap arrow is exactly $\sum (u_i-v_i{)}^2$ — the same shape as our score. So minimising $S$ becomes: find the arrow closest to $\mathbf{y}$. Distance is something we have strong geometric intuition for.

PICTURE. Left: three numbers $y_1,y_2,y_3$. Right: the same three numbers read as coordinates of one arrow $\mathbf{y}$ in 3-D space. Same data, new eyes.

With this, our score is reborn: $S=\Vert \underset{\text{data arrow}}{\underbrace{\mathbf{y}}}-\underset{\text{prediction arrow}}{\underbrace{\hat{\mathbf{y}}}}{\Vert }^2.$

Step 4 — Which arrows can our model reach? The column space

WHAT. Our prediction is $\hat{\mathbf{y}}=\mathbf{X}\mathbf{\beta }$. Here $\mathbf{X}$ is the table of inputs (with a leading column of 1's for the intercept), and $\mathbf{\beta }$ is the list of coefficients we get to choose. Multiplying $\mathbf{X}\mathbf{\beta }$ means: $\mathbf{X}\mathbf{\beta }={\beta }_0\underset{\text{col of 1’s}}{\underbrace{\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]}}+{\beta }_1\underset{\text{predictor }1}{\underbrace{\left[\begin{array}{c}{x}_{11}\\ x_{21}\\ ⋮\\ x_{n1}\end{array}\right]}}+\cdots$ It is a weighted blend of the columns of $\mathbf{X}$, with the $\beta$'s as the mixing weights.

WHY this matters. As we dial the $\beta$'s over all possible values, $\hat{\mathbf{y}}$ sweeps out every blend of those columns — and no more. That set of reachable arrows is a flat sheet (a plane, in general a hyperplane) passing through the origin. Our data arrow $\mathbf{y}$ almost never lies on this sheet (real data is noisy). So we cannot hit $\mathbf{y}$ — we can only get as close as the sheet allows.

PICTURE. Two input columns span a tilted cyan plane (the column space). The data arrow $\mathbf{y}$ (amber) juts off the plane. Every prediction we can make lives on the plane.

Step 5 — The closest point is the foot of the perpendicular

WHAT. On the flat sheet, which point is nearest to $\mathbf{y}$? Drop $\mathbf{y}$ straight down onto the sheet. The landing spot $\hat{\mathbf{y}}$ — the foot of the perpendicular — is the closest reachable point.

WHY perpendicular, not slanted? Slide the landing point a hair in any direction along the sheet. If the connecting arrow $\mathbf{y}-\hat{\mathbf{y}}$ were tilted (not perpendicular), moving toward its tilt would shorten the gap — so you weren't at the minimum yet. Only when the gap arrow stands exactly perpendicular to the sheet can no sideways nudge help. That is the geometric meaning of "least squares".

PICTURE. The amber residual arrow $\mathbf{y}-\hat{\mathbf{y}}$ meets the plane at a clean right angle (little white square). Any other point on the plane (grey) is farther from $\mathbf{y}$ — its dashed connector is longer.

Step 6 — Turning "perpendicular" into the normal equations

WHAT. "The residual is perpendicular to the sheet" means it is perpendicular to every column of $\mathbf{X}$ (the columns spread out the sheet). Two arrows are perpendicular when their dot product is zero. Bundling all those "$=0$" statements together: ${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})=0.$

Term by term:

• ${\mathbf{X}}^{\top }$ — the flipped table; row $j$ of it is column $j$ of $\mathbf{X}$. Multiplying by it computes the dot product of each predictor column with whatever follows.
• $\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }}$ — the residual arrow (data minus prediction).
• $0$ — a column of zeros: every predictor is perpendicular to the residual.

WHY dot product? The dot product $\mathbf{a}\cdot \mathbf{b}=\sum a_i{b}_i$ measures overlap; it is zero exactly when the arrows meet at $90^{\circ }$. It is the perfect tool to state "no leftover overlap between residual and predictors".

PICTURE. Each predictor column (cyan) forms a right angle with the residual (amber). The dot product of each with the residual reads zero.

Multiply the bracket out — distribute ${\mathbf{X}}^{\top }$ across the subtraction: $\underset{\text{predictors}\cdot \text{data}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{y}}}-\underset{\text{predictors}\cdot \text{predictors}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{X}}}\hat{\mathbf{\beta }}=0.$

Step 7 — Solving for the coefficients

WHAT. To isolate $\hat{\mathbf{\beta }}$, undo the multiplication by ${\mathbf{X}}^{\top }\mathbf{X}$. Undoing a matrix multiply is done by its inverse $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ (the matrix version of "divide"). Apply it to both sides: $\hat{\mathbf{\beta }}=\underset{\text{"divide by"}}{\underbrace{({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}}}\underset{\text{right-hand side}}{\underbrace{{\mathbf{X}}^{\top }\mathbf{y}}}.$

WHY an inverse and not real division? You cannot divide by a matrix. The inverse is defined so that $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}({\mathbf{X}}^{\top }\mathbf{X})=\mathbf{I}$ (the identity, the "do-nothing" matrix), which cancels cleanly on the left and leaves $\hat{\mathbf{\beta }}$ alone. See Matrix Inverse.

PICTURE. A flow: the data $\mathbf{y}$ enters, gets dotted against predictors (${\mathbf{X}}^{\top }\mathbf{y}$), then rescaled by $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$, and out comes the weight list $\hat{\mathbf{\beta }}$ — which lands us on the foot of the perpendicular.

Step 8 — The degenerate case: when the inverse does not exist

WHAT. Step 7 quietly assumed $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ exists. It fails to exist when two (or more) predictor columns point the same way — one is a copy or a scaled sum of others. This is perfect collinearity.

WHY it breaks — geometrically. If two columns are parallel, they no longer spread out a full plane; the "sheet" collapses to a lower-dimensional line. Now infinitely many weight-blends land on the same prediction arrow — you can add to one $\beta$ and subtract from its twin with zero visible change. There is no unique answer, so no inverse.

PICTURE. Left (healthy): two columns fan out and pin down one plane; the foot of the perpendicular is unique. Right (collinear): the columns overlap into a single line; the projection point is unpinned — it can slide, so $\hat{\mathbf{\beta }}$ is not unique.

The one-picture summary

Everything above collapses into a single diagram: the data arrow $\mathbf{y}$, the flat sheet of reachable predictions (column space of $\mathbf{X}$), the perpendicular drop giving $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$, and the residual $\mathbf{y}-\hat{\mathbf{y}}$ standing at a right angle — which is the normal equation ${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})=0$, which is the boxed estimator.

Flashcards