Visual walkthrough — Normal equation closed-form solution

This page rebuilds the normal equation as a sequence of pictures. We will not write a single symbol you have not seen drawn first. By the end you will see why the answer is $\mathbf{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$ — not just believe it.

---

Step 1 — The dots and the miss

WHAT. We start with the rawest thing possible: data points $(x,y)$ scattered on a plane, and one candidate straight line drawn through them.

WHY. Before any matrices, you must feel what we are trying to do. Every dot has a true height $y$. The line has a predicted height at the same $x$. The gap between them — the little vertical stick — is the error (also called the residual). Our entire goal is to make all these sticks small.

PICTURE. Look at the red vertical sticks. Each one is $\text{(what the line says)}-\text{(what really happened)}$. The line in the figure is a guess; it is not yet the best.

---

Step 2 — Why "square" the misses?

WHAT. We turn the whole bundle of red sticks into one single number that measures how bad the line is. We square each stick and add them up.

WHY squaring and not just adding? If we simply added the sticks, a stick pointing up ($+3$) and one pointing down ($-3$) would cancel to $0$ — a terrible line would look perfect. We need every miss to count as positive badness. Squaring does exactly that: $(-3{)}^2=9$, $(+3{)}^2=9$. Squaring also punishes big misses much harder than small ones, which is what we want — a wildly wrong point should scream louder.

Why this tool and not absolute value? We could use $|e|$, which also kills signs. But $|e|$ has a sharp corner at zero — it is not smooth, so calculus (finding where the slope is zero) gets messy. The square $e^2$ is a smooth bowl, and its derivative is the clean $2e$. Smoothness is what buys us a closed-form answer later.

PICTURE. Each red stick becomes a red square with that stick as its side. Total badness = total red area.

---

Step 3 — The cost is a bowl, so the bottom is where slope = 0

WHAT. Forget the data plane for a moment. Now we plot the badness number $J$ against the parameters of the line. For a line $y={\theta }_0+{\theta }_{1}x$ there are two knobs, ${\theta }_0$ (height) and ${\theta }_1$ (tilt). Turning the knobs changes $J$.

WHY. Because $J$ is built from squares of things that are linear in $\theta$, $J$ is a quadratic in $\theta$ — and a quadratic in two variables is a smooth bowl (a paraboloid) with exactly one lowest point. There is no second valley, no plateau. This is the deep reason a one-shot answer exists: we are not searching a messy landscape, we are dropping to the bottom of a single bowl.

Why "set the derivative to zero"? At the very bottom of a bowl the surface is momentarily flat in every direction — walk any way and you go up. "Flat in every direction" is exactly "all partial derivatives (the gradient) equal zero." So finding the best line = solving ${\nabla }_{\mathbf{\theta }}J=0$. (Compare with Gradient Descent, which instead rolls down this same bowl step by step.)

PICTURE. The bowl over the $({\theta }_0,{\theta }_1)$ floor; the red dot marks the unique flat bottom.

---

Step 4 — Stack everything into matrices

WHAT. Writing $\sum$ over every point is tiring. We pack all points into a grid $\mathbf{X}$ (the design matrix), all targets into a column $\mathbf{y}$, all knobs into a column $\mathbf{\theta }$.

WHY. One matrix multiply $\mathbf{X}\mathbf{\theta }$ produces all $m$ predictions at once. The whole bundle of red sticks becomes a single error vector $\mathbf{e}=\mathbf{X}\mathbf{\theta }-\mathbf{y}$. Then "total squared miss" is just that vector dotted with itself — one clean expression instead of a sum.

Why the column of 1's? The intercept ${\theta }_0$ must multiply something in every row. We give it a fake feature that is always $1$, so $1\cdot {\theta }_0={\theta }_0$ survives in every prediction.

PICTURE. The grid $\mathbf{X}$ (leftmost column all $1$'s, in red), times the knob column, lands on the prediction column.

---

Step 5 — Differentiate the bowl, land on the perpendicular fact

WHAT. Expand the squared length and take its gradient with respect to $\mathbf{\theta }$, then set it to $0$.

WHY expand? Because $(\mathbf{a}-\mathbf{b}{)}^T(\mathbf{a}-\mathbf{b})={\mathbf{a}}^T\mathbf{a}-2{\mathbf{a}}^T\mathbf{b}+{\mathbf{b}}^T\mathbf{b}$ turns the cost into three friendly pieces we know how to differentiate.

$$J=\frac{1}{2m}(\underset{\text{quadratic in }\theta }{\underbrace{{\mathbf{\theta }}^T{\mathbf{X}}^T\mathbf{X}\mathbf{\theta }}}-\underset{\text{linear in }\theta }{\underbrace{2{\mathbf{\theta }}^T{\mathbf{X}}^T\mathbf{y}}}+\underset{\text{constant}}{\underbrace{{\mathbf{y}}^T\mathbf{y}}})$$

Differentiate term by term (these are the matrix versions of $\frac{d}{dx}(a{x}^2)=2ax$ and $\frac{d}{dx}(bx)=b$):

$${\nabla }_{\mathbf{\theta }}J=\frac{1}{m}(\underset{\text{from }2\times \frac{1}{2}}{\underbrace{{\mathbf{X}}^T\mathbf{X}\mathbf{\theta }}}-\underset{\text{from linear term}}{\underbrace{{\mathbf{X}}^T\mathbf{y}}})$$

Set to zero and the $\frac{1}{m}$ dies:

$\boxed{{\mathbf{X}}^T\mathbf{X}\mathbf{\theta }={\mathbf{X}}^T\mathbf{y}}$

WHAT this says geometrically. Rearrange to ${\mathbf{X}}^T(\mathbf{X}\mathbf{\theta }-\mathbf{y})=0$, i.e. ${\mathbf{X}}^T\mathbf{e}=0$. In words: the error vector is perpendicular to every column of $\mathbf{X}$. That is why it is called the normal equation — "normal" means perpendicular.

PICTURE. The flat sheet is the column space of $\mathbf{X}$ (all lines we can possibly build). The truth $\mathbf{y}$ floats off the sheet. The best prediction $\mathbf{X}\mathbf{\theta }$ is the shadow of $\mathbf{y}$ on the sheet, and the red error joins them at a right angle.

---

Step 6 — Solve for the knobs

WHAT. Peel $\mathbf{\theta }$ out of ${\mathbf{X}}^T\mathbf{X}\mathbf{\theta }={\mathbf{X}}^T\mathbf{y}$ by multiplying both sides by the inverse of ${\mathbf{X}}^T\mathbf{X}$.

WHY an inverse? ${\mathbf{X}}^T\mathbf{X}$ is a small square $(n+1)\times (n+1)$ matrix sitting in front of $\mathbf{\theta }$. Undoing a matrix multiply is exactly what the inverse does, provided the matrix is not degenerate.

PICTURE. The two sides of the equation as a scale; multiplying both pans by $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ leaves $\mathbf{\theta }$ alone on the left.

---

Step 7 — The degenerate case: when the bowl becomes a trough

WHAT. Sometimes ${\mathbf{X}}^T\mathbf{X}$ has no inverse. This happens when two feature columns are secretly the same information (linearly dependent), so $\mathbf{X}$ is rank-deficient.

WHY it breaks. If a feature is redundant, many different knob settings give the identical line. The bowl of Step 3 flattens along one direction into a trough — a whole valley floor of equally-best answers, not a single point. A trough has no unique bottom, so no unique $\mathbf{\theta }$, so $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ cannot exist (determinant $=0$).

Concrete trap (Example 2 from the parent). With $x_2=x_1+2$ always, column $2=$ column $1+2\times$ column of $1$'s. Then $\det ({\mathbf{X}}^T\mathbf{X})=0$ and the formula fails. Infinitely many exact fits exist, e.g. $\mathbf{\theta }=(2,2,1{)}^T$ gives $2+2(1)+3=7$, $2+2(2)+4=10$, $2+2(3)+5=13$ — perfect, zero error.

The fix. Use the Moore-Penrose Pseudoinverse (computed via SVD, or stabilize with QR Decomposition). Among all trough-floor solutions it returns the minimum-norm one. Or add a tiny penalty — that is Ridge Regression, which tilts the trough back into a bowl.

PICTURE. Left: healthy bowl, one red bottom. Right: trough, a red line of equal minima.

---

The one-picture summary

Everything above is one geometric fact: drop the truth $\mathbf{y}$ perpendicularly onto the flat sheet of achievable predictions; the landing point is $\mathbf{X}\mathbf{\theta }$, and the perpendicular error gives ${\mathbf{X}}^T\mathbf{e}=0$, which unfolds into the normal equation.

Recall Feynman retelling — say it like a story

We threw dots on a page and drew a line. For each dot we measured the vertical miss — the red stick. We squared the sticks so ups and downs both count as bad and big misses shout louder, then averaged them into one badness number $J$. Plotting $J$ against the line's two knobs gave a smooth bowl, and the best line lives at the bowl's flat bottom where the slope vanishes. We packed all points into a grid $\mathbf{X}$ so one multiply predicts everything and the misses become one arrow $\mathbf{e}$. Setting the slope to zero said ${\mathbf{X}}^T\mathbf{e}=0$: the leftover error points straight out from the sheet of all possible lines — that's the perpendicular (normal) rule, and it's just the fact that the shortest path to a surface is straight down. Solving that gives $\mathbf{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$. And if two features secretly repeat, the bowl flattens into a trough with infinitely many equal bottoms — then we call the pseudoinverse to pick the smallest one.

Recall Quick self-test

Why square the residuals instead of adding them? ::: Plain sums let $+3$ and $-3$ cancel, hiding a bad line; squaring makes every miss positive and punishes big misses more, while staying smooth for calculus. What does "normal" in normal equation mean? ::: Perpendicular — the error vector is perpendicular (normal) to the column space of $\mathbf{X}$, i.e. ${\mathbf{X}}^T\mathbf{e}=0$. When can't you invert ${\mathbf{X}}^T\mathbf{X}$? ::: When $\mathbf{X}$ is rank-deficient (linearly dependent feature columns); the cost bowl becomes a trough with infinitely many minima. What replaces the inverse then? ::: The Moore-Penrose pseudoinverse (via SVD), returning the minimum-norm least-squares solution.