Visual walkthrough — Linear regression — normal equation, gradient descent derivation

This page rebuilds the normal equation one picture at a time, starting from a scatter of dots and ending at the single formula

$${\mathbf{w}}^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}.$$

Every symbol here is earned before it is used. If you have never seen a matrix, a dot product, or a derivative, start at line one — we build all of it.

---

Step 1 — The dots and the line we are hunting

WHAT. We are given measurements. Each measurement is a pair: an input $x_i$ (say, altitude) and an output $y_i$ (say, fuel-burn rate). The little $i$ is just a name-tag: $i=1$ is the first dot, $i=2$ the second, and so on up to $i=n$ (there are $n$ dots).

We want a line

$$\hat{y}=wx+b.$$

• $\hat{y}$ (read "y-hat") is the line's prediction — the height the line claims at input $x$.
• $w$ is the slope: how steeply the line rises. Change $x$ by 1, $\hat{y}$ changes by $w$.
• $b$ is the bias (or intercept): the height of the line where $x=0$.

WHY a line? Because it is the simplest honest guess: constant slope, no curves. Everything harder is built on top of this.

PICTURE. Dots in blue, one candidate line in yellow. For each dot, the vertical red gap between the dot and the line is the error — how wrong the line is there.

---

Step 2 — Measuring total wrongness: why we square

WHAT. For dot $i$ the error is $e_i=y_i-{\hat{y}}_i$ (true height minus predicted height). We add up all errors — but squared:

$$J(w,b)=\frac{1}{2n}\sum _{i=1}^n\underset{\text{one red gap}}{\underbrace{(y_i-{\hat{y}}_i)}}{}^2.$$

Reading the symbols:

• ${\sum }_{i=1}^n$ means "add up the following for every dot from $1$ to $n$".
• $(y_i-{\hat{y}}_i{)}^2$ is one red gap, squared — always positive.
• $\frac{1}{2n}$ is a tidy-up factor: dividing by $n$ makes it an average, and the $\frac{1}{2}$ will cancel a $2$ that appears when we differentiate later. It changes where the minimum's value is, never where the minimum sits.
• $J$ is the total badness score. Smaller $J$ = better line.

WHY square, not just add? Two reasons, both shown in the figure. (1) A dot above the line has a positive gap, one below has a negative gap — if we added them raw, they would cancel and a terrible line could score zero. Squaring makes every gap count as positive. (2) The squared curve is smooth (a parabola), so calculus — which needs slopes to exist everywhere — works cleanly.

PICTURE. Left: raw gaps cancelling (bad). Right: each gap turned into a red square whose area is the penalty; big gaps get punished much harder than small ones.

---

Step 3 — Packing all the dots into a matrix

WHAT. Writing $\sum$ over and over is clumsy. We stack the data into a matrix $\mathbf{X}$ and a vector $\mathbf{y}$.

• A vector is a column of numbers, e.g. $\mathbf{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]$ — one target per row.
• A matrix is a grid of numbers. We make $\mathbf{X}$ with one row per dot. To also capture the bias $b$, we glue a column of $1$s in front:

$$\mathbf{X}=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_n\end{array}\right],\mathbf{w}=\left[\begin{array}{c}b\\ w\end{array}\right].$$

Now $\mathbf{Xw}$ — matrix times vector — is the whole column of predictions at once. Row $i$ of the product is $1\cdot b+x_i\cdot w=b+w{x}_i={\hat{y}}_i$. The column of $1$s is what turns $b$ into a real column so it rides along in the multiplication.

WHY. So the entire cost becomes one clean object:

$$J(\mathbf{w})=\frac{1}{2n}\Vert \mathbf{y}-\mathbf{Xw}{\Vert }^2.$$

Here $\Vert \mathbf{v}{\Vert }^2$ (the squared norm) means "add up the squares of the entries of $\mathbf{v}$" — exactly our sum of squared gaps, now in one symbol.

PICTURE. The dot-by-dot table on the left collapses into the boxed grid $\mathbf{X}$, column of $1$s highlighted, arrow to $\mathbf{Xw}=\hat{\mathbf{y}}$.

---

Step 4 — The whole geometry lives in one flat sheet

WHAT. Here is the picture that makes the formula inevitable. Think of $\mathbf{y}$ as a single arrow in space (one axis per dot — $n$ axes). Every possible line we could draw gives some $\mathbf{Xw}$, another arrow. As we turn the dials $b$ and $w$, the tip of $\mathbf{Xw}$ sweeps out a flat sheet through the origin: the column space of $\mathbf{X}$ (all combinations of $\mathbf{X}$'s columns).

WHY this matters. Our target arrow $\mathbf{y}$ almost never lies on that sheet (real data doesn't fit a line perfectly). So "best line" = "the point on the sheet closest to $\mathbf{y}$". Distance in this space is exactly $\Vert \mathbf{y}-\mathbf{Xw}\Vert$ — our error. Minimizing squared error = finding the closest point on the sheet.

PICTURE. The green sheet is the column space. $\mathbf{y}$ (blue) floats above it. Candidate predictions $\mathbf{Xw}$ live on the sheet.

---

Step 5 — Closest point = drop a perpendicular

WHAT. The closest point on a flat sheet to an outside point is found by dropping straight down — perpendicular to the sheet. Call that foot of the perpendicular $\mathbf{X}{\mathbf{w}}^{\ast }$. The leftover arrow,

$$\mathbf{r}=\mathbf{y}-\mathbf{X}{\mathbf{w}}^{\ast },$$

the residual (red), must be perpendicular to the sheet.

WHY. If the residual leaned even slightly along the sheet, we could slide a bit that way and get closer — so we would not yet be at the minimum. At the true minimum the residual points straight out. This is the geometric twin of "the derivative is zero at the bottom".

Perpendicular to the whole sheet means perpendicular to every column of $\mathbf{X}$. A dot product of zero means perpendicular, so:

$${\mathbf{X}}^T(\mathbf{y}-\mathbf{X}{\mathbf{w}}^{\ast })=0.$$

Every entry of ${\mathbf{X}}^T\mathbf{r}$ is one column of $\mathbf{X}$ dotted with $\mathbf{r}$; setting them all to zero says $\mathbf{r}\perp$ each column.

PICTURE. Red residual meets the green sheet at a right-angle mark; the yellow columns of $\mathbf{X}$ all lie flat in the sheet, each at $90^{\circ }$ to the residual.

---

Step 6 — Same answer by calculus (the slope-is-zero view)

WHAT. Let us re-derive Step 5 with algebra, so the two pictures agree. Expand the cost using the dot-product facts from Step 3:

$$J(\mathbf{w})=\frac{1}{2n}(\underset{\text{const}}{\underbrace{{\mathbf{y}}^T\mathbf{y}}}-\underset{\text{linear in }\mathbf{w}}{\underbrace{2{\mathbf{w}}^T{\mathbf{X}}^T\mathbf{y}}}+\underset{\text{quadratic in }\mathbf{w}}{\underbrace{{\mathbf{w}}^T{\mathbf{X}}^T\mathbf{Xw}}}).$$

WHY. As a function of the dials $\mathbf{w}$, this is a bowl (a paraboloid) — the quadratic term guarantees it curves upward. A bowl has exactly one lowest point, and at the bottom its slope is zero in every direction. The slope-in-every-direction is the gradient ${\nabla }_{\mathbf{w}}J$: a vector whose entries are the partial derivatives (the sensitivity of $J$ to nudging each dial). Setting it to $0$:

$${\nabla }_{\mathbf{w}}J=\frac{1}{n}({\mathbf{X}}^T\mathbf{X}\mathbf{w}-{\mathbf{X}}^T\mathbf{y})=0⟹{\mathbf{X}}^T\mathbf{X}\mathbf{w}={\mathbf{X}}^T\mathbf{y}.$$

This normal equation is identical to Step 5 (expand ${\mathbf{X}}^T(\mathbf{y}-\mathbf{Xw})=0$). Geometry and calculus land on the same line.

PICTURE. The bowl-shaped cost surface over the $(b,w)$ plane; a ball rolls to the single bottom where the tangent plane is flat.

---

Step 7 — Solve for the winning dials

WHAT. From ${\mathbf{X}}^T\mathbf{X}\mathbf{w}={\mathbf{X}}^T\mathbf{y}$, undo the matrix on the left by multiplying by its inverse $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ (the matrix version of dividing):

Worked check (the parent's Example 1). Data $(1,2),(2,4),(3,5)$:

$${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right],{\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}11\\ 25\end{array}\right],{\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}11\\ 25\end{array}\right]=\left[\begin{array}{c}0.667\\ 1.5\end{array}\right].$$

So $b=\frac{2}{3}\approx 0.667$, $w=1.5$: the fitted line is $\hat{y}=1.5x+0.667$.

PICTURE. The three dots with the solved yellow line, residuals shown, and the fitted numbers labelled.

---

Step 8 — The degenerate case: when the inverse dies

WHAT. Everything above needed $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ to exist. It fails when two columns of $\mathbf{X}$ carry the same information — e.g. altitude in metres and altitude in feet. Then the green sheet is squashed: infinitely many lines tie for best, and ${\mathbf{X}}^T\mathbf{X}$ is singular (no inverse).

WHY it breaks. Geometrically the columns are parallel, so the "sheet" is really a thinner slab — the foot of the perpendicular is not unique. Algebraically $\det ({\mathbf{X}}^T\mathbf{X})=0$, and you cannot divide.

The fix. Use the Moore–Penrose pseudoinverse ${\mathbf{X}}^{+}$, which picks the smallest valid $\mathbf{w}$, or add a $\lambda$ nudge (see ridge regression): $({\mathbf{X}}^T\mathbf{X}+\lambda \mathbf{I}{)}^{-1}$, which is always invertible for $\lambda >0$. Also, wildly different feature scales make ${\mathbf{X}}^T\mathbf{X}$ nearly singular numerically — cured by Feature Scaling.

PICTURE. Left: healthy full sheet, unique foot. Right: collapsed slab, a whole line of tied feet — the ambiguity.

---

The one-picture summary

Dots → squared gaps → column-space sheet → drop the perpendicular → normal equation → solved line. One figure, the whole journey.

Recall Feynman retelling — say it like a story

We scattered some dots and wanted the straightest, fairest line through them. "Fair" meant: add up how far off it is at every dot, but square each miss so ups and downs can't secretly cancel and so the math stays smooth. We packed all the dots into a grid $\mathbf{X}$ (with a column of $1$s so the line's height $b$ comes along for free) and all the answers into a column $\mathbf{y}$. Then came the beautiful trick: think of $\mathbf{y}$ as an arrow, and every possible line as an arrow lying on a flat sheet. The best line is the shadow of $\mathbf{y}$ on that sheet — you drop straight down, and the leftover error points dead-perpendicular to the sheet. "Perpendicular to every column" written in symbols is ${\mathbf{X}}^T(\mathbf{y}-\mathbf{Xw})=0$, which rearranges into ${\mathbf{w}}^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$. Calculus agrees: the cost is a bowl, and the bottom of a bowl is where every slope is zero — same equation. The only catch is when two columns say the same thing; then the sheet collapses, the inverse dies, and we lean on the pseudoinverse or a tiny ridge nudge. For our three dots it spat out $\hat{y}=1.5x+0.667$, no guessing required.

Recall Quick self-test

Why do we square the errors instead of adding them raw? ::: So positive and negative gaps can't cancel, and so the cost is a smooth parabola/bowl with derivatives everywhere. What does ${\mathbf{X}}^T(\mathbf{y}-\mathbf{X}{\mathbf{w}}^{\ast })=0$ mean geometrically? ::: The residual (error arrow) is perpendicular to every column of $\mathbf{X}$, i.e. to the whole column-space sheet. When does $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ fail to exist? ::: When columns of $\mathbf{X}$ are redundant (multicollinearity), making ${\mathbf{X}}^T\mathbf{X}$ singular. For data $(1,2),(2,4),(3,5)$, what line does the normal equation give? ::: $\hat{y}=1.5x+0.667$.