Visual walkthrough — Ordinary least squares derivation

This is the visual walkthrough child of the OLS derivation. The parent proved the Normal Equation with algebra. Here we draw it — every step is a picture, every symbol earns its place before it appears. If you have never seen a matrix, a transpose, or a gradient, start here. We build all of it.

---

Step 1 — The data, the guess, and the miss

WHAT. We start with plain dots on a grid. Each dot is one training example: a horizontal position $x_i$ (the input we know) and a vertical position $y_i$ (the answer we want to predict). We then draw a candidate straight line and measure how badly it misses each dot.

WHY. Before any formula, we must be crystal-clear on what "wrong" means. Everything downstream is just "make this wrongness as small as possible."

PICTURE (Step-1 figure): blue data dots, a yellow guess line, and pink vertical sticks marking each miss.

A straight line is written $\hat{y}={\theta }_0+{\theta }_{1}x$.

The red vertical stick from a dot up (or down) to the line is the residual:

$r_i=\underset{\text{line’s guess}}{\underbrace{{\hat{y}}_i}}-\underset{\text{true dot}}{\underbrace{y_i}}$

If the line is above the dot, $r_i$ is positive; below, negative. Look at the red sticks in the figure — some point up, some down.

---

Step 2 — Why we square the misses

WHAT. We turn each red stick into a square: a little box whose side is the residual. The total "badness" is the sum of the areas of these boxes.

WHY. Two problems with just adding raw residuals: (1) up-misses and down-misses would cancel and pretend the line is perfect; (2) we want big misses to hurt more than small ones. Squaring fixes both — every square is positive, and a miss of $4$ makes a box of area $16$ while a miss of $2$ makes only $4$.

PICTURE (Step-2 figure): each pink miss redrawn as a shaded square whose area is the squared error; total shaded area is the cost.

The best line is the one that makes the total box area smallest. That is the entire optimisation. (See Linear Regression Fundamentals for the model itself.)

---

Step 3 — Stacking everything into one matrix picture

WHAT. Writing out ${\theta }_0+{\theta }_1{x}_i$ for every $i$ is tedious. We stack the dots into a table $X$, the answers into a column $y$, and the two knobs into a column $\theta$. Then all predictions at once are the single product $X\theta$.

WHY. With one clean object we can use calculus once instead of chasing ${\theta }_0$ and ${\theta }_1$ separately. This is where linear algebra pays for itself.

PICTURE (Step-3 figure): the table $X$, the knob column $\theta$, and their product $X\theta$ shown side by side, with the column of 1s highlighted.

$X=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_m\end{array}\right],\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\end{array}\right],X\theta =\left[\begin{array}{c}{\theta }_0+{\theta }_1{x}_1\\ ⋮\\ {\theta }_0+{\theta }_1{x}_m\end{array}\right]$

Before we measure the length of the residual we need one new symbol, the transpose.

Now the cost from Step 2 collapses to a length:

$J(\theta )=\frac{1}{2m}\Vert X\theta -y{\Vert }^2,\Vert v{\Vert }^2=v^{T}v={\sum }_k{v}_k^2.$

Here $\Vert X\theta -y\Vert$ is the length of the residual column — Pythagoras in $m$ dimensions. Small length = good line.

---

Step 4 — The geometry: predictions live on a flat sheet

WHAT. Think of $y$ as a single arrow in a high-dimensional space (one axis per dot). As we turn the two knobs ${\theta }_0,{\theta }_1$, the prediction arrow $X\theta$ can only reach points on a flat sheet (the column space of $X$). We cannot reach $y$ itself unless it happens to lie on the sheet.

WHY. This is the picture that makes the answer obvious. Minimising $\Vert X\theta -y\Vert$ means: find the point on the sheet closest to $y$. Geometry gives that instantly — it is the shadow (projection) of $y$ onto the sheet.

PICTURE (Step-4 figure): the pink arrow $y$ floating above a blue tilted sheet, its yellow shadow $X{\theta }^{\ast }$ on the sheet, and the white residual arrow standing straight up between them.

The closest point is where the residual arrow is perpendicular to the sheet. Any other spot on the sheet is farther from $y$ (slanted stick is longer than the vertical drop). That perpendicularity is the whole secret of the next step.

---

Step 5 — "Perpendicular" written as an equation

WHAT. Perpendicular means "no shadow along the sheet's directions." The sheet's directions are exactly the columns of $X$. So the residual must be perpendicular to every column of $X$ at once.

WHY. A geometric statement ("perpendicular") becomes an algebraic one ("dot product $=0$"), which we can actually solve.

PICTURE (Step-5 figure): a blue column-direction arrow and a white residual arrow meeting at a right-angle marker, captioned "dot product = 0."

Perpendicular arrows have dot product zero. The transpose from Step 3 packages "dot with every column of $X$" into one product $X^T(\cdots )$:

$X^T(X{\theta }^{\ast }-y)=0$

This is called the Normal Equation because "normal" is the old geometry word for perpendicular (see the upright arrow in Step 4). Rearrange:

$X^{T}X{\theta }^{\ast }=X^{T}y.$

---

Step 6 — Same answer from calculus (the valley floor)

WHAT. We repeat Step 5's result, this time by rolling downhill on the cost surface instead of using geometry. Plot $J(\theta )$ over the two knobs: it is a bowl. The bottom is where the ground is flat — the gradient is zero.

WHY. Two independent routes landing on the same equation is the strongest evidence it is right. Calculus also generalises when we later add regularisation (see Ridge Regression).

PICTURE (Step-6 figure): the bowl-shaped cost surface $J(\theta )$ over the two knobs, with a pink dot marking the flat bottom where the gradient vanishes.

Differentiating $J(\theta )=\frac{1}{2m}[{\theta }^T{X}^{T}X\theta -2y^{T}X\theta +y^{T}y]$ term by term:

${\nabla }_{\theta }J=\frac{1}{m}(\underset{\text{from }{\theta }^T{X}^{T}X\theta }{\underbrace{X^{T}X\theta }}-\underset{\text{from }-2y^{T}X\theta }{\underbrace{X^{T}y}}).$

Set the gradient to zero — the $\frac{1}{2}$ from Step 2 exactly cancels the $2$ from the square's derivative (that was its whole purpose) — and we recover

$X^{T}X{\theta }^{\ast }=X^{T}y.$

Same equation as Step 5. Geometry and calculus agree. (Rolling downhill iteratively instead of jumping to the bottom is Gradient Descent.)

---

Step 7 — Solve, and the bowl-vs-trough edge case

WHAT. If $X^{T}X$ can be undone (inverted), multiply both sides by its inverse to isolate ${\theta }^{\ast }$. If it cannot be inverted, the bowl is not a bowl — it is a flat-bottomed trough, and infinitely many lines tie for best.

WHY. A formula is only useful if we know when it breaks. The trough case is the difference between "one answer" and "no unique answer."

PICTURE (Step-7 figure): left, a genuine bowl with one lowest point (unique best line); right, a flat-bottomed trough whose whole valley floor ties for best (singular case).

Let $m$ be the number of dots and let $n$ be the number of feature columns in $X$ (so $X$ has $n+1$ columns once the 1s column is added). The formula works exactly when the columns of $X$ are linearly independent (nothing is a copy or blend of the others) — call this full column rank.

Left panel (good): columns of $X$ point in genuinely different directions → true bowl → one lowest point → one best line.

Right panel (degenerate): the columns are not independent (multicollinearity) — for example one feature is a scaled copy of another. Then the two sheet-directions collapse onto the same line, the bowl flattens along a valley, $\det (X^{T}X)=0$, and $(X^{T}X{)}^{-1}$ does not exist.

---

Step 8 — A tiny numeric run you can check by hand

WHAT. Use three dots $(1,2),(2,4),(3,5)$ and crank the formula once, end to end.

WHY. Numbers make the abstract machine concrete, and confirm the boxes really do get smallest.

$X=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right],y=\left[\begin{array}{c}2\\ 4\\ 5\end{array}\right]\Rightarrow X^{T}X=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right],X^{T}y=\left[\begin{array}{c}11\\ 25\end{array}\right].$

$\det (X^{T}X)=3\cdot 14-6\cdot 6=6,(X^{T}X{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right].$

${\theta }^{\ast }=(X^{T}X{)}^{-1}{X}^{T}y=\left[\begin{array}{c}2/3\\ 3/2\end{array}\right]\Rightarrow \hat{y}=\frac{2}{3}+\frac{3}{2}x.$

Check at $x=2$: $\hat{y}=\frac{2}{3}+3=3.67$, close to the true $y=4$. The line threads the dots — exactly the "smallest boxes" line from Step 2.

---

The one-picture summary

PICTURE (summary figure): on the left the fitted line with its pink residual sticks; on the right the same story as geometry — $y$'s shadow on the sheet with the perpendicular residual — jointly giving ${\theta }^{\ast }=(X^{T}X{)}^{-1}{X}^{T}y$.

Left: dots + best line + the red residual sticks. Right: the same story as geometry — $y$'s shadow lands on the sheet, and the residual stands perpendicular, giving $X^T(X{\theta }^{\ast }-y)=0$, i.e. ${\theta }^{\ast }=(X^{T}X{)}^{-1}{X}^{T}y$.

Recall Feynman retelling (say it to a 12-year-old)

We scatter dots and want the fairest straight line. "Fair" = the total area of little squares built on each vertical miss should be as small as possible. Squares (not raw gaps) because we do not want ups and downs to cancel, and we want big misses punished hard. We pack the dots into a table $X$, the answers into a column $y$, and our two dials (height and tilt) into $\theta$; then all predictions are just $X\theta$. Picture $y$ floating in the air and all reachable predictions living on a flat sheet — the closest we can get is $y$'s shadow, and the leftover gap points straight up off the sheet. "Straight up" means the gap has no overlap with the sheet's directions, which are the columns of $X$; written down (using the transpose to dot with every column at once), that is $X^T(\text{gap})=0$. Solving gives ${\theta }^{\ast }=(X^{T}X{)}^{-1}{X}^{T}y$. Rolling downhill on a bowl-shaped cost gives the exact same equation, so we trust it. The one catch: if two dials secretly do the same thing (a copied feature) or we have more dials than dots, the bowl flattens into a trough and there is no single best line — then we drop the copy or add a gentle tilt (Ridge) to scoop the trough back into a bowl.

Recall Quick self-test

Why is it called the Normal equation? ::: "Normal" is geometry-speak for perpendicular: at the best line the residual is perpendicular to the column space of $X$, giving $X^T(X{\theta }^{\ast }-y)=0$. What does $v^T$ (transpose) let us do in one multiplication? ::: Lay a column on its side so a row-times-column product adds up the squares, $v^{T}v={\sum }_k{v}_k^2$ = length squared. What role does the column of 1s in $X$ play? ::: It carries the intercept ${\theta }_0$; without it the line is forced through the origin. When does $(X^{T}X{)}^{-1}$ fail to exist? ::: When $X$'s columns are not independent (duplicate/scaled features, or $n+1>m$) — the bowl becomes a flat trough with infinitely many best $\theta$. In the trough case, does a closest point still exist? ::: Yes — the perpendicular projection always exists; it is the labelling $\theta$ that becomes non-unique. Where did the $\frac{1}{2}$ in the cost go? ::: It cancels the factor of 2 from differentiating the square, so the gradient stays clean.

Connections: Gradient Descent (the iterative cousin), Ridge Regression (fixing the trough), Maximum Likelihood Estimation (why squared error under Gaussian noise), QR Decomposition & Pseudoinverse (solving without forming $X^{T}X$), Feature Scaling, Bias-Variance Tradeoff.