Worked examples — Ordinary least squares derivation

This is the hands-on companion to the Ordinary least squares derivation parent note. There we derived the normal equation. Here we run it — on every kind of input the world can hand you. We will not skip a single case: clean data, noisy data, the "line through the origin" trap (fitting with no intercept), the broken (non-invertible) case, more-features-than-points, and a real word problem.

Everything below rests on one boxed result from the parent note. Let us restate it in plain words so no symbol is unearned.

Recall What the symbols mean (read this before anything else)

• $X$ ::: the design matrix. One row per data point. The first column is all $1$s (this column is the intercept). The remaining columns are the feature values.
• $y$ ::: the target column — the true answers we are trying to predict, stacked into one tall column.
• $\theta$ ::: the parameter column we are solving for. ${\theta }_0$ is the intercept (height where the line crosses the vertical axis), ${\theta }_1,{\theta }_2,\dots$ are the slopes.
• $X^T$ ::: $X$ flipped on its diagonal (rows become columns). We need it because the normal equation demands it.
• $(X^{T}X{)}^{-1}$ ::: the inverse of the square matrix $X^{T}X$ — the matrix that "undoes" it, like $\frac{1}{a}$ undoes $a$.
• $m$ ::: the number of data points (how many rows $X$ has — how many examples you collected).
• $n$ ::: the number of real features (not counting the $1$s column). With the intercept, $\theta$ has $n+1$ entries.

---

The scenario matrix

Every OLS problem you will meet falls into one of these cells. The examples that follow are labelled with the cell they cover, and together they fill the whole grid. Here $m$ is the number of data points and $n$ the number of real features (defined in the recall box above).

Cell	Scenario	What is special	Example
A	Exact fit, 1 feature	Points lie on a perfect line, residuals = 0	Ex 1
B	Noisy fit, 1 feature	Best line, non-zero residuals, positive slope	Ex 2
C	Negative slope	Sign check — line goes down	Ex 3
D	Zero slope (flat data)	${\theta }_1=0$, degenerate direction	Ex 4
E	Two real features	Full multivariate normal equation	Ex 5
F	Singular $X^{T}X$ (collinear)	Inverse does not exist — the failure case	Ex 6
G	$m<n+1$ (too few points)	Underdetermined — pseudoinverse rescue	Ex 7
H	Real-world word problem + units	Interpret $\theta$ with units, extrapolate	Ex 8
I	No intercept (line through origin)	Drop the $1$s column — the "origin trap"	Ex 9

We build a picture of the whole idea first, then knock out the cells one by one.

Look at the figure: OLS drops vertical dashed drop-lines from each point to the line, squares each length, and slides the line until the total squared drop is smallest. Every example below is just arithmetic that finds that best line.

---

Cell A — Exact fit, one feature

Step 1 — build $X$ and $y$. $X=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right],y=\left[\begin{array}{c}3\\ 5\\ 7\end{array}\right]$ Why this step? The first column of $1$s carries the intercept ${\theta }_0$; the second column holds the $x$ values. Every row is one data point.

Step 2 — compute $X^{T}X$. $X^{T}X=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right]$ Why this step? The normal equation needs this square matrix; its top-left entry is just "how many points" ($=3$).

Step 3 — compute $X^{T}y$. $X^{T}y=\left[\begin{array}{ccc}1& 1& 1\\ 1& 2& 3\end{array}\right]\left[\begin{array}{c}3\\ 5\\ 7\end{array}\right]=\left[\begin{array}{c}15\\ 34\end{array}\right]$ Why this step? This is the right-hand side of $X^{T}X\theta =X^{T}y$.

Step 4 — invert and solve. $\det =3\cdot 14-6\cdot 6=6$, so $(X^{T}X{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right].$ ${\theta }^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}15\\ 34\end{array}\right]=\left[\begin{array}{c}1\\ 2\end{array}\right].$ Why this step? $(X^{T}X{)}^{-1}$ undoes $X^{T}X$, isolating $\theta$.

Answer: ${\theta }_0=1,{\theta }_1=2$, so $\hat{y}=1+2x$.

Verify: plug in $x=2$: $1+2(2)=5=y$. ✓ All three points hit exactly — residuals are all $0$, exactly as forecast for perfectly collinear data.

---

Cell B — Noisy fit, positive slope

Step 1 — matrices. $X=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right],y=\left[\begin{array}{c}2\\ 4\\ 5\end{array}\right]$ Why this step? Same layout as Ex 1; only $y$ changed.

Step 2 — $X^{T}X$ and $X^{T}y$. $X^{T}X=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right],X^{T}y=\left[\begin{array}{c}11\\ 25\end{array}\right]$ Why this step? $X^{T}X$ depends only on the $x$-values, so it is identical to Ex 1; only $X^{T}y$ changes.

Step 3 — solve. ${\theta }^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}11\\ 25\end{array}\right]=\left[\begin{array}{c}2/3\\ 3/2\end{array}\right].$ Why this step? One matrix–vector multiply gives both parameters at once.

Answer: $\hat{y}=\frac{2}{3}+\frac{3}{2}x$.

Verify: the residuals are $r_1=2-(2/3+3/2)=-1/6$, $r_2=4-(2/3+3)=1/3$, $r_3=5-(2/3+9/2)=-1/6$. Their sum is $-1/6+1/3-1/6=0$. ✓ The residuals sum to zero — this is the geometric "orthogonality" fact from the parent note: the residual vector is perpendicular to the $1$s column.

---

Cell C — Negative slope (sign check)

Step 1 — matrices. $X=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right],y=\left[\begin{array}{c}6\\ 4\\ 2\end{array}\right]$ Why this step? Note $x$ starts at $0$ — perfectly legal, it just changes $X^{T}X$.

Step 2 — $X^{T}X$, $X^{T}y$. $X^{T}X=\left[\begin{array}{cc}3& 3\\ 3& 5\end{array}\right],X^{T}y=\left[\begin{array}{c}12\\ 8\end{array}\right]$ Why this step? $\det =15-9=6\ne 0$, so the inverse exists.

Step 3 — solve.

\theta^*=\tfrac16\begin{bmatrix}5&-3\\-3&3\end{bmatrix}\begin{bmatrix}12\\8\end{bmatrix}=\begin{bmatrix}6\\-2\end{bmatrix}.$$ *Why this step?* The negative off-diagonal entries push $\theta_1$ negative — exactly the downhill trend. **Answer:** $\hat y = 6 - 2x$. **Verify:** at $x=0,1,2$ we predict $6,4,2$ — an *exact* fit here, residuals $0$. Slope is $-2$, negative as forecast. ✓ --- ## Cell D — Zero slope (flat, degenerate direction) > [!example] Example 4 — the target does not depend on $x$ > Fit $(1,5),\ (2,5),\ (3,5)$. > > **Forecast:** $y$ is stuck at $5$ no matter what $x$ is. What is $\theta_1$? What is $\theta_0$? **Step 1 — matrices.** $$X=\begin{bmatrix}1&1\\1&2\\1&3\end{bmatrix},\quad y=\begin{bmatrix}5\\5\\5\end{bmatrix}$$ *Why this step?* $X$ is fine and full-rank; only $y$ is flat. **Step 2 — $X^TX$, $X^Ty$.** $$X^TX=\begin{bmatrix}3&6\\6&14\end{bmatrix},\qquad X^Ty=\begin{bmatrix}15\\30\end{bmatrix}$$ *Why this step?* $X^Ty=[\sum y,\ \sum x_i y_i]^T = [15, 30]^T$. **Step 3 — solve.** $$\theta^*=\tfrac16\begin{bmatrix}14&-6\\-6&3\end{bmatrix}\begin{bmatrix}15\\30\end{bmatrix}=\begin{bmatrix}5\\0\end{bmatrix}.$$ *Why this step?* The slope collapses to exactly $0$ — OLS correctly reports "$x$ carries no information." **Answer:** $\hat y = 5 + 0\cdot x$. A flat line at height $5$. **Verify:** every prediction is $5$, matching all targets; residuals all $0$. Here "the slope has no room to help": tilting the line even slightly lifts one end above the flat data and drops the other below, so the squared drops grow. That is what we mean by a **degenerate slope direction** — every wiggle only makes things worse, so $\theta_1=0$ is the unique best choice. ✓ --- ## Cell E — Two real features > [!example] Example 5 — genuine multivariate fit > Fit $\hat y=\theta_0+\theta_1 x_1+\theta_2 x_2$ to > $(x_1,x_2,y)=(0,0,1),\ (1,0,3),\ (0,1,2),\ (1,1,4)$. > > **Forecast:** raising $x_1$ by $1$ seems to add about $2$ to $y$; raising $x_2$ by $1$ adds about $1$; the base (both zero) is $1$. Guess $\theta_0,\theta_1,\theta_2$. **Step 1 — build $X$ (three columns now).** $$X=\begin{bmatrix}1&0&0\\1&1&0\\1&0&1\\1&1&1\end{bmatrix},\quad y=\begin{bmatrix}1\\3\\2\\4\end{bmatrix}$$ *Why this step?* Column 1 = intercept, column 2 = feature $x_1$, column 3 = feature $x_2$. Four points, three unknowns — overdetermined, exactly what OLS loves. I chose clean $0/1$ features so the algebra stays readable. **Step 2 — compute $X^TX$.** $$X^TX=\begin{bmatrix}4&2&2\\2&2&1\\2&1&2\end{bmatrix}$$ *Why this step?* Symmetric $3\times3$. Its top-left entry is $m=4$ (the number of points); the rest are sums of feature products. **Step 3 — compute $X^Ty$.** $$X^Ty=\begin{bmatrix}1+3+2+4\\3+4\\2+4\end{bmatrix}=\begin{bmatrix}10\\7\\6\end{bmatrix}$$ *Why this step?* Row 1 sums all $y$; row 2 sums $y$ where $x_1=1$; row 3 sums $y$ where $x_2=1$. **Step 4 — solve $X^TX\,\theta = X^Ty$ by elimination.** We solve the system, labelling the equations R1, R2, R3: $$\text{R1: }4\theta_0+2\theta_1+2\theta_2=10,\quad \text{R2: }2\theta_0+2\theta_1+\theta_2=7,\quad \text{R3: }2\theta_0+\theta_1+2\theta_2=6.$$ Compute $\text{R1}-2\cdot\text{R2}$. The $\theta_0$ terms cancel ($4-4=0$), the $\theta_2$ terms cancel ($2-2=0$), leaving only the $\theta_1$ terms: $(2-4)\theta_1 = 10-14$, i.e. $-2\theta_1=-4$, so $\theta_1=2$. Compute $\text{R1}-2\cdot\text{R3}$. Now the $\theta_0$ terms cancel ($4-4=0$) and the $\theta_1$ terms cancel ($2-2=0$), leaving only the $\theta_2$ terms: $(2-4)\theta_2 = 10-12$, i.e. $-2\theta_2=-2$, so $\theta_2=1$. Put $\theta_1=2,\theta_2=1$ into R2: $2\theta_0+4+1=7\Rightarrow 2\theta_0=2\Rightarrow \theta_0=1$. *Why this step?* Each subtraction is chosen to cancel two of the three unknowns at once, isolating a single parameter — so we never had to invert the $3\times3$ by hand. (For big systems this is exactly the job of [[QR Decomposition]].) **Answer:** $\theta^*=(\,\theta_0,\theta_1,\theta_2\,)=(1,\,2,\,1)$, so $\hat y = 1 + 2x_1 + x_2$. **Verify:** predictions at the four points are $1,\ 3,\ 2,\ 4$ — **exactly** the targets, so every residual is $0$ and the normal equation $X^T(X\theta-y)=0$ holds trivially. ✓ Notice the fitted numbers match our forecast: base $1$, $+2$ per unit of $x_1$, $+1$ per unit of $x_2$. --- ## Cell F — Singular $X^TX$ (collinear features, the failure case) > [!example] Example 6 — two features that are secretly one > $x_2 = 2x_1$ always: data $(x_1,x_2,y)=(1,2,3),\ (2,4,5),\ (3,6,7)$. > > **Forecast:** can we split "how much comes from $x_1$" vs "from $x_2$"? Think before computing. **Step 1 — build $X$.** $$X=\begin{bmatrix}1&1&2\\1&2&4\\1&3&6\end{bmatrix}$$ *Why this step?* Column 3 is exactly $2\times$ column 2 — the two features move in lockstep. **Step 2 — form $X^TX$ and check its determinant.** $$X^TX=\begin{bmatrix}3&6&12\\6&14&28\\12&28&56\end{bmatrix},\qquad \det(X^TX)=0.$$ *Why this step?* A zero determinant means **no inverse exists** — the normal equation has no unique solution. **Step 3 — interpret the collapse.** Because $x_2=2x_1$, any trade $\theta_1\!\to\!\theta_1+2c,\ \theta_2\!\to\!\theta_2-c$ leaves every prediction unchanged. Infinitely many $\theta$ tie for best. *Why this step?* This is the geometric meaning of "==singular==": the column space is lower-dimensional than the number of columns. The picture below shows exactly this collapse. **Answer:** OLS **fails** — there is no single $\theta^*$. **Verify:** $\det=0$ confirms non-invertibility (checked below). Look at the figure: every data point sits on **one** straight line, so the two feature columns do not point in independent directions — they trace the same line. There is no genuine "second axis" for the second feature to live on, which is why the matrix cannot be inverted. ![[deepdives/dd-ai-ml-2.2.03-d3-s02.png]] > [!intuition] The two rescue tools, sketched > - **Ridge regression** — add a tiny $\lambda>0$ down the diagonal: replace $X^TX$ by $X^TX+\lambda I$. This nudges every diagonal entry up so the determinant is no longer $0$, and the inverse exists again. Geometrically it gently "props up" the flat direction so a unique lowest point appears. Full story in [[Ridge Regression]]. > - **Pseudoinverse** — when the inverse truly does not exist, the [[Pseudoinverse]] still returns *one* answer: the smallest-length $\theta$ among the infinitely many tied winners. It picks the tie-break "keep the parameters as small as possible." See also [[Feature Scaling]] for spotting collinear columns before they break the fit. --- ## Cell G — Fewer points than parameters (underdetermined) > [!example] Example 7 — 1 point, 2 unknowns > Fit $\hat y=\theta_0+\theta_1 x$ to a **single** point $(2,5)$. > > **Forecast:** how many lines pass through one point? (Infinitely many.) **Step 1 — matrices.** $$X=\begin{bmatrix}1&2\end{bmatrix},\quad y=\begin{bmatrix}5\end{bmatrix}$$ *Why this step?* Here $m=1$ (one data point) and $n+1=2$ unknowns ($\theta_0,\theta_1$): **more unknowns than equations**. **Step 2 — form $X^TX$.** $$X^TX=\begin{bmatrix}1&2\\2&4\end{bmatrix},\qquad \det=1\cdot4-2\cdot2=0.$$ *Why this step?* Singular again — but for a different reason than Cell F: here we simply have too little data, so $m<n+1$. **Step 3 — rescue with the pseudoinverse.** The plain inverse does not exist, so we call on the [[Pseudoinverse]] $X^{+}$, which returns the **shortest** $\theta$ that fits. For a single-row $X$ it is $X^{+}=X^T(XX^T)^{-1}$ (note: $XX^T$ here, a $1\times1$ number, not $X^TX$). Compute $XX^T = 1^2+2^2 = 5$, so $$\theta^{+}=X^{+}y=X^T(XX^T)^{-1}y=\begin{bmatrix}1\\2\end{bmatrix}\cdot\tfrac15\cdot 5=\begin{bmatrix}1\\2\end{bmatrix}.$$ *Why this step?* When $X^TX$ is singular there is no unique inverse, but the pseudoinverse still hands back **one** well-defined answer — the parameter vector of smallest length $\theta_0^2+\theta_1^2$ among all that fit. This is the rescue tool we introduced in Cell F, now actually used. **Answer:** the pseudoinverse picks $\theta^{+}=(1,2)$ — one specific line $\hat y = 1 + 2x$ out of infinitely many that fit. **Verify:** it fits the point: $1+2\cdot2=5=y$ ✓. And it is genuinely non-unique — another legal fit is $\theta_0=3,\theta_1=1$ giving $3+2=5$ ✓; the pseudoinverse simply prefers the smaller-length $(1,2)$ (length$^2 = 5$) over $(3,1)$ (length$^2 = 10$). $\det(X^TX)=0$ confirms the plain inverse is unavailable. --- ## Cell H — Real-world word problem with units > [!example] Example 8 — pricing coffee by cups sold > A café records daily *cups sold* $x$ and *revenue* $y$ (in dollars): $(10,\$35),\ (20,\$55),\ (30,\$85)$. Fit $\hat y=\theta_0+\theta_1 x$ and predict revenue for $25$ cups. Interpret both parameters **with units**. > > **Forecast:** revenue rises roughly $\$2.50$ per cup; a fixed daily base near $\$10$. Guess. **Step 1 — matrices.** $$X=\begin{bmatrix}1&10\\1&20\\1&30\end{bmatrix},\quad y=\begin{bmatrix}35\\55\\85\end{bmatrix}$$ *Why this step?* Column of $1$s = fixed base revenue; second column = cups sold. **Step 2 — $X^TX$, $X^Ty$.** $$X^TX=\begin{bmatrix}3&60\\60&1400\end{bmatrix},\quad X^Ty=\begin{bmatrix}175\\4300\end{bmatrix}$$ *Why this step?* $\det = 3\cdot1400-60\cdot60=4200-3600=600\neq0$ — invertible, so a unique fit exists. **Step 3 — invert $X^TX$.** For a $2\times2$ matrix $\begin{bmatrix}a&b\\b&d\end{bmatrix}$ the inverse is $\tfrac{1}{ad-b^2}\begin{bmatrix}d&-b\\-b&a\end{bmatrix}$, so $$(X^TX)^{-1}=\tfrac{1}{600}\begin{bmatrix}1400&-60\\-60&3\end{bmatrix}.$$ *Why this step?* This is the "$\tfrac{1}{a}$ undoes $a$" idea for matrices; we now multiply it by $X^Ty$. **Step 4 — solve.** $$\theta^*=\tfrac1{600}\begin{bmatrix}1400&-60\\-60&3\end{bmatrix}\begin{bmatrix}175\\4300\end{bmatrix} =\tfrac1{600}\begin{bmatrix}1400\cdot175-60\cdot4300\\-60\cdot175+3\cdot4300\end{bmatrix} =\tfrac1{600}\begin{bmatrix}5000\\1500\end{bmatrix}=\begin{bmatrix}25/3\\5/2\end{bmatrix}.$$ *Why this step?* One matrix–vector product gives both parameters at once. **Answer:** $\theta_0=25/3\approx\$8.33$ and $\theta_1=5/2=\$2.50$, so $\hat y = \tfrac{25}{3} + \tfrac52 x$. **Step 5 — interpret with units.** - $\theta_1 = \$2.50\ \text{per cup}$ — each extra cup sold adds \$2.50 of revenue (slope = dollars ÷ cups). - $\theta_0 = \$8.33$ — the modelled base revenue at **zero** cups (the intercept, in dollars). **Step 6 — predict at $x=25$ cups.** $$\hat y = \frac{25}{3} + \frac{5}{2}\cdot 25 = \frac{25}{3} + \frac{125}{2} = \frac{50+375}{6}=\frac{425}{6}\approx \$70.83.$$ *Why this step?* Plug the desired $x=25$ into the fitted line. **Verify:** midpoint sanity check — $25$ cups sits between $20\ (\$55)$ and $30\ (\$85)$, so revenue should land between them; $\$70.83$ does. ✓ Units check: $[\text{cups}]\times[\$/\text{cup}]+[\$]=[\$]$. ✓ --- ## Cell I — No intercept (the "line through the origin" trap) > [!example] Example 9 — dropping the $1$s column > Fit $\hat y = \theta_1 x$ (**no** intercept, so the line is *forced through the origin*) to $(1,3),\ (2,5),\ (3,7)$ — the same data as Example 1, where the true best line was $\hat y = 1 + 2x$. > > **Forecast:** we are forbidding the intercept, so the line *must* pass through $(0,0)$. Will the single slope come out to $2$ (the with-intercept slope) or something different? Guess before computing. **Step 1 — build $X$ with **no** $1$s column.** $$X=\begin{bmatrix}1\\2\\3\end{bmatrix},\quad y=\begin{bmatrix}3\\5\\7\end{bmatrix}$$ *Why this step?* By omitting the column of $1$s we remove $\theta_0$ entirely; only a single slope $\theta_1$ survives, and the model is $\hat y=\theta_1 x$. This is exactly the mistake the parent note warns about — the line is nailed to the origin. **Step 2 — form $X^TX$ (now a single number).** $$X^TX = 1^2+2^2+3^2 = 14,\qquad X^Ty = 1\cdot3+2\cdot5+3\cdot7 = 34.$$ *Why this step?* With one column, $X^TX$ collapses to the scalar $\sum x_i^2$ and $X^Ty$ to $\sum x_i y_i$; the "matrix inverse" is just ordinary division. **Step 3 — solve.** $$\theta_1 = \frac{X^Ty}{X^TX} = \frac{34}{14} = \frac{17}{7}\approx 2.43.$$ *Why this step?* Dividing isolates the single slope — the closed-form for a through-origin fit. **Answer:** $\hat y = \tfrac{17}{7}\,x \approx 2.43\,x$. **Verify:** predictions at $x=1,2,3$ are $17/7\approx2.43,\ 34/7\approx4.86,\ 51/7\approx7.29$ versus targets $3,5,7$. The residuals are $3-17/7=4/7$, $5-34/7=1/7$, $7-51/7=-2/7$; their sum is $4/7+1/7-2/7=3/7\neq 0$. **The residuals no longer sum to zero** — the tell-tale sign of a missing intercept. Notice the slope came out $2.43$, *not* the correct $2$: forcing the line through the origin tilts it up to compensate for the intercept it cannot express. Moral: **always include the $1$s column** unless you truly know the line passes through $(0,0)$. --- ## Recall checkpoint Cover the right side and answer. Which cell has a **zero determinant because features repeat**? ::: Cell F (collinearity) — Example 6. Which cell has a **zero determinant because there is too little data** ($m<n+1$)? ::: Cell G — Example 7. Which tool gives *one* answer when $X^TX$ cannot be inverted? ::: The [[Pseudoinverse]] — it returns the smallest-length $\theta$ that fits (used in Example 7). In every *solvable, intercept-included* example the residuals shared one property — what? ::: They **sum to zero** (residual ⟂ the $1$s column). Drop the intercept (Cell I) and this breaks. When $\theta_1=0$ but $X$ is full-rank, what does it tell you? ::: The feature carries no predictive information (flat target, Cell D) — the fit is still unique. > [!mnemonic] Two ways the inverse dies > **"Copies or scarcity."** Copies = repeated/collinear columns (Cell F). Scarcity = fewer points than parameters, $m<n+1$ (Cell G). Either way, $\det(X^TX)=0$ and you reach for [[Ridge Regression]] or the [[Pseudoinverse]].