Worked examples — Normal equation closed-form solution

This is the drill-ground for the normal equation. The parent note derived the formula

$$\mathbf{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}.$$

Here we throw every kind of dataset at it — clean, degenerate, more-equations-than-unknowns, fewer, and a real word problem — so you never meet a case you haven't already seen solved.

---

The scenario matrix

Every dataset you can hand the normal equation falls into one of these cells. The examples below cover all of them.

Cell	Situation	What decides it	Does the plain formula work?
A	Overdetermined, full rank, imperfect fit	$m>n+1$, columns independent, noisy points	✅ unique least-squares line
B	Exact fit possible ($m=n+1$)	points lie exactly on a line	✅ residual is zero
C	Zero-slope / degenerate target	all $y$ equal, or all $x$ equal	✅ but geometry is special
D	Rank-deficient (redundant features)	one feature $=$ combo of others	❌ ${\mathbf{X}}^T\mathbf{X}$ singular → use pseudoinverse
E	Underdetermined ($m<n+1$)	fewer points than unknowns	❌ infinitely many exact fits → min-norm via ${\mathbf{X}}^{+}$ (see Underdetermined vs Overdetermined Systems)
F	Real-world word problem	units, meaning of $\theta$	✅ interpret the numbers
G	Exam twist: scaling / conditioning	huge or tiny feature values	✅ but numerically fragile → Feature Scaling

---

Example 1 — Cell A: overdetermined, noisy, full rank

Forecast: the points roughly climb by $\approx 1.6$ per step and start near $1$ — guess the answer before reading on.

1. Build $\mathbf{X}$ and $\mathbf{y}$.

$$\mathbf{X}=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\\ 1& 3\end{array}\right],\mathbf{y}=\left[\begin{array}{c}1\\ 3\\ 4\\ 6\end{array}\right]$$

Why this step? $m=4$ points, $n=1$ feature, so $\mathbf{X}$ is $4\times 2$: first column of 1's (bias), second column the $x$'s.

2. Form the little square matrix ${\mathbf{X}}^T\mathbf{X}$.

$${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}4& 6\\ 6& 14\end{array}\right]$$

Why this step? We can only invert a square matrix. The top-left $4$ counts the rows; $6=0+1+2+3$; $14=0^2+1^2+2^2+3^2$.

3. Form ${\mathbf{X}}^T\mathbf{y}$.

$${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}1+3+4+6\\ 0+3+8+18\end{array}\right]=\left[\begin{array}{c}14\\ 29\end{array}\right]$$

Why this step? This is the right-hand side — it captures how the targets line up with each column.

4. Determinant then inverse. $\det =4(14)-6(6)=56-36=20$.

$$({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{20}\left[\begin{array}{cc}14& -6\\ -6& 4\end{array}\right]$$

Why this step? $\det \ne 0$ confirms full column rank — the plain formula is legal here.

5. Multiply through.

$$\mathbf{\theta }=\frac{1}{20}\left[\begin{array}{cc}14& -6\\ -6& 4\end{array}\right]\left[\begin{array}{c}14\\ 29\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}196-174\\ -84+116\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}22\\ 32\end{array}\right]=\left[\begin{array}{c}1.1\\ 1.6\end{array}\right]$$

Result: $y=1.1+1.6x$.

Verify: predictions are $1.1,2.7,4.3,5.9$; residuals $-0.1,0.3,-0.3,0.1$ sum to $0$ — exactly what "the residual is orthogonal to the column space" forces. The fit does not pass through every point (Cell A = imperfect), and that is correct.

---

Example 2 — Cell B: exact fit, $m=n+1$

Forecast: these are not perfectly collinear ($2,4,5$ is not an arithmetic run) — so predict a near-fit, not an exact one.

1. From the parent note, $\mathbf{\theta }=\left[\begin{array}{c}2/3\\ 3/2\end{array}\right]$, i.e. $y=0.667+1.5x$. Why re-use it? To connect drill and theory — same machinery.

2. Predict: $x=1\to 2.167$, $x=2\to 3.667$, $x=3\to 5.167$. Why this step? Checking residuals tells us which cell we're really in.

Verify: residuals $-0.167,0.333,-0.167$; they are non-zero, so this is still Cell A-flavoured. True Cell B needs genuinely collinear points, e.g. $(1,2),(2,4),(3,6)$ → $y=2x$ exactly, residual $0$. We check that in =VERIFY=.

---

Example 3 — Cell C: degenerate target (all $y$ equal)

Forecast: the truth is obviously $y=5$, so guess ${\theta }_1=0,{\theta }_0=5$.

1. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 4\end{array}\right],\mathbf{y}=\left[\begin{array}{c}5\\ 5\\ 5\end{array}\right]$. Why this step? Standard setup; the interesting part is the flat $\mathbf{y}$.

2. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 7\\ 7& 21\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}15\\ 35\end{array}\right]$. Why this step? $\det =3(21)-49=63-49=14\ne 0$ — full rank, formula applies. Varying $x$ keeps the columns independent even though $y$ is constant.

3. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{14}\left[\begin{array}{cc}21& -7\\ -7& 3\end{array}\right]$, then

$$\mathbf{\theta }=\frac{1}{14}\left[\begin{array}{cc}21& -7\\ -7& 3\end{array}\right]\left[\begin{array}{c}15\\ 35\end{array}\right]=\frac{1}{14}\left[\begin{array}{c}315-245\\ -105+105\end{array}\right]=\left[\begin{array}{c}5\\ 0\end{array}\right].$$

Result: $y=5+0\cdot x$ — a perfectly horizontal line.

Verify: slope is exactly $0$, intercept exactly $5$, residuals all $0$. The machinery gracefully returns a flat line when the data is flat. (Contrast Cell D: it is a degenerate feature, not a degenerate target, that breaks invertibility.)

---

Example 4 — Cell D: rank-deficient features (the trap)

Forecast: the plain formula will explode — predict "no inverse".

1. $\mathbf{X}=\left[\begin{array}{ccc}1& 1& 3\\ 1& 2& 4\\ 1& 3& 5\end{array}\right]$. Notice ${\text{col}}_2={\text{col}}_1+2\cdot {\text{col}}_0$. Why this step? Spotting a linear relation among columns is the whole point of Cell D.

2. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{ccc}3& 6& 12\\ 6& 14& 26\\ 12& 26& 50\end{array}\right]$. Why this step? We compute it to demonstrate its determinant is $0$.

3. Determinant check: $\det ({\mathbf{X}}^T\mathbf{X})=0$ (row/column dependence carries through). Why this step? Zero determinant ⇒ singular ⇒ $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ does not exist ⇒ plain formula illegal.

4. Fix: use the pseudoinverse ${\mathbf{X}}^{+}$ (via SVD/QR), which returns the minimum-norm solution. One exact fit is $\mathbf{\theta }=(2,2,1)$: $2+2x_1+x_2$. Why this step? Among infinitely many perfect fits, ${\mathbf{X}}^{+}$ picks the shortest $\Vert \mathbf{\theta }\Vert$.

Verify: $(2,2,1)$ gives $2+2+3=7,2+4+4=10,2+6+5=13$ ✅ zero error, but the solution is not unique — e.g. $(0,3,\frac{...}{})$-type combos also fit. The determinant equals $0$; that is the alarm bell.

---

Example 5 — Cell E: underdetermined ($m<n+1$)

Forecast: one equation, two unknowns → infinitely many lines through the point.

1. $\mathbf{X}=[12]$ (size $1\times 2$), $\mathbf{y}=[6]$. Why this step? $m=1<n+1=2$ ⇒ underdetermined.

2. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}1& 2\\ 2& 4\end{array}\right]$, $\det =4-4=0$ → singular. Why this step? Underdetermined systems also give a singular ${\mathbf{X}}^T\mathbf{X}$.

3. Min-norm fix: $\mathbf{\theta }={\mathbf{X}}^{+}\mathbf{y}={\mathbf{X}}^T(\mathbf{X}{\mathbf{X}}^T{)}^{-1}\mathbf{y}$. Here $\mathbf{X}{\mathbf{X}}^T=[1+4]=[5]$, so

$$\mathbf{\theta }=\left[\begin{array}{c}1\\ 2\end{array}\right]\cdot \frac{6}{5}=\left[\begin{array}{c}1.2\\ 2.4\end{array}\right].$$

Why this step? When rows are the short dimension, $\mathbf{X}{\mathbf{X}}^T$ is invertible; this gives the shortest-length solution.

Verify: $1.2+2.4(2)=1.2+4.8=6$ ✅ passes through the point, and $\Vert \mathbf{\theta }{\Vert }^2=1.44+5.76=7.2$ is minimal among all lines through $(2,6)$.

---

Example 6 — Cell F: real-world word problem

Forecast: perfectly linear ($y=2x+1$?) — guess price \6000$at$x=2.5$.

1. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\\ 1& 4\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}3\\ 5\\ 7\\ 9\end{array}\right]$. Why this step? Real data → same machinery; units: ${\theta }_0$ in $1000s, ${\theta }_1$ in $1000s per 100 ft².

2. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}4& 10\\ 10& 30\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}24\\ 70\end{array}\right]$, $\det =120-100=20$. Why this step? Full rank ⇒ proceed.

3. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{20}\left[\begin{array}{cc}30& -10\\ -10& 4\end{array}\right]$, and

$$\mathbf{\theta }=\frac{1}{20}\left[\begin{array}{cc}30& -10\\ -10& 4\end{array}\right]\left[\begin{array}{c}24\\ 70\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}720-700\\ -240+280\end{array}\right]=\left[\begin{array}{c}1\\ 2\end{array}\right].$$

Result: $y=1+2x$. At $x=2.5$: $y=1+5=6$ ⇒ $6000.

Verify: each point fits exactly ($1+2=3,\dots ,1+8=9$), residual $0$; prediction $6000 matches the forecast, units consistent ($1000s).

---

Example 7 — Cell G: exam twist (scaling / conditioning)

Forecast: the shape is identical, so ${\theta }_1$ should shrink by $1000$ and ${\theta }_0$ stay near $1$.

1. Fit gives ${\theta }_1=2/1000=0.002$, ${\theta }_0=1$. Why this step? Slope has units "per $x$"; scaling $x$ up by $1000$ scales slope down by $1000$.

2. But ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}4& 10000\\ 10000& 3\times 10^7\end{array}\right]$: entries span $4$ to $3\times 10^7$. Why this step? Huge range ⇒ ill-conditioned matrix ⇒ inversion loses precision.

Verify: $y=1+0.002x$ reproduces $1+0.002(1000)=3$ ✅ etc. The math is right, but the condition number is enormous — this is exactly why we apply Feature Scaling or switch to QR Decomposition / Gradient Descent on real data.

Recall Which cell breaks the plain inverse?

Cells D (redundant features) and E (too few points) ::: both make ${\mathbf{X}}^T\mathbf{X}$ singular → use the pseudoinverse.

Recall Why does an all-equal

$y$ (Cell C) still work? Because the features still vary, so the columns of $\mathbf{X}$ stay independent ::: only feature/column dependence kills invertibility, not a flat target.

---

Connections

• Parent: Normal equation closed-form solution · Hinglish: 2.2.05 Normal equation closed-form solution (Hinglish)
• Foundations: Linear Regression Fundamentals · alternative solver: Gradient Descent
• Degenerate cases: Moore-Penrose Pseudoinverse · Underdetermined vs Overdetermined Systems · numerically-stable route: QR Decomposition
• Preprocessing & regularisation: Feature Scaling · Ridge Regression · Polynomial Regression