Worked examples — Least squares — normal equations, QR approach

Before we start, one reminder of the vocabulary so no symbol is unearned:

One more symbol we will lean on later:

The scenario matrix

Every least-squares question you meet is one (or a blend) of these cells. The last column names the example that clears it.

We work these in order. Cells C1, C2, C6, C7 get a figure because geometry is doing the talking.

Example 1 — Overdetermined, full rank (cell C1)

The figure below plots the four coral data points and the lavender best-fit line we are about to compute; the mint verticals are the residuals — the up-and-down misses least squares makes as small as possible.

Step 1 — build $A$ and $\mathbf{b}$. $A=\left(\begin{array}{cc}1& -1\\ 1& 0\\ 1& 1\\ 1& 2\end{array}\right),\mathbf{b}=\left(\begin{array}{c}1\\ 0\\ 0\\ -3\end{array}\right).$ Why this step? Each row says "predict $y_i$ by $c\cdot 1+d\cdot t_i$". Column one is all-ones (the intercept $c$); column two holds the $t_i$ (the slope $d$).

Step 2 — form $A^{\top }A$ and $A^{\top }\mathbf{b}$. $A^{\top }A=\left(\begin{array}{cc}4& 2\\ 2& 6\end{array}\right),A^{\top }\mathbf{b}=\left(\begin{array}{c}-2\\ -7\end{array}\right).$ Why this step? $A^{\top }A$ entries are column dot products: $1\cdot 1=4$, $1\cdot \mathbf{t}=(-1+0+1+2)=2$, $\mathbf{t}\cdot \mathbf{t}=(1+0+1+4)=6$. $A^{\top }\mathbf{b}$ dots each column with $\mathbf{b}$.

Step 3 — solve the $2\times 2$ system. $\left(\begin{array}{cc}4& 2\\ 2& 6\end{array}\right)\left(\genfrac{}{}{0ex}{}{c}{d}\right)=\left(\genfrac{}{}{0ex}{}{-2}{-7}\right).$ Determinant $=4\cdot 6-2\cdot 2=20$. By Cramer: $c=\frac{(-2)(6)-(2)(-7)}{20}=\frac{2}{20}=0.1,d=\frac{(4)(-7)-(2)(-2)}{20}=\frac{-24}{20}=-\mathrm{1.2.}$ Why this step? $A^{\top }A$ is invertible (full column rank), so $\hat{\mathbf{x}}=(A^{\top }A{)}^{-1}{A}^{\top }\mathbf{b}$ is unique.

Best line: $y=0.1-1.2t$. Slope is negative — forecast confirmed. In the figure the lavender line indeed tilts downward.

Verify: predictions at $t=-1,0,1,2$ are $1.3,0.1,-1.1,-2.3$; residual $\mathbf{r}=(-0.3,-0.1,1.1,-0.7)$. Check orthogonality: $1\cdot \mathbf{r}=0$ and $\mathbf{t}\cdot \mathbf{r}=0$. ✓ (Both are checked in VERIFY.)

Example 2 — Consistent system, zero residual (cell C2)

In the figure the three coral points sit dead-on the lavender line, so there are no mint residual sticks to draw — a picture of a zero residual.

Step 1 — set up. $A=\left(\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right),\mathbf{b}=\left(\begin{array}{c}2\\ 5\\ 8\end{array}\right).$ Why this step? Exactly as before, each row encodes "predict $y_i$ by $c+d{t}_i$": the all-ones first column carries the intercept $c$, the second column carries the inputs $t_i=0,1,2$ for the slope $d$. Only $\mathbf{b}$ (the measured $y$-values) changes from Ex 1.

Step 2 — normal equations. $A^{\top }A=\left(\begin{array}{cc}3& 3\\ 3& 5\end{array}\right)$, $A^{\top }\mathbf{b}=\left(\begin{array}{c}15\\ 21\end{array}\right)$. Why this step? We project $\mathbf{b}$ onto $\text{Col}(A)$ by forcing the residual perpendicular to both columns; the entries are again column dot products ($1\cdot 1=3$, $1\cdot \mathbf{t}=3$, $\mathbf{t}\cdot \mathbf{t}=5$) and $A^{\top }\mathbf{b}$ dots each column with $\mathbf{b}$.

Step 3 — solve. From $3c+3d=15$ and $3c+5d=21$: subtract → $2d=6\Rightarrow d=3$, then $c=2$. Why this step? Two linear equations in two unknowns; subtracting the first from the second cancels $c$ (both have coefficient $3$), leaving a single equation in $d$. Back-substitute $d$ into the first to recover $c$ — the same elimination you'd use on any $2\times 2$ system.

Line: $y=2+3t$ — the exact line, as forecast.

Verify: predictions $(2,5,8)=\mathbf{b}$, so $\mathbf{r}=0$. When $\mathbf{b}\in \text{Col}(A)$, the projection is $\mathbf{b}$ and least squares degenerates to an ordinary solve. ✓

Example 3 — Same data via QR (cell C3)

Step 1 — orthonormalise. ${\mathbf{q}}_1=\frac{1}{\sqrt{3}}(1,1,1)$ since $\Vert {\mathbf{a}}_1\Vert =\sqrt{3}$. Projection coefficient ${\mathbf{q}}_1^{\top }{\mathbf{a}}_2=\frac{0+1+2}{\sqrt{3}}=\sqrt{3}$. Remove it: ${\mathbf{a}}_2-\sqrt{3}{\mathbf{q}}_1=(0,1,2)-(1,1,1)=(-1,0,1)$. Normalise: ${\mathbf{q}}_2=\frac{1}{\sqrt{2}}(-1,0,1)$. Why this step? Orthonormal columns turn projection into simple dot products — no matrix inverse needed.

Step 2 — read off $R=Q^{\top }A$. $R=\left(\begin{array}{cc}\sqrt{3}& \sqrt{3}\\ 0& \sqrt{2}\end{array}\right).$ Why this step? $R_{11}=\Vert {\mathbf{a}}_1\Vert =\sqrt{3}$, $R_{12}={\mathbf{q}}_1^{\top }{\mathbf{a}}_2=\sqrt{3}$, $R_{22}=\Vert (-1,0,1)\Vert =\sqrt{2}$.

Step 3 — compute $Q^{\top }\mathbf{b}$ and back-substitute. $Q^{\top }\mathbf{b}=\left(\begin{array}{c}\frac{2+5+8}{\sqrt{3}}\\ \frac{-2+0+8}{\sqrt{2}}\end{array}\right)=\left(\begin{array}{c}\frac{15}{\sqrt{3}}\\ \frac{6}{\sqrt{2}}\end{array}\right)=\left(\begin{array}{c}5\sqrt{3}\\ 3\sqrt{2}\end{array}\right).$ Solve $R\hat{\mathbf{x}}=Q^{\top }\mathbf{b}$ bottom-up: $\sqrt{2}d=3\sqrt{2}\Rightarrow d=3$; then $\sqrt{3}c+\sqrt{3}(3)=5\sqrt{3}\Rightarrow c=2$. Why this step? $R$ is upper triangular so the last equation has one unknown — back-substitution is fast and doesn't square the condition number.

Verify: $c=2,d=3$, matching Ex 2 exactly. ✓

Example 4 — Rank-deficient: infinitely many solutions (cell C4)

Step 1 — spot the rank. $\text{Col}(A)$ is a single line spanned by $(1,1,1)$; rank $=1$, not $2$. Why this step? If columns don't span two independent directions, we can't pin down two coordinates.

Step 2 — normal equations. $A^{\top }A=\left(\begin{array}{cc}3& 6\\ 6& 12\end{array}\right),\det =3\cdot 12-6\cdot 6=0.$ Singular — no inverse. $A^{\top }\mathbf{b}=\left(\genfrac{}{}{0ex}{}{6}{12}\right)$. Why this step? The normal equations $A^{\top }A\hat{\mathbf{x}}=A^{\top }\mathbf{b}$ are still the right machinery even when columns are dependent — we form them exactly as usual (entries are column dot products, $A^{\top }\mathbf{b}$ dots each column with $\mathbf{b}$). Computing the determinant tells us in advance that this $A^{\top }A$ cannot be inverted, which is the whole point of this case.

Step 3 — describe ALL solutions. The system $3x_1+6x_2=6$ (the second row is just $2\times$ this) gives $x_1+2x_2=2$. Let $s$ be any real number and set $x_2=s$; then $x_1=2-2s$, so every $\hat{\mathbf{x}}=(2-2s,s)$ with $s\in \mathbb{R}$ is a valid least-squares solution. Why this step? One equation, two unknowns → the second unknown is free, so a whole line of answers, indexed by the free parameter $s$.

Step 4 — the projection is still unique. For any such $\hat{\mathbf{x}}$, $A\hat{\mathbf{x}}=(x_1+2x_2)(1,1,1)=2(1,1,1)=(2,2,2)$. Why this step? Different $\hat{\mathbf{x}}$'s along the ambiguous direction all map to the same point of $\text{Col}(A)$ — the projection of $\mathbf{b}$. The pseudoinverse picks the minimum-norm representative.

Verify: residual $\mathbf{r}=\mathbf{b}-(2,2,2)=(-1,0,1)$; check $1\cdot \mathbf{r}=0$. Minimum-norm choice minimises $\Vert (2-2s,s){\Vert }^2$: derivative $2(2-2s)(-2)+2s=0\Rightarrow 10s=8\Rightarrow s=0.8$, giving $\hat{\mathbf{x}}=(0.4,0.8)$. ✓

Example 5 — Word problem with real units (cell C5)

Step 1 — one-column $A$. $A=\left(\begin{array}{c}1\\ 2\\ 3\\ 4\end{array}\right),\mathbf{b}=\left(\begin{array}{c}1.9\\ 4.1\\ 5.8\\ 8.2\end{array}\right).$ Why this step? Model has one unknown $k$ and passes through the origin, so there's no all-ones column — just the $x$ values. This is regression through the origin.

Step 2 — scalar normal equation. With one column, $A^{\top }A$ and $A^{\top }\mathbf{b}$ are scalars: $A^{\top }A=1+4+9+16=30,A^{\top }\mathbf{b}=1.9+8.2+17.4+32.8=\mathrm{60.3.}$ Why this step? The dot product $\mathbf{x}\cdot \mathbf{x}$ weights larger stretches more — they carry more "leverage".

Step 3 — solve. $k=\frac{60.3}{30}=2.01$ N/cm. Why this step? With only one unknown the normal equation collapses to a single scalar equation $30k=60.3$; dividing both sides by the scalar $A^{\top }A$ isolates $k$ — the one-dimensional version of "invert $A^{\top }A$".

Verify: units check — $(\text{N⋅cm})/({\text{cm}}^2)=$ N/cm. ✓ Predicted forces $(2.01,4.02,6.03,8.04)$ hug the data; residuals small. ✓ ($k=2.01$ checked in VERIFY.)

Example 6 — Degenerate limit: coincident data (cell C6)

In the figure the two clashing coral points sit stacked at $t=2$ ($y=4$ and $y=6$); the butter dot marks where the lavender fit-line actually passes — right between them at $y=5$, the vertical average shown by the mint stick.

Step 1 — set up. $A=\left(\begin{array}{cc}1& 0\\ 1& 2\\ 1& 2\end{array}\right),\mathbf{b}=\left(\begin{array}{c}0\\ 4\\ 6\end{array}\right).$ Why this step? Two rows share the same $t$; the design is still full column rank (columns $(1,1,1)$ and $(0,2,2)$ are independent), so a unique line exists.

Step 2 — normal equations. $A^{\top }A=\left(\begin{array}{cc}3& 4\\ 4& 8\end{array}\right)$, $A^{\top }\mathbf{b}=\left(\begin{array}{c}10\\ 20\end{array}\right)$, $\det =24-16=8$. $c=\frac{10\cdot 8-4\cdot 20}{8}=\frac{0}{8}=0,d=\frac{3\cdot 20-4\cdot 10}{8}=\frac{20}{8}=\mathrm{2.5.}$

Step 3 — interpret. Line $y=0+2.5t$. At $t=2$ it predicts $5$ — exactly the average of the two clashing readings $4$ and $6$, as forecast. Why this step? When several rows share the same input, least squares can only pick one output there, and the distance-minimising choice is their mean.

Verify: predictions $(0,5,5)$, residual $(0,-1,1)$; $1\cdot \mathbf{r}=0$, $\mathbf{t}\cdot \mathbf{r}=0$. ✓

Example 7 — Projection matrix, split $\mathbf{b}$ (cell C7)

The figure is a 2D schematic: the lavender line is $\text{Col}(A)$, the coral arrow is $\mathbf{b}$, the mint arrow is its shadow $P\mathbf{b}$ on the line, and the dashed slate segment is the residual — meeting the line at a right angle (marked), which is exactly the orthogonality least squares enforces.

Step 1 — build $P$ from the definition. Recall $P=A(A^{\top }A{)}^{-1}{A}^{\top }$. Here $A$ is the single column $(1,1,1{)}^{\top }$, so $A^{\top }A=1+1+1=3$ (a scalar), its inverse is $\frac{1}{3}$, and $A{A}^{\top }$ is the $3\times 3$ all-ones matrix. Substituting, $P=A(A^{\top }A{)}^{-1}{A}^{\top }=\frac{A{A}^{\top }}{3}=\frac{1}{3}\left(\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right).$ Why this step? With one column, the general formula $P=A(A^{\top }A{)}^{-1}{A}^{\top }$ collapses to $\frac{A{A}^{\top }}{\Vert A{\Vert }^2}$ — the orthogonal projection onto that line. Each row of $P$ is $(\frac{1}{3},\frac{1}{3},\frac{1}{3})$, i.e. "take the average of the three entries" — precisely what projecting onto $(1,1,1)$ should do.

Step 2 — idempotence check. $P^2=P$ because averaging an already-constant vector changes nothing, and $P^{\top }=P$ since the all-ones matrix is symmetric. Why this step? Projecting twice = projecting once; these two identities ($P^2=P$, $P^{\top }=P$) are the defining fingerprints of any projection matrix, confirming we built $P$ correctly.

Step 3 — split $\mathbf{b}$ into fit + residual. $\hat{\mathbf{b}}=P\mathbf{b}=\left(\frac{1+2+3}{3}\right)(1,1,1)=(2,2,2),\mathbf{r}=\mathbf{b}-\hat{\mathbf{b}}=(-1,0,1).$ Why this step? $\hat{\mathbf{b}}$ lies in the line (the fit — it is a multiple of $(1,1,1)$), $\mathbf{r}$ is perpendicular to it (the miss); the two pieces add back to $\mathbf{b}$, splitting it into "explained" plus "leftover".

Verify: ${\hat{\mathbf{b}}}^{\top }\mathbf{r}=(2,2,2)\cdot (-1,0,1)=0$ ✓; $\Vert \mathbf{b}{\Vert }^2=14=\Vert \hat{\mathbf{b}}{\Vert }^2+\Vert \mathbf{r}{\Vert }^2=12+2$ (Pythagoras). ✓

Example 8 — Exam twist: fit forced through a fixed point (cell C8)

Step 1 — reduce the model. Forcing $c=0$ removes the intercept column, leaving a single column of $t$-values: $A=\left(\begin{array}{c}1\\ 2\\ 3\end{array}\right),\mathbf{b}=\left(\begin{array}{c}3\\ 5\\ 8\end{array}\right).$ Why this step? A hard constraint "pass through the origin" isn't an extra equation to bolt on — it deletes the unknown $c$, shrinking $A$ from two columns to one. Fewer unknowns, same least-squares machinery.

Step 2 — scalar solve. With one column, $A^{\top }A=1+4+9=14$ and $A^{\top }\mathbf{b}=3+10+24=37$, so $d=\frac{A^{\top }\mathbf{b}}{A^{\top }A}=\frac{37}{14}\approx \mathrm{2.643.}$ Why this step? Same origin-regression scalar formula as Ex 5: one unknown means the normal equation is a single scalar equation $14d=37$, solved by dividing.

Step 3 — interpret. Best origin line: $y=\frac{37}{14}t$, slope inside the forecast band $2.5$–$2.67$.

Verify: $d=37/14=2.6428\dots$; predictions $(2.64,5.29,7.93)$, residual $(0.36,-0.29,0.07)$, check $\mathbf{t}\cdot \mathbf{r}=0$. ✓

Active Recall