Worked examples — Linear regression — normal equation, gradient descent derivation

This deep dive is the hands-on companion to the parent topic. There we derived the two ways to fit a line. Here we run those machines on every kind of input you could meet — clean data, perfect fits, broken (singular) data, badly scaled data, real-world word problems, and an exam twist.

This page is self-contained: everything we use is defined right here first.

Recall The two formulas we will hammer

Closed form (one shot): ${\mathbf{w}}^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$. Iterative: ${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\frac{\alpha }{n}{\mathbf{X}}^T(\mathbf{X}{\mathbf{w}}_t-\mathbf{y})$. Here $\alpha$ is the learning rate (step size). The $\frac{1}{n}$ appears because the gradient of the average cost carries that same $\frac{1}{n}$ — we step against that averaged slope.

---

The scenario matrix

A line-fitting problem can be thrown at you in a limited number of flavours. Here is the full menu. Every flavour gets at least one fully worked example below.

Cell	Case class	What makes it special	Example
A	Positive slope, imperfect fit	ordinary case, errors don't cancel	Ex 1
B	Perfectly collinear points	error is exactly zero; fit passes through all	Ex 2
C	Negative slope	sign of $w$ is negative — catches sign slips	Ex 3
D	Zero slope (flat)	best line is horizontal, $w=0$	Ex 4
E	Singular / degenerate ${\mathbf{X}}^T\mathbf{X}$	inverse does not exist (all $x$ equal)	Ex 5
F	Gradient descent, step-by-step	iterative path, learning rate behaviour	Ex 6
G	Learning rate too large (divergence)	limiting behaviour of $\alpha$	Ex 7
H	Badly scaled features	why Feature Scaling matters	Ex 8
I	Real-world word problem (aerospace)	units, interpretation	Ex 9
J	Exam twist: proof/geometry	orthogonality of the residual	Ex 10

---

Example 1 — Cell A: positive slope, imperfect fit

Forecast: the points rise left-to-right, so guess $w\approx 1.5$ and a small positive $b$. Write your guess down before reading on.

1. Build $\mathbf{X}$ and $\mathbf{y}$. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right],\mathbf{y}=\left[\begin{array}{c}2\\ 4\\ 5\end{array}\right]$ Why this step? The first column of $1$s lets the bias $b$ live inside $\mathbf{w}=[b,w{]}^T$, so one formula handles both knobs.

2. Form ${\mathbf{X}}^T\mathbf{X}$ and ${\mathbf{X}}^T\mathbf{y}$. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right],{\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}11\\ 25\end{array}\right]$ Why this step? The normal equation ${\mathbf{X}}^T\mathbf{X}\mathbf{w}={\mathbf{X}}^T\mathbf{y}$ needs exactly these two objects. ${\mathbf{X}}^T\mathbf{X}$ collapses all the inputs into a $2\times 2$ summary (row·column sums: top-left is $1+1+1=3$, bottom-right is $1+4+9=14$); ${\mathbf{X}}^T\mathbf{y}$ pairs each input column with the answers ($1\cdot 2+1\cdot 4+1\cdot 5=11$, and $1\cdot 2+2\cdot 4+3\cdot 5=25$).

3. Invert the $2\times 2$. Determinant $=3\cdot 14-6\cdot 6=6$. $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]$ Why this step? To isolate $\mathbf{w}$ we must undo the multiply-by-${\mathbf{X}}^T\mathbf{X}$; for a $2\times 2$ we swap the diagonal, negate the off-diagonal, divide by the determinant.

4. Multiply out. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}11\\ 25\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}4\\ 9\end{array}\right]=\left[\begin{array}{c}0.6\overline{6}\\ 1.5\end{array}\right]$ So $b=0.667,w=1.5$.

Verify: ${\hat{y}}_1=1.5(1)+0.667=2.167$, ${\hat{y}}_2=3.667$, ${\hat{y}}_3=5.167$. Residuals $-0.167,+0.333,-0.167$ sum to $\approx 0$ — exactly what a least-squares fit forces. Slope matched our forecast. ✓

Figure — read it like this: the black dots are the three data points, the red line is the fitted $\hat{y}=1.5x+0.667$, and each dashed vertical segment is a residual (how far a dot sits above or below the red line). Notice the middle dot is above the line while the outer two are below — that visual balance is the residuals summing to zero.

---

Example 2 — Cell B: perfectly collinear data

Forecast: if the points are already on a line, the "best" line is that line, and the total error should be exactly zero.

1. Build. $\mathbf{X}=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}1\\ 3\\ 5\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 3\\ 3& 5\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}9\\ 13\end{array}\right]$. Why this step? Same normal-equation recipe as Ex 1 — top-left $=1+1+1=3$, bottom-right $=0+1+4=5$; nothing about "perfect data" changes the procedure.
3. Determinant $=3\cdot 5-3\cdot 3=6$, so $({\mathbf{X}}^T\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right].$ Why this step? Same isolate-$\mathbf{w}$ reason as Ex 1: invert to undo the multiply.
4. Solve. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right]\left[\begin{array}{c}9\\ 13\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}6\\ 12\end{array}\right]=\left[\begin{array}{c}1\\ 2\end{array}\right]$. So $b=1,w=2$.

Verify: $\hat{y}=2x+1$ gives exactly $1,3,5$ — residual vector is $0$, cost $J=0$. The fit is exact, matching the forecast. ✓

---

Example 3 — Cell C: negative slope

Forecast: downhill data → $w<0$. Each step of $+1$ in $x$ drops $y$ by $2$, so expect $w\approx -2$, $b\approx 7$.

1. Build. $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}5\\ 3\\ 1\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right]$ (same $x$'s as Ex 1!), ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}9\\ 14\end{array}\right]$. Why this step? ${\mathbf{X}}^T\mathbf{X}$ depends only on the inputs, so we reuse Ex 1's inverse; only ${\mathbf{X}}^T\mathbf{y}$ changes ($1\cdot 5+1\cdot 3+1\cdot 1=9$, $1\cdot 5+2\cdot 3+3\cdot 1=14$).
3. Solve. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}9\\ 14\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}42\\ -12\end{array}\right]=\left[\begin{array}{c}7\\ -2\end{array}\right]$. So $b=7,w=-2$.

Verify: $\hat{y}=-2x+7$ → $5,3,1$: an exact fit, and $w=-2<0$ confirms the negative slope our eyes predicted. ✓ Lesson: a negative answer is not an error — the sign encodes direction.

---

Example 4 — Cell D: zero slope (flat line)

Forecast: $y$ ignores $x$ entirely, so the best line is flat: $w=0$, $b=4$.

1. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 14\end{array}\right]$, ${\mathbf{X}}^T\mathbf{y}=\left[\begin{array}{c}12\\ 24\end{array}\right]$. Why this step? The $x$'s are the same as Ex 1 so ${\mathbf{X}}^T\mathbf{X}$ is unchanged; ${\mathbf{X}}^T\mathbf{y}$ sums the answers against each column ($4+4+4=12$, $1\cdot 4+2\cdot 4+3\cdot 4=24$). We build these two because the normal equation is fed exactly them.
2. Solve. ${\mathbf{w}}^{\ast }=\frac{1}{6}\left[\begin{array}{cc}14& -6\\ -6& 3\end{array}\right]\left[\begin{array}{c}12\\ 24\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}24\\ 0\end{array}\right]=\left[\begin{array}{c}4\\ 0\end{array}\right]$. Why this step? The gradient with respect to $w$ is zero only when the slope contributes nothing — the algebra hands us exactly $w=0$.

Verify: $\hat{y}=4$ everywhere, residuals all $0$. A flat target genuinely produces a flat fit. ✓

---

Example 5 — Cell E: singular / degenerate input

Forecast: all points sit on a vertical line $x=2$. A function $\hat{y}=wx+b$ cannot be vertical. Expect the machine to break — the inverse should not exist.

1. Build. $\mathbf{X}=\left[\begin{array}{cc}1& 2\\ 1& 2\\ 1& 2\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}3\\ 5\\ 9\end{array}\right]$.
2. Summaries. ${\mathbf{X}}^T\mathbf{X}=\left[\begin{array}{cc}3& 6\\ 6& 12\end{array}\right]$. Why this step? Same recipe; note the $x$-column is exactly $2\times$ the $1$-column, a warning sign we'll feel in the next step.
3. Determinant. $3\cdot 12-6\cdot 6=36-36=0$. Why this step? A zero determinant means the matrix has no inverse — the normal equation formula $({\mathbf{X}}^T\mathbf{X}{)}^{-1}$ is undefined. This is multicollinearity: the bias column and the $x$-column carry the same information (one is $2\times$ the other), so we cannot separate $b$ from $w$.

Verify: the determinant is literally $0$, confirming the forecast. Fixes: drop the redundant column, or add a tiny penalty — see Regularization (Ridge, Lasso), which replaces ${\mathbf{X}}^T\mathbf{X}$ with ${\mathbf{X}}^T\mathbf{X}+\lambda \mathbf{I}$ so the determinant is never zero. ✓

Figure — read it like this: the red vertical line is where all three black dots live ($x=2$). No sloped-or-flat line of the form $\hat{y}=wx+b$ can trace a vertical — that impossibility is exactly the zero determinant expressed as a picture.

---

Example 6 — Cell F: gradient descent, two honest steps

Forecast: starting from zero, the first gradient is large and negative (predictions are far below the data), so $\mathbf{w}$ jumps up. Expect it to crawl toward the true answer $[0.667,1.5]$ from Ex 1.

1. Iteration 0 — predict then measure error. $\hat{\mathbf{y}}=\mathbf{X}{\mathbf{w}}_0=[0,0,0{]}^T$, so error $\mathbf{e}=\mathbf{X}{\mathbf{w}}_0-\mathbf{y}=[-2,-4,-5{]}^T$. Why this step? The gradient $\nabla J=\frac{1}{n}{\mathbf{X}}^T\mathbf{e}$ needs the error first — we must know how wrong we are before deciding which way to step.
2. Gradient. $\nabla J=\frac{1}{3}{\mathbf{X}}^T\mathbf{e}=\frac{1}{3}\left[\begin{array}{ccc}1& 1& 1\\ 1& 2& 3\end{array}\right]\left[\begin{array}{c}-2\\ -4\\ -5\end{array}\right]=\frac{1}{3}\left[\begin{array}{c}-11\\ -25\end{array}\right]=\left[\begin{array}{c}-3.667\\ -8.333\end{array}\right].$
3. Update. ${\mathbf{w}}_1={\mathbf{w}}_0-\alpha \nabla J=[0,0]-0.1[-3.667,-8.333]=[0.3667,0.8333]$. Why this step? We step against the gradient (downhill on the cost surface); $\alpha$ scales how far.
4. Iteration 1. $\hat{\mathbf{y}}=\mathbf{X}{\mathbf{w}}_1=[1.2,2.0333,2.8667{]}^T$, error $\mathbf{e}=[-0.8,-1.9667,-2.1333{]}^T$. Gradient $=\frac{1}{3}{\mathbf{X}}^T\mathbf{e}$: first component $\frac{1}{3}(-0.8-1.9667-2.1333)=\frac{1}{3}(-4.9)=-1.6333$; second component $\frac{1}{3}(1(-0.8)+2(-1.9667)+3(-2.1333))=\frac{1}{3}(-11.1333)=-3.7111$. Update ${\mathbf{w}}_2={\mathbf{w}}_1-0.1[-1.6333,-3.7111]=[0.5300,1.2044]$. Why this step? Exactly the same rule repeated — descent is just "measure error, step downhill" over and over.

Verify: ${\mathbf{w}}_0=[0,0]\to {\mathbf{w}}_1=[0.367,0.833]\to {\mathbf{w}}_2=[0.530,1.204]$. Each step moves toward the closed-form target $[0.667,1.5]$ — exactly the crawl we forecast. ✓

Figure — read it like this: the axes are the two knobs ($b$ horizontal, $w$ vertical). The black path hops $w_0\to w_1\to w_2$, and the red star is the exact answer. See how each hop shrinks and bends toward the star — that is convergence.

---

Example 7 — Cell G: learning rate too large

Forecast: a giant step overshoots the valley and lands further up the other wall. The cost $J$ should get worse, and repeated steps blow up.

1. One update. ${\mathbf{w}}_1=[0,0]-2[-3.667,-8.333]=[7.333,16.667]$. Why this step? $\alpha =2$ multiplies the step; the sign is still correct (downhill), but the magnitude is far too big.
2. Check the cost. Using $J(\mathbf{w})=\frac{1}{2n}\Vert \mathbf{Xw}-\mathbf{y}{\Vert }^2$ with $n=3$: $J_0=\frac{1}{6}(2^2+4^2+5^2)=\frac{45}{6}=7.5$. With ${\mathbf{w}}_1$: $\hat{\mathbf{y}}=[24,40.667,57.333]$, residuals $\approx [22,36.667,52.333]$, $J_1=\frac{1}{6}(22^2+36.667^2+52.333^2)\approx 762$. Why this step? Comparing $J$ before and after tells us if the step helped. Here it exploded.

Verify: $J_1\approx 762\gg J_0=7.5$ — cost rose ~100×, confirming divergence. Rule of thumb: if $J$ increases, halve $\alpha$. Convex Optimization guarantees a valley exists; too-big steps just can't find it. ✓

---

Example 8 — Cell H: badly scaled features

Forecast: the metre-feature's gradient component will dwarf the fraction's, so one knob races while the other barely moves — the descent zig-zags.

1. Gradient component per feature. For one point, $\nabla J=(\hat{y}-y)\mathbf{x}=(0-10)(8000,0.5)=(-80000,-5)$. Why this step? The gradient is the error times the feature value; a big feature value makes a big component.
2. Ratio. $\frac{80000}{5}=16000$. Why this step? The two knobs feel forces differing by a factor of $16000$; any single $\alpha$ is wrong for one of them.

Verify: the component ratio $16000$ matches the feature-magnitude ratio $8000/0.5=16000$ exactly — proof the imbalance comes purely from scale. Fix: standardise features first — see Feature Scaling. After scaling, both components are comparable and descent goes straight downhill. ✓

---

Example 9 — Cell I: aerospace word problem

Forecast: we already fit this data (Ex 1): $\hat{y}=1.5x+0.667$. At $x=4$ expect $\approx 6.7$ kg/s.

1. Reuse the fit. $w=1.5\text{(kg/s per V)}$, $b=0.667\text{kg/s}$. Why this step? Calibration = fit once, then use the line as a lookup for new readings.
2. Predict at $x=4$. $\hat{y}=1.5(4)+0.667=6.667\text{kg/s}$. Why this step? We extrapolate one volt beyond the data — allowed because the physical relation is linear here.

Verify: units check — slope is kg/s per volt, times volts gives kg/s; bias is kg/s; sum is kg/s. ✓ Answer $6.667$ kg/s matches the forecast. (In practice we'd flag $x=4$ as slightly outside the $1$–$3$ V calibration range.)

---

Example 10 — Cell J: exam twist (geometry proof)

Forecast: the normal equation was born from the condition ${\mathbf{X}}^T(\mathbf{y}-\mathbf{Xw})=0$, so this should hold exactly — geometry, not luck.

1. Compute residuals. From Ex 1, $\hat{\mathbf{y}}=[2.167,3.667,5.167]$, so $\mathbf{r}=\mathbf{y}-\hat{\mathbf{y}}=[-0.167,0.333,-0.167]$. Why this step? Perpendicularity is a statement about $\mathbf{r}$, so we need $\mathbf{r}$ in numbers first.
2. Dot with column 1 (the $1$s). $(-0.167)+(0.333)+(-0.167)=0$. Why this step? Dotting with the all-ones column tests perpendicularity to the bias direction — geometrically it says the residuals must sum to zero (no leftover vertical offset the bias could still absorb).
3. Dot with column 2 (the $x$'s). $1(-0.167)+2(0.333)+3(-0.167)=-0.167+0.667-0.5=0$. Why this step? Dotting with the $x$-column tests perpendicularity to the slope direction — it says the errors carry no leftover linear trend the slope could still exploit. Both dots being zero means ${\mathbf{X}}^T\mathbf{r}=0$, completing the proof.

Verify: both dot products are $0$, so ${\mathbf{X}}^T\mathbf{r}=0$. Geometrically, $\mathbf{X}{\mathbf{w}}^{\ast }$ is the shadow (orthogonal projection) of $\mathbf{y}$ onto the plane spanned by $\mathbf{X}$'s columns; the leftover $\mathbf{r}$ points straight out of that plane. This is the heart of Least Squares Estimation. ✓

Figure — read it like this: the black horizontal arrow $\mathbf{X}{\mathbf{w}}^{\ast }$ lies inside the column space (drawn as the black line); the tall black arrow is $\mathbf{y}$; the red arrow is the residual $\mathbf{r}$ joining the tip of $\mathbf{X}{\mathbf{w}}^{\ast }$ up to $\mathbf{y}$. The little square shows the red arrow meets the column space at a right angle — that right angle is ${\mathbf{X}}^T\mathbf{r}=0$.

---

Recall Self-test

Which cell has no inverse and why? ::: Cell E — all $x$ equal makes $\det ({\mathbf{X}}^T\mathbf{X})=0$ (bias and feature columns are proportional). If gradient descent's cost $J$ rises, what do you do? ::: Halve the learning rate $\alpha$ (Cell G). What does a negative $w$ mean? ::: Downhill data — $y$ decreases as $x$ increases (Cell C). Why must residuals be orthogonal to $\mathbf{X}$'s columns? ::: Otherwise a leftover trend could still be removed, so the fit wasn't optimal (Cell J).