Worked examples — Multiple regression

This page is one long practice hall. We walk through every kind of situation a multiple-regression problem can hand you — clean data, degenerate data, collinear data, a real-world story, and an exam trick — and solve each one from the ground up. If a symbol appears here that you have not met, we build it first.

Before we start, one reminder from the parent note: the whole game is solving $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ where $\mathbf{X}$ is the table of predictors (with a leading column of $1$'s), $\mathbf{y}$ is the outcome column, and $\hat{\mathbf{\beta }}$ is the list of numbers we want: the intercept and the slopes.

We will also need three "sum of squares" quantities to judge fit. Build them now so they are never a surprise later:

---

The scenario matrix

Every problem below lands in one of these cells. The point is that after this page you have seen each cell at least once, so nothing on an exam is truly new.

Cell	What makes it special	Example
A. Clean full-rank fit	predictors independent, ${\mathbf{X}}^{\top }\mathbf{X}$ invertible	Ex 1
B. Single predictor sanity	multiple-regression machinery must reduce to simple regression	Ex 2
C. Negative slope	a predictor that lowers $y$	Ex 3
D. Prediction + interpretation	plug a new point into a fitted plane, read a partial slope	Ex 4
E. Degenerate: perfect collinearity	${\mathbf{X}}^{\top }\mathbf{X}$ singular, inverse does not exist	Ex 5
F. Near-collinearity (VIF blow-up)	inverse exists but coefficients explode	Ex 6
G. Fit quality: $R^2$ and adjusted $R^2$	including the "useless extra predictor" limiting case	Ex 7
H. Real-world word problem	translate a story into $\mathbf{X},\mathbf{y}$	Ex 8
I. Exam twist	centring / rescaling changes coefficients but not fit	Ex 9

---

Ex 1 — Cell A: a clean full-rank fit

Forecast: guess — is the slope positive, and is it bigger or smaller than $1$? (The dots go generally up.)

1. Build $\mathbf{X}$. Leading column of $1$'s, then the $x$ column: $\mathbf{X}=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\\ 1& 4\end{array}\right],\mathbf{y}=\left[\begin{array}{c}2\\ 3\\ 5\\ 4\end{array}\right].$ Why this step? The $1$-column is how the intercept enters the algebra — it multiplies ${\beta }_0$ in every row.

2. Form ${\mathbf{X}}^{\top }\mathbf{X}$ and ${\mathbf{X}}^{\top }\mathbf{y}$ — and see where every entry comes from. ${\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}n& \sum x_i\\ \sum x_i& \sum x_i^2\end{array}\right]=\left[\begin{array}{cc}4& 10\\ 10& 30\end{array}\right].$ Here $n=4$ (four rows), $\sum x_i=1+2+3+4=10$, and $\sum x_i^2=1+4+9+16=30$. ${\mathbf{X}}^{\top }\mathbf{y}=\left[\begin{array}{c}\sum y_i\\ \sum x_i{y}_i\end{array}\right]=\left[\begin{array}{c}2+3+5+4\\ (1)(2)+(2)(3)+(3)(5)+(4)(4)\end{array}\right]=\left[\begin{array}{c}14\\ 39\end{array}\right].$ So $14=\sum y_i$ and $\sum x_i{y}_i=2+6+15+16=39$. Why this step? Multiplying ${\mathbf{X}}^{\top }$ (a matrix whose first row is all $1$'s and second row is the $x$'s) by a column literally adds things up: a row of $1$'s times a column is a plain sum; the $x$-row times a column weights each entry by $x_i$. Those are exactly the sums simple regression uses.

3. Solve the $2\times 2$ system. For a $2\times 2$ matrix $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$ the inverse is $\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$. Here $ad-bc=4\cdot 30-10\cdot 10=120-100=20$. $\hat{\mathbf{\beta }}=\frac{1}{20}\left[\begin{array}{cc}30& -10\\ -10& 4\end{array}\right]\left[\begin{array}{c}14\\ 39\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}30(14)-10(39)\\ -10(14)+4(39)\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}420-390\\ -140+156\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}30\\ 16\end{array}\right]=\left[\begin{array}{c}1.5\\ 0.8\end{array}\right].$ So ${\hat{\beta }}_0=1.5,{\hat{\beta }}_1=0.8$.

Why flag the determinant? $ad-bc=20\ne 0$, so the inverse genuinely exists (Matrix Inverse) — the machinery is sound. The safe habit is always to survive a plug-back check (Ex 2).

Verify: the VERIFY block solves the exact system symbolically: the correct line is $\hat{y}=1.5+0.8x$, giving $\hat{y}=(2.3,3.1,3.9,4.7)$ with residuals summing to $0$.

The figure below plots these four points, draws the fit line $1.5+0.8x$ in yellow, and marks each residual (the pink vertical stick from a dot to the line). Look at the pink sticks: they are the vertical gaps least squares makes as small as possible, and their signed lengths add to zero.

---

Ex 2 — Cell B: the machine must reduce to simple regression

Forecast: the honest slope should be less than 1 (the dots rise but not on a perfect $45^{\circ }$ line). Guess: around $0.8$?

1. Centre the data. $\bar{x}=2.5,\bar{y}=3.5$. Deviations $x_i-\bar{x}=(-1.5,-0.5,0.5,1.5)$, $y_i-\bar{y}=(-1.5,-0.5,1.5,0.5)$. Why this step? Centring removes the intercept from the slope calculation — it isolates the "tilt". The simple-regression slope formula is the single-predictor special case of $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$, so if the two disagree we have made an arithmetic slip.

2. Numerator (covariation) $=(-1.5)(-1.5)+(-0.5)(-0.5)+(0.5)(1.5)+(1.5)(0.5)=2.25+0.25+0.75+0.75=4$.

3. Denominator (spread of $x$) $=1.5^2+0.5^2+0.5^2+1.5^2=2.25+0.25+0.25+2.25=5$.

4. Slope ${\hat{\beta }}_1=4/5=0.8$, and intercept ${\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}=3.5-0.8(2.5)=3.5-2=1.5$.

Why this matches Ex 1: two independent routes — the matrix inverse in Ex 1 and the centred slope formula here — both land on ${\hat{\beta }}_0=1.5,{\hat{\beta }}_1=0.8$. That agreement is the strongest sanity check we have.

Verify: with $\hat{y}=1.5+0.8x$, predictions are $(2.3,3.1,3.9,4.7)$, residuals $(-0.3,-0.1,1.1,-0.7)$ which sum to $0$. ✓

---

Ex 3 — Cell C: a genuinely negative slope

Forecast: more maintenance → fewer defects, so the slope should be negative. Guess its size.

1. Sums. $n=4,\sum x=6,\sum x^2=14,\sum y=24,\sum xy=0\cdot 9+1\cdot 7+2\cdot 5+3\cdot 3=26$. Why this step? These four sums are all ${\mathbf{X}}^{\top }\mathbf{X}$ and ${\mathbf{X}}^{\top }\mathbf{y}$ need — exactly the pattern $[n,\sum x;\sum x,\sum x^2]$ and $[\sum y;\sum xy]$ we built in Ex 1 Step 2.

2. Normal equations. $\left[\begin{array}{cc}4& 6\\ 6& 14\end{array}\right]\hat{\mathbf{\beta }}=\left[\begin{array}{c}24\\ 26\end{array}\right].$

3. Determinant $=4\cdot 14-6\cdot 6=56-36=20\ne 0$, so $\hat{\mathbf{\beta }}=\frac{1}{20}\left[\begin{array}{cc}14& -6\\ -6& 4\end{array}\right]\left[\begin{array}{c}24\\ 26\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}14(24)-6(26)\\ -6(24)+4(26)\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}336-156\\ -144+104\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}180\\ -40\end{array}\right]=\left[\begin{array}{c}9\\ -2\end{array}\right].$ So $\hat{y}=9-2x$.

Why negative works fine: least squares never demands positive slopes; the sign just follows the data's tilt.

Verify: $\hat{y}=(9,7,5,3)$ exactly matches $y$ — the points are perfectly collinear, so all residuals are $0$ and $R^2=1$. Units: defects per (maintenance hour) $=-2$, meaning 2 fewer defects per extra hour. ✓

---

Ex 4 — Cell D: two predictors, predict and interpret

Forecast: plugging in is a weighted sum; the sleep effect per hour is just ${\hat{\beta }}_2$.

1. (a) Substitute. $\hat{y}=5+8(3)+3(7)=5+24+21=50$. Why this step? The model is linear, so prediction is literally the dot product of the coefficient vector with $(1,x_1,x_2)$.

2. (b) Partial slope. Change $x_2$ by $+1$, hold $x_1$ fixed: $\Delta \hat{y}={\hat{\beta }}_2\cdot 1=3$. Why "hold $x_1$ fixed": ${\hat{\beta }}_2$ is the partial slope — sleep's unique contribution after study is accounted for (Orthogonal Projection is why this "after" is clean).

Verify: recompute (a) with a small nudge: at $x_1=4,x_2=7$, $\hat{y}=5+32+21=58$, which is $50+8$ — consistent with ${\hat{\beta }}_1=8$ per study hour. Units check: points, and points-per-hour. ✓

---

Ex 5 — Cell E: perfect collinearity, the inverse dies

Forecast: two columns carrying the same information — the machine should choke.

1. Look at the columns. The $x_2$ column is exactly $100\times$ the $x_1$ column. One is a copy of the other (scaled). Why this matters: the column space of $\mathbf{X}$ is "flat" in the $x_1$–$x_2$ plane — the predictors do not span a genuine 2D direction.

2. Compute the determinant explicitly. Take a tiny two-row example with just the two length columns (drop the intercept for a clean $2\times 2$ to make the algebra visible): $x_1=(1,2),x_2=(100,200)$, $\mathbf{X}=\left[\begin{array}{cc}1& 100\\ 2& 200\end{array}\right],{\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}{1}^2+2^2& 1\cdot 100+2\cdot 200\\ 100\cdot 1+200\cdot 2& 100^2+200^2\end{array}\right]=\left[\begin{array}{cc}5& 500\\ 500& 50000\end{array}\right].$ Now the determinant, entry by entry: $\det =(5)(50000)-(500)(500)=250000-250000=0.$ So $ad-bc=0$ exactly — the two products cancel because the second column is $100\times$ the first, which forces $b=100a$ and $d=100c$ patterns that make $ad$ and $bc$ equal.

3. No inverse. Since the determinant is $0$, the formula $\frac{1}{ad-bc}[\dots ]$ divides by zero: $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ does not exist. Infinitely many $\hat{\mathbf{\beta }}$ minimise $S$ equally — the plane can tilt freely in the redundant direction at zero cost. Why: dividing by a "flat" matrix is like dividing by zero — see Multicollinearity & VIF and Positive Semidefinite Matrices (${\mathbf{X}}^{\top }\mathbf{X}$ is only semidefinite here, not positive definite).

Verify: the VERIFY block confirms $\det ({\mathbf{X}}^{\top }\mathbf{X})=0$ for the full three-column version too. Fix: drop one of the two length columns; the remaining fit is unique. ✓

The figure below draws the two predictor columns as arrows from the origin. Because $x_2=100x_1$, the pink arrow lies exactly on top of the blue one's direction (only its length differs). Two arrows pointing the same way cannot frame a 2-D patch of area — and zero area is a zero determinant.

---

Ex 6 — Cell F: near-collinearity, coefficients explode

Forecast: the inverse exists (determinant tiny but nonzero) yet the answer is unstable.

1. Why unstable: $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ divides by a near-zero determinant, magnifying tiny data changes into huge coefficient swings. Why this step? This is the practical face of Multicollinearity & VIF.

2. Concrete blow-up. Solve the least-squares fit for ${\hat{\beta }}_1,{\hat{\beta }}_2$ once, then change the last outcome from $y_4=10$ to $y_4=11$ and solve again. Both slopes shift substantially from a mere $1$-unit data nudge — the exact rational before/after values, and the size of the swing, are computed in the VERIFY block. Why this happens: $x_1$ and $x_2$ differ only in the last entry, so the fit must lean heavily on that one distinguishing coordinate; a tiny wobble there whipsaws both slopes.

3. The VIF summary. Regress $x_2$ on $x_1$; the near-perfect fit gives $R_{x_2\mid x_1}^2$ close to $1$, so the variance inflation factor $\text{VIF}=\frac{1}{1-R_{x_2\mid x_1}^2}$ is huge. At $R_{x_2\mid x_1}^2=0.9999$ this equals $1/0.0001=10000$; standard errors scale with $\sqrt{\text{VIF}}\approx 100$. High VIF is the early-warning gauge for exactly the whipsaw in Step 2.

Verify: VERIFY confirms (i) $\det ({\mathbf{X}}^{\top }\mathbf{X})$ is small-but-positive here (inverse exists, unlike Ex 5); (ii) the two coefficient vectors differ noticeably after the $1$-unit nudge; and (iii) $1/(1-0.9999)=10000$. ✓

---

Ex 7 — Cell G: $R^2$, adjusted $R^2$, and the useless predictor limit

Forecast: $R^2$ always rises when you add a variable; adjusted $R^2$ should fall for junk.

1. (a) $R^2$. $R^2=1-SSE/SST=1-20/100=0.8$. Why this step? $R^2$ is the fraction of variance explained: the leftover error $SSE$ as a share of the total wobble $SST$, subtracted from $1$.

2. Adjusted $R^2$ with $n=10,p=2$: $R_{adj}^2=1-(1-R^2)\frac{n-1}{n-p-1}=1-0.2\cdot \frac{9}{7}=1-0.2571\dots =\mathrm{0.7429.}$ Why this step? Dividing by degrees of freedom $n-p-1=7$ penalises each extra predictor.

3. (b) Add junk. Now $p=3$, $SSE=19.9$, so $R^2=1-19.9/100=0.801$ (rose slightly, as always). But $R_{adj}^2=1-\frac{19.9/(10-3-1)}{100/(10-1)}=1-\frac{19.9/6}{100/9}=1-\frac{3.3167}{11.111}=1-0.2985=\mathrm{0.7015.}$ Adjusted $R^2$ fell from $0.7429$ to $0.7015$.

Why: the useless predictor cost a degree of freedom ($n-p-1$ dropped $7\to 6$) but barely reduced error — a bad trade.

Verify: VERIFY recomputes all four numbers ($0.8,0.7429,0.801,0.7015$) and confirms the adjusted value dropped. ✓

---

Ex 8 — Cell H: a real-world word problem

Forecast: warmer days sell more (positive ${\beta }_1$), weekends sell more (positive ${\beta }_2$).

1. Set up three equations. $\{\begin{array}{l}{\beta }_0+20{\beta }_1+0{\beta }_2=100\\ {\beta }_0+30{\beta }_1+0{\beta }_2=140\\ {\beta }_0+20{\beta }_1+1{\beta }_2=160\end{array}$ Why this step? With exactly as many points as unknowns and independent rows, the plane passes through all three — no least-squares averaging needed.

2. Subtract eqn 1 from eqn 2: $10{\beta }_1=40\Rightarrow {\beta }_1=4$ ($4 revenue per °C).

3. Subtract eqn 1 from eqn 3: ${\beta }_2=60$ ($60 extra on a weekend).

4. Back-substitute into eqn 1: ${\beta }_0+80=100\Rightarrow {\beta }_0=20$.

Verify: plug $(30,0)$: $20+4(30)+0=140$ ✓; plug $(20,1)$: $20+80+60=160$ ✓. Units: dollars, dollars-per-degree, dollars-per-weekend. Interpretation of ${\beta }_2=60$: the weekend bump holding temperature fixed — a clean partial effect. ✓

---

Ex 9 — Cell I: the exam twist (centring changes coefficients, not fit)

Forecast: the slope should stay $-2$ (steepness is unaffected by shifting the axis); the intercept must change so the line still passes through the same points.

1. Slope is invariant to shifting $x$. Replacing $x$ by $x-c$ leaves $\sum (x_i-\bar{x}{)}^2$ and the covariation unchanged, so ${\hat{\beta }}_1^{\prime }=-2$. Why this step? Centring slides the vertical axis left/right; it does not tilt the line, and the slope formula depends only on deviations from the mean, which shifting cannot alter.

2. New intercept = fitted value at $x^{\prime }=0$, i.e. at $x=\bar{x}=1.5$: $\hat{y}=9-2(1.5)=6$. So ${\hat{\beta }}_0^{\prime }=6$. Why: when $x^{\prime }=0$ we sit at the mean of $x$; a least-squares line always passes through the point $(\bar{x},\bar{y})$, and $\bar{y}=(9+7+5+3)/4=6$. Hence the centred intercept equals $\bar{y}$.

3. Fit unchanged. Predictions $\hat{y}=6-2x^{\prime }=6-2(x-1.5)=6-2x+3=9-2x$ — algebraically identical to Ex 3. So $R^2$, every residual, and every prediction are exactly the same; only the labels ${\hat{\beta }}_0^{\prime }$ vs ${\hat{\beta }}_0$ moved.

Verify: VERIFY confirms ${\hat{\beta }}_0^{\prime }=6,{\hat{\beta }}_1^{\prime }=-2$ and that the centred predictions equal the original $(9,7,5,3)$. Lesson: rescaling/centring predictors reshuffles coefficients but never the quality of fit — beware exam questions that hide this behind a scary-looking coefficient change.

---

Recall Which cell was hardest for you?

Perfect collinearity (Ex 5) fails ::: because ${\mathbf{X}}^{\top }\mathbf{X}$ is singular ($\det =0$), so $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ does not exist and $\hat{\mathbf{\beta }}$ is not unique. Adjusted $R^2$ falls for a useless predictor ::: because $n-p-1$ shrinks (lost degree of freedom) while $SSE$ barely drops — a bad trade. Centring $x$ changes the intercept to ::: $\bar{y}$ (the fitted value at the mean of $x$), while the slope and the fit are unchanged.