Exercises — Multiple regression

(a) The intercept is the constant term: ${\hat{\beta }}_0=12$ (predicted $y$ when both predictors are $0$). (b) The partial slope of $x_1$ is its multiplier: ${\hat{\beta }}_1=-4$. It means $y$ drops by 4 for each 1-unit rise in $x_1$, holding $x_2$ fixed. (c) Plug in — the model is a weighted sum: $\hat{y}=12-4(2)+0.5(10)=12-8+5=9.$

• $\mathbf{X}$ has one row per observation and $p+1=3$ columns (the 1-column + $x_1$ + $x_2$): $4\times 3$.
• ${\mathbf{X}}^{\top }\mathbf{X}$ is $(3\times 4)(4\times 3)=3\times 3$ (square, one row/col per coefficient).
• ${\mathbf{X}}^{\top }\mathbf{y}$ is $(3\times 4)(4\times 1)=3\times 1$ (a column, matching $\hat{\mathbf{\beta }}$).

Why square? Being square is a necessary condition to even talk about an inverse — but it is not sufficient. ${\mathbf{X}}^{\top }\mathbf{X}$ is also invertible only when its determinant is nonzero, which happens exactly when $\mathbf{X}$ has full column rank (linearly independent columns, proved in Ex 5.1). A square-but-rank-deficient matrix has $\det =0$ and no inverse (Matrix Inverse).

Step 1 — build $\mathbf{X}$ (leading 1-column then the $x$ column): $\mathbf{X}=\left[\begin{array}{cc}1& 0\\ 1& 1\\ 1& 2\end{array}\right],\mathbf{y}=\left[\begin{array}{c}1\\ 3\\ 4\end{array}\right].$ Step 2 — form ${\mathbf{X}}^{\top }\mathbf{X}$. Row-of-ones $\cdot$ itself $=n=3$; ones $\cdot$ $x=\sum x=3$; $x\cdot x=\sum x^2=0+1+4=5$: ${\mathbf{X}}^{\top }\mathbf{X}=\left[\begin{array}{cc}3& 3\\ 3& 5\end{array}\right].$ Step 3 — form ${\mathbf{X}}^{\top }\mathbf{y}$. $\sum y=8$; $\sum x_i{y}_i=0+3+8=11$: ${\mathbf{X}}^{\top }\mathbf{y}=\left[\begin{array}{c}8\\ 11\end{array}\right].$ Step 4 — invert the $2\times 2$. Why this formula works: for a $2\times 2$ matrix $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$, the inverse is $\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]$. The scalar $ad-bc=\det$ measures the (signed) area the matrix's columns span; dividing by it undoes the scaling. The rearranged matrix (swap the diagonal, negate the off-diagonal) is the adjugate — you can check $\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right]=(ad-bc)I$ by direct multiplication, which is why dividing by $\det$ then gives $I$. Here $\det =3\cdot 5-3\cdot 3=6\ne 0$ (so the inverse exists): $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right].$ Step 5 — solve: $\hat{\mathbf{\beta }}=\frac{1}{6}\left[\begin{array}{cc}5& -3\\ -3& 3\end{array}\right]\left[\begin{array}{c}8\\ 11\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}40-33\\ -24+33\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}7\\ 9\end{array}\right]=\left[\begin{array}{c}7/6\\ 3/2\end{array}\right].$ So ${\hat{\beta }}_0=\frac{7}{6}\approx 1.167$ and ${\hat{\beta }}_1=\frac{3}{2}=1.5$. (This is exactly Ordinary Least Squares on one predictor — same machinery, matrix-shaped.)

One hour $=60$ minutes, so the minutes value is $60\times$ the hours value: $x_1^{\text{min}}=60x_1$. To keep every prediction identical, the coefficient must shrink by the same factor: $8x_1=8\left(\frac{x_1^{\text{min}}}{60}\right)=\frac{8}{60}{x}_1^{\text{min}}\approx 0.1333x_1^{\text{min}}.$ New coefficient $=\frac{8}{60}=\frac{2}{15}\approx 0.133$. The fit is unchanged — only the number relabelling the same slope changed.

Original ($p=3$): $R^2=1-\frac{SSE}{SST}=1-\frac{100}{400}=\mathrm{0.75.}$ $R_{adj}^2=1-(1-R^2)\frac{n-1}{n-p-1}=1-0.25\cdot \frac{19}{16}=1-0.296875=\mathrm{0.703125.}$ After adding the junk predictor ($p=4$, $SSE=99$): $R_{\text{new}}^2=1-\frac{99}{400}=0.7525(\text{went }\text{up}\text{, as it always does}).$ $R_{adj,\text{new}}^2=1-(1-0.7525)\frac{19}{15}=1-0.2475\cdot 1.26\overline{6}\approx \mathrm{0.6865.}$ What happened: plain $R^2$ rose ($0.75\to 0.7525$) even though the predictor is garbage — R² can never fall when you add a column. But adjusted $R^2$ fell ($0.7031\to 0.6865$) because the denominator $n-p-1$ shrank from $16$ to $15$, penalising the wasted degree of freedom. Adjusted $R^2$ correctly flags the junk.

When $x_1$ and $x_2$ are nearly parallel (left panel of the figure), the columns of $\mathbf{X}$ almost lie on one line, so ${\mathbf{X}}^{\top }\mathbf{X}$ is near-singular: its determinant is close to $0$. The inverse is computed as $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}=\frac{1}{\det ({\mathbf{X}}^{\top }\mathbf{X})}\mathrm{adj}({\mathbf{X}}^{\top }\mathbf{X})$, where $\mathrm{adj}$ is the adjugate (the matrix of cofactors, transposed). The cofactor entries stay of ordinary size, but they are all divided by a determinant that is heading to $0$ — so the entries of the inverse blow up. It is not a clean scalar proportionality; the point is that the shared factor $1/\det$ magnifies every entry. Hence tiny changes in the data produce huge swings in $\hat{\mathbf{\beta }}$. Geometrically (right panel), tilting the fitted plane steeply "up" along $x_1$ and equally steeply "down" along $x_2$ barely moves the surface where the data sit, so the data can't pin the two slopes down: many wildly different $({\hat{\beta }}_1,{\hat{\beta }}_2)$ fit almost equally well.

This is multicollinearity. It is quantified by the VIF (variance inflation factor): for predictor $j$, ${\text{VIF}}_j=\frac{1}{1-R_j^2},$ where $R_j^2$ is the $R^2$ obtained by regressing $x_j$ on all the other predictors. If $x_1$ and $x_2$ are nearly parallel, each explains the other almost perfectly, so $R_j^2\to 1$ and ${\text{VIF}}_j\to \infty$ — the factor by which the variance of ${\hat{\beta }}_j$ is inflated relative to the no-collinearity case. A common rule of thumb flags $\text{VIF}>10$.

Orthogonality. Start from the normal equations ${\mathbf{X}}^{\top }\mathbf{X}\hat{\mathbf{\beta }}={\mathbf{X}}^{\top }\mathbf{y}$. Move everything to one side: ${\mathbf{X}}^{\top }\mathbf{y}-{\mathbf{X}}^{\top }\mathbf{X}\hat{\mathbf{\beta }}={\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})={\mathbf{X}}^{\top }\mathbf{e}=0.$ ${\mathbf{X}}^{\top }\mathbf{e}=0$ literally says the dot product of each column of $\mathbf{X}$ with $\mathbf{e}$ is zero — i.e. $\mathbf{e}$ is perpendicular to every predictor direction. That is orthogonal projection: $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ is the shadow of $\mathbf{y}$ on the column space, $\mathbf{e}$ is the perpendicular drop (Figure s02, red arrow). Residuals sum to zero. With an intercept, $\mathbf{X}$'s first column is all ones, $1=(1,\dots ,1{)}^{\top }$. Its row of ${\mathbf{X}}^{\top }\mathbf{e}=0$ reads $1^{\top }\mathbf{e}={\sum }_{i=1}^n{e}_i=0.$ So the positive and negative vertical gaps exactly cancel — a direct consequence of including the 1-column.

Center everything by subtracting $\bar{y}$. Write:

• total vector $\mathbf{y}-\bar{y}1$ (length${}^2$ = $SST$),
• explained vector $\hat{\mathbf{y}}-\bar{y}1$ (length${}^2$ = $SSR$),
• residual vector $\mathbf{y}-\hat{\mathbf{y}}=\mathbf{e}$ (length${}^2$ = $SSE$). These add: $(\mathbf{y}-\bar{y}1)=(\hat{\mathbf{y}}-\bar{y}1)+\mathbf{e}$. Take squared length: $\Vert \mathbf{y}-\bar{y}1{\Vert }^2=\Vert \hat{\mathbf{y}}-\bar{y}1{\Vert }^2+\Vert \mathbf{e}{\Vert }^2+2(\hat{\mathbf{y}}-\bar{y}1{)}^{\top }\mathbf{e}.$ The cross term vanishes: $\hat{\mathbf{y}}-\bar{y}1$ lives in the column space of $\mathbf{X}$ (both $\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ and $\bar{y}1$ are combinations of $\mathbf{X}$'s columns), and $\mathbf{e}\perp$ that whole space by Ex 4.1. So $2(\cdots {)}^{\top }\mathbf{e}=0$, giving $\underset{SST}{\underbrace{\Vert \mathbf{y}-\bar{y}1{\Vert }^2}}=\underset{SSR}{\underbrace{\Vert \hat{\mathbf{y}}-\bar{y}1{\Vert }^2}}+\underset{SSE}{\underbrace{\Vert \mathbf{e}{\Vert }^2}}.$ This is exactly $a^2+b^2=c^2$ for a right triangle whose right angle is at $\hat{\mathbf{y}}$ — the same right angle from Figure s02.

PSD. For any vector $\mathbf{v}$, ${\mathbf{v}}^{\top }({\mathbf{X}}^{\top }\mathbf{X})\mathbf{v}=(\mathbf{Xv}{)}^{\top }(\mathbf{Xv})=\Vert \mathbf{Xv}{\Vert }^2\ge 0.$ A squared length is never negative, so ${\mathbf{X}}^{\top }\mathbf{X}$ is positive semidefinite. Always. Positive definite iff full column rank. $\Vert \mathbf{Xv}{\Vert }^2=0$ forces $\mathbf{Xv}=0$.

• If columns are linearly independent, the only $\mathbf{v}$ with $\mathbf{Xv}=0$ is $\mathbf{v}=0$. So ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}>0$ for all $\mathbf{v}\ne 0$ → positive definite → invertible → unique $\hat{\mathbf{\beta }}$.
• If columns are dependent (perfect collinearity), some $\mathbf{v}\ne 0$ gives $\mathbf{Xv}=0$, so ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}=0$ with $\mathbf{v}\ne 0$ → only semidefinite, $\det =0$, not invertible. The formula $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ doesn't exist; infinitely many $\hat{\mathbf{\beta }}$ minimise $S$.

The objective is $S(\mathbf{\beta })=\Vert \mathbf{y}-\mathbf{X}\mathbf{\beta }{\Vert }^2$, a quadratic whose Hessian (matrix of second derivatives) is $2{\mathbf{X}}^{\top }\mathbf{X}$. By Ex 5.1 this is positive definite when columns are independent. A quadratic with a positive-definite Hessian is strictly convex: it curves upward in every direction, so it has exactly one lowest point and no other stationary points. That single point is where the gradient $-2{\mathbf{X}}^{\top }\mathbf{y}+2{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }$ vanishes — i.e. the normal equations — giving the unique $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$. This is exactly what the Gauss–Markov Theorem needs to even speak of "the" estimator. Compare with Simple Linear Regression, where the same convexity gives the one best line.

One-line answers before you leave. Ex 2.1 gives ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ equal to what? ::: $7/6$ and $3/2$. In Ex 3.1, why does adjusted $R^2$ fall when plain $R^2$ rises? ::: The degrees-of-freedom penalty $n-p-1$ shrinks faster than $SSE$ drops for a useless predictor. What single equation proves residual $\perp$ every column of $\mathbf{X}$? ::: ${\mathbf{X}}^{\top }\mathbf{e}=0$ (the normal equations rearranged). When is ${\mathbf{X}}^{\top }\mathbf{X}$ positive definite (not just PSD)? ::: Exactly when $\mathbf{X}$ has linearly independent columns (full column rank).