Question bank — Multiple regression

False. It is the effect of $x_j$ holding the other predictors fixed — its partial (unique) contribution, not its isolated effect.

True — but note the condition is sample orthogonality (${\mathbf{x}}_1^{\top }{\mathbf{x}}_2=0$ in the actual design matrix), not merely population "uncorrelated." Only then do the slopes decouple so the estimate for $x_1$ is identical with or without $x_2$; finite-sample correlation between columns generally breaks this.

False. Coefficients carry the units of their predictor; rescale hours to minutes and the coefficient shrinks $60\times$ with no change in real importance. Compare standardised coefficients or t-statistics.

For $R_{\text{in}}^2$ (in-sample, with intercept), no — it is trapped in $[0,1]$ by the Pythagorean split. But $R_{\text{oos}}^2$ on held-out data (or a no-intercept model) can have $SSE>SST$, giving a negative value: the model predicts worse than the flat mean line.

False for $R_{\text{in}}^2$ (in-sample, with intercept): it never decreases when you add a column, since the fit can only stay equal or improve. (On held-out data, or in no-intercept models, the out-of-sample version can fall.) That is exactly why we need ==adjusted $R^2$==, which can fall in-sample.

True — that is the geometric meaning of the normal equations ${\mathbf{X}}^{\top }(\mathbf{y}-\mathbf{X}\hat{\mathbf{\beta }})=0$: least squares drops a perpendicular onto the column space. See Orthogonal Projection.

False. The estimator $\hat{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ needs no normality. Normality is only assumed later for exact t-tests and confidence intervals. The Gauss–Markov optimality needs only zero-mean, constant-variance, uncorrelated errors.

True. Invertibility means the predictors are not perfectly collinear, so the columns are independent, the projection has one representation, and $\hat{\mathbf{\beta }}$ is unique.

True. For any vector $\mathbf{v}$, ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}=\Vert \mathbf{Xv}{\Vert }^2\ge 0$, which is the definition of PSD. It is strictly positive definite (hence invertible) exactly when $\mathbf{X}$ has full column rank.

False. Under Gauss–Markov the OLS estimates stay unbiased; multicollinearity inflates their variance, making them unstable and hard to trust — not systematically wrong on average.

Wrong grouping: it must be $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$. $\mathbf{X}{\mathbf{X}}^{\top }$ is $n\times n$ and generally singular; ${\mathbf{X}}^{\top }\mathbf{X}$ is the small $(p+1)\times (p+1)$ matrix that is actually invertible. See Matrix Inverse.

That forces the hyperplane through the origin (${\beta }_0=0$), i.e. it must predict $y=0$ when all $x=0$ — almost always wrong and it corrupts every other slope.

Perfect correlation makes a column a linear combination of others, so ${\mathbf{X}}^{\top }\mathbf{X}$ is singular (determinant zero) and has no inverse; no unique solution exists until you drop or combine a predictor.

A high $R^2$ measures association/fit, not causation. Causal claims need experimental design or explicit causal assumptions, not a good fit.

The plain sum of residuals is zero by construction (when an intercept is present) and lets positives cancel negatives. We minimise the sum of squared residuals, which cannot cancel.

In-sample $R_{\text{in}}^2$ always rises (or ties) when you add predictors, so this comparison is empty. Compare adjusted $R^2$, or better, out-of-sample error via cross-validation.

A negative partial slope means: after accounting for the other predictors, more sleep is associated with lower $y$ in this sample. Sign flips are a classic symptom of multicollinearity, not corrupt data — check the VIF.

Squares are smooth (differentiable everywhere), penalise large misses harder, and yield a unique closed-form minimum via the normal equations. Absolute values give a non-smooth problem with no simple formula.

Because $S(\mathbf{\beta })$ is a quadratic with Hessian $2{\mathbf{X}}^{\top }\mathbf{X}$, which is PSD, so any stationary point is a global minimum. When $\mathbf{X}$ has full column rank the Hessian is positive definite (strictly convex), making that minimiser the unique one; if rank is deficient, minima form a flat set.

A simple regression slope absorbs both $x_j$'s direct effect and whatever it correlates with; the multiple-regression slope removes the part explained by the other predictors, isolating $x_j$'s unique contribution.

In every row that 1 multiplies ${\beta }_0$, so ${\beta }_0$ is added to every prediction — it is the vertical offset of the hyperplane, present regardless of the $x$ values.

Adjusted $R^2$ divides each sum of squares by its degrees of freedom; a useless predictor costs a degree of freedom, inflating $SSE/(n-p-1)$ and lowering the adjusted value even as raw $R^2$ nudges up.

Near-collinearity makes ${\mathbf{X}}^{\top }\mathbf{X}$ nearly singular, so $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$ has huge entries; since $\text{Var}(\hat{\mathbf{\beta }})={\sigma }^2({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}$, those large entries directly inflate coefficient variances.

$\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}$ is the closest point to $\mathbf{y}$ inside the column space of $\mathbf{X}$; "closest" in Euclidean distance means dropping a perpendicular, which is precisely orthogonal projection.

The Pythagorean argument needs the mean vector $\bar{y}1$ to lie in the column space; the 1's column is what guarantees this. Drop it and the residual is no longer orthogonal to $\bar{y}1$, so the cross-term does not vanish and the three sums stop adding up.

With full rank the hyperplane passes through every point, $SSE=0$ and $R^2=1$ — but this is pure interpolation with zero degrees of freedom for error; it tells you nothing about generalisation.

Then $R^2=1-SSE/SST$ divides by zero — $R^2$ is undefined (degenerate). Geometrically there is no variation to explain: the best fit is the flat line $\hat{y}=\bar{y}$ with $SSE=0$ too, so the ratio is $0/0$. Report that the response has no variance rather than a meaningless $R^2$.

Its column is proportional to the 1's column, creating perfect collinearity; ${\mathbf{X}}^{\top }\mathbf{X}$ becomes singular and the intercept/slope split is undefined until you remove the constant column.

The fitted values equal the mean $\bar{y}$ for every point — the best hyperplane is flat, so no predictor explains any variation and $SSR=0$.

They coincide: with orthogonal columns each slope is computed independently, so multiple regression gives exactly the same coefficients as running each simple regression alone.

$\mathbf{X}$ cannot have full column rank, ${\mathbf{X}}^{\top }\mathbf{X}$ is singular, and infinitely many $\hat{\mathbf{\beta }}$ achieve $SSE=0$; OLS has no unique solution and you need regularisation or fewer predictors.

Yes — exactly when the predictors are collinear, so the column space is spanned in more than one way; the projection $\hat{\mathbf{y}}$ is unique but its coordinates $\hat{\mathbf{\beta }}$ are not.

Multiple regression reduces to simple linear regression: the hyperplane collapses to a straight line and $({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}$ gives the familiar slope-and-intercept pair.