Multiple linear regression: Question bank — AI-ML deep dive

This deck attacks the ideas behind multiple linear regression, not the arithmetic. Read the prompt, say your answer out loud with a reason, then reveal. If your answer was "yes/no" with no because, count it wrong — every trap here is about the why.

Symbols and pictures this deck assumes

Before the traps, let us pin down every symbol and give each key idea one picture, so the deck is self-contained. Nothing below is used later without appearing here first.

Definition The vocabulary (defined once, used everywhere)

$m$ = number of training examples (rows of data). $n$ = number of features (columns like size, bedrooms). One extra "feature" column of $1$ 's carries the intercept, so $β$ has $n + 1$ entries.
$X$ = the design matrix: one row per example, one column per feature, plus that leading column of $1$ 's. So $X$ is $m \times (n + 1)$ , and the whole model's predictions are the single product $\hat{y} = X β$ .
$β$ = the column of $n + 1$ coefficients (intercept $β_{0}$ first, then one weight per feature).
$\overset{y}{^}$ = the model's prediction; $y$ = the true target; $y$ = the whole column of true targets.
$λ$ = the ridge penalty strength (a single non-negative number you choose). It appears in Ridge Regression as the amount of $λ I$ added to $X^{⊤} X$ : bigger $λ$ means "shrink the coefficients harder," $λ = 0$ recovers ordinary least squares.
MSE = Mean Squared Error, the average of the squared misses. In formula form: $MSE = J (β) = \frac{1}{m} \sum_{i = 1}^{m} (y^{(i)} - \overset{y}{^}^{(i)})^{2} .$ The single letter $J$ is just a shorthand name for this MSE-as-a-function-of- $β$ ; every " $\partial J / \partial β_{j}$ " later means "how the average squared miss changes as we nudge weight $β_{j}$ ."

True or false — justify

True or false: the "linear" in multiple linear regression means the relationship between features and target must be a straight line.

False. "Linear" means linear in the coefficients

β

— the model

\overset{y}{^} = β_{0} + β_{1} x_{1} + \dots

is a weighted sum of the

β

's. You can feed it

x^{2}

lo g x

as features (that's Polynomial Regression) and it is still a linear model.

True or false: adding more features to the model can never increase the training MSE.

True. Extra features can only be given coefficient

0

, which reproduces the smaller model, so the best fit never gets worse on training data. This is exactly why training error is a bad guide to feature choice — it invites Overfitting.

True or false: if

X^{⊤} X

is invertible, the Normal Equation gives the unique global minimum of the MSE.

True. The Hessian of

J

\frac{2}{m} X^{⊤} X

; when

X^{⊤} X

is invertible it is positive definite, so the cost bowl curves up in every direction (strictly convex) and its single stationary point is the global minimum.

True or false: scaling one feature from metres to centimetres changes the model's predictions.

False. The coefficient simply rescales by the inverse factor (

β

shrinks by

100

x

grows by

100

), so

β x

and therefore

\overset{y}{^}

are unchanged. Scaling matters for Gradient Descent speed (it makes the bowl round rather than a stretched ellipse) and for Ridge Regression penalties, not for the fitted predictions of ordinary least squares.

True or false: a coefficient

β_{j}

tells you the effect of feature

j

in isolation.

False. It tells you the effect of

x_{j}

holding all other features fixed. Because features are correlated in real data, that "all else equal" world may never actually occur, so the coefficient is a conditional effect, not a marginal one.

True or false: if two features are perfectly correlated, the model is useless.

False for prediction, true for interpretation. Perfect collinearity makes

X^{⊤} X

singular (the flat-floored trench above) so individual coefficients are unidentifiable, but the fitted plane and its predictions can still be fine (a pseudoinverse or Ridge Regression recovers a usable

β

True or false: a negative intercept

β_{0}

signals a bug in the fit.

False.

β_{0}

is only "the prediction when all features are zero." If all-zero is far outside your data (e.g. a house of

0

sq ft), that point is a meaningless extrapolation and its sign carries no physical meaning.

True or false: a high

R^{2}

guarantees the model will predict well on new data.

False.

R^{2}

measures fit on the data you already have; it never decreases as you add features. Good out-of-sample behaviour needs a validation set, and adjusted $R^2$ partially penalises the free-parameter inflation.

Spot the error

Spot the error: "We have

5

examples and

8

features, so we'll just use the Normal Equation

(X^{⊤} X)^{- 1} X^{⊤} y

With

m = 5

examples fewer than

n = 8

features,

X^{⊤} X

(size

9 \times 9

with the bias column) has rank at most

5

, so it is singular and cannot be inverted. Use regularization, a pseudoinverse, or drop features.

Spot the error: "MSE has a minimum, so I take

\partial J / \partial β_{j}

and set it to zero — but I only did this for

β_{1}

, and got the answer."

Recall

J

is the MSE. The minimum requires the gradient to be zero in every direction at once. Setting only one partial to zero solves for one coefficient given the others; you must solve the whole system

X^{⊤} X β = X^{⊤} y

simultaneously.

Spot the error: "

X β

where

X

has no leading column of ones — the intercept is inside

β

anyway."

Without the ones column there is no feature to multiply

β_{0}

by, so the model is forced through the origin (

\overset{y}{^} = 0

when all

x = 0

). The ones column is precisely what turns the intercept into "just another coefficient."

Spot the error: "I standardised my target

y

but not my features before running least squares, to make the coefficients comparable."

For ordinary least squares this mixes two different fixes. Coefficient comparability comes from scaling the features; scaling

y

alone just rescales all coefficients together. Feature scaling (Feature Scaling) is what puts

β_{j}

on a common footing.

Spot the error: "Since

J = \frac{1}{m} \sum (y - X β)^{2}

, its gradient is

\nabla J = - 2 X^{⊤} y + 2 X^{⊤} X β

, and I then solve

β = X^{⊤} y / (X^{⊤} X)

Two errors. First, because

J

carries the

\frac{1}{m}

, the true gradient is

\nabla J = \frac{2}{m} (X^{⊤} X β - X^{⊤} y)

— dropping the

\frac{1}{m}

is wrong (it happens not to change the root since we set it to zero, but the stated formula is still incorrect). Second, you cannot "divide" by a matrix: setting the gradient to zero gives

X^{⊤} X β = X^{⊤} y

, whose solution needs the inverse

β = (X^{⊤} X)^{- 1} X^{⊤} y

, not scalar division.

Spot the error: "The gradient came out as

\frac{2}{m} (X^{⊤} X β - X^{⊤} y)

, so I set the

\frac{2}{m}

to zero to find the minimum."

The constant factor

\frac{2}{m}

is never zero (it just scales the gradient); you set the vector part

X^{⊤} X β - X^{⊤} y

to zero, which yields the Normal Equation.

Spot the error: "

y^{⊤} X β

and

β^{⊤} X^{⊤} y

are different terms, so the expansion has four distinct terms."

Both are

1 \times 1

scalars, and a scalar equals its own transpose, so

y^{⊤} X β = β^{⊤} X^{⊤} y

. They combine into the single

- 2 β^{⊤} X^{⊤} y

term.

Why questions

Why do we minimise the sum of squared errors instead of the sum of absolute errors?

Squaring makes the cost

J

smooth and differentiable everywhere, giving a clean closed-form gradient and a single convex bowl; it also penalises large misses disproportionately. Absolute error is non-differentiable at zero and has no neat Normal Equation.

Why does adding

λ I

X^{⊤} X

(Ridge) rescue a singular problem?

For any eigenvector

v

with

X^{⊤} X v = σ v

, we get

(X^{⊤} X + λ I) v = (σ + λ) v

, so every eigenvalue rises by exactly

λ

(the penalty strength). A zero eigenvalue (the flat trench floor) becomes

λ > 0

, so the matrix turns strictly positive definite and hence invertible. See Ridge Regression.

Why is

X^{⊤} X

guaranteed to be symmetric?

Because

(X^{⊤} X)^{⊤} = X^{⊤} (X^{⊤})^{⊤} = X^{⊤} X

. Symmetry is what lets us use the matrix-calculus rule

\frac{\partial}{\partial β} β^{⊤} A β = 2 A β

Why might we prefer Gradient Descent over the Normal Equation even when

X^{⊤} X

is invertible?

Gradient Descent rolls downhill on the cost bowl step by step instead of inverting a matrix. Inverting an

n \times n

matrix costs roughly

O (n^{3})

, brutal for thousands of features; gradient descent scales far better and streams over data, at the price of choosing a learning rate and iterating.

Why does the Normal Equation involve

X^{⊤} y

rather than just

y

X^{⊤} y

collects, for each feature, the dot product of that feature's column with the target — i.e. "how strongly does this feature co-vary with

y

." That is the quantity the optimal weights must match.

Why can the same data give a positive simple-regression slope for a feature but a negative coefficient in the multiple model?

Because the multiple model holds other correlated features constant. A confounder can create a spurious marginal trend that reverses once its influence is removed — a form of Simpson's paradox; compare with Simple Linear Regression.

Edge cases

What happens if you fit the model with exactly

m = n + 1

examples (one per coefficient) and all points are in "general position"?

The plane passes through every point with zero residual — the system is exactly determined. Training MSE is

0

, but this tells you nothing about generalisation; it is the boundary just before over-determination.

What does the Normal Equation output if you accidentally include a feature that is constant (same value for every example)?

That column is a multiple of the ones column, creating perfect collinearity, so

X^{⊤} X

becomes singular and the plain inverse fails. Drop the constant feature or use a pseudoinverse.

What is the model's prediction structure when you have only the intercept (no features at all)?

With just the ones column, least squares sets

β_{0}

to the mean of

y

, so

\overset{y}{^} = \overset{y}{ˉ}

for every input. This is the baseline that

R^{2}

compares against.

If two features are strongly (but not perfectly) collinear, does the Normal Equation still run?

Yes —

X^{⊤} X

is technically invertible, but nearly singular (a nearly flat trench), so the inverse has huge entries and coefficients become wildly unstable to tiny data changes. Predictions may still be okay while interpretations are worthless; regularisation stabilises it.

What happens to the coefficients if you duplicate an entire column of

X

Perfect collinearity again: the two identical features are indistinguishable, so infinitely many

(β_{a}, β_{b})

pairs with a fixed sum fit equally well.

X^{⊤} X

is singular; only the combined effect is identifiable.

Recall One-line self-test

The single most common trap ::: " $β_{j}$ is the effect of $x_{j}$ " — always add "holding the other features constant," and remember it may be conditional on a world your data never actually contains.