Correlation analysis and multicollinearity: Question bank

This is a rapid-fire trap bank for the parent topic. Each line is a question ::: answer reveal. Cover the answer, commit to a guess, then check — the value is in catching why your instinct misfires, not in scoring points.

Before you start, a one-line refresher of the two symbols that show up everywhere below:

Recall What do

$r$ and VIF each measure? $r$ is the Pearson correlation coefficient — one number in $[- 1, 1]$ for a pair of features, measuring linear co-movement. VIF (Variance Inflation Factor) is a number $\geq 1$ for one feature, measuring how predictable it is from all the others together.

True or false — justify

A perfect correlation of

r = 0

means the two features are independent.

False.

r = 0

only rules out a linear relationship;

Y = X^{2}

over a symmetric range gives

r \approx 0

while

Y

is fully determined by

X

If every pairwise correlation between features is small (say

∣ r ∣ < 0.3

), the dataset is free of multicollinearity.

False. Multicollinearity can be multi-way:

X_{3} = X_{1} + X_{2}

makes

X_{3}

perfectly predictable while each pairwise

r

can stay modest. That is exactly why we need VIF, not just the correlation matrix.

A VIF of 1 means the feature is useless for prediction.

False. VIF

= 1

means the feature is uncorrelated with the other features (

R_{j}^{2} = 0

), i.e. it carries independent information — that is a good thing, not a useless one.

High multicollinearity always lowers the model's

R^{2}

(predictive fit).

False. Multicollinearity leaves overall fit/

R^{2}

largely intact; it wrecks the individual coefficient estimates (huge standard errors, sign flips), not the predictions.

Correlation is symmetric, so VIF must be symmetric too.

False.

r_{X Y} = r_{Y X}

, but VIF is a per-feature quantity from regressing one feature on all the rest — VIF

_{2} \neq =

VIF

_{1}

in general. See VIF logic in the parent note.

The condition number and the correlation matrix always agree on whether collinearity is "severe."

Mostly but not guaranteed. Condition number

κ (X)

captures the global near-singularity of the whole design matrix, which can be triggered by combinations no single pairwise

r

reveals.

Standardizing features to unit variance changes their correlations.

False. Correlation is already scale-invariant (that is the whole point of dividing covariance by

σ_{X} σ_{Y}

), so linear rescaling leaves

r

untouched. It does change the condition number of the raw design matrix, though.

Removing one of two perfectly correlated features loses information.

False. If

r = \pm 1

exactly, one feature is a linear function of the other, so it adds zero new information — dropping it is lossless. This is the basis of feature selection for redundant columns.

Multicollinearity is a problem you must fix before doing PCA.

False. PCA thrives on correlation — it rotates correlated features into uncorrelated principal components, so collinearity is the raw material PCA consumes, not an obstacle to it.

Spot the error

r = 0.9

, so

X

causes 90% of

Y

Two errors.

r

measures association, not causation; and

r

is not a percentage of anything — the variance explained is

r^{2} = 0.81

, not

0.9

"VIF

_{j} = \frac{1}{1 - R _{j}}

The denominator uses

R_{j}^{2}

(coefficient of determination), not

R_{j}

: VIF

_{j} = \frac{1}{1 - R _{j}^{2}}

. Dropping the square changes both the scale and the sign-insensitivity.

"The condition number of

X^{T} X

κ (X)

No — because

X^{T} X

squares the singular values,

cond (X^{T} X) = κ (X)^{2}

. The rule-of-thumb thresholds (

> 30

> 100

) refer to

κ (X)

, so applying them to

X^{T} X

's eigenvalue ratio flags far too many matrices.

r

can be

1.4

for extremely strong relationships."

Impossible. By Cauchy–Schwarz,

∣ Cov (X, Y) ∣ \leq σ_{X} σ_{Y}

, forcing

- 1 \leq r \leq 1

. Any value outside means a computation bug.

"To measure how

X_{2}

inflates variance, I regress

Y

(the target) on

X_{2}

Wrong target. VIF regresses the feature

X_{2}

on all the other features — the response

Y

never enters a VIF calculation.

"Two features have

r = - 1

, so they are unrelated."

The opposite:

r = - 1

is the strongest possible linear relationship (perfectly anti-correlated). It is

r = 0

that signals no linear relation.

"Ridge removes multicollinearity from my data."

Ridge doesn't remove collinearity from the data; it tolerates it by adding a penalty that stabilizes

(X^{T} X + λ I)^{- 1}

, shrinking coefficients so small data changes no longer cause wild swings.

Why questions

Why divide covariance by

σ_{X} σ_{Y}

instead of using covariance directly?

Covariance has units of

(X -units) \times (Y -units)

, so its magnitude depends on scale and is uncomparable across pairs. Dividing removes units and bounds the result to

[- 1, 1]

, making strengths comparable. See covariance.

Why does VIx blow up to infinity as

R_{j}^{2} \to 1

VIF

_{j} = 1/ (1 - R_{j}^{2})

; as the feature becomes perfectly predictable from the others,

R_{j}^{2} \to 1

, the denominator

\to 0

, and the coefficient variance divides by (nearly) zero — the estimate becomes infinitely uncertain.

Why does multicollinearity cause coefficient sign flips?

When features are near-collinear,

X^{T} X

is near-singular, so

(X^{T} X)^{- 1}

is huge and unstable; tiny data perturbations move the fitted coefficients enormously, sometimes across zero. This ties to the no-perfect-collinearity assumption.

Why can a model have high

R^{2}

but no individually "significant" coefficients under multicollinearity?

The features jointly explain

Y

well (high

R^{2}

), but because they overlap, the data can't attribute the effect to any single one — each coefficient's standard error is inflated, so none clears its significance test.

Why does adding a redundant feature hurt the variance side of the tradeoff?

A redundant feature adds an extra unstable parameter without new information, raising the model's estimation variance while doing nothing for bias — a pure variance cost.

Why does Lasso handle multicollinearity differently from Ridge?

Lasso's

L_{1}

penalty can drive redundant coefficients exactly to zero, effectively selecting one feature from a correlated group, whereas Ridge shrinks them together toward small but nonzero values.

Why is the condition number defined with singular values rather than raw entries of

X

Singular values measure how much the matrix stretches or squashes vectors along its principal directions; their ratio

σ_{m a x} / σ_{m i n}

directly quantifies how much input error in

y

gets amplified when solving for

\hat{β}

Edge cases

What is

r

when one feature is constant (zero variance)?

Undefined —

σ_{X} = 0

puts a zero in the denominator. A constant column has no variation to co-move with, so correlation simply doesn't exist; drop the column.

What VIF does a perfectly collinear feature get, and what does software report?

In theory VIF

= \infty

(

R_{j}^{2} = 1

). In practice the regression is rank-deficient, so software may report a giant finite number or error out — treat any VIF above ~10 as a red flag regardless.

Two features are identical duplicates. What is their

r

, and what happens to

X^{T} X

r = 1

exactly, and

X^{T} X

becomes singular (two identical columns), so

(X^{T} X)^{- 1}

does not exist — ordinary least squares has no unique solution.

With only

n = 2

data points, what is the correlation between any two features?

Always exactly

\pm 1

— two points always lie on a straight line, so

r

is degenerate and meaningless. You need

n

comfortably larger than the number of features for stable estimates.

A feature is a nonlinear transform of another, e.g.

X_{2} = e^{X_{1}}

. Will correlation flag the redundancy?

Not reliably.

r

only sees the linear part, so a strong nonlinear dependence can show a modest

r

while the feature is genuinely redundant — always inspect scatter plots, not just

r

What is the condition number of a design matrix with perfectly orthogonal (uncorrelated) standardized features?

κ (X) = 1

— all singular values are equal, so nothing is amplified. This is the best-conditioned case and the ideal target when engineering features.

If two features have

r = 0

but the model still behaves unstably, what should you check?

A three-or-more-way linear dependence (e.g.

X_{3} = X_{1} + X_{2}

) invisible to pairwise

r

— compute VIF and the condition number, which catch multi-way collinearity that the correlation matrix misses.