Exercises — Correlation analysis and multicollinearity

What to look for: off-diagonal entries near $\pm 1$ mean two features are essentially the same vector (angle near $0^{\circ }$ or $180^{\circ }$).

• $r_{12}=0.98$ → features 1 and 2 are almost perfectly aligned → redundant pair. Keep one, drop the other (see Feature Selection).
• Feature 3 has $r_{13}=0.10,r_{23}=0.12$ — near $0$, so it is nearly orthogonal to both → carries independent information. Answer: redundant pair = $(X_1,X_2)$; independent feature = $X_3$.

${\text{VIF}}_j=\frac{1}{1-R_j^2}$. Setting $\text{VIF}=1$ forces $1-R_j^2=1$, so $R_j^2=0$. $R_j^2$ is how well the other features predict $X_j$. $R_j^2=0$ means they predict it not at all — $X_j$ is linearly uncorrelated with every other predictor. No inflation of its coefficient's variance.

False. $r$ measures only the linear (straight-line) part of the relationship — the cosine of the deviation-vectors. Two features can sit on a perfect parabola ($Y=X^2$ over symmetric $X$) and have $r=0$ while being fully dependent. $r=0$ means "no linear dependence," not "independent."

Center them. $\bar{x}=2,\bar{y}=2$.

• $u=[-1,0,1]$, $v=[0,-1,1]$.
• Dot product: $(-1)(0)+(0)(-1)+(1)(1)=1$.
• $\sqrt{\sum u^2}=\sqrt{1+0+1}=\sqrt{2}$; $\sqrt{\sum v^2}=\sqrt{0+1+1}=\sqrt{2}$. $r=\frac{1}{\sqrt{2}\cdot \sqrt{2}}=\frac{1}{2}=0.5$

${\text{VIF}}_j=\frac{1}{1-0.90}=\frac{1}{0.10}=10$ Variance inflates by VIF; standard error is $\sqrt{\text{Variance}}$, so it inflates by $\sqrt{10}\approx 3.16$. Reading: VIF $=10$ sits exactly on the common "severe multicollinearity" threshold. You are about $3.2\times$ less certain of this coefficient than if $X_j$ were clean.

$\kappa (X)=\frac{{\sigma }_{\max }}{{\sigma }_{\min }}=\frac{40}{2}=20$ Rule of thumb: $\kappa >30$ moderate, $\kappa >100$ severe. Here $\kappa =20<30$ → mild/acceptable. See Condition Number.

Singular values of $X$ = square roots of eigenvalues of $X^{T}X$. So $\kappa (X)=\sqrt{\frac{{\lambda }_{\max }}{{\lambda }_{\min }}}=\sqrt{\frac{900}{4}}=\sqrt{225}=15$ Watch the square root — using $900/4=225$ directly would be $\text{cond}(X^{T}X)=\kappa (X{)}^2$, not $\kappa (X)$.

Adding the constant $-7$ shifts $X_2$ but centering subtracts the mean, so the constant vanishes: the deviation vector of $X_2$ is $3\times$ the deviation vector of $X_1$. A vector and a positive multiple of itself point the same way — angle $0^{\circ }$ — so $r_{12}=\cos 0^{\circ }=+1.$ The scale factor $3>0$ is cancelled by the length-normalization in the denominator. (If it were $-3$, they'd point opposite → $r=-1$.)

• Angle: $r=\cos \theta \Rightarrow \theta =\arccos (0.6)\approx 53.13^{\circ }$. In the figure, that's the arc between the black and red arrows.
• Explained variance: projecting the red vector onto the black one keeps only the part of $Y$ that a straight line from $X$ can reproduce. The length of that projection, relative to the full red vector, is $\cos \theta =r$; the variance (length squared) it reproduces is therefore $r^2$. So $R^2=r^2=0.36$, i.e. 36% explained. The remaining $64\%$ is the perpendicular part of the red arrow — genuinely unexplained by the linear fit.

Diagnosis: multicollinearity. The features together predict $Y$ well (high $R^2$), but because they overlap heavily, the model cannot decide which one deserves credit — so each coefficient has a huge standard error and looks insignificant individually. Symptoms to check: high VIF ($>10$) on several features, and a large condition number $\kappa (X)>30$. This is the classic "significant model, insignificant parts" signature.

• Ridge (${\ell }_2$ penalty) tends to spread the coefficient across correlated features, shrinking them together toward similar small values — it keeps the whole group.
• Lasso (${\ell }_1$ penalty) tends to pick one feature from the correlated group and zero the rest, doing implicit feature selection — but the choice can be unstable (which one it keeps flips with small data changes, precisely because they're near-identical). Why: the ${\ell }_1$ penalty's sharp corners land the solution on an axis (a zero coefficient), while ${\ell }_2$'s round contour never forces exact zeros. Correlation makes the group interchangeable, so Lasso arbitrates arbitrarily.

Let centered deviation-vectors be $u=(u_1,\dots ,u_n)$ and $v=(v_1,\dots ,v_n)$. Cauchy–Schwarz states $\left|{\sum }_i{u}_i{v}_i\right|\le \sqrt{{\sum }_i{u}_i^2}\sqrt{{\sum }_i{v}_i^2}.$ The left side is $|\text{numerator of }r|$; the right side is the denominator of $r$. Dividing both sides by the (positive) right side: $\frac{|{\sum }_i{u}_i{v}_i|}{\sqrt{\sum u_i^2}\sqrt{\sum v_i^2}}=|r|\le 1\Rightarrow -1\le r\le 1.$ Equality ($|r|=1$) holds exactly when $u$ and $v$ are proportional, $v=au$ — i.e. $Y=aX+b$, a perfect line. This is the cosine bound: $|\cos \theta |\le 1$.

Where the denominator comes from — no black box. In OLS the coefficient covariance is $\text{Var}(\hat{\beta })={\sigma }^2(X^{T}X{)}^{-1}$, and $\text{Var}({\hat{\beta }}_j)$ is its $j$-th diagonal entry. The key fact is what that diagonal entry equals, and here is the geometric reason.

Step 1 — split $X_j$ into "explained by others" + "unique." Regress column $X_j$ on all the other feature-columns. This splits $X_j$ into two orthogonal pieces: $X_j=\underset{\text{part the others can build}}{\underbrace{{\hat{X}}_j}}+\underset{\text{residual: }{X}_j\text{’s own direction}}{\underbrace{r_j}}$ The residual $r_j$ is, by construction, perpendicular to every other feature — it is the only direction in $X_j$ that no other feature can imitate.

Step 2 — only the unique part can estimate ${\hat{\beta }}_j$. In multiple regression, ${\hat{\beta }}_j$ is determined entirely by projecting $Y$ onto that unique residual direction $r_j$. So its variance depends only on how long $r_j$ is, i.e. on ${\text{RSS}}_j={\sum }_i{r}_{j,i}^2$ (the residual sum of squares). A standard OLS identity (the Frisch–Waugh result) makes this exact: $\text{Var}({\hat{\beta }}_j)=\frac{{\sigma }^2}{{\text{RSS}}_j}.$ Intuition: a short residual means $X_j$ has almost no direction of its own → tiny denominator → huge variance. This is the WHY of the $(X^{T}X{)}^{-1}$ blow-up in plain geometry: inverting a near-singular matrix is exactly "dividing by a tiny residual length."

Step 3 — rewrite ${\text{RSS}}_j$ via $R_j^2$. By definition of $R_j^2$ from that same regression, $R_j^2=1-\frac{{\text{RSS}}_j}{{\text{TSS}}_j},{\text{TSS}}_j={\sum }_i(x_{ij}-{\bar{x}}_j{)}^2\Rightarrow {\text{RSS}}_j={\text{TSS}}_j(1-R_j^2).$ Step 4 — substitute: $\text{Var}({\hat{\beta }}_j)=\frac{{\sigma }^2}{{\text{TSS}}_j(1-R_j^2)}=\underset{\text{simple-regression variance}}{\underbrace{\frac{{\sigma }^2}{{\sum }_i(x_{ij}-{\bar{x}}_j{)}^2}}}\cdot \underset{{\text{VIF}}_j}{\underbrace{\frac{1}{1-R_j^2}}}.$ Limit. As $X_j$ becomes perfectly predictable from the others, $R_j^2\to 1$, so $1-R_j^2\to 0$ and ${\text{VIF}}_j\to \infty$: the variance blows up. Geometrically, the unique residual $r_j$ shrinks to zero length, leaving nothing to estimate its own coefficient with.

Orthonormal centered columns mean $X^{T}X=I$ (identity). Then:

• VIF: each $X_j$ is uncorrelated with the others, so regressing $X_j$ on them gives $R_j^2=0$, hence ${\text{VIF}}_j=\frac{1}{1-0}=1$ for all $j$.
• Condition number: eigenvalues of $I$ are all $1$, so ${\lambda }_{\max }={\lambda }_{\min }=1$ and $\kappa (X)=\sqrt{\frac{1}{1}}=1.$ $\kappa =1$ is the best possible — a perfectly well-conditioned problem, no error amplification. This is exactly the state PCA engineers (orthogonal components) and what standardizing + decorrelating aims for.

Read a correlation matrix for redundancy? ::: L1 ✓ Compute $r$, VIF, $\kappa$ from raw numbers? ::: L2 ✓ Explain a high-$R^2$-yet-insignificant model? ::: L3 ✓ Choose drop vs ridge vs PCA by redundancy strength? ::: L4 ✓ Prove the $r$ bound and derive VIF? ::: L5 ✓