Correlation analysis and multicollinearity

Overview

Correlation analysis examines the statistical relationship between features, while multicollinearity occurs when predictor variables are highly correlated with each other. Both are critical for feature selection and model interpretability.

Figure — Correlation analysis and multicollinearity

Core Concepts

Definition Correlation Coefficient

The Pearson correlation coefficient $r$ between features $X$ and $Y$ measures linear relationship: $r_{XY} = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i - \bar{y})^2}}$

Range: $-1 \leq r \leq 1$

$r = 1$ : perfect positive linear relationship
$r =0$ : no linear relationship
$r = -1$ : perfect negative linear relationship Why this formula? It's the covariance (how much X and Y vary together) normalized by their individual standard deviations (their "room to vary"). This makes $r$ unit-free and comparable across feature pairs.

Derivation: Why Correlation Measures Linear Dependence

Start from first principles: We want a measure that:

Is symmetric: $\text{measure}(X,Y) = \text{measure}(Y,X)$
Is scale-invariant: doubling $X$ 's units shouldn't change the relationship strength
Captures "co-movement": when $X$ is above its mean, $Y$ tends to be above/below its mean

Step 1: Measure co-movement with covariance: $\text{Cov}(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - E[X]E[Y]$

Why this step? $(X - \mu_X)$ is positive when $X$ is above average, negative when below. Their product $(X - \mu_X)(Y - \mu_Y)$ is:

Positive when both are above/below average (same direction)
Negative when one is above, one below (opposite direction)

Averaging this product gives the "typical co-movement."

Step 2: Make it scale-invariant by normalizing: $r = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$

Why this step? Covariance has units of $(X\text{-units}) \times (Y\text{-units})$ . Dividing by both standard deviations removes units and bounds $r$ to $[-1, 1]$ (by Cauchy-Schwarz inequality).

Proof of bounds: By Cauchy-Schwarz, $|\text{Cov}(X,Y)| \leq \sigma_X \sigma_Y$ , so $|r| \leq 1$ . Equality holds when $Y = aX + b$ (perfect linear relationship).

Measuring Multicollinearity

Formula Variance Inflation Factor (VIF)

For feature $X_j$ , the VIF quantifies how much its variance inflates due to colinearity: $\text{VIF}_j = \frac{1}{1 - R_j^2}$ where $R_j^2$ is the coefficient of determination from regressing $X_j$ on all other features.

Derivation from variance: In simple regression $Y = \beta_0 + \beta_1 X + \epsilon$ , the variance of $\hat{\beta}_1$ is: $\text{Var}(\hat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2}$

With multiple predictors and colinearity, this becomes: $\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{\sum_{i=1}^{n}(x_{ij} - \bar{x}_j)^2} \cdot \frac{1}{1 - R_j^2} = \text{Var}_{\text{simple}} \times \text{VIF}_j$

Why? The denominator $(1 - R_j^2)$ shrinks as $X_j$ becomes more predictable from others. When $R_j^2 \to 1$ (perfect collinearity), $\text{VIF}_j \to \infty$ and variance explodes.

Interpretation:

$\text{VIF} = 1$ : no correlation with other features
$\text{VIF} < 5$ : acceptable
$\text{VIF} > 10$ : severe multicollinearity (common threshold)

Formula Condition Number

For the design matrix $X$ , the condition number is the ratio of its largest to smallest singular value: $\kappa(X) = \frac{\sigma_{\max}(X)}{\sigma_{\min}(X)}$

Equivalently, in terms of eigenvalues of $X^TX$ : since the singular values of $X$ are the square roots of the eigenvalues of $X^TX$ , $\kappa(X) = \sqrt{\frac{\lambda_{\max}(X^TX)}{\lambda_{\min}(X^TX)}}$

Why the square root? The rule-of-thumb thresholds refer to $\kappa(X)$ (the condition number of $X$ itself), NOT to $\lambda_{\max}/\lambda_{\min}$ of $X^TX$ . Because $X^TX$ squares the singular values, $\text{cond}(X^TX) = \kappa(X)^2$ , so using the eigenvalue ratio of $X^TX$ directly with the same cutoff would be inconsistent (it would flag far more matrices).

Why this matters: When solving $X^T X \hat{\beta} = X^T y$ , the condition number measures how errors in $y$ get amplified. If $\kappa(X) = 1000$ , a 0.1% error in $y$ can cause up to 100% error in $\hat{\beta}$ .

Rule of thumb: $\kappa(X) > 30$ indicates moderate multicollinearity; $\kappa(X) > 100$ indicates severe multicollinearity.

Methods and Examples

Worked example Computing Correlation Matrix

Dataset: 4 features $X_1, X_2, X_3, X_4$ with $n=5$ samples

X1 = [1, 2, 3, 4, 5]
X2 = [2, 4, 6, 8, 10]  # Perfect correlation with X1
X3 = [5, 4, 3, 2, 1]   # Perfect negative with X1
X4 = [3, 1, 4, 1, 5]   # Random

Step 1: Calculate $\bar{x}_1 = 3$ , and $\sum_i (x_{1,i}-\bar{x}_1)^2 = 4+1+0+1+4 = 10$ . Similarly $\bar{x}_2 = 6$ and $\sum_i (x_{2,i}-\bar{x}_2)^2 = 16+4+0+4+16 = 40$ .

Step 2: Compute $r_{12}$ : $r_{12} = \frac{(1-3)(2-6) + (2-3)(4-6) + (3-3)(6-6) + (4-3)(8-6) + (5-3)(10-6)}{\sqrt{10} \cdot \sqrt{40}}$ $= \frac{(-2)(-4) + (-1)(-2) + 0 + (1)(2) + (2)(4)}{\sqrt{400}} = \frac{8+2+0+2+8}{20} = \frac{20}{20} = 1.0$

Why this step? The numerator is the raw sum of cross-deviation products $\sum_i (x_{1,i}-\bar{x}_1)(x_{2,i}-\bar{x}_2) = 20$ . The denominator is $\sqrt{\sum(x_1-\bar{x}_1)^2}\cdot\sqrt{\sum(x_2-\bar{x}_2)^2} = \sqrt{10}\cdot\sqrt{40} = \sqrt{400} = 20$ . Since $X_2 = 2X_1$ exactly, the correlation is exactly $1.0$ . (Note: the sample covariance would be $20/(n-1)=20/4=5$ , and $\sigma_1\sigma_2 = \sqrt{2.5}\cdot\sqrt{10}=5$ , giving the same $r=5/5=1$ . The normalization cancels whether we use population or sample scaling.)

Step 3: Compute $r_{14}$ (with $X_4 = [3,1,4,1,5]$ , $\bar{x}_4 = 2.8$ ): $\sum_i (x_{1,i}-\bar{x}_1)(x_{4,i}-\bar{x}_4) = (-2)(0.2)+(-1)(-1.8)+(0)(1.2)+(1)(-1.8)+(2)(2.2)$ $= -0.4 + 1.8 + 0 - 1.8 + 4.4 = 4.0$ $\sum_i (x_{4,i}-\bar{x}_4)^2 = 0.04 + 3.24 + 1.44 + 3.24 + 4.84 = 12.8$ $r_{14} = \frac{4.0}{\sqrt{10}\cdot\sqrt{12.8}} = \frac{4.0}{\sqrt{128}} = \frac{4.0}{11.31} \approx 0.354$

Why this step? $X_4$ has a mild positive association with $X_1$ , giving $r_{14} \approx 0.35$ —not zero, but far weaker than the perfect $X_1$ – $X_2$ relationship. Since $X_2 = 2X_1$ , we have $r_{24} = r_{14} \approx 0.35$ , and since $X_3 = 6-X_1$ , we have $r_{34} = -r_{14} \approx -0.35$ .

Full symmetric matrix (diagonal = 1, symmetric off-diagonals): $R = \begin{bmatrix} 1.00 & 1.00 & -1.00 & 0.35 \\ 1.00 & 1.00 & -1.00 & 0.35 \\ -1.00 & -1.00 & 1.00 & -0.35 \\ 0.35 & 0.35 & -0.35 & 1.00 \end{bmatrix}$ Insight: $X_1, X_2, X_3$ are perfectly colinear (each is a linear function of the others)—keep only one! $X_4$ carries genuinely independent information.

Worked example VIF Calculation

Setup: Predicting house price with features:

$X_1$ : square footage (1000-3000 sqft)
$X_2$ : number of bedrooms (2-5)
$X_3$ : number of bathrooms (1-3)
$X_4$ : age of house (0-50 years)

Step 1: Regress $X_2$ on $X_1, X_3, X_4$ : $X_2 = 0.5 + 0.001\cdot X_1 + 0.8 \cdot X_3 - 0.01 \cdot X_4$

Get $R_2^2 = 0.85$ (bedrooms are highly predictable from size and bathrooms).

Step 2: Calculate VIF: $\text{VIF}_2 = \frac{1}{1 - 0.85} = \frac{1}{0.15} = 6.67$ Why this matters: The standard error of the "bedrooms" coefficient is inflated by $\sqrt{6.67} \approx 2.58$ times. We're 2.6× less certain about bedrooms' true effect.

VIF for all features:

$\text{VIF}_1 = 4.2$ (sqft correlated with bedrooms/bathrooms)
$\text{VIF}_2 = 6.7$ (bedrooms highly correlated)
$\text{VIF}_3 = 5.1$ (bathrooms correlated)
$\text{VIF}_4 = 1.3$ (age independent)

Action: Drop $X_2$ (bedrooms) since it's redundant with $X_1$ and $X_3$ .

Worked example Detecting Nonlinear Relationships

Limitation of Pearson correlation: Only detects linear relationships. Data: $X = [-2, -1, 0, 1, 2]$ , $Y = [4, 1, 0, 1, 4]$ (perfect quadratic $Y = X^2$ )

Pearson correlation ( $\bar{x} = 0$ , $\bar{y} = 2$ ): $\text{numerator} = \sum_i x_i(y_i - 2) = (-2)(2) + (-1)(-1) + (0)(-2) + (1)(-1) + (2)(2) = -4+1+0-1+4 = 0$ $\Rightarrow r = 0$ Why zero? Since $\bar{x} = 0$ , $(x_i - \bar{x}) = x_i$ . The product $x_i(y_i - \bar{y})$ is antisymmetric in $x$ (because $Y = X^2$ is symmetric), so positive and negative terms cancel exactly.

Solution: Use Spearman rank correlation (based on ranks, not values) or plot scatter to see the U-shape. For feature engineering, create $X^2$ as a new feature.

Common Pitfalls

Detection Workflow

Step-by-step:

Pairwise check: Compute $|r_{ij}|$ for all feature pairs. Flag if $> 0.8$ .
VIF check: For each feature, regress on others, compute VIF. Flag if $> 10$ .
Global check: Compute $\kappa(X)$ (condition number of the design matrix). If $> 30$ , matrix is ill-conditioned.
Action: Drop one feature from each correlated pair, or use PCA, or apply L2 regularization.

Remediation Strategies

Drop one feature from each highly correlated pair (keeps $X_1$ , drops $X_2$ if $r_{12} = 0.95$ )
Combine features: Create $X_{\text{new}} = X_1 + X_2$ (e.g., "total_rooms = bedrooms + bathrooms")
Principal Component Analysis (PCA): Transform to uncorrelated components
Regularization: Use Ridge ( $L_2$ penalty) or Lasso ( $L_1$ , also does selection)
Domain knowledge: Keep the feature with clearer causal interpretation

Connections

Covariance Matrix: Correlation is normalized covariance
Feature Selection: Correlation is one criterion for dropping features
Principal Component Analysis (PCA): Transforms correlated features to orthogonal components
Ridge Regression: Handles multicollinearity via $L_2$ penalty
Lasso Regression: Does feature selection under multicollinearity
Linear Regression Assumptions: Multicollinearity violates "no perfect colinearity" assumption
Variance-Bias Tradeoff: High VIF inflates variance of coefficient estimates
Condition Number: Matrix theory perspective on multicollinearity

Recall Explain to a 12-Year-Old

Imagine you're trying to figure out what makes a video game fun by looking at different features: graphics quality, sound quality, story depth, and number of explosions.

Correlation is like asking: "Do games with better graphics also tend to have better sound?" If yes, they're correlated. You measure this with a number from -1 to 1:

1 means: Better graphics ALWAYS means better sound (perfect match)
0 means: Graphics and sound have nothing to do with each other
-1 means: Better graphics ALWAYS means worse sound (opposite)

Multicollinearity is when two features are so similar that using both is like counting the same thing twice. Example: "graphics quality" and "texture resolution" are almost the same thing! If you include both, your analysis gets confused—it can't tell which one really matters because they always move together.

Why is this bad? Imagine you're trying to figure out if graphics or story makes games fun. If graphics and story always improve together in your data, you can't separate their effects. It's like trying to figure out if flour or sugar makes cake tasty when you always add them together—you can't tell!

The fix: Drop one of the similar features (just keep graphics, drop texture resolution), or combine them into one "visual quality" score.

Flashcards

What does a correlation coefficient of -0.85 between features X and Y indicate?

Strong negative linear relationship: as X increases, Y tends to decrease proportionally. The features are highly corre

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, correlation analysis ka core idea ye hai ki hum check karte hain do features ek doosre ke saath kitna related hain. Jaise ghar ki price predict karni ho, toh "square footage" aur "number of rooms" — dono naturally connected hain, kyunki bade ghar mein zyada rooms hote hain. Yahi cheez Pearson correlation coefficient $r$ measure karta hai, jo -1 se +1 ke beech rehta hai. Formula bas covariance (dono features saath mein kitna vary karte hain) ko unki individual standard deviations se divide karta hai, taaki result unit-free ho jaye aur alag-alag feature pairs compare kar sako. Simple intuition: jab X apne average se upar hota hai aur Y bhi upar jaata hai, toh positive correlation; opposite direction mein negative.

Ab multicollinearity kya hai? Ye tab hota hai jab tumhare predictor variables aapas mein hi bahut zyada correlated ho jaate hain. Problem ye hai ki agar tum dono redundant features model mein daal do, toh model confuse ho jaata hai — coefficients unstable ho jaate hain, chhoti si data change pe wildly badal jaate hain, aur sign tak flip ho jaata hai. Interpretability bhi khatam — price area se badhta hai ya rooms se, ye clear nahi rehta. Technically, jab $X^TX$ matrix nearly singular ho jaata hai, toh uska inverse compute karna hi mushkil ho jaata hai, aur high $R^2$ hone ke baawajood individual coefficients insignificant nikalte hain.

Isko detect karne ke liye hum VIF (Variance Inflation Factor) use karte hain, jiska formula hai $\text{VIF}_j = \frac{1}{1 - R_j^2}$ . Yahaan $R_j^2$ tab milta hai jab tum ek feature ko baaki saare features pe regress karo — agar wo easily predict ho jaata hai (high $R_j^2$ ), toh VIF bada ho jaata hai, matlab wo feature redundant hai. Ye sab why-matters isliye hai kyunki feature selection karte time tumhe pehle hi redundancy pakadni hoti hai, warna model unstable ban jaayega aur uske results par bharosa nahi kar paoge. Toh clean, non-redundant features chunna ek accha aur samajhne-laayak model banane ka pehla step hai.

Correlation analysis and multicollinearity

Overview

Core Concepts

Derivation: Why Correlation Measures Linear Dependence

Measuring Multicollinearity

Methods and Examples

Common Pitfalls

Detection Workflow

Remediation Strategies

Connections

Flashcards

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Data Preprocessing & Feature Engineering

Connections