Correlation analysis and multicollinearity — Hinglish notes

2.1.15 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Correlation analysis statistical relationship examine karta hai features ke beech, jabki multicollinearity tab hoti hai jab predictor variables aapas mein highly correlated hote hain. Dono hi feature selection aur model interpretability ke liye critical hain.

Figure — Correlation analysis and multicollinearity

Core Concepts

Derivation: Correlation Linear Dependence Kyun Measure Karta Hai

Pehle principles se shuru karo: Hum ek aisa measure chahte hain jo:

Symmetric ho: $measure (X, Y) = measure (Y, X)$
Scale-invariant ho: $X$ ki units double karne se relationship strength nahi badlni chahiye
"Co-movement" capture kare: jab $X$ apne mean se upar ho, to $Y$ apne mean se upar/neeche hone ki tendency rakhta ho

Step 1: Covariance se co-movement measure karo: $Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] = E [X Y] - E [X] E [Y]$

Ye step kyun? $(X - μ_{X})$ positive hota hai jab $X$ average se upar ho, negative jab neeche ho. Unka product $(X - μ_{X}) (Y - μ_{Y})$ :

Positive hota hai jab dono average se upar/neeche hों (same direction)
Negative hota hai jab ek upar ho, ek neeche (opposite direction)

Is product ka average "typical co-movement" deta hai.

Step 2: Normalize karke scale-invariant banao: $r = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}$

Ye step kyun? Covariance ki units hoti hain $(X -units) \times (Y -units)$ . Dono standard deviations se divide karne par units hat jaati hain aur $r$ ko $[- 1, 1]$ mein bound kar deta hai (Cauchy-Schwarz inequality se).

Bounds ka proof: Cauchy-Schwarz se, $∣ Cov (X, Y) ∣ \leq σ_{X} σ_{Y}$ , isliye $∣ r ∣ \leq 1$ . Equality tab hoti hai jab $Y = a X + b$ (perfect linear relationship).

Multicollinearity Measure Karna

Formula Variance Inflation Factor (VIF)

Feature $X_{j}$ ke liye, VIF quantify karta hai ki colinearity ki wajah se uska variance kitna inflate hota hai: $VIF_{j} = \frac{1}{1 - R _{j}^{2}}$ jahan $R_{j}^{2}$ coefficient of determination hai jo $X_{j}$ ko baaki saare features par regress karne se milta hai.

Variance se derivation: Simple regression $Y = β_{0} + β_{1} X + ϵ$ mein, $\hat{β}_{1}$ ka variance hota hai: $Var (\hat{β}_{1}) = \frac{σ ^{2}}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2}}$

Multiple predictors aur colinearity ke saath, ye ban jaata hai: $Var (\hat{β}_{j}) = \frac{σ ^{2}}{\sum _{i = 1}^{n} ( x _{ij} - x ˉ _{j} ) ^{2}} \cdot \frac{1}{1 - R _{j}^{2}} = Var_{simple} \times VIF_{j}$

Kyun? Denominator $(1 - R_{j}^{2})$ shrink hota hai jaise $X_{j}$ doosron se zyada predictable hota jaata hai. Jab $R_{j}^{2} \to 1$ (perfect collinearity), $VIF_{j} \to \infty$ aur variance explode ho jaata hai.

Interpretation:

$VIF = 1$ : doosre features ke saath koi correlation nahi
$VIF < 5$ : acceptable
$VIF > 10$ : severe multicollinearity (common threshold)

Formula Condition Number

Design matrix $X$ ke liye, condition number uske largest aur smallest singular value ka ratio hai: $κ (X) = \frac{σ _{m a x} ( X )}{σ _{m i n} ( X )}$

Equivalently, $X^{T} X$ ke eigenvalues ke terms mein: kyunki $X$ ke singular values, $X^{T} X$ ke eigenvalues ke square roots hote hain, $κ (X) = \frac{λ _{m a x} ( X ^{T} X )}{λ _{m i n} ( X ^{T} X )}$

Square root kyun? Rule-of-thumb thresholds $κ (X)$ (yaani $X$ ka condition number) ko refer karte hain, $X^{T} X$ ke $λ_{m a x} / λ_{m i n}$ ko nahi. Kyunki $X^{T} X$ singular values ko square kar deta hai, $cond (X^{T} X) = κ (X)^{2}$ , isliye $X^{T} X$ ke eigenvalue ratio ko directly same cutoff ke saath use karna inconsistent hoga (ye bahut zyada matrices ko flag kar deta).

Ye kyun matters karta hai: $X^{T} X \hat{β} = X^{T} y$ solve karte waqt, condition number measure karta hai ki $y$ mein errors kitni amplify hoti hain. Agar $κ (X) = 1000$ hai, to $y$ mein 0.1% error, $\hat{β}$ mein 100% tak error cause kar sakti hai.

Rule of thumb: $κ (X) > 30$ moderate multicollinearity indicate karta hai; $κ (X) > 100$ severe multicollinearity indicate karta hai.

Methods aur Examples

Worked example Correlation Matrix Compute Karna

Dataset: 4 features $X_{1}, X_{2}, X_{3}, X_{4}$ with $n = 5$ samples

X1 = [1, 2, 3, 4, 5]
X2 = [2, 4, 6, 8, 10]  # Perfect correlation with X1
X3 = [5, 4, 3, 2, 1]   # Perfect negative with X1
X4 = [3, 1, 4, 1, 5]   # Random

Step 1: Calculate karo $\overset{x}{ˉ}_{1} = 3$ , aur $\sum_{i} (x_{1, i} - \overset{x}{ˉ}_{1})^{2} = 4 + 1 + 0 + 1 + 4 = 10$ . Similarly $\overset{x}{ˉ}_{2} = 6$ aur $\sum_{i} (x_{2, i} - \overset{x}{ˉ}_{2})^{2} = 16 + 4 + 0 + 4 + 16 = 40$ .

Step 2: $r_{12}$ compute karo: $r_{12} = \frac{( 1 - 3 ) ( 2 - 6 ) + ( 2 - 3 ) ( 4 - 6 ) + ( 3 - 3 ) ( 6 - 6 ) + ( 4 - 3 ) ( 8 - 6 ) + ( 5 - 3 ) ( 10 - 6 )}{10 \cdot 40}$ $= \frac{( - 2 ) ( - 4 ) + ( - 1 ) ( - 2 ) + 0 + ( 1 ) ( 2 ) + ( 2 ) ( 4 )}{400} = \frac{8 + 2 + 0 + 2 + 8}{20} = \frac{20}{20} = 1.0$

Ye step kyun? Numerator cross-deviation products ka raw sum hai $\sum_{i} (x_{1, i} - \overset{x}{ˉ}_{1}) (x_{2, i} - \overset{x}{ˉ}_{2}) = 20$ . Denominator hai $\sum (x_{1} - \overset{x}{ˉ}_{1})^{2} \cdot \sum (x_{2} - \overset{x}{ˉ}_{2})^{2} = 10 \cdot 40 = 400 = 20$ . Kyunki $X_{2} = 2 X_{1}$ exactly hai, correlation exactly $1.0$ hai. (Note: sample covariance hogi $20/ (n - 1) = 20/4 = 5$ , aur $σ_{1} σ_{2} = 2.5 \cdot 10 = 5$ , same $r = 5/5 = 1$ milta hai. Normalization cancel ho jaata hai chahe population ya sample scaling use karo.)

Step 3: $r_{14}$ compute karo (with $X_{4} = [3, 1, 4, 1, 5]$ , $\overset{x}{ˉ}_{4} = 2.8$ ): $\sum_{i} (x_{1, i} - \overset{x}{ˉ}_{1}) (x_{4, i} - \overset{x}{ˉ}_{4}) = (- 2) (0.2) + (- 1) (- 1.8) + (0) (1.2) + (1) (- 1.8) + (2) (2.2)$ $= - 0.4 + 1.8 + 0 - 1.8 + 4.4 = 4.0$ $\sum_{i} (x_{4, i} - \overset{x}{ˉ}_{4})^{2} = 0.04 + 3.24 + 1.44 + 3.24 + 4.84 = 12.8$ $r_{14} = \frac{4.0}{10 \cdot 12.8} = \frac{4.0}{128} = \frac{4.0}{11.31} \approx 0.354$

Ye step kyun? $X_{4}$ ka $X_{1}$ ke saath mild positive association hai, jisse $r_{14} \approx 0.35$ milta hai—zero nahi, lekin perfect $X_{1}$ – $X_{2}$ relationship se bahut weaker. Kyunki $X_{2} = 2 X_{1}$ hai, hamare paas $r_{24} = r_{14} \approx 0.35$ hai, aur kyunki $X_{3} = 6 - X_{1}$ hai, hamare paas $r_{34} = - r_{14} \approx - 0.35$ hai.

Full symmetric matrix (diagonal = 1, symmetric off-diagonals): $R = 1.00 1.00 - 1.00 0.35 1.00 1.00 - 1.00 0.35 - 1.00 - 1.00 1.00 - 0.35 0.35 0.35 - 0.35 1.00$ Insight: $X_{1}, X_{2}, X_{3}$ perfectly colinear hain (har ek doosron ka linear function hai)—sirf ek rakho! $X_{4}$ genuinely independent information carry karta hai.

Worked example VIF Calculation

Setup: House price predict karna in features ke saath:

$X_{1}$ : square footage (1000-3000 sqft)
$X_{2}$ : number of bedrooms (2-5)
$X_{3}$ : number of bathrooms (1-3)
$X_{4}$ : age of house (0-50 years)

Step 1: $X_{2}$ ko $X_{1}, X_{3}, X_{4}$ par regress karo: $X_{2} = 0.5 + 0.001 \cdot X_{1} + 0.8 \cdot X_{3} - 0.01 \cdot X_{4}$

$R_{2}^{2} = 0.85$ milta hai (bedrooms size aur bathrooms se highly predictable hain).

Step 2: VIF calculate karo: $VIF_{2} = \frac{1}{1 - 0.85} = \frac{1}{0.15} = 6.67$ Ye kyun matters karta hai: "bedrooms" coefficient ka standard error $6.67 \approx 2.58$ times inflate ho gaya hai. Hum bedrooms ke true effect ke baare mein 2.6× kam certain hain.

Saare features ke liye VIF:

$VIF_{1} = 4.2$ (sqft correlated hai bedrooms/bathrooms se)
$VIF_{2} = 6.7$ (bedrooms highly correlated hain)
$VIF_{3} = 5.1$ (bathrooms correlated hain)
$VIF_{4} = 1.3$ (age independent hai)

Action: $X_{2}$ (bedrooms) drop karo kyunki ye $X_{1}$ aur $X_{3}$ ke saath redundant hai.

Common Pitfalls

Detection Workflow

Step-by-step:

Pairwise check: Saare feature pairs ke liye $∣ r_{ij} ∣$ compute karo. Agar $> 0.8$ ho to flag karo.
VIF check: Har feature ke liye, doosron par regress karo, VIF compute karo. Agar $> 10$ ho to flag karo.
Global check: $κ (X)$ (design matrix ka condition number) compute karo. Agar $> 30$ hai, to matrix ill-conditioned hai.
Action: Har correlated pair se ek feature drop karo, ya PCA use karo, ya L2 regularization apply karo.

Remediation Strategies

Har highly correlated pair se ek feature drop karo (agar $r_{12} = 0.95$ hai to $X_{1}$ rakho, $X_{2}$ drop karo)
Features combine karo: $X_{new} = X_{1} + X_{2}$ create karo (jaise, "total_rooms = bedrooms + bathrooms")
Principal Component Analysis (PCA): Uncorrelated components mein transform karo
Regularization: Ridge ( $L_{2}$ penalty) ya Lasso ( $L_{1}$ , selection bhi karta hai) use karo
Domain knowledge: Jo feature clearer causal interpretation rakhta ho use rakho

Connections

Covariance Matrix: Correlation normalized covariance hai
Feature Selection: Features drop karne ke liye Correlation ek criterion hai
Principal Component Analysis (PCA): Correlated features ko orthogonal components mein transform karta hai
Ridge Regression: $L_{2}$ penalty ke zariye multicollinearity handle karta hai
Lasso Regression: Multicollinearity ke under feature selection karta hai
Linear Regression Assumptions: Multicollinearity "no perfect colinearity" assumption violate karta hai
Variance-Bias Tradeoff: High VIF coefficient estimates ke variance ko inflate karta hai
Condition Number: Multicollinearity ka matrix theory perspective

Recall Ek 12-Saal ke Bacche ko Explain Karo

Socho tum figure out karne ki koshish kar rahe ho ki koi video game fun kyun hoti hai, alag-alag features dekh kar: graphics quality, sound quality, story depth, aur number of explosions.

Correlation aise puuchna hai: "Kya better graphics wale games mein better sound bhi hota hai?" Agar haan, to ye correlated hain. Tum ise -1 se 1 tak ke number se measure karte ho:

1 ka matlab hai: Better graphics ka HAMESHA better sound matlab hai (perfect match)
0 ka matlab hai: Graphics aur sound ka ek doosre se koi lena dena nahi
-1 ka matlab hai: Better graphics ka HAMESHA worse sound matlab hai (opposite)

Multicollinearity tab hoti hai jab do features itni similar hoti hain ki dono use karna ek hi cheez do baar count karne jaisa hai. Example: "graphics quality" aur "texture resolution" almost same cheez hain! Agar tum dono include karo, to tumhara analysis confuse ho jaata hai—ye nahi bata sakta ki kaunsa actually matter karta hai kyunki dono hamesha saath chalte hain.

Ye bura kyun hai? Socho tum figure out karne ki koshish kar rahe ho ki graphics ya story games ko fun banati hai. Agar tumhare data mein graphics aur story hamesha saath improve hote hain, to tum unke effects alag nahi kar sakte. Ye aise hai jaise figure out karna ki cake mein flour ya sugar tasty banata hai jab tum dono hamesha saath daalte ho—tum bata nahi sakte!

Fix: Similar features mein se ek drop karo (sirf graphics rakho, texture resolution drop karo), ya unhe ek "visual quality" score mein combine karo.

Flashcards

Features X aur Y ke beech -0.85 ka correlation coefficient kya indicate karta hai?

Strong negative linear relationship: jaise X badhta hai, Y proportionally decrease hoti hai. Features highly corre