Intuition The one core ideaCorrelation and multicollinearity are both about overlap : when two columns of numbers rise and fall together, they carry the same information twice . This whole topic is a toolkit for measuring that overlap so we can spot — and remove — redundant features before they make a model unstable.
Before we can measure overlap, we need to be fluent in every symbol the parent note throws at us. This page builds each one from nothing, in an order where every symbol is earned before it is used. Nothing here assumes you have seen statistics before.
Common mistake A notation warning before we startOne letter, X two different things : a single feature column, and the whole table of features stacked side by side. To avoid confusion, this page uses the bold capital X only for the whole table (the "design matrix", Section 6) and the plain capital X
Everything starts with a table of numbers . Each row is one sample (one house, one patient, one photo). Each column is one feature — one measured quantity, like square footage.
Definition Feature and its notationA feature is a single column of measurements. We call it X x 1 , x 2 , … , x n
n the number of samples (how many rows).x i i
Look at the figure: the column of blue dots is the feature X x i i pointer that says "which row" — x 3 every row and then summed up.
Mnemonic Capital vs smallPlain capital X = one whole column (one feature). small x i = one number inside it. Bold X = the entire table (comes later).
Before we can talk about "above average" or "below average", we need the average itself.
Definition Mean (average)The mean of feature X x ˉ x ˉ = n 1 ∑ i = 1 n x i
The symbol ∑ "add these up" . The bits around it are instructions:
Definition The summation symbol∑ ∑ i = 1 n ( something with i ) i 1 i = n add all the results together ."
In the figure, x ˉ Why the topic needs it: every measure of relationship asks "is this point above or below its own average?" — so the average is the reference line we measure from .
This single quantity carries the sign that everything downstream depends on:
x i − x ˉ > 0 above average.x i − x ˉ < 0 below average.x i − x ˉ = 0 exactly on the average line.
Why the topic needs it: correlation is built from pairs of deviations. When we ask "do X Y X Y
We now measure how spread out a feature is.
Intuition Why square the deviations?If we averaged the raw deviations ( x i − x ˉ ) zero (that is what "balance point" means!). Squaring makes every term positive, so nothing cancels — big spreads produce big variance. Squaring (rather than taking absolute values) is chosen because it makes the algebra — and the Cauchy–Schwarz bound later — clean.
Definition Standard deviationσ X standard deviation is the square root of variance:
σ X = σ X 2 = n 1 ∑ i = 1 n ( x i − x ˉ ) 2 σ X typical distance from its mean — its "room to vary".
The symbol σ Why the topic needs it: correlation divides by σ X σ Y σ
1/ n 1/ ( n − 1 ) n population formula, which treats your data as the whole world . In practice you will often see n − 1 1 sample formula, which treats your data as a sample drawn from a bigger world and corrects a slight underestimate. For correlation r it does not matter which you use : the same factor (n n − 1 and the bottom of the fraction and cancels. So we keep the simpler n 1
Common mistake Zero-variance features break the formulaIf a feature never changes (all x i σ X = 0 divides by σ X — such as the correlation coefficient built in Section 5, or standardization σ X x i − x ˉ divide by zero , so the result is undefined , not "no relationship". Always drop constant columns before computing correlations.
Now we bring in a second feature Y y i y ˉ
Cov ( X , Y ) = n 1 ∑ i = 1 n ( x i − x ˉ ) ( y i − y ˉ ) for the same row and averages the products.
Read the figure quadrant by quadrant (the crossing lines are x ˉ y ˉ
Top-right (both above average): ( + ) ( + ) = + Bottom-left (both below average): ( − ) ( − ) = + Top-left / bottom-right (one above, one below): ( + ) ( − ) = −
So covariance is positive when points cluster on the diagonal (move together), negative when they cluster on the anti-diagonal (move oppositely), and near zero when they scatter into all four quadrants equally.
E [ ⋅ ] E [( X − μ X ) ( Y − μ Y )] E [ ⋅ ] expected value — the theoretical long-run average — and μ X E [ ⋅ ] n 1 ∑ μ X x ˉ
Why the topic needs covariance: it is the raw "co-movement" number. But its size depends on units, so we normalise it next. Stacking every pairwise covariance into a grid gives the covariance matrix, which we notate and connect to X T X
r = + 1 r = − 1 r = 0 linear pattern (points may still form a curve!).
r = 0 straight-line relationships. A perfect U-shape (parabola) can have r = 0 r = 0
The reason the value can never escape [ − 1 , 1 ] Cauchy–Schwarz inequality , which guarantees ∣ Cov ( X , Y ) ∣ ≤ σ X σ Y r
The multicollinearity half of the topic lives inside linear regression. First, one indexing convention we will lean on throughout this section.
Definition The feature indexj p j j = 1 , 2 , … , p p the number of features (columns). So X j j X 1 , X 2 , … , X p i j i rows/samples up to n j columns/features up to p
β j β ^ j β j true weight the model gives feature X j X j j j = 1 , … , p β ^ j estimate of it computed from data. The hat always means "our best guess of", not the true value.
Definition The target vectory y thing we are trying to predict — one number per row (e.g. the house price). It is a column with entries y 1 , y 2 , … , y n answer , not an input. Regression hunts for the weights β ^ y
R j 2 R j 2 coefficient of determination when we regress feature X j on all the other features (j 1 , … , p R j 2 = 1 − S S tot S S res = 1 − ∑ i = 1 n ( x ij − x ˉ j ) 2 ∑ i = 1 n ( x ij − x ^ ij ) 2 x ij j i x ^ ij other features predict for it, x ˉ j j S S res residual sum of squares (leftover error) and S S tot total sum of squares (original spread). It answers: "what fraction (0 to 1) of X j
R j 2 = 0 X j R j 2 → 1 everything → X j
This feeds the Variance Inflation Factor : VIF j = 1 − R j 2 1 R j 2 → 1 → 0 → ∞
Definition The design matrixX X T Σ design matrix X bold , meaning the whole table of n p X T transpose — the table tipped on its side so that rows become columns and columns become rows (see figure below). The product X T X p × p j k j k x ˉ j n covariance matrix Σ Σ = n 1 X T X (columns mean-centred) , Σ j k = Cov ( X j , X k ) . X T X is the covariance matrix up to the scale factor n Covariance Matrix .
Definition Condition numberκ ( X ) condition number measures how close X T X singular value of X X κ ( X ) = σ m i n ( X ) σ m a x ( X ) = λ m i n ( X T X ) λ m a x ( X T X ) , λ m a x , λ m i n eigenvalues of X T X X T X κ Condition Number .
In the figure, the transpose simply flips the table across its diagonal: the entry in row a b X b a X T
variance and std dev sigma
Multicollinearity topic 2.1.15
Each arrow means "you must understand the tail before the head makes sense." Correlation needs both variance and covariance; multicollinearity needs correlation plus the matrix picture.
Test yourself — reveal only after answering out loud.
What does the subscript i x i The row number — which sample the value comes from.
What does the summation symbol ∑ i = 1 n Add up the expression for every row from i = 1 i = n
Why do we compute x i − x ˉ It tells us, with a sign, whether a sample is above (+ −
Why do we square deviations to get variance? Raw deviations always sum to zero (they cancel); squaring makes them all positive so spread is measured, not cancelled.
Does it matter whether variance uses 1/ n 1/ ( n − 1 ) r No — the factor appears in top and bottom of the fraction and cancels; it only changes the raw variance/covariance numbers.
What happens to correlation if a feature has zero variance? It is undefined — you would divide by σ X = 0
What does σ X The typical distance of the points from the mean line — the feature's "room to vary".
When is Cov ( X , Y ) When the two features tend to be above/below their means together (points cluster on the rising diagonal).
Why divide covariance by σ X σ Y r To cancel units and bound the result to [ − 1 , 1 ]
Does r = 0 No — it only means no straight-line relationship; a curved relationship can still give r = 0
What is the difference between the indices i j i n j p
What does the hat in β ^ j "Our data-based estimate of", as opposed to the unknown true value β j
What is the target vector y The column of answers we are trying to predict — one number per row.
What is the formula for R j 2 R j 2 = 1 − S S res / S S tot X j other features can explain (0 = independent, near 1 = redundant).
How does X T X Σ With mean-centred columns, Σ = n 1 X T X n
How is the condition number κ ( X ) As
σ m a x / σ m i n , the ratio of largest to smallest singular value of
X (equivalently
λ m a x / λ m i n of
X T X ).
What does the plain X X Plain X X X T
See also: Feature Selection , Principal Component Analysis (PCA) , Ridge Regression , Lasso Regression , Linear Regression Assumptions , Variance-Bias Tradeoff , Condition Number , and the parent 2.1.15 Correlation analysis and multicollinearity (Hinglish) .
