Correlation analysis and multicollinearity: Visual walkthrough

This page rebuilds the central idea of the parent topic from absolute zero, in pictures. We start with two columns of numbers and end with the reason a model's coefficients go haywire when features march in step. Every step shows you what we do, why we do it, and what it looks like.

We will use only these plain ideas, each built before it is used:

a feature = one column of numbers (one measurement per sample);
the mean $\overset{x}{ˉ}$ = the balance point of a column;
and later, a vector = an arrow whose length and direction we can see.

Throughout, $n$ is the number of samples (rows) and, once we have more than two features, $p$ is the number of features (columns), indexed $j = 1, 2, \dots, p$ . So $X_{j}$ means "the $j$ -th feature column," and $VIF_{j}$ , $R_{j}^{2}$ are that column's numbers.

Step 1 — Two columns of numbers, and their balance points

WHAT. Put two features side by side. Feature $X$ and feature $Y$ each have $n$ samples: $x_{1}, \dots, x_{n}$ and $y_{1}, \dots, y_{n}$ . First we mark the mean of each — the number where the column would balance like a seesaw: $\overset{x}{ˉ} = add them up, divide by how many \frac{1}{n} i = 1 \sum n x_{i}$

Here $\sum$ (the tall Greek "S") just means "add up all of these," and $n$ is how many samples we have.

WHY. Everything about relationship is about movement away from the usual value. A number by itself (" $x_{5} = 4$ ") tells us nothing until we know whether $4$ is above or below normal. The mean is our "normal."

PICTURE. Look at the two number lines below. The orange dots are the $X$ values; the teal dots are the $Y$ values. The dashed vertical line on each is the mean. Notice which dots sit to the right of the balance line (above average) and which sit to the left (below average).

Figure — Correlation analysis and multicollinearity

Step 2 — The product of deviations: agreement vs. disagreement

WHAT. For each sample $i$ , multiply the two deviations together: $p_{i} = is X high or low? (x_{i} - \overset{x}{ˉ}) \cdot is Y high or low? (y_{i} - \overset{y}{ˉ})$

WHY. We want a single number per sample that says "did $X$ and $Y$ agree here?" Multiplication of two signs does exactly that — think of the sign rule you already know:

$x_{i} - \overset{x}{ˉ}$	$y_{i} - \overset{y}{ˉ}$	product $p_{i}$	meaning
$+$ (high)	$+$ (high)	$+$	agree (both above)
$-$ (low)	$-$ (low)	$+$	agree (both below)
$+$ (high)	$-$ (low)	$-$	disagree
$-$ (low)	$+$ (high)	$-$	disagree

So a positive product means the two features moved the same way at that sample; a negative product means they moved opposite ways. If either is exactly average, the product is $0$ — that sample casts no vote.

PICTURE. Each sample is a point on an $X$ – $Y$ grid. The two mean lines split the grid into four quadrants. Green shading = agreement quadrants (product positive); red shading = disagreement quadrants (product negative). A point far into a corner (big $∣ d^{X} ∣$ and big $∣ d^{Y} ∣$ ) votes loudly; a point near the crossing votes softly.

Step 3 — Average the votes: covariance

WHAT. Add all the per-sample votes $p_{i}$ and divide by the count. That average is the covariance: $Cov (X, Y) = \frac{1}{n} \sum_{i = 1}^{n} vote from sample i (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$

WHY. One sample can be a fluke. Averaging over all $n$ samples gives the typical co-movement: is the crowd mostly voting "agree" (positive), "disagree" (negative), or split (near zero)? This is the covariance between two features.

PICTURE. We collapse all the green/red votes into a single tilted cloud. A cloud that leans up-to-the-right has positive covariance (agreement wins). A cloud leaning up-to-the-left has negative covariance. A round, tilt-free cloud averages to zero.

Step 4 — Normalise the units away: the correlation coefficient

WHAT. Divide covariance by how much each feature spreads on its own. The spread of a feature is its standard deviation $σ$ — the typical distance of its values from their mean. We define one for $X$ and one for $Y$ : $σ_{X} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}, σ_{Y} = \frac{1}{n} \sum_{i = 1}^{n} (y_{i} - \overset{y}{ˉ})^{2}$ Each is "root-mean-square deviation": square every deviation (so signs cannot cancel), average them, take the square root to return to the original units. With both spreads in hand, the Pearson correlation coefficient $r$ is: $r_{X Y} = \frac{Cov ( X , Y ) how they move together}{X ’s own spread σ _{X} Y ’s own spread σ _{Y}}$

WHY this specific division? Covariance has units of $(X -units) \times (Y -units)$ . The spread $σ_{X}$ carries the $X$ -units and $σ_{Y}$ carries the $Y$ -units. Dividing by $σ_{X} σ_{Y}$ cancels both units exactly, leaving a pure number. And that pure number is trapped inside $[- 1, 1]$ — no matter the data — by the Cauchy–Schwarz inequality. So doubling units, changing dollars to cents, none of it moves $r$ .

PICTURE. The same cloud, but now we rescale each axis by its own $σ$ (turning each feature into "how many spreads away from average"). In these fair units, $r$ is simply how flat the cloud has been squashed onto a line: a perfectly thin line means $r = \pm 1$ ; a round blob means $r = 0$ .

Step 5 — Every case of $r$ , corner to corner

WHAT. We now walk through all the values $r$ can take, so you never meet a cloud you cannot read.

WHY. A tool you only understand at one value ( $r = 1$ ) is a trap. The judge of a good picture is: can you name the value for any cloud shown to you? Here is the whole range.

PICTURE. Five clouds, left to right: $r = - 1$ , $r = - 0.5$ , $r = 0$ , $r = + 0.5$ , $r = + 1$ . Read the slope and tightness of each.

$r = + 1$ — perfect up line. $Y$ is exactly $a X + b$ with $a > 0$ . Redundant feature.
$0 < r < 1$ — leans up, some scatter. Partial overlap of information.
$r = 0$ — no linear tilt. Careful: this only means no straight-line relationship; a perfect U-shape (parabola) also gives $r = 0$ while being fully dependent. $r$ is blind to curves.
$- 1 < r < 0$ — leans down. Opposite movers.
$r = - 1$ — perfect down line. Still redundant: $X_{3} = 6 - X_{1}$ from the parent's example gives exactly $- 1$ .

Step 6 — Now the danger: two features that are (almost) the same arrow

WHAT. Treat each feature-column as a vector — an arrow in a space with one axis per sample (so with $n$ samples the arrow lives in $n$ -dimensional space; we draw a flattened 2D cartoon of it). Two features with $r$ near $\pm 1$ are two arrows pointing nearly the same way (or exactly opposite). This is the situation the parent calls multicollinearity.

WHY. A model must split "credit" between the two features: how much of the target does each explain? If the arrows point in genuinely different directions, the split is clean. If the arrows nearly coincide, the model cannot tell them apart — it can shovel a giant positive coefficient onto one and a giant negative onto the other, and their near-identical arrows almost cancel, leaving the prediction unchanged. Many wildly different coefficient pairs give nearly the same fit.

PICTURE. Two near-parallel arrows $X_{1}$ and $X_{2}$ . The target arrow $y$ is decomposed onto them. Watch the tiny sliver of angle between $X_{1}$ and $X_{2}$ : as it shrinks, the coefficients needed to reach $y$ blow up.

Step 7 — From two arrows to many: the leftover piece of a feature

WHAT. With more than two features we cannot draw the picture, but the idea extends cleanly. Ask of feature $X_{j}$ (one of the $p$ feature columns): how much of it is genuinely new, versus rebuildable from the other $p - 1$ features? Line up all the other feature-arrows and find the best combination of them that approximates $X_{j}$ — that is exactly "regress $X_{j}$ on the others." What is left over is the part of $X_{j}$ perpendicular to all of them: its unique direction.

WHY. In Step 6 with two features, the "leftover" was just the sliver of angle between $X_{1}$ and $X_{2}$ . With many features the same sliver becomes the perpendicular distance from $X_{j}$ to the flat space spanned by the others. The smaller that leftover, the more $X_{j}$ is a near-copy of the pack — and the more the model's credit-splitting wobbles, exactly as before.

PICTURE. Feature-arrow $X_{j}$ split into two pieces: a shadow lying in the plane of the other features (the rebuildable part) and a short perpendicular stub sticking out (the unique part). As the stub shrinks, instability grows.

Step 8 — Putting a number on the instability: VIF and the condition number

WHAT. Two matching numbers measure how bad the near-parallel situation is.

The Variance Inflation Factor for feature $X_{j}$ : $VIF_{j} = \frac{1}{1 - R _{j}^{2}}$

WHY does the formula look like this? From Step 7, $1 - R_{j}^{2}$ is the fraction of $X_{j}$ that is unique — the length² of the perpendicular stub relative to the whole arrow. The regression coefficient for $X_{j}$ is determined only by that stub, because the shadow part is shared with the other features and carries no information they don't already have. A short stub means a small effective "lever," and the variance of an estimated coefficient scales with $1/ (stub length^{2})$ . Writing the stub fraction as $1 - R_{j}^{2}$ gives directly $Var (\hat{β}_{j}) = baseline Var_{if X_{j} were alone} \times = VIF_{j} \frac{1}{1 - R _{j}^{2}} .$ So VIF is literally "one over the fraction of $X_{j}$ that is unique." When the stub vanishes ( $R_{j}^{2} \to 1$ ), the lever length $\to 0$ and $VIF_{j} \to \infty$ : the variance explodes — the numeric face of the knife-edge in Step 6.

Now the design matrix $X$ : stack your $n$ samples as rows and your $p$ features as columns, so $X$ is the whole data table as one matrix. Its singular values $σ_{m a x} (X)$ and $σ_{m i n} (X)$ are the lengths of the longest and shortest directions the data cloud stretches (found by the SVD — think "the axes of the ellipse that hugs the cloud"). The condition number is their ratio: $κ (X) = \frac{σ _{m a x} ( X )}{σ _{m i n} ( X )}$ A pancake-flat cloud (features nearly parallel) has a tiny $σ_{m i n} (X)$ , so the ratio explodes.

WHY does a big $κ$ mean unstable coefficients? Fitting a linear model means solving $X^{⊤} X \hat{β} = X^{⊤} y$ — undoing $X$ to recover the coefficients. The condition number is precisely how much undoing $X$ magnifies errors: if $κ (X) = 1000$ , then a $0.1%$ wobble in the data $y$ can swing $\hat{β}$ by up to $100%$ . Geometrically, the tiny $σ_{m i n} (X)$ direction is the near-parallel sliver from Step 6 seen for the whole feature set at once: the model must divide by that sliver, and dividing by something tiny blows the answer up. So VIF and $κ$ are two views of one fact — small angle $\Rightarrow$ short stub $\Rightarrow$ high $R_{j}^{2}$ $\Rightarrow$ high VIF $\Rightarrow$ thin cloud $\Rightarrow$ tiny $σ_{m i n}$ $\Rightarrow$ high $κ$ . (See Condition Number for the full error-amplification bound.)

PICTURE. A dial: as $R_{j}^{2}$ climbs from $0$ toward $1$ , the VIF curve rockets upward — flat and safe until about $0.8$ , then a cliff. The thresholds $VIF = 5$ and $VIF = 10$ are marked; alongside it, a thin vs. round data cloud shows the matching $κ$ .

Recall

VIF meaning of 6.67 ::: The standard error of that coefficient is inflated by $6.67 \approx 2.58 \times$ compared to an uncorrelated feature. $R_{j}^{2} = 0.85$ gives what VIF? ::: $1/ (1 - 0.85) = 1/0.15 \approx 6.67$ . Why does a thin data cloud give a huge condition number? ::: $σ_{m i n} (X)$ (the cloud's thinnest direction) becomes tiny, so $σ_{m a x} (X) / σ_{m i n} (X)$ blows up. What happens to VIF and $κ$ for a constant (zero-variance) feature? ::: Both blow up / become undefined — $σ_{m i n} (X) = 0$ makes $κ = \infty$ ; drop the column first.

Step 9 — What to actually do about it

WHAT. Once VIF or $κ$ flags trouble, you have three routes:

Drop the redundant feature — Feature Selection (keep one of $X_{1}, X_{2}, X_{3}$ from the parent's example).
Combine them into new uncorrelated directions — Principal Component Analysis (PCA) rotates the axes to line up with $σ_{m a x} (X)$ and $σ_{m i n} (X)$ , so the new features are orthogonal by construction.
Shrink the coefficients so they cannot explode — Ridge Regression (penalises large coefficients, stabilising the near-singular inverse) or Lasso Regression (which can zero features out entirely). This trades a little bias for a lot less variance — the Variance-Bias Tradeoff.

WHY. The instability came from near-parallel arrows. You either remove an arrow, rotate to a perpendicular set, or forbid the giant opposing coefficients. Each attacks the exact mechanism of Step 6. These fixes rest on the Linear Regression Assumptions holding otherwise.

PICTURE. Two mini-panels: (left) PCA rotating the tilted cloud onto clean perpendicular axes; (right) Ridge pulling a runaway coefficient pair back toward the origin.

The one-picture summary

Everything on this page is one chain: deviations → signed product → averaged covariance → unit-free $r$ → arrow angle → leftover stub → VIF/ $κ$ → fix. This final figure compresses that whole journey into a single flow.

Recall Feynman retelling — say it back in plain words

We started with two columns of numbers and marked each column's usual value, its mean. For every sample we asked two yes/no-ish questions — "is $X$ above usual? is $Y$ above usual?" — and multiplied the answers so that agreement scores positive and disagreement scores negative. Averaging those scores over all samples gave covariance: the crowd's verdict on whether the two features move together. But covariance is polluted by units, so we divided by each feature's own spread; the units cancelled and out popped $r$ , a pure number trapped between $- 1$ and $+ 1$ that reads off as "how flat the cloud has been squashed onto a line" — and, equivalently, as the cosine of the angle between the two feature-arrows. (If a feature never changes, its spread is zero and $r$ is undefined — a constant column has to be thrown out first.) Then we noticed the villain: when $r$ hugs $\pm 1$ , the two feature-columns are nearly the same arrow. A model trying to split credit between two near-identical arrows can throw a giant positive coefficient on one and a giant negative on the other — they almost cancel, the fit barely changes, so the coefficients wobble wildly with any data nudge. To handle more than two features we asked how much of each of the $p$ features is genuinely new — the perpendicular stub left after the others rebuild it — and called the rebuilt fraction $R_{j}^{2}$ . We then measured the wobble two ways: VIF $= 1/ (1 - R_{j}^{2})$ , which is one over the unique fraction of a feature, and the condition number $κ (X) = σ_{m a x} (X) / σ_{m i n} (X)$ , which is how pancake-flat the whole data cloud is — and they tell the same story twice, because solving for the coefficients means dividing by that thinnest direction. Finally we fixed it by dropping a redundant arrow, rotating to perpendicular axes with PCA, or forbidding the giant coefficients with Ridge and Lasso. One idea, start to finish: correlation is agreement-scoring made unit-free, and multicollinearity is what happens when two features are secretly the same arrow.