Correlation analysis and multicollinearity: Worked examples

This page is the drill floor for Correlation Analysis and Multicollinearity. We take every situation the topic can hand you — perfect positive, perfect negative, zero, degenerate (constant) features, a real word problem, and an exam trap — and grind each one to a checked number.

Nothing here uses a symbol you have not met. But a handful of symbols do a lot of work, so let us pin them down before the first example, in plain words.

Definition The symbols we lean on

$\overset{x}{ˉ}$ (read "x-bar") = the average of the $x$ -values. Add them up, divide by how many. It is the "balance point" of the numbers.
$\overset{y}{ˉ}$ (read "y-bar") = exactly the same idea for the $y$ -values: the average of the $Y$ list, its balance point.
$r$ = the correlation coefficient: a single number between $- 1$ and $+ 1$ that says how tightly two features move together in a straight line. $+ 1$ = lockstep up, $- 1$ = perfect see-saw, $0$ = no straight-line link.
$VIF_{j}$ = Variance Inflation Factor for feature number $j$ (the subscript $j$ just names which feature we are examining — feature 1, feature 2, ...). It says how many times more uncertain we become about that feature's effect because the other features already explain it. $1$ = perfectly clean, big = redundant. Its formula is built in Example 6 before we use it.

The figure below draws this for five dots. The dashed lavender lines are the two averages $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ ; they split the plane into four boxes. Green dots sit in an "agree" box (both above average, or both below) and add a positive product; coral dots sit in a "disagree" box and add a negative product. Correlation is the running total of these green-minus-coral contributions, then rescaled to land in $[- 1, 1]$ .

Figure — Correlation analysis and multicollinearity

The scenario matrix

Every problem this topic throws lives in one of these cells. The examples that follow are labelled with the cell they cover.

Cell	Scenario class	What makes it tricky	Covered by
A	Perfect positive ( $r = + 1$ )	Scaled copy of a feature	Ex 1
B	Perfect negative ( $r = - 1$ )	Reflected copy — sign of numerator flips	Ex 2
C	Genuinely zero ( $r = 0$ )	Numerator cancels; watch for nonlinear traps	Ex 3
D	Partial / mixed sign	Real messy data, $0<	r
E	Degenerate input	A constant feature → division by zero	Ex 5
F	Limiting behaviour	$R_{j}^{2} \to 1$ so $VIF \to \infty$	Ex 6
G	Real-world word problem	Translate business words → numbers	Ex 7
H	Exam twist	High individual $r$ but a hidden third feature; condition number	Ex 8

The master recipe (used in every numeric example)

Example 1 — Cell A: Perfect positive correlation

Worked example Statement

$X = [10, 20, 30, 40]$ , $Y = [3, 6, 9, 12]$ . Find $r_{X Y}$ .

Forecast: $Y$ is exactly $0.3 \times X$ . Before computing — guess the value of $r$ . (Write it down.)

Step 1 — averages. $\overset{x}{ˉ} = \frac{10 + 20 + 30 + 40}{4} = 25$ , $\overset{y}{ˉ} = \frac{3 + 6 + 9 + 12}{4} = 7.5$ . Why this step? Everything is measured relative to the average, so we need the balance point first.

Step 2 — deviations. $x - \overset{x}{ˉ} = [- 15, - 5, 5, 15]$ , $y - \overset{y}{ˉ} = [- 4.5, - 1.5, 1.5, 4.5]$ . Why this step? These are the "distances from balance point" — the raw material of the agreement picture.

Step 3 — the three sums. $S_{x y} = (- 15) (- 4.5) + (- 5) (- 1.5) + (5) (1.5) + (15) (4.5) = 67.5 + 7.5 + 7.5 + 67.5 = 150$ $S_{xx} = 225 + 25 + 25 + 225 = 500, S_{y y} = 20.25 + 2.25 + 2.25 + 20.25 = 45$ Why this step? $S_{x y}$ is total agreement; $S_{xx}, S_{y y}$ measure each feature's own spread so we can strip out units.

Step 4 — combine. $r = \frac{150}{500 45} = \frac{150}{22500} = \frac{150}{150} = 1.0$ . Why this step? Dividing the agreement sum by the two spreads strips out units and forces the answer into $[- 1, 1]$ ; only then is the number comparable across feature pairs.

Verify: Every deviation pair has the same sign (both negative, then both positive), so every product is positive → maximum possible agreement → $r = 1$ . Matches the forecast. Because $Y = 0.3 X$ is a straight line with positive slope, $r$ must be exactly $+ 1$ , and scaling by $0.3$ cancels in the normalization.

Example 2 — Cell B: Perfect negative correlation

Worked example Statement

$X = [1, 2, 3, 4, 5]$ , $Y = [10, 8, 6, 4, 2]$ . Find $r_{X Y}$ .

Forecast: As $X$ climbs, $Y$ falls in lockstep. Sign of $r$ ? Magnitude?

Step 1 — averages. $\overset{x}{ˉ} = 3$ , $\overset{y}{ˉ} = 6$ .

Step 2 — deviations. $x - \overset{x}{ˉ} = [- 2, - 1, 0, 1, 2]$ , $y - \overset{y}{ˉ} = [4, 2, 0, - 2, - 4]$ . Why this step? Notice they are mirror images — where $X$ is below average, $Y$ is above.

Step 3 — sums. $S_{x y} = (- 2) (4) + (- 1) (2) + 0 + (1) (- 2) + (2) (- 4) = - 8 - 2 + 0 - 2 - 8 = - 20$ $S_{xx} = 4 + 1 + 0 + 1 + 4 = 10, S_{y y} = 16 + 4 + 0 + 4 + 16 = 40$ Why this step? Now every agreement product is negative (opposite signs), so $S_{x y}$ is negative — the see-saw signature.

Step 4 — combine. $r = \frac{- 20}{10 40} = \frac{- 20}{400} = \frac{- 20}{20} = - 1.0$ . Why this step? The same normalization as before; here it delivers exactly $- 1$ because the negative agreement is as strong as it can possibly get.

Verify: $Y = 12 - 2 X$ (check: $X = 1 \to 10$ , ✓). A negative-slope straight line forces $r = - 1$ . The only difference from Ex 1 is the sign of the numerator — the picture is the same triangle, reflected.

The figure below shows the two extremes side by side: on the left the perfect up-slope ( $r = + 1$ ), on the right the perfect down-slope ( $r = - 1$ ). Every dot lies exactly on the lavender line, which is what "perfect" means — no wobble at all.

Example 3 — Cell C: Genuine zero (and the nonlinear trap)

Worked example Statement

$X = [- 2, - 1, 0, 1, 2]$ , $Y = [4, 1, 0, 1, 4]$ (that is $Y = X^{2}$ ). Find $r_{X Y}$ .

Forecast: $Y$ clearly depends on $X$ — but does linear correlation see it? Guess $r$ .

Step 1 — averages. $\overset{x}{ˉ} = 0$ , $\overset{y}{ˉ} = \frac{4 + 1 + 0 + 1 + 4}{5} = 2$ .

Step 2 — deviations. $x - \overset{x}{ˉ} = [- 2, - 1, 0, 1, 2]$ , $y - \overset{y}{ˉ} = [2, - 1, - 2, - 1, 2]$ .

Step 3 — numerator. $S_{x y} = (- 2) (2) + (- 1) (- 1) + (0) (- 2) + (1) (- 1) + (2) (2) = - 4 + 1 + 0 - 1 + 4 = 0$ Why this step? The left half contributes $- 4 + 1 = - 3$ ; the right half contributes $- 1 + 4 = + 3$ . They cancel by symmetry.

Step 4 — combine. $r = \frac{0}{10 \cdot S _{y y}} = 0$ . Why this step? Once the numerator is $0$ , the normalization cannot change it — any nonzero spread divided into zero stays zero, so $r = 0$ regardless of $S_{y y}$ .

Verify (the trap): $r = 0$ does not mean "unrelated" — here $Y$ is perfectly determined by $X$ ! Pearson $r$ only detects straight-line links. This is the single most tested misconception. See Linear Regression Assumptions for why we still care about linearity.

Example 4 — Cell D: Partial / mixed correlation

Worked example Statement

$X = [1, 2, 3, 4, 5]$ , $Y = [2, 1, 4, 3, 5]$ . Find $r_{X Y}$ .

Forecast: $Y$ roughly rises with $X$ but wobbles. Guess $r$ between $0$ and $1$ .

Step 1 — averages. $\overset{x}{ˉ} = 3$ , $\overset{y}{ˉ} = \frac{2 + 1 + 4 + 3 + 5}{5} = 3$ .

Step 2 — deviations. $x - \overset{x}{ˉ} = [- 2, - 1, 0, 1, 2]$ , $y - \overset{y}{ˉ} = [- 1, - 2, 1, 0, 2]$ .

Step 3 — sums. $S_{x y} = (- 2) (- 1) + (- 1) (- 2) + (0) (1) + (1) (0) + (2) (2) = 2 + 2 + 0 + 0 + 4 = 8$ $S_{xx} = 10, S_{y y} = 1 + 4 + 1 + 0 + 4 = 10$

Step 4 — combine. $r = \frac{8}{10 10} = \frac{8}{10} = 0.8$ . Why this step? Normalizing turns the raw agreement $8$ into a comparable score; because the two spreads are equal ( $S_{xx} = S_{y y} = 10$ ) the denominator is simply $10$ , landing $r$ at $0.8$ — strong but not perfect.

Verify: $0 < 0.8 < 1$ ✓ — a strong-but-imperfect positive link. The " $4$ at $X = 3$ then $3$ at $X = 4$ " wobble is exactly what stops it reaching $1$ . Sanity: $∣ r ∣ \leq 1$ always holds (Cauchy–Schwarz). This is the everyday case in real datasets.

Example 5 — Cell E: Degenerate input (a constant feature)

Worked example Statement

$X = [7, 7, 7, 7]$ (a constant column), $Y = [1, 2, 3, 4]$ . Find $r_{X Y}$ .

Forecast: $X$ never changes. Can "how they move together" even be defined? Guess.

Step 1 — averages. $\overset{x}{ˉ} = 7$ , $\overset{y}{ˉ} = 2.5$ .

Step 2 — deviations. $x - \overset{x}{ˉ} = [0, 0, 0, 0]$ . Every deviation is zero. Why this step? If $X$ never leaves its average, it has no "distance from balance point" to contribute — the raw material of correlation is missing.

Step 3 — sums. $S_{xx} = 0 + 0 + 0 + 0 = 0$ , and $S_{x y} = 0$ too.

Step 4 — combine. $r = \frac{0}{0 \cdot S _{y y}} = \frac{0}{0}$ — undefined. Why this step? $σ_{X} = 0$ , so we are dividing by zero. Correlation asks "when $X$ is above its average, what does $Y$ do?" — but $X$ is never above its average. The question is meaningless.

Verify: A constant feature carries no information and must be dropped before any correlation or regression. In code, libraries return NaN here. This is the degenerate/zero-variance case — always screen for it first. Related: Feature Selection removes such columns.

Example 6 — Cell F: Limiting behaviour of VIF

Before the numbers, we owe two definitions this example depends on.

Worked example Statement

Feature $X_{3}$ is regressed on the other features, giving $R_{3}^{2} = 0.98$ . Then a data update pushes it to $R_{3}^{2} = 0.999$ . Track $VIF_{3}$ .

Forecast: As $R_{3}^{2} \to 1$ , does VIF stay finite? Guess the two values.

Step 1 — VIF at $R_{3}^{2} = 0.98$ . $VIF_{3} = \frac{1}{1 - 0.98} = \frac{1}{0.02} = 50$ Why this step? $1 - R_{3}^{2}$ is the "fraction of $X_{3}$ that is still its own." When only $2%$ is unique, the variance is inflated $50 \times$ .

Step 2 — VIF at $R_{3}^{2} = 0.999$ . $VIF_{3} = \frac{1}{1 - 0.999} = \frac{1}{0.001} = 1000$ Why this step? We shrank the unique slice from $2%$ to $0.1%$ — a $20 \times$ smaller denominator — so VIF jumps $20 \times$ , from $50$ to $1000$ .

Step 3 — the limit. As $R_{3}^{2} \to 1^{-}$ , $1 - R_{3}^{2} \to 0^{+}$ , so $VIF_{3} \to + \infty$ . Why this step? Perfect collinearity means $X_{3}$ is a linear combo of the others — the design matrix becomes singular and the coefficient's variance literally explodes.

Verify: Standard-error inflation is $VIF$ : at $R_{3}^{2} = 0.98$ that is $50 \approx 7.07 \times$ ; at $0.999$ it is $1000 \approx 31.6 \times$ . Both exceed the $VIF > 10$ "severe" threshold. Fixes: drop the feature, or shrink with Ridge Regression / Lasso Regression. See Variance-Bias Tradeoff.

The figure plots the VIF curve. Notice how flat it is on the left (VIF barely above $1$ for $R_{j}^{2}$ up to about $0.7$ ) and how it rockets upward near the right wall $R_{j}^{2} = 1$ — the coral line marks the $VIF = 10$ "severe" cutoff, and both green dots sit far above it.

Example 7 — Cell G: Real-world word problem

Worked example Statement

A café tracks 6 days: temperature (°C) $T = [10, 15, 20, 25, 30, 35]$ and iced-coffee sales (cups) $S = [20, 35, 50, 65, 80, 95]$ . The owner asks: "Is heat driving sales, and how strongly?"

Forecast: Every $+ 5$ °C adds exactly $+ 15$ cups. Guess $r$ .

Step 1 — averages. $\overset{ˉ}{T} = \frac{10 + 15 + 20 + 25 + 30 + 35}{6} = 22.5$ , $\overset{ˉ}{S} = \frac{20 + 35 + 50 + 65 + 80 + 95}{6} = 57.5$ .

Step 2 — deviations. $T - \overset{ˉ}{T} = [- 12.5, - 7.5, - 2.5, 2.5, 7.5, 12.5]$ , $S - \overset{ˉ}{S} = [- 37.5, - 22.5, - 7.5, 7.5, 22.5, 37.5]$ . Why this step? Notice $S$ -deviations $= 3 \times T$ -deviations exactly — the tell-tale of a perfect line.

Step 3 — sums. $S_{T S} = (- 12.5) (- 37.5) + (- 7.5) (- 22.5) + (- 2.5) (- 7.5) + (2.5) (7.5) + (7.5) (22.5) + (12.5) (37.5)$ $= 468.75 + 168.75 + 18.75 + 18.75 + 168.75 + 468.75 = 1312.5$ $S_{T T} = 156.25 + 56.25 + 6.25 + 6.25 + 56.25 + 156.25 = 437.5, S_{S S} = 3^{2} \cdot S_{T T} = 9 \cdot 437.5 = 3937.5$

Step 4 — combine. $r = \frac{1312.5}{437.5 3937.5} = \frac{1312.5}{1722656.25} = \frac{1312.5}{1312.5} = 1.0$ . Why this step? Because every $S$ -deviation is exactly $3 \times$ the matching $T$ -deviation, the denominator collapses to the same $1312.5$ as the numerator, forcing $r = 1$ — the signature of a perfect straight line whatever the slope.

Verify: $S = 3 T - 10$ (check $T = 10 \to 20$ ✓). Perfect line ⇒ $r = 1$ . Business caution: correlation is not causation — a heatwave could also lengthen café opening hours. But statistically heat and sales move in perfect lockstep here.

Example 8 — Cell H: Exam twist (hidden collinearity + condition number)

This example uses two new words. Let us earn them first.

Definition Singular values and the condition number

$κ$ Think of the data matrix $X$ as stretching space along a few perpendicular directions. The singular values $σ_{m a x}$ and $σ_{m i n}$ are the biggest and smallest stretch factors it applies — how long the longest and shortest axes of the stretched cloud are. If some direction gets squashed to nothing, its stretch factor is $σ_{m i n} = 0$ .

The condition number (Greek letter $κ$ , "kappa") is their ratio: $κ = \frac{σ _{m a x}}{σ _{m i n}} .$ Why this ratio? It measures how lopsided the stretch is. $κ \approx 1$ means a nicely round, well-behaved data cloud; a huge $κ$ means one direction is almost flat — tiny input errors get amplified enormously when we invert. If $σ_{m i n} = 0$ the cloud is truly flat in some direction and $κ = \infty$ : the matrix is singular (non-invertible).

Worked example Statement

Three features: $X_{1} = [1, 2, 3, 4]$ , $X_{2} = [2, 4, 6, 8]$ , and $X_{3} = X_{1} + X_{2} = [3, 6, 9, 12]$ . Individually $r_{12} = 1$ . A student says "just drop $X_{2}$ and we're fine." Trap: is the remaining ${X_{1}, X_{3}}$ set clean? What does the condition number say?

Forecast: Guess whether $X_{3}$ adds new information after dropping $X_{2}$ .

Step 1 — check $r_{13}$ . $\overset{x}{ˉ}_{1} = 2.5$ , $\overset{x}{ˉ}_{3} = 7.5$ ; deviations $[- 1.5, - 0.5, 0.5, 1.5]$ and $[- 4.5, - 1.5, 1.5, 4.5]$ . $S_{13} = (- 1.5) (- 4.5) + (- 0.5) (- 1.5) + (0.5) (1.5) + (1.5) (4.5) = 6.75 + 0.75 + 0.75 + 6.75 = 15$ $S_{11} = 2.25 + 0.25 + 0.25 + 2.25 = 5, S_{33} = 20.25 + 2.25 + 2.25 + 20.25 = 45$ $r_{13} = \frac{15}{5 45} = \frac{15}{225} = \frac{15}{15} = 1.0$ Why this step? Because $X_{3} = 3 X_{1}$ (once $X_{2} = 2 X_{1}$ is substituted), $X_{3}$ is still a scaled copy — dropping $X_{2}$ did not remove the collinearity.

Step 2 — condition number of ${X_{1}, X_{3}}$ . Since $X_{3} = 3 X_{1}$ , the two columns are linearly dependent, so the smaller singular value $σ_{m i n} = 0$ , giving $κ = \frac{σ _{m a x}}{σ _{m i n}} = \frac{σ _{m a x}}{0} = \infty.$ Why this step? A zero smallest singular value = a flat, squashed column space — $X^{T} X$ is singular and cannot be inverted. See Condition Number and Covariance Matrix.

Step 3 — the real fix. Keep exactly one of ${X_{1}, X_{2}, X_{3}}$ . They span a single direction; two of them are redundant.

Verify: $r_{13} = 1$ ✓ confirms the student's fix failed. $κ = \infty$ ✓ confirms singularity. Sanity in coefficient space: if $X_{1}, X_{3}$ both enter, infinitely many $(β_{1}, β_{3})$ pairs give the same prediction (since $β_{1} X_{1} + β_{3} \cdot 3 X_{1} = (β_{1} + 3 β_{3}) X_{1}$ ) — the classic un-identifiability of perfect multicollinearity. Principal Component Analysis (PCA) would collapse these into one component automatically.

The figure draws $X_{1}$ and $X_{3}$ as arrows: they point in exactly the same direction (just different lengths), so the data cloud is a flat line — the visual meaning of $κ = \infty$ .

Recall

Recall Which cell of the matrix does each result live in?

$r = + 1$ from a scaled copy ( $Y = 0.3 X$ ) ::: Cell A (perfect positive) $r = - 1$ from a reflected feature ( $Y = 12 - 2 X$ ) ::: Cell B (perfect negative) $r = 0$ but $Y = X^{2}$ (nonlinear) ::: Cell C (linear-only blindness) $r = 0.8$ , a strong-but-imperfect real link ::: Cell D (partial / mixed) A constant column giving undefined $r$ ::: Cell E (degenerate, $σ = 0$ ) $VIF \to \infty$ as $R_{j}^{2} \to 1$ ::: Cell F (limiting behaviour) A perfect café temperature-vs-sales line ( $r = 1$ ) ::: Cell G (real-world word problem) Dropping one collinear feature still leaves $r = 1$ , $κ = \infty$ ::: Cell H (hidden collinearity)

Recall What does

$VIF$ mean physically? The factor by which a coefficient's standard error is inflated ::: e.g. $VIF = 6.67 \Rightarrow 6.67 \approx 2.58 \times$ less certain about that effect.

Recall Why does dropping one of three perfectly collinear features not always fix multicollinearity?

Because the survivor may still be a scaled copy of the one you kept ::: e.g. $X_{3} = 3 X_{1}$ , so $r = 1$ and $κ = \infty$ persist — you must keep exactly one.