Covariance and correlation: Question bank — AI-ML deep dive

This is a question bank of traps. Each line is a question ::: answer reveal. Cover the answer, commit to a response, then check. The goal is not arithmetic (see the worked-example decks for that) — it is to catch the conceptual slips that this topic is famous for. Parent: 1.3.09 Covariance and correlation.

True or false — justify

ρ = 0

then

X

and

Y

are independent.

False.

ρ = 0

only kills the linear trend; a curved dependence like

Y = X^{2}

can be perfectly deterministic yet have

ρ = 0

. Independence is strictly stronger than uncorrelatedness.

X

and

Y

are independent then

Cov (X, Y) = 0

True. Independence gives

E [X Y] = E [X] E [Y]

, so

Cov = E [X Y] - E [X] E [Y] = 0

. This direction always holds — only the converse fails.

A covariance of

500

means a stronger relationship than a covariance of

5

False. Covariance carries the units of

X \times Y

; rescale

X

from metres to millimetres and the number jumps by

1000

with no change in the actual relationship. Only

ρ

compares strength fairly.

ρ = - 1

means

X

and

Y

are unrelated.

False.

ρ = - 1

is the strongest possible relationship — a perfect descending line. It is

ρ = 0

, not

ρ = - 1

, that signals no linear relationship.

Correlation can exceed

1

if the variables are very strongly linked.

False. The Cauchy–Schwarz bound forces

ρ \in [- 1, 1]

for any pair; there is no "super-correlation." A reported

r > 1

always means a computational error.

Cov (X, X)

equals the standard deviation of

X

False.

Cov (X, X) = E [(X - μ_{X})^{2}] = Var (X)

— the variance, not the standard deviation. Variance is a covariance of a variable with itself.

Correlation is symmetric:

ρ_{X, Y} = ρ_{Y, X}

True. The definition multiplies

(X - μ_{X}) (Y - μ_{Y})

, and multiplication commutes, so swapping the roles changes nothing. Direction of "cause" is invisible to

ρ

ρ_{X, Y}

is high, changing

X

will change

Y

False. Correlation is a statement about co-movement in observed data, not about intervention. A lurking common cause (or pure coincidence) can produce high

ρ

with zero causal link — see 3.2.06-Multicolinearity for where this bites in models.

Adding a constant

c

to every value of

X

changes

Cov (X, Y)

False. Covariance uses deviations from the mean; shifting

X

shifts its mean by the same amount, so

(X - μ_{X})

is unchanged. Location shifts leave covariance (and

ρ

) untouched.

Multiplying

X

3

leaves the correlation

ρ_{X, Y}

unchanged (for positive scaling).

True. Both

Cov

and

σ_{X}

scale by

3

, and they cancel in the ratio. Correlation is scale-invariant, which is exactly why we normalise.

ρ_{X, Y} = 0.9

, then

90%

of points lie on the fitted line.

False.

ρ

is not a percentage of points. In regression it is

ρ^{2}

(here

0.81

) that is the fraction of variance explained — and even that is about variance, not point counts.

Spot the error

"Cov

(X, Y) = E [X Y] - E [X + Y]

The subtracted term should be the product

E [X] E [Y]

, not the expectation of the sum. The correct form is

E [X Y] - E [X] E [Y]

"Since

ρ = Cov (X, Y) / (σ_{X} σ_{Y})

and

σ_{X}, σ_{Y} \geq 0

, correlation is always non-negative."

The denominator is non-negative, but the numerator (covariance) can be negative, so

ρ

can be negative. The sign of

ρ

is inherited entirely from the sign of the covariance.

"Independence follows because we computed

Cov = 0

A zero covariance only certifies no linear association; it never certifies independence. To claim independence you must show

P (X, Y) = P (X) P (Y)

everywhere (see 1.2.04-Independence).

"For a perfect line

Y = 2 X + 1

, correlation is

2

because the slope is

2

Correlation ignores slope magnitude; any exact increasing line gives

ρ = + 1

regardless of steepness. The slope lives in the regression coefficient, not in

ρ

"Sample covariance divides by

n

The unbiased sample covariance divides by

n - 1

(Bessel's correction) because one degree of freedom was spent estimating the two means. Dividing by

n

gives a biased-low estimate.

Y = X^{2}

has zero covariance, so

E [X Y]

must also be zero for all such setups."

The zero came from

X

being symmetric about 0 (so

E [X^{3}] = 0

and

E [X] = 0

). Shift

X

off-centre and

Cov (X, X^{2})

is generally non-zero — the trick depends on symmetry, not on the squaring alone.

"A correlation matrix can have

1.2

on the diagonal if a feature is very variable."

Every diagonal entry is

ρ_{X, X} = 1

exactly, because a variable is perfectly correlated with itself. Variability changes off-diagonal entries, never the diagonal.

Why questions

Why do we divide covariance by

σ_{X} σ_{Y}

rather than by, say,

σ_{X} + σ_{Y}

Because the product of standard deviations has the same units as covariance (

X \times Y

), so the ratio is dimensionless; a sum would not cancel units and Cauchy–Schwarz would not bound the result to

[- 1, 1]

Why does independence force

E [X Y] = E [X] E [Y]

Under independence the joint splits,

P (x, y) = P (x) P (y)

, so the double sum

\sum x y P (x, y)

factors into

(\sum x P (x)) (\sum y P (y))

— the product of the two marginal means.

Why can two datasets share the same

ρ

yet look completely different?

ρ

compresses the whole cloud into one linear-trend number; it is blind to curvature, clusters, and outliers. This is exactly why you plot the data (Anscombe's quartet) instead of trusting

ρ

alone.

Why does PCA care about the covariance matrix rather than correlations by default?

PCA finds directions of maximum variance, which the covariance matrix encodes in its own units; when features have wildly different scales you standardise first — which is equivalent to using correlations. See 2.4.01-Principal-component-analysis.

Why is high correlation between two features a warning sign in linear regression?

Strongly correlated predictors carry near-duplicate information, making the fitted coefficients unstable and hard to interpret — this is multicollinearity, covered in 3.2.06-Multicolinearity and relevant to 4.1.03-Feature-selection.

Why does correlation say nothing about the steepness of a relationship?

Because both numerator and denominator scale with

Y

's spread, any rescaling of

Y

cancels out;

ρ

measures how tightly points hug a line, not how fast

Y

climbs along it.

Edge cases

What is

ρ

when

X

is a constant (zero variance)?

Undefined.

σ_{X} = 0

puts a zero in the denominator; a constant has no variation to co-vary with, so correlation is meaningless there.

What is

Cov (X, Y)

when the two variables never overlap in variation — e.g.

Y

is constant?

Zero.

(Y - μ_{Y}) = 0

for every sample, so every product term vanishes. A constant is uncorrelated with everything, and this is a genuine zero (not the "hidden nonlinear" kind).

What happens to

ρ

as data points approach a perfect line from a slightly noisy cloud?

ρ \to \pm 1

continuously; the sign matches the line's direction. The limiting value

\pm 1

is reached exactly when every point satisfies

Z_{Y} = \pm Z_{X}

(the Cauchy–Schwarz equality case).

Can covariance be negative while both variables only take positive values?

Yes. Signs come from deviations

(x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})

, not from raw values; if large-

X

pairs with small-

Y

, those deviation products are negative and can dominate.

If you compute

r

from just

n = 2

points, what do you get?

Any two distinct points lie exactly on a line, so

r = \pm 1

mechanically — a meaningless "perfect" correlation. Small samples inflate correlation; you need enough points for

r

to mean anything.

What is

ρ_{X, Y}

Y = - X

exactly?

$- 1$ . They move in perfect opposition:

Cov (X, - X) = - Var (X)

and

σ_{- X} = σ_{X}

, giving

- Var (X) / (σ_{X} σ_{X}) = - 1

Recall Quick self-check

Zero correlation rules out only linear relationships, never dependence in general. The bound $ρ \in [- 1, 1]$ comes from the Cauchy–Schwarz inequality applied to standardized variables. Correlation is invariant to shifts and positive scaling of either variable.