Covariance and correlation: Exercises — AI-ML deep dive

This page is your self-test dojo for the parent topic. Work each problem before opening the solution. Every symbol below was built in the parent note — if a piece feels shaky, revisit 1.3.05-Expectation-and-variance (means, variance) and 1.3.08-Joint-and-marginal-distributions (joint tables) first.

Quick reminder of the tools we keep reusing (all proven in the parent):

Level 1 — Recognition

L1.1 — Sign of a relationship

A scatter cloud of points slopes downhill from top-left to bottom-right (as $X$ grows, $Y$ shrinks). Is $Cov (X, Y)$ positive, negative, or zero?

Recall Solution

What we look at: the direction of the trend, not its steepness. Downhill means: when $X$ is above its mean, $Y$ tends to be below its mean. So the product $(x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$ is (positive)×(negative) = negative for most points. Averaging many negative products gives a negative number. Answer: negative covariance. The figure below makes all three signs visible side by side.

The figure below shows three scatter clouds. Left (pink): negative covariance — the pink points and the pink arrow slope downhill, matching this problem. Middle (blue): near-zero covariance — a shapeless cloud with no tilt. Right (yellow): positive covariance — points climb uphill. Compare the tilt direction, not the spread, to read the sign.

L1.2 — Which quantity is unit-free?

You measure covariance and correlation between height (metres) and weight (kg). One of these numbers would change if you switched height to centimetres. Which one stays the same?

Recall Solution

Covariance carries units of $X \times Y$ (metre·kg). Switch metres→centimetres and every $x_{i} - \overset{x}{ˉ}$ gets multiplied by $100$ , so covariance scales by $100$ . Correlation divides by $σ_{X} σ_{Y}$ , and $σ_{X}$ also scales by $100$ . The two 100's cancel. Answer: correlation $ρ$ is unchanged (dimensionless). Covariance changes.

Level 2 — Application

L2.1 — Covariance from a joint table

$X$	$Y$	$P (X, Y)$
0	1	0.3
0	3	0.2
2	1	0.1
2	3	0.4

Compute $Cov (X, Y)$ .

Recall Solution

Step 1 — marginal means. Group the probabilities by each value. $E [X] = 0 (0.3 + 0.2) + 2 (0.1 + 0.4) = 0 + 2 (0.5) = 1$ $E [Y] = 1 (0.3 + 0.1) + 3 (0.2 + 0.4) = 1 (0.4) + 3 (0.6) = 0.4 + 1.8 = 2.2$ Why: a marginal mean sums a variable's value against all joint rows containing it.

Step 2 — the cross term $E [X Y]$ . Multiply $x \cdot y$ by its probability, then sum. $E [X Y] = (0) (0.3) + (0) (0.2) + (2 \cdot 1) (0.1) + (2 \cdot 3) (0.4) = 0 + 0 + 0.2 + 2.4 = 2.6$

Step 3 — assemble. $Cov (X, Y) = 2.6 - (1) (2.2) = 2.6 - 2.2 = 0.4$ Interpretation: positive — $X$ and $Y$ lean upward together.

L2.2 — Sample correlation from four points

Data: $(1, 3), (2, 3), (3, 5), (4, 5)$ . Find the sample correlation $r$ .

Recall Solution

Step 1 — means. $\overset{x}{ˉ} = \frac{1 + 2 + 3 + 4}{4} = 2.5$ , $\overset{y}{ˉ} = \frac{3 + 3 + 5 + 5}{4} = 4$ .

Step 2 — deviations and products.

$x_{i}$	$y_{i}$	$x_{i} - \overset{x}{ˉ}$	$y_{i} - \overset{y}{ˉ}$	product	$(x_{i} - \overset{x}{ˉ})^{2}$	$(y_{i} - \overset{y}{ˉ})^{2}$
1	3	$- 1.5$	$- 1$	$1.5$	$2.25$	$1$
2	3	$- 0.5$	$- 1$	$0.5$	$0.25$	$1$
3	5	$0.5$	$1$	$0.5$	$0.25$	$1$
4	5	$1.5$	$1$	$1.5$	$2.25$	$1$

Sums: $\sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ}) = 4$ , $\sum (x_{i} - \overset{x}{ˉ})^{2} = 5$ , $\sum (y_{i} - \overset{y}{ˉ})^{2} = 4$ .

Step 3 — why the $\frac{1}{n - 1}$ disappears. The full sample formula would divide the numerator sum by $n - 1$ and each variance sum by $n - 1$ too. Writing $c = \frac{1}{n - 1}$ : $r = \frac{c \sum ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{c \sum ( x _{i} - x ˉ ) ^{2} c \sum ( y _{i} - y ˉ ) ^{2}} = \frac{c \cdot ( num sum )}{c c \cdot ( denom )} = \frac{c}{c} \cdot \frac{num sum}{denom}$ The $c \cdot c = c$ in the denominator exactly cancels the $c$ on top. So $r$ depends only on the raw sums: $r = \frac{4}{5 4} = \frac{4}{2 5} = \frac{2}{5} \approx 0.894$ Interpretation: strong positive — but not perfect, because $Y$ stays flat while $X$ moves inside each pair.

Level 3 — Analysis

L3.1 — Rescaling and shifting

You already found $Cov (X, Y) = 0.4$ in L2.1. Now define $U = 3 X + 7$ and $V = - 2 Y + 1$ . What is $Cov (U, V)$ , and what is $ρ_{U, V}$ compared with $ρ_{X, Y}$ ?

Recall Solution

Rule (from linearity): additive constants shift the mean but not the deviations, so they drop out. Multiplicative constants pull straight out: $Cov (a X + b, c Y + d) = a c Cov (X, Y)$ Apply: $a = 3, c = - 2$ : $Cov (U, V) = (3) (- 2) (0.4) = - 2.4$ Correlation: standard deviations scale by $∣ a ∣$ and $∣ c ∣$ : $ρ_{U, V} = \frac{a c Cov ( X , Y )}{∣ a ∣ σ _{X} ∣ c ∣ σ _{Y}} = \frac{a c}{∣ a ∣∣ c ∣} ρ_{X, Y} = sign (a c) ρ_{X, Y}$ Here $a c < 0$ , so $ρ_{U, V} = - ρ_{X, Y}$ . The +7 and +1 do nothing; only the sign flips.

L3.2 — Reading a correlation off a line

Points $(0, 1), (1, 3), (2, 5), (3, 7)$ lie exactly on the line $y = 2 x + 1$ . Without a table, state $r$ and justify.

Recall Solution

Key fact: correlation $= \pm 1$ exactly when all points sit on one straight line. The sign matches the slope's sign. Here the slope $+ 2 > 0$ , so $r = + 1$ .

Heuristic proof (why a perfect line forces $∣ r ∣ = 1$ ): write $y_{i} = m x_{i} + b$ with $m > 0$ . Then $\overset{y}{ˉ} = m \overset{x}{ˉ} + b$ , so each $y$ -deviation is a scaled copy of the $x$ -deviation: $y_{i} - \overset{y}{ˉ} = m (x_{i} - \overset{x}{ˉ})$ Substitute into the $r$ formula: $r = \frac{\sum m ( x _{i} - x ˉ ) ^{2}}{\sum ( x _{i} - x ˉ ) ^{2} \sum m ^{2} ( x _{i} - x ˉ ) ^{2}} = \frac{m \sum ( x _{i} - x ˉ ) ^{2}}{∣ m ∣ \sum ( x _{i} - x ˉ ) ^{2}} = \frac{m}{∣ m ∣} = + 1$ The $\sum (x_{i} - \overset{x}{ˉ})^{2}$ cancels top and bottom, and $m /∣ m ∣ = sign (m)$ . So the size of $m$ leaves no trace — only its sign survives.

Why the slope's size $2$ doesn't matter: correlation measures how tightly points hug a line, not how steep the line is. Perfect hug ⇒ $∣ r ∣ = 1$ regardless of slope (as long as slope $\neq = 0$ ).

Level 4 — Synthesis

L4.1 — Correlation of a sum with a part

Let $X$ and $Z$ be independent with $Var (X) = Var (Z) = 1$ . Define $Y = X + Z$ . Find $ρ_{X, Y}$ .

Recall Solution

Step 1 — covariance splits over sums. Because covariance is linear in each slot: $Cov (X, Y) = Cov (X, X + Z) = Cov (X, X) + Cov (X, Z)$ Now $Cov (X, X) = Var (X) = 1$ , and independence gives $Cov (X, Z) = 0$ (see 1.2.04-Independence). So $Cov (X, Y) = 1$ .

Step 2 — variance of $Y$ . For independent parts variances add: $Var (Y) = Var (X) + Var (Z) = 1 + 1 = 2 \Rightarrow σ_{Y} = 2$ And $σ_{X} = 1$ .

Step 3 — assemble. $ρ_{X, Y} = \frac{1}{( 1 ) ( 2 )} = \frac{1}{2} \approx 0.707$ Interpretation: $X$ makes up "half the ingredients" of $Y$ , so they share a moderate positive correlation, not a perfect one — the noise $Z$ dilutes it.

L4.2 — Building a covariance matrix

For the same $X, Z$ (independent, unit variance) and $Y = X + Z$ , write the $2 \times 2$ covariance matrix of the vector $(X, Y)$ .

Recall Solution

Layout: the covariance matrix $Σ$ has variances on the diagonal, covariances off-diagonal: $Σ = [Var (X) Cov (Y, X) Cov (X, Y) Var (Y)]$ Plug in $Var (X) = 1$ , $Cov (X, Y) = 1$ (from L4.1), $Var (Y) = 2$ , and $Cov (Y, X) = Cov (X, Y)$ by symmetry: $Σ = [1112]$ Why symmetric? $Cov (X, Y) = Cov (Y, X)$ always — swapping the two factors inside $E [(X - μ_{X}) (Y - μ_{Y})]$ changes nothing. This matrix is exactly what 2.4.01-Principal-component-analysis eigen-decomposes.

Level 5 — Mastery

L5.1 — Prove zero correlation need not mean independence

Let $X$ take values $- 1, 0, + 1$ each with probability $\frac{1}{3}$ , and set $Y = X^{2}$ . Show $Cov (X, Y) = 0$ , then show $X$ and $Y$ are not independent.

Recall Solution

Step 1 — the means. $E [X] = \frac{1}{3} (- 1) + \frac{1}{3} (0) + \frac{1}{3} (1) = 0$ Step 2 — the cross term. $Y = X^{2}$ , so $X Y = X^{3}$ : $E [X Y] = E [X^{3}] = \frac{1}{3} (- 1)^{3} + \frac{1}{3} (0)^{3} + \frac{1}{3} (1)^{3} = \frac{1}{3} (- 1 + 0 + 1) = 0$ Step 3 — covariance. $Cov (X, Y) = E [X Y] - E [X] E [Y] = 0 - (0) E [Y] = 0$ So they are uncorrelated.

Step 4 — but not independent. Independence demands $P (X = x, Y = y) = P (X = x) P (Y = y)$ for all pairs. Test $X = 0, Y = 1$ : $P (X = 0, Y = 1) = 0 (because X = 0 \Rightarrow Y = 0)$ $P (X = 0) P (Y = 1) = \frac{1}{3} \cdot \frac{2}{3} = \frac{2}{9} \neq = 0$ Since $0 \neq = \frac{2}{9}$ , they are dependent. Knowing $X = 0$ tells you $Y = 0$ for certain — total information, yet zero linear correlation. This is why 4.1.03-Feature-selection cannot rely on correlation alone to drop features.

L5.2 — Derive that $ρ = \pm 1$ forces a straight line

Using standardized variables $Z_{X} = \frac{X - μ _{X}}{σ _{X}}$ and $Z_{Y} = \frac{Y - μ _{Y}}{σ _{Y}}$ (each mean $0$ , variance $1$ ), show that if $ρ = 1$ then $Y$ is an exact increasing linear function of $X$ .

Recall Solution

Step 1 — build a clever non-negative quantity. Consider the variance of the difference: $Var (Z_{X} - Z_{Y}) = Var (Z_{X}) + Var (Z_{Y}) - 2 Cov (Z_{X}, Z_{Y})$ Step 2 — substitute knowns. Each standardized variable has variance $1$ , and $Cov (Z_{X}, Z_{Y}) = E [Z_{X} Z_{Y}] = ρ$ . So: $Var (Z_{X} - Z_{Y}) = 1 + 1 - 2 ρ = 2 (1 - ρ)$ Step 3 — set $ρ = 1$ . Then $Var (Z_{X} - Z_{Y}) = 0$ . A random variable with zero variance is constant (equal to its mean, which is $0 - 0 = 0$ ). Therefore: $Z_{X} - Z_{Y} = 0 \Rightarrow Z_{Y} = Z_{X} \Rightarrow \frac{Y - μ _{Y}}{σ _{Y}} = \frac{X - μ _{X}}{σ _{X}}$ Step 4 — solve for $Y$ . $Y = μ_{Y} + \frac{σ _{Y}}{σ _{X}} (X - μ_{X})$ a straight line with positive slope $σ_{Y} / σ_{X}$ . (Repeating with $ρ = - 1$ uses $Var (Z_{X} + Z_{Y}) = 2 (1 + ρ) = 0$ , giving $Z_{Y} = - Z_{X}$ — a line of negative slope.) $■$

Recall Self-check ledger (answers only)

L1.1 negative ::: negative covariance L1.2 which is unit-free ::: correlation $ρ$ L2.1 covariance ::: $0.4$ L2.2 correlation ::: $\approx 0.894$ L3.1 Cov(U,V) ::: $- 2.4$ L3.2 correlation ::: $+ 1$ L4.1 $ρ_{X, Y}$ ::: $1/ 2 \approx 0.707$ L5.1 covariance ::: $0$ (but dependent)