Exercises — Covariance and correlation

What the picture shows: when $x$ is above its mean (right of the vertical dashed line) the points sit below their mean (below the horizontal dashed line), and vice-versa. So one variable being high pairs with the other being low. Deviation product: $(x-{\mu }_X)$ and $(y-{\mu }_Y)$ have opposite signs for most points, so their product $(x-{\mu }_X)(y-{\mu }_Y)$ is negative. Averaging negatives gives a negative average. Answer: $\mathrm{Cov}(X,Y)<0$, and since $\rho$ is $\mathrm{Cov}$ divided by the two positive spreads ${\sigma }_X{\sigma }_Y$, it keeps the sign: $\rho <0$.

Recognise: we have means and the mean of the product — that is exactly the working formula $\mathrm{Cov}=E[XY]-E[X]E[Y]$. Compute: $7-(2)(3)=7-6=1$. Answer: $\mathrm{Cov}(X,Y)=1$.

Set $Y=X$ in the definition: $\mathrm{Cov}(X,X)=E[(X-{\mu }_X)(X-{\mu }_X)]=E[(X-{\mu }_X{)}^2]=\mathrm{Var}(X)$. Answer: it is the variance of $X$. (Covariance generalises variance — see Variance and Standard Deviation.)

Marginals (sum over the other variable): $P(X=1)=P(1,0)+P(1,1)=0.2+0.3=0.5$, so $E[X]=0.5$. By symmetry $E[Y]=0.5$. $E[XY]$: only the corner $(1,1)$ has $XY=1$; all other corners have $XY=0$. So $E[XY]=1\cdot 0.3=0.3$. Covariance: $0.3-(0.5)(0.5)=0.3-0.25=0.05$. Positive ⇒ mild agreement. Variances: $X$ is Bernoulli$(0.5)$, so $\mathrm{Var}(X)=0.5\cdot 0.5=0.25$, giving ${\sigma }_X=0.5$; likewise ${\sigma }_Y=0.5$. Correlation: $\rho =\frac{0.05}{0.5\cdot 0.5}=\frac{0.05}{0.25}=0.2$. Answer: $\mathrm{Cov}=0.05$, $\rho =0.2$.

Means: $\bar{x}=\frac{2+4+6+8+10}{5}=6$, $\bar{y}=\frac{1+2+5+4+8}{5}=4$. Deviation products $\sum (x_i-\bar{x})(y_i-\bar{y})$: $(-4)(-3)+(-2)(-2)+(0)(1)+(2)(0)+(4)(4)=12+4+0+0+16=32$. $S_{xx}=\sum (x_i-\bar{x}{)}^2=16+4+0+4+16=40$. $S_{yy}=\sum (y_i-\bar{y}{)}^2=9+4+1+0+16=30$. Why these sums? Sample $r=\frac{\sum (x-\bar{x})(y-\bar{y})}{\sqrt{S_{xx}{S}_{yy}}}$ — the same $E[XY]-E[X]E[Y]$ idea, but on data (the $\frac{1}{n}$ factors cancel top and bottom). Answer: $r=\frac{32}{\sqrt{40\cdot 30}}=\frac{32}{\sqrt{1200}}\approx 0.924$. Strong positive.

Tool (property 3): $\mathrm{Var}(X\pm Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)\pm 2\mathrm{Cov}(X,Y)$. Sum: $4+9+2(-2)=13-4=9$. Difference: $4+9-2(-2)=13+4=17$. Answer: $\mathrm{Var}(X+Y)=9$, $\mathrm{Var}(X-Y)=17$.

Bilinearity (property 2): $\mathrm{Cov}(1000X,Y)=1000\cdot \mathrm{Cov}(X,Y)=1000\cdot 3=3000$. New spread of $X^{\prime }$: ${\sigma }_{X^{\prime }}=1000{\sigma }_X=2000$. New correlation: ${\rho }_{X^{\prime }Y}=\frac{3000}{2000\cdot 5}=\frac{3000}{10000}=0.3$. Original correlation: ${\rho }_{XY}=\frac{3}{2\cdot 5}=0.3$. Insight: covariance exploded $1000\times$ (from $3$ to $3000$) but correlation is identical ($0.3$). Rescaling changes units, never the true strength — exactly why we prefer $\rho$. Answer: $\mathrm{Cov}(X^{\prime },Y)=3000$, ${\rho }_{X^{\prime }Y}=0.3={\rho }_{XY}$.

Means: $E[X]=\frac{1}{4}(-2-1+1+2)=0$ (symmetric about $0$). $E[XY]$: $XY=X|X|$, values $(-2)(2),(-1)(1),(1)(1),(2)(2)=-4,-1,1,4$. Average $=\frac{1}{4}(-4-1+1+4)=0$. Covariance: $E[XY]-E[X]E[Y]=0-0\cdot E[Y]=0$. But dependent: knowing $X=2$ forces $Y=2$; knowing $Y=1$ tells you $X\in \{-1,1\}$. So $P(Y=y\mid X=x)\ne P(Y=y)$ — they share information. Why covariance misses it: covariance only detects the linear part of a relationship (see Independence of Random Variables). Here the link $Y=|X|$ is symmetric, so its "up-slope" and "down-slope" halves cancel. Answer: $\mathrm{Cov}=0$, yet not independent. ∎

Prediction: $Y$ is a perfect increasing linear function of $X$, so the Cauchy–Schwarz equality case gives $\rho =+1$. Verify: $\mathrm{Cov}(X,3X-7)=3\mathrm{Cov}(X,X)=3\mathrm{Var}(X)$ (shift $-7$ drops out). Spread: ${\sigma }_Y=\sqrt{\mathrm{Var}(3X-7)}=\sqrt{9\mathrm{Var}(X)}=3{\sigma }_X$. Then $\rho =\frac{3\mathrm{Var}(X)}{{\sigma }_X\cdot 3{\sigma }_X}=\frac{3{\sigma }_X^2}{3{\sigma }_X^2}=1$. (If the slope were negative, say $Y=-3X+7$, the same steps give $\rho =-1$.) Answer: ${\rho }_{XY}=+1$.

Covariance: $\mathrm{Cov}(X,Y)=\mathrm{Cov}(X,X+Z)=\mathrm{Cov}(X,X)+\mathrm{Cov}(X,Z)$ by bilinearity. Independence ⇒ $\mathrm{Cov}(X,Z)=0$, and $\mathrm{Cov}(X,X)=\mathrm{Var}(X)=1$. So $\mathrm{Cov}(X,Y)=1$. Spread of $Y$: $\mathrm{Var}(Y)=\mathrm{Var}(X)+\mathrm{Var}(Z)+2\mathrm{Cov}(X,Z)=1+1+0=2$, so ${\sigma }_Y=\sqrt{2}$. Correlation: ${\rho }_{XY}=\frac{1}{{\sigma }_X{\sigma }_Y}=\frac{1}{1\cdot \sqrt{2}}=\frac{1}{\sqrt{2}}\approx 0.707$. Answer: ${\rho }_{XY}=\frac{1}{\sqrt{2}}\approx 0.707$. (A part is correlated with its noisy total, but never perfectly, because $Z$ dilutes it.)

From L2.2: $S_{xy}=32$, $S_{xx}=40$, $S_{yy}=30$, $\bar{x}=6$, $\bar{y}=4$. Slope: $b=\frac{32}{40}=0.8$. Intercept: $a=\bar{y}-b\bar{x}=4-0.8\cdot 6=4-4.8=-0.8$. Goodness of fit: $r^2={\left(\frac{S_{xy}}{\sqrt{S_{xx}{S}_{yy}}}\right)}^2=\frac{S_{xy}^2}{S_{xx}{S}_{yy}}=\frac{32^2}{40\cdot 30}=\frac{1024}{1200}\approx 0.853$. Meaning: about $85.3\%$ of the up-and-down spread in $y$ is explained by the straight-line dependence on $x$ (see Linear Regression). Answer: $b=0.8$, $a=-0.8$, $r^2\approx 0.853$.

Definition: $\Sigma =\left(\begin{array}{cc}\mathrm{Var}(X)& \mathrm{Cov}(X,Y)\\ \mathrm{Cov}(Y,X)& \mathrm{Var}(Y)\end{array}\right)$. Fill in: $\Sigma =\left(\begin{array}{cc}0.25& 0.05\\ 0.05& 0.25\end{array}\right).$ Checks: diagonal holds the variances $0.25,0.25$; off-diagonal entries are equal ($0.05=0.05$) because $\mathrm{Cov}(X,Y)=\mathrm{Cov}(Y,X)$ (symmetry). ✓ Answer: the matrix above.

Numerator (bilinearity, property 2): shifts vanish, scales pull out: $\mathrm{Cov}(aX+b,cY+d)=ac\mathrm{Cov}(X,Y).$ Denominator (spreads): ${\sigma }_{aX+b}=\sqrt{\mathrm{Var}(aX+b)}=\sqrt{a^2\mathrm{Var}(X)}=|a|{\sigma }_X$, and likewise ${\sigma }_{cY+d}=|c|{\sigma }_Y$. Assemble: ${\rho }^{\prime }=\frac{ac\mathrm{Cov}(X,Y)}{|a||c|{\sigma }_X{\sigma }_Y}=\frac{ac}{|ac|}\cdot \frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\mathrm{sign}(ac){\rho }_{XY}.$ Reading it: stretching either axis keeps the strength $|\rho |$ fixed; flipping an axis ($a<0$ or $c<0$) flips the sign. This is why correlation is the honest, unit-free strength score. ∎

Set-up: let $U=X-{\mu }_X$, $V=Y-{\mu }_Y$, so $E[U^2]={\sigma }_X^2$, $E[V^2]={\sigma }_Y^2$, $E[UV]=\mathrm{Cov}(X,Y)$. A square is never negative: for every real $t$, $0\le E[(U-tV{)}^2]=E[U^2]-2tE[UV]+t^{2}E[V^2].$ Read it as a quadratic in $t$: it is $q(t)=E[V^2]t^2-2E[UV]t+E[U^2]$, opening upward and never dipping below $0$. A quadratic that stays $\ge 0$ has discriminant $\le 0$: $(2E[UV]{)}^2-4E[V^2]E[U^2]\le 0\Rightarrow E[UV{]}^2\le E[U^2]E[V^2].$ Translate back: $\mathrm{Cov}(X,Y{)}^2\le {\sigma }_X^2{\sigma }_Y^2$, hence $\left|\frac{\mathrm{Cov}}{{\sigma }_X{\sigma }_Y}\right|=|\rho |\le 1$. Equality: the discriminant is $0$ exactly when $q(t)$ has a real double root $t^{\star }$, i.e. $E[(U-t^{\star }V{)}^2]=0$, meaning $U=t^{\star }V$ with probability $1$ — that is, $X-{\mu }_X$ is an exact multiple of $Y-{\mu }_Y$. Geometrically $Y$ is a perfect linear function of $X$, which is the $\rho =\pm 1$ case. ∎ (See Cauchy–Schwarz Inequality.)