Visual walkthrough — Covariance and correlation

This page rebuilds the central result of Covariance and correlation from nothing. We will earn the number $\rho$ (the correlation) one picture at a time. By the end you will see why $\rho$ always lives between $-1$ and $+1$, and why $0$ does not mean "unrelated".

We only assume you know what a random variable is (a number whose value depends on chance) and what an average / expectation is (the balance-point of all its possible values). Everything else we build here.

---

Step 1 — Two numbers that wobble together

WHAT. We have two random quantities. Call them $X$ (say, house size) and $Y$ (say, price). Each sample of the world gives us a pair $(x_i,y_i)$ — one dot on a flat picture. This picture of dots is a joint distribution made visible.

WHY. Before we can measure "do they move together?", we need to see the moving. The dot cloud is the raw data of teamwork.

PICTURE. Below, each dot is one observation. The two dashed lines are the means ${\mu }_X$ (a vertical line at $x={\mu }_X$) and ${\mu }_Y$ (a horizontal line at $y={\mu }_Y$) — the balance-points of $X$ and $Y$ alone.

---

Step 2 — Move the origin to the centre of the cloud

WHAT. For every dot we compute two deviations:

$$\underset{\text{how far right/left of centre}}{\underbrace{(x_i-{\mu }_X)}}\underset{\text{how far up/down of centre}}{\underbrace{(y_i-{\mu }_Y)}}$$

WHY. "Together-ness" is about being on the same side of the centre, not about raw position. A dot at $x=1000$ tells us nothing until we know the middle is $999$ (barely above) or $10$ (far above). Subtracting the mean re-centres the whole cloud on $(0,0)$.

PICTURE. The dashed cross becomes the new origin. Every dot now lives in one of four quadrants. Look at the sign of each deviation:

• Top-right: $(x_i-{\mu }_X)>0$ and $(y_i-{\mu }_Y)>0$ → product $>0$.
• Bottom-left: both negative → negative × negative $=$ product $>0$.
• Top-left / bottom-right: one positive, one negative → product $<0$.

---

Step 3 — Multiply the deviations: the sign carries the message

WHAT. For each dot form the product of its two deviations:

$$p_i=\underset{\text{horizontal}}{\underbrace{(x_i-{\mu }_X)}}\underset{\text{vertical}}{\underbrace{(y_i-{\mu }_Y)}}$$

WHY. Multiplication is exactly the tool that turns "same side / opposite side" into a sign. We chose the product (not the sum, not the distance) because only a product flips sign when exactly one factor is negative — precisely the top-left / bottom-right case we want to count as "disagreeing". Think of $p_i$ as the signed area of the rectangle from the centre to the dot.

PICTURE. Each dot gets a shaded rectangle. Warm (magenta) rectangles are positive $p_i$; violet ones are negative. A cloud that leans up-right is mostly warm.

---

Step 4 — Average the products: this IS covariance

WHAT. Average all the little signed rectangles:

$$\mathrm{Cov}(X,Y)=\mathbb{E}\left[\underset{\text{horiz. dev.}}{\underbrace{(X-{\mu }_X)}}\underset{\text{vert. dev.}}{\underbrace{(Y-{\mu }_Y)}}\right]$$

WHY. One dot is noise. The average rectangle tells us which quadrants dominate. If warm rectangles win, the average is positive; if violet wins, negative; if they tie, zero.

PICTURE. Three clouds, three average-rectangle verdicts: positive, zero, negative.

The shortcut formula, derived here. The rectangle form is the picture, but there is a faster way to compute. Expand the product inside the average, using that ${\mu }_X$ and ${\mu }_Y$ are fixed numbers (constants), and that the average of a sum is the sum of the averages (linearity of expectation):

$$\mathbb{E}[(X-{\mu }_X)(Y-{\mu }_Y)]=\mathbb{E}[XY-X{\mu }_Y-{\mu }_{X}Y+{\mu }_X{\mu }_Y]$$ $$=\mathbb{E}[XY]-\underset{={\mu }_Y{\mu }_X}{\underbrace{{\mu }_Y\mathbb{E}[X]}}-\underset{={\mu }_X{\mu }_Y}{\underbrace{{\mu }_X\mathbb{E}[Y]}}+\underset{\text{a constant, avg of itself}}{\underbrace{{\mu }_X{\mu }_Y}}$$

Here we pulled the constants ${\mu }_X,{\mu }_Y$ outside their averages, then used $\mathbb{E}[X]={\mu }_X$ and $\mathbb{E}[Y]={\mu }_Y$. Two of the last three terms are $+{\mu }_X{\mu }_Y$ and one subtracts $2{\mu }_X{\mu }_Y$, so they collapse to $-{\mu }_X{\mu }_Y$:

The trouble: this number's size is meaningless. Stretch the $X$-axis by $1000$ (millimetres instead of metres) and every rectangle stretches $1000\times$. We must normalise.

---

Step 5 — Squash both axes to a fair, unit-free grid

WHAT. Replace $X$ and $Y$ by their standardised versions:

$$Z_X=\frac{X-{\mu }_X}{{\sigma }_X},Z_Y=\frac{Y-{\mu }_Y}{{\sigma }_Y}$$

where ${\sigma }_X=\sqrt{\mathrm{Var}(X)}$ is the standard deviation — the typical distance of $X$ from its own mean.

WHY. Dividing by ${\sigma }_X$ measures each deviation in units of its own typical spread. A dot "$2\sigma$ to the right" means the same thing whether $X$ is in millimetres or miles. This kills the unit problem from Step 4.

PICTURE. The stretched cloud (left) becomes a round cloud (right) where each axis is now measured in units of one standard deviation — a step of $1$ along $Z_X$ means "one typical spread of $X$". Both $Z_X$ and $Z_Y$ now have mean $0$ and spread $1$.

---

Step 6 — Why $\rho$ can never leave the corridor $[-1,+1]$

WHAT. We show $-1\le \rho \le 1$ by looking at a single guaranteed-non-negative quantity:

$$\mathbb{E}[(Z_X-Z_Y{)}^2]\ge 0$$

WHY. A square is never negative, and an average of never-negative things is never negative. This one honest fact is the whole cage. Expand it:

$$\mathbb{E}[Z_X^2]-2\mathbb{E}[Z_X{Z}_Y]+\mathbb{E}[Z_Y^2]\ge 0$$

Now use the two facts we built in Step 5: $\mathbb{E}[Z_X^2]=1$ and $\mathbb{E}[Z_Y^2]=1$ (each standardised variable has variance $1$). And $\mathbb{E}[Z_X{Z}_Y]=\rho$. Substitute:

$$1-2\rho +1\ge 0⟹2-2\rho \ge 0⟹\rho \le 1.$$

Repeat with $(Z_X+Z_Y{)}^2\ge 0$ to get $2+2\rho \ge 0\Rightarrow \rho \ge -1$.

PICTURE. The two parabolas $(Z_X\mp Z_Y{)}^2$ never dip below the axis; their averages pin $\rho$ inside the corridor.

The corridor endpoints, made visible. Equality (touching $-1$ or $+1$) happens exactly when $Z_Y=\pm Z_X$, i.e. the dots lie on a perfect straight line. The next picture shows both extreme cases as scatter plots: $\rho =+1$ (rising line) and $\rho =-1$ (falling line). These are the walls of the corridor.

---

Step 7 — The edge case: $\rho =0$ does NOT mean "unrelated"

WHAT. Let $X$ be uniform on $[-1,1]$ and set $Y=X^2$. Then

$$\mathbb{E}[XY]=\mathbb{E}[X^3]=0,\mathbb{E}[X]=0\Rightarrow \mathrm{Cov}(X,Y)=0\Rightarrow \rho =0.$$

WHY. Our rectangles only detect lines. For a symmetric U-shape, the warm rectangles on the right cancel the warm ones on the left in a way that zeroes the average — even though $Y$ is completely pinned down by $X$. This is the boundary of what $\rho$ can see, and every honest reader must meet it.

PICTURE. A perfect parabola of dots: obviously dependent, yet $\rho =0$. The left half's rectangles exactly cancel the right half's.

---

The one-picture summary

Everything above, in one frame: raw dots → subtract means → sign the rectangles → average → squash to fair units → read $\rho$ off the corridor.

Recall Feynman retelling — say it back in plain words

We start with a scatter of dot-pairs. First we find the middle of the cloud and slide it to the origin — now every dot is "left/right of centre" and "up/down of centre". For each dot we multiply those two distances; the product is positive when the dot agrees with an up-right trend and negative when it disagrees. We average all those signed rectangles — that average is covariance: a direction of teamwork. But its size lies (change units, size changes), so we shrink each axis by its own typical spread (one standard deviation), giving a fair round grid. The average rectangle on that grid is correlation $\rho$. Because the average of a square is never negative, $\rho$ can never escape $-1$ to $+1$; it hits the ends only when the dots sit on a perfect line. If either variable never moves ($\sigma =0$), $\rho$ isn't even defined — there's nothing to compare. And finally: if the dots make a curve like a U, the left-side and right-side rectangles cancel — so $\rho =0$ even though the variables are joined at the hip. $\rho$ measures straight-line teamwork, nothing more.

---

Where this leads

• The rectangle-averaging idea generalises to a whole covariance matrix, the engine of PCA.
• $\rho$ near $\pm 1$ between features is the warning sign of multicollinearity in linear regression, and a lever in feature selection.

Recall

What does the sign of the deviation-product $(x_i-{\mu }_X)(y_i-{\mu }_Y)$ encode? ::: Whether the dot agrees (positive, up-right or down-left) or disagrees (negative, up-left or down-right) with a rising trend. Why divide covariance by ${\sigma }_X{\sigma }_Y$? ::: To make it unit-free/scale-invariant, so its size is comparable across variable pairs. Which single inequality cages $\rho$ inside $[-1,1]$? ::: $\mathbb{E}[(Z_X\mp Z_Y{)}^2]\ge 0$ (a square's average can't be negative). When is $\rho$ undefined? ::: When ${\sigma }_X=0$ or ${\sigma }_Y=0$ (a variable with no spread — division by zero). Does $\rho =0$ prove independence? ::: No — it only rules out a linear relationship; curves like $Y=X^2$ give $\rho =0$ while being fully dependent.