Foundations — Covariance and correlation

Before you can read the parent note Covariance and correlation you must own every symbol it throws at you. This page builds each one from nothing, in the order they depend on each other. Never worry that a symbol looks scary — by the time we use it, you will have seen its picture.

1. The random variable $X$

Plain words: think of $X$ as a labelled slot that gets filled with a number each time you run the experiment. Roll a die → $X$ becomes the face showing. Measure a house → $X$ becomes its square footage.

Picture: imagine a number line, and each trial drops a dot somewhere on it. Over many trials you get a cloud of dots.

Why the topic needs it: covariance compares two such slots, $X$ and $Y$, filled at the same time. Without the idea of a random number, there is nothing to compare. (Build more at 1.3.01-Random-variables.)

2. The expectation $\mathbb{E}[X]$ and the mean ${\mu }_X$

Plain words: if you ran the experiment a million times and averaged all the numbers, that average is $\mathbb{E}[X]$. The two symbols $\mathbb{E}[X]$ and ${\mu }_X$ mean exactly the same thing — $\mathbb{E}[\cdot ]$ is the operation "take the average of", and ${\mu }_X$ is the name of the answer for $X$.

Picture: place a finger under the number line where the dots would perfectly balance like a see-saw. That balance point is ${\mu }_X$.

Why the tool $\mathbb{E}$ and not a plain average? Because we rarely have all the data — we have probabilities. $\mathbb{E}$ is the average weighted by probability, which works even for infinite or continuous possibilities where you cannot literally list every trial. (See 1.3.05-Expectation-and-variance.)

3. The deviation $X-{\mu }_X$

Picture: draw an arrow from the balance point ${\mu }_X$ to where a dot landed. Arrow pointing right = positive deviation; pointing left = negative deviation.

Why the topic needs it: covariance never asks "was $X$ big?" — it asks "was $X$ above or below its usual, and was $Y$ above or below its usual, at the same time?" That "above/below usual" is exactly the deviation. Multiplying two deviations is the heart of the whole idea (next section).

4. Variance $\text{Var}(X)$ and standard deviation ${\sigma }_X$

Plain words: variance measures how spread out the cloud is. We square the deviations so that below-average (negative) and above-average (positive) misses both count as "spread" instead of cancelling.

Why take the square root to get ${\sigma }_X$? Squaring changed the units (metres became metres²). The square root undoes that, so ${\sigma }_X$ is back in the same units as $X$ — a genuine "typical distance from the mean".

Picture: ${\sigma }_X$ is the typical width of the cloud, a ruler you lay along the number line.

Why the topic needs it: correlation divides covariance by ${\sigma }_X{\sigma }_Y$. To divide by spread, you must first have spread — that is $\sigma$. (More: 1.3.05-Expectation-and-variance.)

5. The joint pair $(X,Y)$ and $P(X,Y)$

Plain words: now every trial produces two numbers, so each dot lives on a 2-D plane, not a line. The dot's horizontal position is $X$, its vertical position is $Y$.

Picture: a scatter plot. Each experiment = one dot at coordinates $(x,y)$.

Why the topic needs it: covariance is a two-variable idea. You cannot see "do they move together" from two separate number lines — you must see the pairs side by side, which is exactly the scatter plot. (Build this at 1.3.08-Joint-and-marginal-distributions.)

6. The product of deviations $(X-{\mu }_X)(Y-{\mu }_Y)$

This is the single cleverest symbol on the parent page, so we picture it carefully.

Take one dot at $(x,y)$. Draw the vertical line $X={\mu }_X$ and the horizontal line $Y={\mu }_Y$. These two lines split the plane into four quadrants, meeting at the "average point" $({\mu }_X,{\mu }_Y)$.

For a dot, look at the sign of each deviation and multiply:

This is why the parent defines $\text{Cov}(X,Y)=\mathbb{E}[(X-{\mu }_X)(Y-{\mu }_Y)].$ Every symbol in it you now own: $\mathbb{E}$ (section 2), the deviations (section 3), their product (this section).

7. The correlation symbol $\rho$ and its scale

Plain words: covariance's size depends on the units of $X$ and $Y$, so a big number could just mean "big units", not "strong link". Dividing by ${\sigma }_X{\sigma }_Y$ cancels the units and squeezes the answer into the fixed range $[-1,+1]$, where $\pm 1$ means "perfectly on a line" and $0$ means "no straight-line trend". This makes $\rho$ comparable across any pair of variables.

Why the topic needs it: it is the rating on a universal scale the parent's intuition promised.

8. Sample symbols: $\bar{x}$, $n$, $\sum$

When you only have real data (finitely many measured pairs) rather than a probability formula, the parent switches to lower-case sample symbols:

• $n$ — the count of data points you collected.
• $x_i$ — the $i$-th data value; the little $i$ is just a counter running $1,2,\dots ,n$.
• ${\sum }_{i=1}^n$ — the summation sign: "add up the thing to my right for $i$ from $1$ to $n$."
• $\bar{x}$ — "x-bar", the sample mean, $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i$. It is the data-only stand-in for ${\mu }_X$.

Why the topic needs them: in machine learning you never see the true ${\mu }_X$ — you estimate it from a spreadsheet of $n$ rows. The bar and the $\sum$ are how theory becomes computable.

Prerequisite map

Where these tools go next

• Covariance of many features at once builds the matrix behind 2.4.01-Principal-component-analysis.
• The same "move together" idea drives the slope in 3.2.05-Linear-regression and the warning of 3.2.06-Multicolinearity.
• Zero covariance is a weaker cousin of 1.2.04-Independence — never confuse them.
• Correlation strength is one criterion in 4.1.03-Feature-selection.

Equipment checklist