Foundations — Covariance and correlation

This page is the toolbox room. Before you enter Covariance and Correlation, we lay out every symbol, picture, and idea that note quietly assumes. Read top to bottom — each rung of the ladder rests on the one below it.

0. The raw material: a random variable $X$

Picture a dartboard experiment. Every throw is an outcome; the rule "measure how far the dart landed from the centre, in cm" turns each throw into a number. That number-valued rule is $X$.

• The small letter $x$ means one particular value $X$ can take.
• The capital letter $X$ means the whole random rule, before we know the result.

We need this because covariance compares two such rules, $X$ and $Y$, measured on the same experiment (same dart throw gives both a horizontal position $X$ and a vertical position $Y$).

Prerequisite depth lives in Expectation of Random Variables and Variance and Standard Deviation — we rebuild just what we need here.

1. Probability $P$: how likely each value is

Before we can average, we need the symbol that says how likely.

Picture a loaded die whose faces $1,\dots ,6$ have unequal chances. The list of six heights (one per face), each between $0$ and $1$ and summing to $1$, is the pmf. We need $P$ because every average below is a probability-weighted one — no probabilities, no averaging.

2. The average: $E[\cdot ]$, and the mean $\mu$

For a discrete variable that takes value $x$ with probability $P(X=x)$: $E[X]={\sum }_{x}xP(X=x).$

Read this literally: walk through every value $x$ the variable can take, weight it by how likely it is, add them up. The $\sum$ (capital Greek "S", for "sum") just means "add over all cases".

The balance-point idea deserves a picture.

Why a separate symbol? Because we will subtract it so often — $X-{\mu }_X$ — that writing $E[X]$ each time would clutter the page.

3. The deviation: $X-{\mu }_X$

This is the heart of everything to come. Look at the sign:

• $X-{\mu }_X>0$ ⇒ $X$ came out above average (a high day).
• $X-{\mu }_X<0$ ⇒ $X$ came out below average (a low day).
• $X-{\mu }_X=0$ ⇒ $X$ landed exactly on average.

Why subtract the mean? Covariance cares about movement around the centre, not the centre itself. If everyone earns ₹1000 more, nobody's "high day / low day" pattern changes — subtracting ${\mu }_X$ throws away the location and keeps only the wiggle.

4. Squaring one deviation: Variance $\mathrm{Var}(X)$ and ${\sigma }_X$

Before comparing two variables, understand the one-variable case the parent note builds on.

Why square? A raw deviation is sometimes $+$, sometimes $-$; averaging them cancels to zero (that is what "balance point" means). Squaring makes every deviation positive, so wiggles can no longer cancel. Big wiggles get counted; direction is thrown away.

We need ${\sigma }_X$ and ${\sigma }_Y$ because correlation divides covariance by ${\sigma }_X{\sigma }_Y$ to cancel units. More in Variance and Standard Deviation.

5. Multiplying TWO deviations: covariance and its shortcut

Now the leap from one variable to two. Multiply the two deviations $(X-{\mu }_X)(Y-{\mu }_Y)$ and read the sign of the product — this is the single trick the whole topic rests on.

Covariance is then just the average of this product: $\mathrm{Cov}(X,Y)=E[(X-{\mu }_X)(Y-{\mu }_Y)].$ If agreements outweigh disagreements, the average is positive. That is the entire definition, now fully earned.

6. The correlation symbol $\rho$, its $[-1,1]$ leash, and where it breaks

• $\rho =+1$: perfect agreement (points fall on an upward line).
• $\rho =-1$: perfect disagreement (points fall on a downward line).
• $\rho =0$: no linear teamwork.

Why a leash? Raw covariance can be any size and carries units (kg·cm), so "big covariance" tells you nothing about strength. The leash makes $\rho$ a fair, unitless report card.

7. Two supporting ideas the topic leans on

Prerequisite map

The diagram below is drawn with mermaid (a plain-text way to specify boxes and arrows). Read each box as a concept and each arrow as "must understand before": start at the bottom-source boxes and follow the arrows upward — the flow ends at the parent topic. If your reader shows it as raw text, just trace the pairs in words: random variable → expectation → mean → deviation, and from deviation the ladder splits into variance and product of deviations, both feeding covariance, which together with variance feeds correlation.

In plain words: probabilities let us average; averaging gives the mean; the mean lets us measure deviations; deviations (squared) give variance and (paired, multiplied) give covariance; covariance divided by the two sigmas gives correlation — the doorway into Covariance and Correlation. The sideways ideas also feed Linear Regression and the Covariance Matrix.

Equipment checklist

Test yourself — cover the right side.

Connections

• Expectation of Random Variables — the $E[\cdot ]$ machine every formula uses.
• Variance and Standard Deviation — the one-variable case covariance generalises.
• Independence of Random Variables — why independence forces $\mathrm{Cov}=0$.
• Cauchy–Schwarz Inequality — the reason $\rho$ stays in $[-1,1]$.
• Linear Regression — uses these same deviations to fit a line.
• Covariance Matrix — bundles all pairwise covariances together.
• Covariance and Correlation — the parent topic this page prepares you for.