Exploratory data analysis (EDA) workflow: Visual walkthrough

We build the correlation number, $r$ , in 8 steps. Each step is one picture.

Step 1 — Two columns of numbers become a cloud of dots

WHAT. We have $n$ paired measurements. For each customer $i$ we know two numbers: their age $x_{i}$ and their income $y_{i}$ . Nothing else. $n$ just means "how many rows we have".

WHY. Before any formula, we need a shape to reason about. A table of numbers has no shape your eye can grab. A scatter plot — one dot per row, placed at horizontal position $x_{i}$ and vertical position $y_{i}$ — turns the table into a picture. Now "do they move together?" becomes "does the cloud tilt?".

PICTURE. Each red dot is one customer. Read the dot's shadow on the bottom axis to get $x_{i}$ , its shadow on the left axis to get $y_{i}$ .

Figure — Exploratory data analysis (EDA) workflow

There is no tilt to measure yet, because we have no center to tilt around. That is Step 2.

Step 2 — Find the center of the cloud

WHAT. Compute the average horizontal position $\overset{x}{ˉ}$ and the average vertical position $\overset{y}{ˉ}$ . The point $(\overset{x}{ˉ}, \overset{y}{ˉ})$ is the balance point of the cloud.

WHY. "Move together" only makes sense relative to typical. A customer isn't "high income" in the absolute — they're high compared to the crowd. To ask whether being above-average in age goes with being above-average in income, we first need to know where "average" sits. The bar over a letter (read "x-bar") means "the mean of all the x's".

PICTURE. The red crosshair marks $(\overset{x}{ˉ}, \overset{y}{ˉ})$ . It splits the plot into four quadrants. Notice roughly the same number of dots sit left and right of the vertical line, and above/below the horizontal line — that is what "balance point" means.

Now we can re-describe every dot not by where it is, but by how far it is from center.

Step 3 — Re-measure every dot as a deviation from center

WHAT. For each dot, form two new numbers: how far right of center it is, $(x_{i} - \overset{x}{ˉ})$ , and how far above center, $(y_{i} - \overset{y}{ˉ})$ . These are called deviations.

WHY. We just slid the origin to the balance point. Now the sign of a deviation carries meaning:

$(x_{i} - \overset{x}{ˉ}) > 0$ → this row is older than average.
$(x_{i} - \overset{x}{ˉ}) < 0$ → younger than average.

Same for income vertically. This is the single move that unlocks everything: correlation is a question about signs of deviations.

PICTURE. One dot is highlighted. Its horizontal red arrow is $(x_{i} - \overset{x}{ˉ})$ ; its vertical red arrow is $(y_{i} - \overset{y}{ˉ})$ . Both point away from the crosshair.

Step 4 — Multiply the two deviations: the quadrant vote

WHAT. For each dot, multiply its two deviations together to get one number: $(x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$ .

WHY — this is the heart of correlation. Why multiply, and not add or subtract? Because a product of two signed numbers asks exactly the right question: "are these two deviations on the same side of zero?"

Trace the sign through all four quadrants — every case is covered:

Quadrant	$(x_{i} - \overset{x}{ˉ})$	$(y_{i} - \overset{y}{ˉ})$	product
top-right (old and rich)	$+$	$+$	$+$
bottom-left (young and poor)	$-$	$-$	$+$
top-left (young but rich)	$-$	$+$	$-$
bottom-right (old but poor)	$+$	$-$	$-$

A dot that agrees with "they rise together" (top-right or bottom-left) casts a positive vote. A dot that contradicts it casts a negative vote. A dot sitting exactly on a center line casts a zero vote (one factor is $0$ ). Multiplication is the voting machine.

PICTURE. The four quadrants are labelled with the sign the product takes there. The red-shaded quadrants are the "positive vote" ones.

Step 5 — Add up all the votes: covariance

WHAT. Sum every dot's vote and divide by $n$ . This average vote is the covariance, written $Cov (X, Y)$ — read "the covariance of variable $X$ with variable $Y$ ", where $X$ and $Y$ are the two whole columns from Step 1.

WHY. One dot's vote is noise. The sum is the crowd's verdict. If positive-vote quadrants win, the total is positive → the cloud tilts up. If negative quadrants win, total is negative → tilts down. If they cancel, the total is near zero → no linear tilt. Dividing by $n$ turns "total vote" into "average vote per dot", so a dataset with more rows isn't automatically judged more correlated.

PICTURE. The best-fit tilt of the cloud emerges. An up-tilting cloud stacks its dots into the two positive quadrants, so the votes pile up positive.

But covariance has a flaw, and Step 6 fixes it.

Step 6 — The scale problem: why covariance can't stand alone

WHAT. Covariance carries units and depends on scale. Measure income in dollars vs. cents and the covariance jumps by a factor of 100 — even though the cloud looks identical.

WHY it's a problem. Covariance answers "which way does it tilt?" but not "how tightly?". Two clouds with the same tilt but wildly different spreads give wildly different covariances. We want a number that means the same thing whether age is in years or months, income in dollars or cents. We need to strip out the units.

PICTURE. Two clouds, identical shape and tilt — the left in dollars, the right in cents. The red covariance value balloons on the right for the same relationship. That's the bug.

Step 7 — Divide by the spreads: correlation is born

WHAT. Divide covariance by the product of the two standard deviations $σ_{X}$ and $σ_{Y}$ . A standard deviation is the typical size of a deviation — the natural "ruler" for each axis. We must pin down its formula now, because the whole cancellation depends on it:

WHY divide by exactly this? Covariance is "votes measured in axis-units". $σ_{X}$ is the size of one horizontal unit-of-spread; $σ_{Y}$ the vertical one. Dividing by both cancels both units, leaving a pure, unitless number. It also caps the result: it is impossible for the votes to sum to more than the total spread allows, which forces the answer into the range $[- 1, + 1]$ . Equivalently, we first rescale each axis by its own $σ$ (turning both into unitless z-scores), then take covariance — same thing.

PICTURE. The dollar cloud and the cent cloud from Step 6, now each axis divided by its $σ$ . They become the same picture, and the red $r$ value is identical. Units gone, scale gone.

Step 8 — The edge cases: what makes $r$ hit $\pm 1$ , $0$ , or break

WHAT & WHY. Cover every degenerate scenario so you're never surprised:

$r = + 1$ — all dots on one upward straight line. Every vote is positive, the divide-by-spread caps it exactly at $1$ (Step 7 example).
$r = - 1$ — all dots on one downward straight line. Every vote negative → $- 1$ .
$r = 0$ — the up-votes and down-votes cancel. Two sub-cases:
1. a random blob (no tilt), or
2. a perfect U-shape / circle — a strong relationship that Pearson cannot see, because it's not linear. $r$ only measures straight-line agreement.
$r$ undefined (division by zero) — one variable is constant ( $σ = 0$ ). If age never varies, $(x_{i} - \overset{x}{ˉ}) = 0$ for every dot, so the ruler is $0$ and we can't divide. There is literally no "spread" to correlate against.

PICTURE. Four mini-clouds side by side: $r = + 1$ , $r = - 1$ , $r = 0$ (blob), $r \approx 0$ (U-shape). The red one is the trap — a clear pattern that $r$ reports as zero.

The one-picture summary

Every stage on one canvas: dots → center → deviations → signed product per quadrant → sum & divide → a number in $[- 1, 1]$ .

Recall Feynman retelling — say it back in plain words

I have two columns of numbers, so I drop one dot per row onto a plot: horizontal = variable one, vertical = variable two. I find the middle of the cloud — the average across and the average up — and plant a crosshair there. Now for each dot I ask two yes/no-ish questions: is it to the right of center, and is it above center? I turn those into signed distances and multiply them. Multiplying is clever: if the dot is high in both (or low in both) I get a positive number — it "agrees" the two things rise together. If it's high in one and low in the other, I get a negative number — it "disagrees". I add up every dot's vote. Positive total = the cloud tilts up; negative = tilts down; near zero = no straight-line tilt. But this total is measured in weird mixed units, so I divide by how spread out each axis is (its standard deviation). That cancels the units and squeezes the answer into $- 1$ to $+ 1$ . Plus-one is a perfect up-line, minus-one a perfect down-line, zero is either a shapeless blob or a curve my straight-line ruler simply can't detect — which is exactly why I always eyeball the scatter plot too.

Recall Quick self-test

Why do we multiply the two deviations instead of adding them? ::: A signed product is positive only when both deviations share a sign (agree) and negative when they disagree — it detects same-side-ness. Addition would mix incompatible units and couldn't flip sign on disagreement. Why divide by $σ_{X} σ_{Y}$ ? ::: To cancel the units of both axes and force $r$ into $[- 1, + 1]$ , making it comparable across variable pairs. Does dividing covariance by $n$ or $n - 1$ change $r$ ? ::: No — the factor cancels between numerator and denominator, so both conventions give the identical $r$ . When is $r$ undefined? ::: When a variable is constant ( $σ = 0$ ) — dividing by zero spread. Does $r = 0$ mean no relationship? ::: No — only no linear relationship; a U-shape can have $r \approx 0$ .

Connections: parent Exploratory data analysis (EDA) workflow · foundations in 2.1.15-Statistical-measuresfor-data-analysis · pair with 2.1.16-Data-visualization-techniques · feeds 2.2.1-Introduction-to-feature-engineering and 2.2.8-Feature-selection-methods · watch 3.1.1-Bias-variance-tradeoff when correlated features inflate variance.