Exploratory data analysis (EDA) workflow: Foundations

This page is a toolbox opening. The parent note throws around $μ$ , $σ$ , $E [\cdot]$ , skewness, covariance, correlation $r$ , and matrix notation as if you already own them. Here we buy each tool from an empty shelf, in the order you actually need them. Nothing is used before it is built.

0. The starting picture: what a dataset even is

Everything below sits on top of one picture. Hold it in your head.

Figure — Exploratory data analysis (EDA) workflow

Why this comes first: every symbol later is about a single column (like the age column) or about two columns together (age vs income). If you don't see the rectangle, the symbols float.

1. $x_{i}$ , $n$ , and $\sum$ — naming the numbers in one column

Pick ONE numerical column, say age. It is a list of numbers.

2. The mean $\overset{x}{ˉ}$ (also written $μ$ ) — the balance point

Now our first real summary number.

Why the topic needs it: the mean is the single number that answers "what's a typical value here?" — the first thing you report in univariate analysis.

Definition Median

The median is the middle value when you line the numbers up in order. Half the data sits below it, half above.

Picture: stand all customers in a line, shortest income to tallest. The person exactly in the middle has the median income.

The even- $n$ edge case: if $n$ is odd, there is one exact middle person, and their value is the median. If $n$ is even, there is no single middle — two people tie for the centre. The rule is to average those two middle values. Example: for the ordered list $2, 4, 6, 8$ ( $n = 4$ ), the two middle values are $4$ and $6$ , so the median is $(4 + 6) /2 = 5$ .

Why keep both mean and median? The mean gets dragged by extreme values (one billionaire lifts the whole average). The median doesn't budge — it only cares about position. Comparing the two is how the parent note detects skew (Section 5).

3. Deviation $(x_{i} - \overset{x}{ˉ})$ — how far each point is from typical

Every spread and relationship formula is built from this one brick.

4. Squaring, $σ^{2}$ , and $σ$ — measuring spread

Definition Population vs sample variance (

$n$ vs $n - 1$ ) The formula above divides by $n$ — this is the population variance, correct when your data is the whole group you care about.

But usually your dataset is just a sample drawn from a bigger population, and you're using it to estimate the true spread. It turns out dividing by $n$ then slightly underestimates the true variance — because you measured deviations from $\overset{x}{ˉ}$ (computed from the same sample), not from the true unknown $μ$ , and $\overset{x}{ˉ}$ always sits a touch closer to your own points. The fix is to divide by $n - 1$ instead: $s^{2} = \frac{1}{n - 1} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}$ This sample variance $s^{2}$ is the unbiased estimator — on average it hits the true value. Its square root $s$ is the sample standard deviation.

Rule of thumb: use $n - 1$ ( $s^{2}$ , $s$ ) when analysing sample data (almost always in EDA — this is pandas .std()'s default); use $n$ ( $σ^{2}$ , $σ$ ) only when the data is the entire population. For large $n$ the difference is tiny.

Why the topic needs it: spread tells you whether a mean is trustworthy. Mean income of $50k with $\sigma=\$ 2 $k i s a t i g h t, r e l iab l e g r o u p; t h es am e m e an w i t h$ \sigma=$40$k is a wild mix. Both are used later in outlier detection.

5. $E [\cdot]$ and moments — the machine behind skewness

The parent note writes $Skewness = \frac{E [( X - μ ) ^{3} ]}{σ ^{3}}$ . Let's earn every piece.

Definition Expectation

$E [\cdot]$ $E [\cdot]$ reads "the expected value of" — it is just a fancy, general word for average. Concretely, for any quantity computed per row, $E [that quantity] = \frac{1}{n} \sum_{i = 1}^{n} (that quantity for row i)$ — you build it for every row, add, and divide by $n$ . So $E$ is nothing new; it's the $\frac{1}{n} \sum$ machine from Sections 1–2 wearing a shorter name.

With that, $E [X] = μ$ is literally "the average of $X$ is the mean," and $E [(X - μ)^{2}]$ is exactly the variance $σ^{2}$ from Section 4 — the average of squared deviations. Same idea, cleaner clothing.

On the capital letters: $X$ (capital) names the whole column as one object — "the age variable." A lowercase $x_{i}$ is one specific value from it. $E [\dots X \dots]$ says "run the $\frac{1}{n} \sum$ over all the $x_{i}$ values of column $X$ ." So $E [(X - μ)^{2}]$ unpacks to $\frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}$ .

Definition Moments

A == $k$ -th moment== is $E [(X - μ)^{k}]$ — the average of the deviation raised to the power $k$ . Each power asks a different question:

$k$	Quantity	What it measures
$1$	$E [(X - μ)]$	always $0$ (Section 3)
$2$	$E [(X - μ)^{2}] = σ^{2}$	spread
$3$	$E [(X - μ)^{3}]$	lopsidedness

This is why the parent's age example (mean $34.5$ , median $32.0$ , so mean is the larger of the two) is called right-skewed — the two-number comparison and the sign of skewness agree.

6. From one column to two: $Cov$ and $r$

So far every tool described one column. Relationships (Stage 3) need two columns, $x$ and $y$ , measured on the same rows.

How these foundations feed the topic

Read it upward: the rectangle gives you columns; columns give you sums; sums give the mean; the mean gives deviations; deviations branch into spread (variance/ $σ$ ) and lopsidedness (skewness), and pair up into covariance and correlation. All of it pours into the parent EDA workflow.

Equipment checklist

Cover the right side and answer each. If any stalls you, reread its section above before opening the parent note.

What does the subscript

i

x_{i}

do?

Points to which value in the column — a walking ticket number from

1

n

What does

\sum_{i = 1}^{n} x_{i}

mean?

Add up every value in the column, from the first to the

n

-th.

Why is

\overset{x}{ˉ}

called a balance point?

It's the pivot where the see-saw of marbles balances; deviations on each side cancel.

How do you compute the median when

n

is even?

Order the values and average the two middle ones — e.g. for

2, 4, 6, 8

the median is

(4 + 6) /2 = 5

Why do the raw deviations

(x_{i} - \overset{x}{ˉ})

average to zero?

Positive and negative arrows cancel exactly — that's what the mean is.

Why do we square deviations for variance?

To remove signs (so they don't cancel) and to punish far-away points more heavily.

When do you divide by

n

n - 1

n

for a whole population (

σ^{2}

);

n - 1

for a sample you're estimating from (

s^{2}

, the unbiased estimator) — the EDA default.

Why take the square root to get

σ

To return to the original units so the spread is human-readable.

What does

E [\cdot]

mean in plain words, concretely?

"The average of" the bracketed quantity — literally

\frac{1}{n} \sum

over all rows.

Why does the cube (

k = 3

) detect skew when the deviation-sum can't?

Cubing keeps the sign, so a long tail's large cubes don't cancel — the sum leans positive or negative.

What does dividing by

σ^{3}

(or

σ_{X} σ_{Y}

) achieve?

Removes units, making the number scale-free and comparable across columns.

What is the difference between covariance and correlation

r

Same "move together" idea, but

r

is standardized into the fixed range

[- 1, + 1]

so its size is meaningful.

Why is

r

undefined for a constant column?

That column has

σ = 0

, so

r = \frac{0}{0}

— a never-moving column can't "move together" with anything.

What does

r_{ij}

with two subscripts represent, and what is

R

r_{ij}

is the correlation between column

i

and column

j

; the full grid of all such pairs (with

1

s on the diagonal) is the correlation matrix

R

Recall Self-test: mean vs median for skew

A column has mean $34.5$ and median $32.0$ . Which way is it skewed and what sign of skewness do you expect? Answer: mean $>$ median → a long right tail → positive skewness.