Exploratory data analysis (EDA) workflow: Exercises

Before we start, one tiny reminder of the two formulas we lean on most. If you have not met them, treat these boxes as the definition.

Formula The two workhorse formulas

Skewness measures lopsidedness of a distribution: $Skewness = \frac{E [ ( X - μ ) ^{3} ]}{σ ^{3}}$ The symbol $E [\cdot]$ is the expectation: it just means "the average value of whatever is inside the brackets, taken over all the data points". So $E [(X - μ)^{3}]$ is literally "the average of the cubed distances from the mean". Here $μ$ is the mean (the balance point) and $σ$ is the standard deviation (typical distance from the mean). Cubing keeps the sign, so the average comes out positive when big values sit far to the right, negative when they sit far to the left — that sign is what tells you which way the long tail points.

Pearson correlation measures straight-line association between two numbers $x$ and $y$ : $r = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2} \sum _{i = 1}^{n} ( y _{i} - y ˉ ) ^{2}}$ Here $\overset{x}{ˉ}$ means "average of the $x$ values", and $n$ is the number of paired points. $r$ always lands between $- 1$ and $+ 1$ .

Level 1 — Recognition

L1.1

A dataset command returns df.shape = (50000, 12). In plain words, what are these two numbers, and which EDA stage produced them?

Recall Solution

df.shape returns (n_rows, n_columns).

50000 = number of rows = number of samples (e.g. customers).
12 = number of columns = number of features (measured properties).

This belongs to Stage 1: Data Collection & Initial Inspection (recapped above) — the very first "how big and what shape is this?" check.

L1.2

For each variable below, say whether you'd start with numerical or categorical univariate analysis: age, blood_type, monthly_income, city_name, number_of_children.

Recall Solution

age → numerical (measured quantity, ordered).
blood_type → categorical (A, B, AB, O — labels, no arithmetic).
monthly_income → numerical.
city_name → categorical (often high-cardinality).
number_of_children → numerical (a count; discrete but still a quantity you can average).

L1.3

A skewness value of $- 1.4$ describes a distribution. Which side is the long tail on, and is the mean above or below the median?

Recall Solution

Skewness $< 0$ ⇒ left-skewed ⇒ the long tail points to the left (toward small values). The tail pulls the mean toward it, so here mean $<$ median.

Level 2 — Application

L2.1

Compute the Pearson correlation $r$ for these paired points: $x = [1, 2, 3, 4, 5]$ , $y = [2, 4, 5, 4, 5]$ . Round to 3 decimals.

Recall Solution

Step 1 — means (the balance points). $\overset{x}{ˉ} = \frac{1 + 2 + 3 + 4 + 5}{5} = 3$ , $\overset{y}{ˉ} = \frac{2 + 4 + 5 + 4 + 5}{5} = 4$ .

Step 2 — centred values $(x_{i} - \overset{x}{ˉ})$ and $(y_{i} - \overset{y}{ˉ})$ : $x$ -dev: $[- 2, - 1, 0, 1, 2]$ , $y$ -dev: $[- 2, 0, 1, 0, 1]$ .

Step 3 — numerator $\sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$ : $(- 2) (- 2) + (- 1) (0) + (0) (1) + (1) (0) + (2) (1) = 4 + 0 + 0 + 0 + 2 = 6$ .

Step 4 — the two spread terms. $\sum (x_{i} - \overset{x}{ˉ})^{2} = 4 + 1 + 0 + 1 + 4 = 10$ . $\sum (y_{i} - \overset{y}{ˉ})^{2} = 4 + 0 + 1 + 0 + 1 = 6$ .

Step 5 — combine: $r = \frac{6}{10 6} = \frac{6}{60} \approx \frac{6}{7.746} \approx 0.775.$ A strong positive linear relationship.

L2.2

A column income has 10000 rows total, 8700 of which are non-null. What percentage is missing?

Recall Solution

Missing count $= 10000 - 8700 = 1300$ . $percent missing = \frac{1300}{10000} \times 100 = 13%.$ Above the common $\sim 5%$ "ignore it" threshold, so this needs a deliberate handling strategy — see 2.1.3-Handling-missing-data.

L2.3

A feature user_id has 9876 unique values in a 10000-row table. Compute its cardinality ratio (unique ÷ rows). What does the number tell you?

Recall Solution

$cardinality ratio = \frac{9876}{10000} = 0.9876 \approx 98.8%.$ Nearly every row is unique, so user_id behaves like an identifier, not a predictive feature. Drop it from modelling inputs.

Level 3 — Analysis

L3.1

For age: mean $= 34.5$ , median $= 32.0$ , standard deviation $= 12.3$ . Without computing exact skewness, state the sign of the skew and one modelling action it suggests.

Recall Solution

Mean $>$ median $(34.5 > 32.0)$ ⇒ positive (right) skew: a long tail of older customers pulls the mean up. Action: consider a log transformation to compress the tail and make the distribution more symmetric, which stabilises many models.

The figure below shows exactly this shape: a histogram of ages (teal bars) with the orange solid line marking the mean and the plum dashed line marking the median. Notice the orange mean line sits to the right of the plum median line — the long right tail, called out by the arrow, is dragging the mean toward the large values. That gap between the two lines is the visual fingerprint of positive skew.

Figure — Exploratory data analysis (EDA) workflow

Figure L3.1 — Right-skewed age distribution: mean (orange) pulled above median (plum) by the long right tail.

L3.2

A correlation heatmap shows $r_{age, income} = 0.65$ , $r_{income, spending} = 0.93$ . Which pair signals a problem for a linear model, and what is that problem called?

Recall Solution

$r = 0.93$ between income and spending is dangerously high. When two predictors are strongly correlated, the model cannot tell their effects apart — this is multicollinearity. Consequence: unstable, hard-to-interpret coefficients. Fix options: drop one feature, or combine them — see 2.2.8-Feature-selection-methods. The $0.65$ age–income link is moderate and generally fine.

L3.3

Grouped statistics of purchase_amount by category:

category	mean	std
Electronics	250	80
Books	45	15

Compute each group's coefficient of variation (std ÷ mean) and say which category is relatively more variable.

Recall Solution

Coefficient of variation (CV) normalises spread by size, so we can compare across different scales: $CV_{Elec} = \frac{80}{250} = 0.32, CV_{Books} = \frac{15}{45} = 0. \overline{3} \approx 0.333.$ Books' absolute std (15) is much smaller, but relative to its own mean it is slightly more variable ( $0.333 > 0.32$ ). Raw std alone would have fooled you.

Level 4 — Synthesis

L4.1

You load a dataset and observe: income is 13% missing, right-skewed (skew $= 1.2$ ), and correlated $r = 0.9$ with net_worth. Write the ordered EDA-driven plan for handling income, and justify the order.

Recall Solution

First, three terms for why data goes missing (you must classify this before filling anything):

MCAR = Missing Completely At Random: the gaps are pure chance, unrelated to any value.
MAR = Missing At Random: the gaps depend on other observed variables (e.g. income missing more often for a certain city, which you can see).
MNAR = Missing Not At Random: the gaps depend on the hidden value itself (e.g. high earners refuse to report income) — the dangerous case.

Order matters — each step feeds the next:

Assess the missingness pattern first (MCAR / MAR / MNAR). If income tends to be missing for high earners, that is MNAR and a naive fill would bias everything downstream. (Stage 4 → 2.1.3-Handling-missing-data.)
Exploit the $r = 0.9$ with net_worth to impute smartly: predict missing income from net_worth rather than dropping rows — the strong correlation makes this reliable.
Only then treat skew (log transform, skew $= 1.2$ is meaningfully right-skewed). Doing this after imputation keeps the transform consistent across real and filled values.
Re-check the $r = 0.9$ : near-duplicate predictors risk multicollinearity (2.2.8-Feature-selection-methods); consider keeping only one for the final model.

Why this order? You must understand the missingness (2.1.3) before filling, fill before transforming (transform assumes complete data), and resolve redundancy last (a modelling concern, 3.1.1-Bias-variance-tradeoff).

L4.2

Two features have identical Pearson $r = 0.0$ , but one scatter is a clear parabola (U-shape) and the other is a random cloud. Explain how both give $r = 0$ , and which EDA tool would distinguish them.

Recall Solution

Pearson $r$ measures linear (straight-line) association only. For a symmetric U-shape, the left half has a negative slope and the right half a positive slope; their contributions to the numerator $\sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$ cancel to zero — even though $x$ perfectly determines $y$ .

The random cloud: no relationship at all, so $r = 0$ honestly.
The parabola: a strong non-linear relationship hidden from $r$ .

Distinguishing tool: the scatterplot itself (from 2.1.16-Data-visualization-techniques). This is precisely why the parent note insists you plot, never trust a single correlation number. A rank measure (Spearman) also won't catch symmetric U-shapes — only the visual does.

The figure below makes the cancellation visible. The plum dots trace the parabola $y = x^{2}$ ; the orange dashed line marks the mean of $y$ . On the left half (teal arrow) the curve slopes down, giving negative deviation-products; on the right half the curve slopes up, giving positive products of equal size. Summed, they cancel exactly — the title confirms $r = 0.00$ despite the perfect dependence you can plainly see.

Figure L4.2 — A perfect U-shape ( $y = x^{2}$ ) yields Pearson $r = 0$ because left- and right-half deviation products cancel.

Level 5 — Mastery

L5.1

Design a complete EDA checklist (in order) for a fresh 500,000-row × 40-column customer churn dataset where the target churned is 8% positive. For each stage name the one biggest risk you are guarding against.

Recall Solution

A full workflow, each stage justified by the specific risk it neutralises:

Load & inspect (shape, info, head) — Stage 1. Risk: wrong dtypes (dates read as text) silently breaking later math. See 2.1.1-Introduction-to-data-preprocessing.
Univariate — numerical & categorical — Stage 2. Risk: undetected skew/heavy tails and high-cardinality ID columns masquerading as features (2.1.15-Statistical-measuresfor-data-analysis).
Outlier scan — part of Stage 4. Risk: extreme values distorting means and correlations (2.1.5-Outlier-detection-and-treatment).
Missingness audit — Stage 4. Risk: MNAR (missing-not-at-random) patterns that would bias imputation (2.1.3-Handling-missing-data).
Bivariate vs target churned — Stage 3. Risk: missing a strong predictor; use grouped stats + correlation-to-target.
Multicollinearity check (correlation matrix) — Stage 3. Risk: redundant predictors destabilising the model (2.2.8-Feature-selection-methods).
Class-imbalance note (8% positive) — Stage 4. Risk: a model that predicts "never churn" scoring 92% accuracy while being useless — flag for resampling/metric choice.
Hypothesis generation — Stage 5. Risk: jumping straight to modelling with untested hunches; instead write down concrete, testable guesses (e.g. "customers with >3 support tickets churn more"), each tied to a feature you'll build in 2.2.1-Introduction-to-feature-engineering.
Communication & documentation — Stage 6. Risk: untracked decisions no one can reproduce; record every finding, transformation, and assumption so the team (and future-you) can audit the pipeline, mindful of the 3.1.1-Bias-variance-tradeoff choices made along the way.

The through-line: understand → clean → relate → guard against imbalance → hypothesise → document, mirroring the parent's six stages but sharpened for a churn task.

L5.2

The 8% positive rate above means a "predict everyone stays" model gets 92% accuracy. In one worked sentence each, (a) show that accuracy number and (b) name a metric that exposes the uselessness.

Recall Solution

(a) Predicting the majority class ("stays") for all rows is correct on the $100% - 8% = 92%$ who indeed stay: $accuracy = \frac{460000}{500000} = 0.92 = 92%.$ (b) Recall on the churn class (true churners caught) $= 0/40000 = 0%$ — it exposes that the model finds none of the actual churners. The F1-score (harmonic mean of precision and recall) collapses to 0 too. This is why, under imbalance, accuracy is a trap and recall/F1 are the honest metrics.

Recall Quick self-test

Q — Right-skew means mean is above or below the median? A — Above the median (long tail pulls the mean toward large values).

Q — Why can $r = 0$ still hide a strong relationship? A — $r$ only measures straight-line trend; a U-shape or curve gives $r = 0$ while $y$ still depends on $x$ .

Q — Coefficient of variation is defined as? A — Standard deviation divided by the mean (a scale-free spread).

Q — On an 8%-positive target, why is 92% accuracy meaningless? A — A model predicting "always negative" scores 92% while catching 0% of the positives.