Exploratory data analysis (EDA) workflow

The Complete EDA Workflow

The EDA workflow follows a structured path from raw data to modeling readiness:

Figure — Exploratory data analysis (EDA) workflow

Stage 1: Data Collection & Initial Inspection

WHY? You need to understand the shape, structure, and basic properties before diving deep.

WHAT to check:

Dimensions: number of rows (samples) and columns (features)
Data types: numerical, categorical, datetime, text
Memory usage: can it fit in RAM?
First/last rows: sanity check on format

HOW:

# Basic inspection commands
df.shape          # (n_rows, n_columns)
df.info()         # types, non-null counts, memory
df.head(10)       # first 10 rows
df.tail(10)       # last 10 rows
df.columns.tolist()  # feature names

Worked example Initial Inspection Example

import pandas as pd
df = pd.read_csv('customer_data.csv')
print(df.shape)  # (1000, 15)
print(df.info())

Output interpretation:

10,000 customers, 15 features
age is int64 (good for numerical)
purchase_date is object (needs datetime conversion)
income has only 9,500 non-null (500 missing values) Why this step? Immediately reveals data type mismatches and missing data, saving hours of debugging later.

Stage 2: Univariate Analysis

WHY? You must understand each variable in isolation before studying interactions.

WHAT to analyze:

For numerical variables:

Central tendency: mean, median, mode
Spread: standard deviation, IQR, range
Shape: skewness, kurtosis
Distribution: histogram, KDE plot, boxplot

$\text{Skewness} = \frac{E[(X - \mu)^3]}{\sigma^3}$

Derivation from first principles:

Start with the concept of moments: $E[(X - \mu)^k]$ measures how data deviates from mean
$k=1$ : deviation (always zero by definition)
$k=2$ : variance $\sigma^2$ (spread)
$k=3$ : raw asymetry measure
Divide by $\sigma^3$ to standardize (make scale-invariant)

Skewness = 0: symmetric Skewness > 0: right tail (mean > median) Skewness < 0: left tail (mean < median)

For categorical variables:

Frequency counts: how many per category
Proportions: relative frequencies
Cardinality: number of unique values
Mode: most common category

Worked example Univariate Analysis: Numerical

import numpy as np
import matplotlib.pyplot as plt
 
# Age distribution
ages = df['age']
print(f"Mean: {ages.mean():.2f}")      # 34.5
print(f"Median: {ages.median():.2f}")  # 32.0
print(f"Std: {ages.std():.2f}")        # 12.3
print(f"Skewness: {ages.skew():.2f}")  # 0.8 (right-skewed)
# Visualization
plt.hist(ages, bins=30, edgecolor='black')
plt.axvline(ages.mean(), color='red', label='Mean')
plt.axvline(ages.median(), color='blue', label='Median')
plt.legend()

Why this matters: Mean (34.5) > Median (32.0) confirms right skew (positive skewness). This tells us there are outliers pulling the mean up (older customers). For modeling, we might need to apply log transformation to normalize.

Worked example Univariate Analysis: Categorical

# Product category distribution
category_counts = df['product_category'].value_counts()
print(category_counts)
 
# Electronics    4500 (45%)
# Clothing       3000 (30%)
# Books          1500 (15%)
# Food           1000 (10%)
# Check for high cardinality
print(f"Unique categories: {df['product_category'].nunique()}")  # 4
print(f"Unique user_ids: {df['user_id'].nunique()}")  # 9876

Why this step? High cardinality (user_id has 9,876 unique values out of 10,000 rows) means it's an identifier, not a useful feature for modeling. Electronics dominates at 45%, creating class imbalance for classification tasks.

Stage 3: Bivariate & Multivariate Analysis

WHY? Real-world patterns emerge from interactions between variables, not individual distributions.

WHAT to examine:

Numerical vs Numerical:

Correlation coefficient (Pearson, Spearman)
Scatterplots with trend lines
Correlation matrix heatmap

$r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i - \bar{y})^2}}$

Derivation:

Start with covariance: $\text{Cov}(X,Y) = E[(X - \mu_X)(Y - \mu_Y)]$
Covariance measures linear relationship but is scale-dependent
To make scale-invariant, divide by the product of standard deviations
$r = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$
Ranges from -1 (perfect negative) to +1 (perfect positive)

Categorical vs Numerical:

Grouped boxplots (compare distributions across categories)
Grouped statistics (mean/median per category)
ANOVA F-test for significance

Categorical vs Categorical:

Contingency tables (crosstabs)
Chi-square test for independence
Stacked bar charts

Worked example Bivariate: Numerical vs

# Correlation between age and income
correlation = df['age'].corr(df['income'])
print(f"Pearson r: {correlation:.3f}")  # 0.654
 
# Scatter plot
plt.scatter(df['age'], df['income'], alpha=0.5)
plt.xlabel('Age (years)')
plt.ylabel('Income ($)')
plt.title(f'Age vs Income (r={correlation:.3f})')
# Add regression line
from scipy.stats import linregress
slope, intercept, r_value, p_value, std_err = linregress(df['age'], df['income'])
line = slope * df['age'] + intercept
plt.plot(df['age'], line, color='red', labelBest fit')

Why this step? $r = 0.654$ indicates strong positive correlation. As age increases, income tends to increase. The relationship is strong enough to consider age as a predictor for income-related targets.

Why the regression line? Visual confirmation that the relationship is approximately linear (not curved). If points fan out or curve, Pearson correlation might be misleading.

Worked example Bivariate: Categorical vs Numerical

# Income distribution by product category
df.boxplot(column='purchase_amount', by='product_category', figsize=(10,6))
plt.ylabel('Purchase Amount ($)')
plt.title('Purchase Amount by Category')
# Group statistics
grouped = df.groupby('product_category')['purchase_amount'].agg(['mean', 'median', 'std'])
print(grouped)

Output:

               mean  median   std
Electronics       250.00  220.00  80.00
Clothing          120.00  110.00  45.00
Books              45.00   40.00  15.00
Food               30.00   28.00  10.00

Why this matters: Electronics has mean $250, 5× higher than Books ($ 45). High std ($80) shows wide variation. If predicting purchase amount, product category is a critical feature.

Formula Correlation Matrix for Multiple Variables

For $p$ variables, the correlation matrix $\mathbf{R}$ is a $p \times p$ symmetric matrix: $\mathbf{R} = \begin{bmatrix} 1 & r_{12} & \cdots & r_{1p} \ r_{21} & 1 & \cdots & r_{2p} \ \vdots & \vdots & \ddots & \vdots \\ r_{p1} & r_{p2} & \cdots & 1 \end{bmatrix}$

Where $r_{ij} = \text{Cor}(X_i, X_j)$ and $r_{ii} = 1$ (variable perfectly correlates with itself).

import seaborn as sns
cor_matrix = df['age', 'income', 'purchase_amount', 'num_visits']].corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', center=0)

Why heatmap? Instantly spot multicollinearity (high correlation between predictors, e.g., $r > 0.9$ ) which can harm regression models.

Stage 4: Data Quality Assessment

WHY? Garbage in, garbage out. Model performance is capped by data quality.

WHAT to check:

Missing values:
- Percentage missing per column
- Pattern of missingness: MCAR (completely random), MAR (random given other variables), MNAR (not random)
```
missing_pct = (df.isnull().sum() / len(df)) * 100
print(missing_pct[missing_pct > 0].sort_values(ascending=False))
```
Outliers:
- Statistical definition: beyond $Q_1 - 1.5 \times IQR$ or $Q_3 + 1.5 \times IQR$
- Z-score method: $|z| > 3$ (more than 3 std devs from mean)
- Domain knowledge: age = 200 years is impossible
$z_i = \frac{x_i - \mu}{\sigma}$

Why Z-score? Assumes normal distribution. Derived from standardization: we measure "how many standard deviations away from mean."

Duplicates:

duplicates = df.duplicated().sum()
df.drop_duplicates(inplace=True)

Inconsistencies:
- Categorical typos: "Male", "male", "M for same category
- Range violations: age = -5, percentage = 150%
- Format issues: dates as strings, mixed currency symbols

Worked example Handling Missing Values

# Check missing patterns
import misingno as msno
msno.matrix(df)  # Visualize where nulls occur
# Quantify
missing_summary = pd.DataFrame({
   'column': df.columns,
   'missing_count': df.isnull().sum(),
   'missing_pct': (df.isnull().sum() / len(df)) * 100
})
print(missing_summary['missing_pct'] > 0])

Output:

column          missing_count  missing_pct
income          500            5.0
education       1200           12.0

Decision tree:

Income 5% missing → Impute with median (robust to outliers)
Education 12% missing → Create "Unknown" category or impute with mode
If >30% missing → Consider dropping the column

Why not always drop? 5% missing is small; dropping500 rows loses valuable data in other columns.

Worked example Outlier Detection

# IQR method
Q1 = df['income'].quantile(0.25)  # 25000
Q3 = df['income'].quantile(0.75)  # 75000
IQR = Q3 - Q1                # 5000
lower_bound = Q1 - 1.5 * IQR      # -5000 (impossible)
upper_bound = Q3 + 1.5 * IQR      # 15000
 
outliers = df[(df['income'] < lower_bound) | (df['income'] > upper_bound)]
print(f"Outliers: {len(outliers)} ({len(outliers)/len(df)*100:.2f}%)")
 
# Visualize
plt.boxplot(df['income'])
plt.ylabel('Income ($)')

Why 1.5 × IQR? Empirically derived rule from Tukey. For normal distribution, captures ~99.3% of data. The 1.5 multiplier balances sensitivity (catching true outliers) vs specificity (avoiding false positives).

Action: Don't automatically delete outliers. Investigate: Are they errors (typo: 1000 instead of 1000) or genuine (CEOs earning $500k+)? Domain knowledge decides.

Stage 5: Hypothesis Generation

WHY? EDA should produce testable hypotheses that guide modeling and feature engineering.

WHAT to do:

List observed patterns
Form "If X, then Y" statements
Prioritize hypotheses by strength of evidence and business impact

Worked example Hypothesis Generation

Based on EDA findings: Hypothesis 1: "Customers aged 30-45 with income >$75k are3× more likely to purchase Electronics"

Evidence: Bivariate analysis showed mean Electronics purchase age = 38, mean income = $82k
Test: Build binary classifier for Electronics purchase, measure age & income feature importance
Impact: High-value segment for targeted marketing

Hypothesis 2: "Purchase amount increases non-linearly with number of site visits"

Evidence: Scatterplot showed polynomial relationship (r² improved from 0.45 to 0.72with quadratic term)
Test: Add num_visits² feature, compare regression performance
Impact: Better sales forecasting

Hypothesis 3: "Missing income values are not random; they correlate with lower education levels"

Evidence: 80% of missing income rows had education = "High School" or "Unknown"
Test: Logistic regression to predict missingness from other features (MAR test)
Impact: Informs imputation strategy (condition on education)

Stage 6: Communication & Documentation

WHY? EDA insights are useless if not communicated to stakeholders (data scientists, product managers, executives).

WHAT to deliver:

Summary report:
- Dataset overview (size, features, target)
- Key findings (distributions, correlations, anomalies)
- Data quality issues and resolutions
- Hypotheses for modeling
Visualizations:
- Annotated plots (highlight key regions)
- Dashboard for interactive exploration
Recommendations:
- Features to engineer
- Rows to drop/keep
- Preprocessing steps needed

Common mistake Common EDA Mistakes

Mistake 1: Correlation = Causation

Why it feels right: High correlation ( $r = 0.9$ ) between ice cream sales and drowning deaths must mean ice cream causes drowning!
Steel-man: The reasoning uses valid statistical evidence (strong correlation).
The fix: Confounding variables. Both increase summer (hot weather). Correlation measures association, not causation. Need controlled experiments or causal inference methods.

Mistake 2: Ignoring non-linear relationships

Why it feels right: Pearson $r = 0.05$ (weak) so "X and Y are unrelated."
Steel-man: Correlation is the standard metric taught.
The fix: Pearson only detects linear relationships. A perfect parabola (Y = X²) can have r≈ 0. Always plot scatterplots. Use Spearman (rank-based) for monotonic non-linear relationships.

Mistake 3: Overlooking sample bias

Why it feels right: "My dataset has 1million rows, statistically significant!"
Steel-man: Large sample sizes do reduce random error.
The fix: Bias ≠ variance. If your training data is all 20-year-olds, no sample size fixes that. Check if data distribution matches deployment population. Survey response bias, survivorship bias, selection bias all persist regardless of size.

Mistake 4: Death by a thousand histograms

Why it feels right: "I'll plot every variable to be thorough."
Steel-man: Comprehensive analysis is good practice.
The fix: Focus on signal. Use summary statistics to filter first (e.g., only plot features with high variance or correlation with target). Visualize 20 key plots, not 200auto-generated ones. Quality > quantity.

Recall ExplainDA to a 12-year-old

Imagine you're a detective who just got a box full of random puzzle pieces (your dataset). Before you start building the puzzle (machine learning model), you need to:

Dump out the pieces and count them (how many rows/columns?)
Sort by color and shape (univariate: what does each variable look like?)
Find which pieces connect (bivariate: do age and income relate?)
Notice missing or broken pieces (data quality: handle missing values, outliers)
Make guesses about the final picture (hypothesis: what patterns will help?)
Show your findings to teammates (communication: share visualizations)

That's EDA! You're understanding your data deeply before jumping into building models. Just like you wouldn't start a puzzle without sorting pieces first, you don't start machine learning without EDA.

Connections

2.1.1-Introduction-to-data-preprocessing - EDA is the first step before cleaning
2.1.15-Statistical-measuresfor-data-analysis - Statistical tools used in EDA
2.1.16-Data-visualization-techniques - Visualization is core to EDA
2.1.3-Handling-missing-data - Missing value handling informed by EDA
2.1.5-Outlier-detection-and-treatment - Outlier detection is part of EDA
2.2.1-Introduction-to-feature-engineering - EDA findings drive feature engineering
2.2.8-Feature-selection-methods - EDA identifies important features
3.1.1-Bias-variance-tradeoff - Sample bias discovered in EDA affects model bias

#flashcards/ai-ml

What are the six stages of the EDA workflow? :: 1) Data Collection & Loading, 2) Univariate Analysis, 3) Bivariate/Multivariate Analysis, 4) Data Quality Assessment, 5) Hypothesis Generation, 6) Communication

What is the formula for Pearson correlation coefficient?

r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i - \bar{y})^2}}

or equivalently

r = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}

Why do we divide the third moment by

\sigma^3

to calculate skewness?

To make it scale-invariant. Raw third moment

E[(X - \mu)^3]

depends on the units of X; dividing by

\sigma^3

standardizes it so skewness is comparable across different scales.

What is the IQR outlier detection rule?

Outliers are values below

Q_1 - 1.5 \times IQR

or above

Q_3 + 1.5 \times IQR

, where

IQR = Q_3 - Q_1

When does Pearson correlation fail to detect relationships?

When the relationship is non-linear. Pearson only measures linear association. A perfect parabola (Y = X²) can have r ≈ 0. Use scatterplots or Spearman rank correlation for non-linear monotonic relationships.

What are the three types of missing data patterns?

1) MCAR (Missing Completely At Random), 2) MAR (Missing At Random given other variables), 3) MNAR (Missing Not At Random - missingness depends on unobserved values)

What does positive skewness tell you about mean vs median?

Positive skewness (right tail) means mean > median. The tail pulls the mean upward while median remains resistant to outliers.

What is the difference between correlation and causation?

Correlation measures statistical association between variables. Causation means one variable directly causes changes in another. Correlation can exist due to confounding variables (both caused by a third factor) without causation.

Why use median instead of mean for imputing missing values?

Median is robust to outliers. If data has extreme values, mean gets pulled toward outliers, giving a distorted central tendency. Median represents the "typical" value better for skewed distributions.

What is multicollinearity and why does it matter?

Multicollinearity is high correlation (r > 0.8-0.9) between predictor variables. It makes regression coefficients unstable and difficult to interpret, inflates standard errors, and causes numerical issues in matrixversion.

What does a correlation matrix heatmap reveal quickly?

1) Which features are redundant (high correlation pairs), 2) Which features correlate with target variable (predictive power), 3) Multicollinearity issues for regression modeling

What is the purpose of grouped boxplots in EDA?

To compare distributions of a numerical variable across categories. Shows differences in central tendency, spread, and outliers between groups, helping identify if the categorical variable influences the numerical one.

Why create a Z-score for outlier detection?

Z-score standardizes data to "number of standard deviations from mean." Z = (x - μ)/σ. Values with |Z| > 3 are unusual (beyond 99.7% of normal distribution). Makes outlier detection scale-invariant.

What is the DUHAC mnemonic for EDA?

Dimensions, Univariate, Hypothesize, Associate, Clean - a framework for systematic exploratory data analysis workflow.

Why not automatically delete outliers?

Outliers might be: 1) Genuine extreme values (CEOs with high income), 2) Valuable edge cases for model robustness, 3) Signal in fraud/anomaly detection. Investigate with domain knowledge before deleting.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo, EDA ko ek detective ki tarah samjho. Jaise doctor kisi bhi treatment se pehle patient ke vitals check karta hai, waise hi tumhe model banane se pehle apne data ko achhe se samajhna padta hai. EDA basically ek systematic process hai jisme tum visualization aur statistics ka use karke apne data se sawaal poochte ho - kya patterns hain, kya broken hai, aur kaun se assumptions safe hain. Iska core idea yahi hai ki bina data ko samjhe agar tum seedhe modeling mein kood gaye, toh baad mein hours barbaad honge debugging mein.

Workflow simple hai - pehle data load karke initial inspection karo (kitne rows-columns hain, data types kya hain, missing values kahan hain). Yeh df.shape, df.info(), aur df.head() jaise commands se hota hai jo turant reveal kar deta hai ki kahan gadbad hai. Phir univariate analysis aata hai jisme tum har variable ko akele study karte ho - numerical ke liye mean, median, standard deviation aur skewness dekhte ho, aur categorical ke liye frequency counts. Skewness ek important concept hai jo batata hai ki data symmetric hai ya kisi ek taraf jhuka hua hai - agar mean median se bada hai toh data right-skewed hai matlab kuch outliers mean ko upar kheench rahe hain.

Yeh sab important kyun hai? Kyunki EDA se hi tumhe pata chalta hai ki feature engineering mein kya karna hai. Jaise agar age right-skewed hai, toh tum log transformation apply kar sakte ho taki data model ke liye better ban jaye. Bina EDA ke tum blindly model banaoge aur galat assumptions ki wajah se results kharab aayenge. Toh yaad rakho - accha data scientist banna hai toh pehle apne data se dosti karo, uska character samjho, tabhi accha model bana paoge.

Exploratory data analysis (EDA) workflow

The Complete EDA Workflow

Stage 1: Data Collection & Initial Inspection

Stage 2: Univariate Analysis

Stage 3: Bivariate & Multivariate Analysis

Stage 4: Data Quality Assessment

Stage 5: Hypothesis Generation

Stage 6: Communication & Documentation

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Data Preprocessing & Feature Engineering

Connections