Exploratory data analysis (EDA) workflow — Hinglish notes

2.1.14 · AI-ML › Data Preprocessing & Feature Engineering

The Complete EDA Workflow

EDA workflow ek structured path follow karta hai — raw data se lekar modeling readiness tak:

Figure — Exploratory data analysis (EDA) workflow

Stage 1: Data Collection & Initial Inspection

WHY? Deep jaane se pehle aapko shape, structure, aur basic properties samajhni hogi.

WHAT to check:

Dimensions: rows (samples) aur columns (features) ki sankhya
Data types: numerical, categorical, datetime, text
Memory usage: kya yeh RAM mein fit ho sakta hai?
First/last rows: format ka sanity check

HOW:

# Basic inspection commands
df.shape          # (n_rows, n_columns)
df.info()         # types, non-null counts, memory
df.head(10)       # first 10 rows
df.tail(10)       # last 10 rows
df.columns.tolist()  # feature names

Worked example Initial Inspection ka Example

import pandas as pd
df = pd.read_csv('customer_data.csv')
print(df.shape)  # (1000, 15)
print(df.info())

Output interpretation:

10,000 customers, 15 features
age int64 hai (numerical ke liye theek hai)
purchase_date object hai (datetime conversion chahiye)
income mein sirf 9,500 non-null hain (500 missing values) Yeh step kyun? Yeh turant data type mismatches aur missing data reveal karta hai, jo baad mein ghanton ki debugging bachata hai.

Stage 2: Univariate Analysis

WHY? Interactions study karne se pehle har variable ko alag se samajhna zaroori hai.

WHAT to analyze:

Numerical variables ke liye:

Central tendency: mean, median, mode
Spread: standard deviation, IQR, range
Shape: skewness, kurtosis
Distribution: histogram, KDE plot, boxplot

$Skewness = \frac{E [( X - μ ) ^{3} ]}{σ ^{3}}$

First principles se derivation:

Moments ke concept se shuru karo: $E [(X - μ)^{k}]$ measure karta hai ki data mean se kitna deviate karta hai
$k = 1$ : deviation (definition se hamesha zero)
$k = 2$ : variance $σ^{2}$ (spread)
$k = 3$ : raw asymmetry measure
$σ^{3}$ se divide karo taaki standardize ho (scale-invariant ban jaye)

Skewness = 0: symmetric Skewness > 0: right tail (mean > median) Skewness < 0: left tail (mean < median)

Categorical variables ke liye:

Frequency counts: har category mein kitne
Proportions: relative frequencies
Cardinality: unique values ki sankhya
Mode: sabse common category

Worked example Univariate Analysis: Numerical

import numpy as np
import matplotlib.pyplot as plt
 
# Age distribution
ages = df['age']
print(f"Mean: {ages.mean():.2f}")      # 34.5
print(f"Median: {ages.median():.2f}")  # 32.0
print(f"Std: {ages.std():.2f}")        # 12.3
print(f"Skewness: {ages.skew():.2f}")  # 0.8 (right-skewed)
# Visualization
plt.hist(ages, bins=30, edgecolor='black')
plt.axvline(ages.mean(), color='red', label='Mean')
plt.axvline(ages.median(), color='blue', label='Median')
plt.legend()

Yeh kyun important hai: Mean (34.5) > Median (32.0) right skew confirm karta hai (positive skewness). Yeh batata hai ki outliers mean ko upar khich rahe hain (zyada umar ke customers). Modeling ke liye, normalize karne ke liye log transformation apply karni pad sakti hai.

Worked example Univariate Analysis: Categorical

# Product category distribution
category_counts = df['product_category'].value_counts()
print(category_counts)
 
# Electronics    4500 (45%)
# Clothing       3000 (30%)
# Books          1500 (15%)
# Food           1000 (10%)
# Check for high cardinality
print(f"Unique categories: {df['product_category'].nunique()}")  # 4
print(f"Unique user_ids: {df['user_id'].nunique()}")  # 9876

Yeh step kyun? High cardinality (user_id mein 10,000 rows mein se 9,876 unique values) matlab yeh ek identifier hai, modeling ke liye useful feature nahi. Electronics 45% pe dominate karta hai, jo classification tasks ke liye class imbalance create karta hai.

Stage 3: Bivariate & Multivariate Analysis

WHY? Real-world patterns variables ke interactions se emerge hote hain, individual distributions se nahi.

WHAT to examine:

Numerical vs Numerical:

Correlation coefficient (Pearson, Spearman)
Trend lines ke saath scatterplots
Correlation matrix heatmap

$r = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2} \sum _{i = 1}^{n} ( y _{i} - y ˉ ) ^{2}}$

Derivation:

Covariance se shuru karo: $Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})]$
Covariance linear relationship measure karta hai lekin scale-dependent hota hai
Scale-invariant banane ke liye, standard deviations ke product se divide karo
$r = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}$
-1 (perfect negative) se +1 (perfect positive) tak range hota hai

Categorical vs Numerical:

Grouped boxplots (categories mein distributions compare karo)
Grouped statistics (mean/median per category)
Significance ke liye ANOVA F-test

Categorical vs Categorical:

Contingency tables (crosstabs)
Independence ke liye Chi-square test
Stacked bar charts

Worked example Bivariate: Numerical vs Numerical

# Correlation between age and income
correlation = df['age'].corr(df['income'])
print(f"Pearson r: {correlation:.3f}")  # 0.654
 
# Scatter plot
plt.scatter(df['age'], df['income'], alpha=0.5)
plt.xlabel('Age (years)')
plt.ylabel('Income ($)')
plt.title(f'Age vs Income (r={correlation:.3f})')
# Add regression line
from scipy.stats import linregress
slope, intercept, r_value, p_value, std_err = linregress(df['age'], df['income'])
line = slope * df['age'] + intercept
plt.plot(df['age'], line, color='red', labelBest fit')

Yeh step kyun? $r = 0.654$ strong positive correlation indicate karta hai. Jaise-jaise age badhti hai, income bhi badhti hai. Relationship itni strong hai ki age ko income-related targets ke liye predictor maana ja sakta hai.

Regression line kyun? Visual confirmation ki relationship approximately linear hai (curved nahi). Agar points fan out karein ya curve karein, toh Pearson correlation misleading ho sakta hai.

Worked example Bivariate: Categorical vs Numerical

# Income distribution by product category
df.boxplot(column='purchase_amount', by='product_category', figsize=(10,6))
plt.ylabel('Purchase Amount ($)')
plt.title('Purchase Amount by Category')
# Group statistics
grouped = df.groupby('product_category')['purchase_amount'].agg(['mean', 'median', 'std'])
print(grouped)

Output:

               mean  median   std
Electronics       250.00  220.00  80.00
Clothing          120.00  110.00  45.00
Books              45.00   40.00  15.00
Food               30.00   28.00  10.00

Yeh kyun important hai: Electronics ka mean $250 hai, B oo k s ($ 45) se 5 guna zyada. High std ($80) wide variation dikhata hai. Agar purchase amount predict karni hai, toh product category ek critical feature hai.

Formula Multiple Variables ke liye Correlation Matrix

$p$ variables ke liye, correlation matrix $R$ ek $p \times p$ symmetric matrix hai: $R = [1 r_{p 1} r_{12} r_{p 2} \dots \dots r_{1 p} r_{21} 1 1 \dots r_{2 p} ⋮ ⋮ ⋱ ⋮]$

Jahan $r_{ij} = Cor (X_{i}, X_{j})$ aur $r_{ii} = 1$ (variable khud se perfectly correlate karta hai).

import seaborn as sns
cor_matrix = df['age', 'income', 'purchase_amount', 'num_visits']].corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', center=0)

Heatmap kyun? Turant multicollinearity spot karo (predictors ke beech high correlation, jaise $r > 0.9$ ) jo regression models ko harm kar sakta hai.

Stage 4: Data Quality Assessment

WHY? Garbage in, garbage out. Model performance data quality se cap hoti hai.

WHAT to check:

Missing values:
- Har column mein percentage missing
- Pattern of missingness: MCAR (completely random), MAR (doosre variables diye hue random), MNAR (random nahi)
```
missing_pct = (df.isnull().sum() / len(df)) * 100
print(missing_pct[missing_pct > 0].sort_values(ascending=False))
```
Outliers:
- Statistical definition: $Q_{1} - 1.5 \times I QR$ se neeche ya $Q_{3} + 1.5 \times I QR$ se upar
- Z-score method: $∣ z ∣ > 3$ (mean se 3 std devs se zyada)
- Domain knowledge: age = 200 saal impossible hai
$z_{i} = \frac{x _{i} - μ}{σ}$

Z-score kyun? Normal distribution assume karta hai. Standardization se derived: hum maapte hain "mean se kitne standard deviations door."

Duplicates:

duplicates = df.duplicated().sum()
df.drop_duplicates(inplace=True)

Inconsistencies:
- Categorical typos: "Male", "male", "M" ek hi category ke liye
- Range violations: age = -5, percentage = 150%
- Format issues: dates as strings, mixed currency symbols

Worked example Missing Values Handle Karna

# Check missing patterns
import misingno as msno
msno.matrix(df)  # Visualize where nulls occur
# Quantify
missing_summary = pd.DataFrame({
   'column': df.columns,
   'missing_count': df.isnull().sum(),
   'missing_pct': (df.isnull().sum() / len(df)) * 100
})
print(missing_summary['missing_pct'] > 0])

Output:

column          missing_count  missing_pct
income          500            5.0
education       1200           12.0

Decision tree:

Income 5% missing → median se Impute karo (outliers ke liye robust)
Education 12% missing → "Unknown" category banao ya mode se impute karo
Agar >30% missing → Column drop karne par vichar karo

Hamesha drop kyun nahi? 5% missing chhota hai; 500 rows drop karne se doosre columns ka valuable data bhi chala jata hai.

Worked example Outlier Detection

# IQR method
Q1 = df['income'].quantile(0.25)  # 25000
Q3 = df['income'].quantile(0.75)  # 75000
IQR = Q3 - Q1                # 5000
lower_bound = Q1 - 1.5 * IQR      # -5000 (impossible)
upper_bound = Q3 + 1.5 * IQR      # 15000
 
outliers = df[(df['income'] < lower_bound) | (df['income'] > upper_bound)]
print(f"Outliers: {len(outliers)} ({len(outliers)/len(df)*100:.2f}%)")
 
# Visualize
plt.boxplot(df['income'])
plt.ylabel('Income ($)')

1.5 × IQR kyun? Tukey ka empirically derived rule. Normal distribution ke liye, ~99.3% data capture karta hai. 1.5 multiplier sensitivity (true outliers pakadna) aur specificity (false positives avoid karna) ke beech balance karta hai.

Action: Outliers automatically delete mat karo. Investigate karo: Kya yeh errors hain (typo: 1000 ki jagah 1000) ya genuine hain (CEOs earning $500k+)? Domain knowledge decide karta hai.

Stage 5: Hypothesis Generation

WHY? EDA ko testable hypotheses produce karni chahiye jo modeling aur feature engineering guide karein.

WHAT to do:

Observed patterns list karo
"Agar X, toh Y" statements banao
Evidence ki strength aur business impact ke hisaab se hypotheses prioritize karo

Worked example Hypothesis Generation

EDA findings ke aadhar par: Hypothesis 1: "30-45 age ke customers jo income >$75k rakhte hain, Electronics purchase karne ki 3 guna zyada probability hai"

Evidence: Bivariate analysis ne dikhaya ki mean Electronics purchase age = 38, mean income = $82k
Test: Electronics purchase ke liye binary classifier banao, age & income feature importance measure karo
Impact: Targeted marketing ke liye high-value segment

Hypothesis 2: "Purchase amount, site visits ki sankhya ke saath non-linearly increase karta hai"

Evidence: Scatterplot ne polynomial relationship dikhaya (quadratic term ke saath r² 0.45 se 0.72 ho gaya)
Test: num_visits² feature add karo, regression performance compare karo
Impact: Behtar sales forecasting

Hypothesis 3: "Missing income values random nahi hain; yeh lower education levels se correlate karte hain"

Evidence: Missing income wale 80% rows mein education = "High School" ya "Unknown" tha
Test: Missingness predict karne ke liye doosre features se logistic regression (MAR test)
Impact: Imputation strategy inform karta hai (education pe condition karo)

Stage 6: Communication & Documentation

WHY? EDA insights tab tak bekar hain jab tak stakeholders (data scientists, product managers, executives) tak communicate na ho.

WHAT to deliver:

Summary report:
- Dataset overview (size, features, target)
- Key findings (distributions, correlations, anomalies)
- Data quality issues aur resolutions
- Modeling ke liye hypotheses
Visualizations:
- Annotated plots (key regions highlight karo)
- Interactive exploration ke liye dashboard
Recommendations:
- Kaunse features engineer karney hain
- Kaunsi rows drop/keep karni hain
- Kaun se preprocessing steps chahiye

Common mistake Common EDA Mistakes

Mistake 1: Correlation = Causation

Kyun sahi lagta hai: Ice cream sales aur drowning deaths ke beech high correlation ( $r = 0.9$ ) matlab ice cream se drowning hoti hai!
Steel-man: Reasoning valid statistical evidence use karta hai (strong correlation).
Fix: Confounding variables. Dono summer mein badhte hain (garmi). Correlation association measure karta hai, causation nahi. Controlled experiments ya causal inference methods chahiye.

Mistake 2: Non-linear relationships ignore karna

Kyun sahi lagta hai: Pearson $r = 0.05$ (weak) toh "X aur Y unrelated hain."
Steel-man: Correlation standard metric hai jo sikhaya jata hai.
Fix: Pearson sirf linear relationships detect karta hai. Ek perfect parabola (Y = X²) ka r ≈ 0 ho sakta hai. Hamesha scatterplots plot karo. Monotonic non-linear relationships ke liye Spearman (rank-based) use karo.

Mistake 3: Sample bias overlook karna

Kyun sahi lagta hai: "Mere dataset mein 1 million rows hain, statistically significant!"
Steel-man: Large sample sizes random error reduce karte hain.
Fix: Bias ≠ variance. Agar aapka training data sirf 20-year-olds ka hai, toh koi bhi sample size usse fix nahi karega. Check karo ki data distribution deployment population se match karti hai ya nahi. Survey response bias, survivorship bias, selection bias size se regardless persist karte hain.

Mistake 4: Death by a thousand histograms

Kyun sahi lagta hai: "Thorough rehne ke liye main har variable plot karunga."
Steel-man: Comprehensive analysis achhi practice hai.
Fix: Signal pe focus karo. Pehle summary statistics se filter karo (jaise sirf wahi features plot karo jinmein high variance ya target se correlation ho). 200 auto-generated nahi, 20 key plots visualize karo. Quality > quantity.

Recall EDA ko ek 12-saal ke bachche ko explain karo

Socho tum ek detective ho jise abhi random puzzle pieces ka ek dabba mila hai (tumhara dataset). Puzzle banana shuru karne se pehle (machine learning model), tumhe yeh karna hoga:

Pieces giraao aur gino (kitne rows/columns hain?)
Colour aur shape se sort karo (univariate: har variable kaisi dikhti hai?)
Dhundho kaun se pieces connect hote hain (bivariate: kya age aur income relate karte hain?)
Missing ya toote pieces notice karo (data quality: missing values, outliers handle karo)
Final picture ke baare mein andaza lagao (hypothesis: kaunse patterns help karenge?)
Apne teammates ko findings dikhao (communication: visualizations share karo)

Yahi hai EDA! Tum models banana shuru karne se pehle apne data ko deeply samajh rahe ho. Jaise tum pieces sort kiye bina puzzle shuru nahi karoge, waise hi EDA ke bina machine learning shuru nahi karte.

Connections

2.1.1-Introduction-to-data-preprocessing - EDA cleaning se pehle ka pehla step hai
2.1.15-Statistical-measuresfor-data-analysis - EDA mein use hone wale statistical tools
2.1.16-Data-visualization-techniques - Visualization EDA ka core hai
2.1.3-Handling-missing-data - Missing value handling EDA se inform hoti hai
2.1.5-Outlier-detection-and-treatment - Outlier detection EDA ka hissa hai
2.2.1-Introduction-to-feature-engineering - EDA findings feature engineering drive karti hain
2.2.8-Feature-selection-methods - EDA important features identify karta hai
3.1.1-Bias-variance-tradeoff - EDA mein discover hone wala sample bias model bias ko affect karta hai

#flashcards/ai-ml

EDA workflow ke chhe stages kya hain? :: 1) Data Collection & Loading, 2) Univariate Analysis, 3) Bivariate/Multivariate Analysis, 4) Data Quality Assessment, 5) Hypothesis Generation, 6) Communication

Pearson correlation coefficient ka formula kya hai?

r = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2} \sum _{i = 1}^{n} ( y _{i} - y ˉ ) ^{2}}

ya equivalently

r = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}

Skewness calculate karne ke liye hum third moment ko

σ^{3}

se kyun divide karte hain?

Isse scale-invariant banane ke liye. Raw third moment

E [(X - μ)^{3}]

X ki units pe depend karta hai;

σ^{3}

se divide karne par yeh standardize ho jaata hai taaki skewness alag-alag scales mein comparable ho.

IQR outlier detection rule kya hai?

Outliers woh values hain jo

Q_{1} - 1.5 \times I QR

se neeche ya

Q_{3} + 1.5 \times I QR

se upar hain, jahan

I QR = Q_{3} - Q_{1}

Pearson correlation relationships detect karne mein kab fail ho jaata hai?

Jab relationship non-linear ho. Pearson sirf linear association measure karta hai. Ek perfect parabola (Y = X²) ka r ≈ 0 ho sakta hai. Non-linear monotonic relationships ke liye scatterplots ya Spearman rank correlation use karo.

Missing data ke teen types kya hain?

1) MCAR (Missing Completely At Random), 2) MAR (Missing At Random given other variables), 3) MNAR (Missing Not At Random - missingness unobserved values pe depend karti hai)

Positive skewness mean aur median ke baare mein kya bataata hai?

Positive skewness (right tail) ka matlab hai mean > median. Tail mean ko upar kheenchi hai jabki median outliers ke liye resistant rehta hai.

Correlation aur causation mein kya fark hai?

Correlation variables ke beech statistical association measure karta hai. Causation ka matlab hai ki ek variable directly doosre mein changes cause karta hai. Correlation confounding variables ki wajah se ho sakta hai (dono ek teesre factor se caused) bina causation ke.

Missing values impute karne ke liye mean ki jagah median kyun use karte hain?

Median outliers ke liye robust hai. Agar data mein extreme values hain, toh mean outliers ki taraf khich jaata hai, jo distorted central tendency deta hai. Median skewed distributions ke liye "typical" value better represent karta hai.

Multicollinearity kya hai aur yeh kyun matter karta hai?

Multicollinearity predictor variables ke beech high correlation (r > 0.8-0.9) hai. Yeh regression coefficients ko unstable aur interpret karna mushkil bana deta hai, standard errors inflate karta hai, aur matrix inversion mein numerical issues cause karta hai.

Correlation matrix heatmap quickly kya reveal karta hai?

1) Kaun se features redundant hain (high correlation pairs), 2) Kaun se features target variable se correlate karte hain (predictive power), 3) Regression modeling ke liye multicollinearity issues

EDA mein grouped boxplots ka kya purpose hai?

Categories mein ek numerical variable ki distributions compare karna. Groups ke beech central tendency, spread, aur outliers mein differences dikhata hai, yeh identify karne mein help karta hai ki categorical variable numerical wale ko influence karta hai ya nahi.

Outlier detection ke liye Z-score kyun create karte hain?

Z-score data ko "mean se standard deviations ki sankhya" mein standardize karta hai. Z = (x - μ)/σ. |Z| > 3 wali values unusual hain (normal distribution ke 99.7% se bahar). Outlier detection ko scale-invariant banata hai.

EDA ke liye DUHAC mnemonic kya hai?

Dimensions, Univariate, Hypothesize, Associate, Clean — systematic exploratory data analysis workflow ke liye ek framework.

Outliers automatically delete kyun nahi karte?

Outliers ho sakte hain: 1) Genuine extreme values (high income wale CEOs), 2) Model robustness ke liye valuable edge cases, 3) Fraud/anomaly detection mein signal. Delete karne se pehle domain knowledge se investigate karo.