Handling missing values (deletion, imputation strategies)

2.1.2 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Missing data sabse common real-world data quality issues mein se ek hai. Yeh samajhna ki data kyun missing hai aur sahi handling strategy choose karna directly model performance aur bias ko affect karta hai.

Figure — Handling missing values (deletion, imputation strategies)

Core Concepts

Definition Types of Missingness (Rubin's Taxonomy)

MCAR (Missing Completely At Random): Missingness ka kisi bhi variable se zero relationship hai. Example: Ek sensor randomly 5% time fail ho jaata hai.

Impact: Deletion se loss of power hota hai lekin koi bias nahi.

MAR (Missing At Random): Missingness observed variables par depend karti hai, missing value par nahi. Example: Younger patients income questions zyada skip karte hain.

Impact: Deletion results ko bias karta hai. Observed variables use karke imputation true distribution recover kar sakta hai.

MNAR (Missing Not At Random): Missingness unobserved value par depend karti hai. Example: High earners income report karne se mana kar dete hain.

Impact: Koi bhi standard technique isko fully correct nahi karti. Domain knowledge ya sensitivity analysis zaroori hai.

Strategy 1: Deletion Methods

Listwise Deletion (Complete Case Analysis)

Pairwise Deletion

Sirf specific missing values ko delete karo jab statistics compute karo, poori rows ko nahi.

Strategy 2: Imputation Methods

Mean/Median/Mode Imputation

Forward Fill / Backward Fill (Time Series)

K-Nearest Neighbors (KNN) Imputation

Worked example KNN Imputation: Worked Example

Dataset: Patients with Height, Weight, Age. Patient A ka Weight missing hai.

| Patient | Height | Weight | Age | |---------|--------|-----| | A | 170 | NaN | 30 | | B | 168 | 70 | 28 | | C | 172 | 75 | 32 | | D | 165 | 65 | 45 |

Step 1: Distances compute karo (Weight ignore karke)

d (A, B) = (170 - 168)^{2} + (30 - 28)^{2} = 8 \approx 2.83

d (A, C) = (170 - 172)^{2} + (30 - 32)^{2} = 8 \approx 2.83

d (A, D) = (170 - 165)^{2} + (30 - 45)^{2} = 250 \approx 15.81

Step 2: $K = 2$ choose karo → Neighbors hain B aur C

Step 3: Weight impute karo

Weight_{A} = \frac{w _{B} \cdot 70 + w _{C} \cdot 75}{w _{B} + w _{C}}

Agar uniform weights use karein: $\frac{70 + 75}{2} = 72.5$ kg Yeh mean se better kyun hai: Patient A height/age mein B aur C se similar hai (dono athletic hain), D se nahi (older, alag body type). Global mean mein D ka weight bhi shamil ho jaata.

Model-Based Imputation (MICE)

Strategy Choose Karna: Decision Framework

Formula Decision Tree

1. Is missing % < 5% and MCAR? 
  → Listwise deletion
2. Is it time series? 
  → Forward/Backward fill

3. Is feature relationship weak (low correlation)?
  → Mean/Median imputation

4. Are features corelated?
  → KNN or MICE

5. Is missing % > 40% in one feature?
  → Consider dropping the feature entirely

6. MNAR suspected?
  → Add "is_missing" indicator feature + impute

Missingness Indicator Features

Practical Implementation Tips

Pehle hamesha explore karo: missingno library se missing patterns plot karo
Target ke hisaab se stratify karo: Check karo ki missingness classes ke beech alag hai ya nahi (selection bias)
Pipeline integration: Data leakage se bachne ke liye pipelines mein sklearn.impute classes use karo
Cross-validation: Imputation parameters (jaise KNN ka $K$ ) sirf training folds par tune kiye jaane chahiye

Recall Ek 12-Saal Ke Bacche Ko Explain Karo

Socho tum apni class mein "Favorite ice cream flavor" ka survey kar rahe ho lekin kuch bacchon ne jawab nahi diya.

Deletion: Un bacchon ke poore surveys phenk do. Problem: Shayad quiet bacchon ne jawab nahi diya, toh ab tum sirf loud bacchon ki suno ge.

Mean imputation: Kaho "Hum pretend karenge ki jinhone jawab nahi diya unhe vanilla (sabse common) pasand hai." Problem: Ab lagta hai vanilla asal mein hai usse bhi zyada popular hai!

KNN imputation: Similar bacchon ko dhundho (same age, same grade, same hobbies) aur guess karo ki unhe wahi pasand hoga jo un similar bacchon ko pasand hai. Yeh smarter hai lekin zyada kaam leta hai.

Best trick: Yeh bhi likh lo ki "Is bacche ne jawab nahi diya" as extra info. Shayad jo bacche jawab nahi dete unme koi common baat hoti hai jo matter karti hai!

Connections

2.1.01-Data-quality-issues-and-detection: Missing data ek quality issue hai jo profiling se detect hoti hai
2.1.03-Outlier-detection-and-treatment: Outliers missingness cause kar sakte hain (sensors extremes par fail ho jaate hain)
2.2.01-Feature-scaling-and-normalization: Leakage se bachne ke liye scaling imputation ke baad karni chahiye
3.1.02-Training-validation-test-split: Imputation training set par fit hoti hai, validation/test par apply hoti hai
4.3.01-Bias-variance-tradeoff: Simple imputation = high bias, MICE = higher variance
5.2.03-Data-leakage-prevention: Split se pehle impute karna test set ki information leak karta hai

#flashcards/ai-ml

Rubin's taxonomy mein missingness ke teen types kaun se hain?

MCAR (Missing Completely At Random), MAR (Missing At Random), MNAR (Missing Not At Random)

Mean imputation variance kyun underestimate karta hai?

Yeh missing values ko mean se replace karta hai, jo distribution ke center par points add karta hai aur spread kam kar deta hai. Variance formula mein mean se squared deviations hote hain.

Listwise deletion kab unbiased hoti hai?

Sirf tab jab data MCAR (Missing Completely At Random) ho. MAR/MNAR ke liye, deletion selection bias introduce karta hai.

KNN imputation woh problem kyun solve karta hai jo mean imputation nahi kar sakta?

KNN local feature relationships respect karta hai—similar samples ki similar values hoti hain. Mean baaki sabhi features ko ignore karta hai.

Missingness indicator feature kya hota hai aur ise kyun use karte hain?

Ek binary flag (0/1) jo mark karta hai ki koi value originally missing thi. Useful hai kyunki missing hone ka fact imputation ke baad bhi predictive ho sakta hai.

MICE (Multiple Imputation by Chained Equations) kaise kaam karta hai?

Iteratively har missing data wale feature ko baaki features use karke model karta hai, convergence tak cycles mein imputations update karta hai. Correlations preserve karta hai.

Pairwise deletion ka trade-off kya hai?

Listwise deletion se zyada data preserve karta hai, lekin alag statistics alag subsets use karte hain, jo potentially inconsistent correlation matrices create karta hai.

Ek feature ko impute karne ki jagah drop kab karna chahiye?

Jab missing % > 40-50% ho aur missingness MNAR ho, ya jab feature ka predictive power observed hone par bhi weak ho.

Time series ke liye forward fill appropriate kyun hai?

Temporal autocorrelation assume karta hai—values time ke saath slowly change hoti hain. Last observation carry forward karna global statistics se zyada accurate hota hai.

[100, 105, NaN, NaN, 112] ke liye forward fill kya produce karta hai?

[100, 105, 105, 105, 112] — dono NaNs last known value (105) se fill ho jaate hain.

Imputation ke context mein data leakage kya hai?

Poore dataset par—test data including—imputer fit karna (jaise mean compute karna ya KNN neighbors dhundhna), phir impute karna. Sirf training data par fit karna zaroori hai.