Missing data is one of the most common real-world data quality issues. Understanding why data is missing and choosing the right handling strategy directly impacts model performance and bias.
Always explore first: Plot missing patterns with the missingno library
Stratify by target: Check if missingness differs between classes (selection bias)
Pipeline integration: Use sklearn.impute classes in pipelines to avoid data leakage
Cross-validation: Imputation parameters (e.g., KNN's K) must be tuned on training folds only
Recall Explain to a 12-Year-Old
Imagine you're doing a class survey on "Favorite ice cream flavor" but some kids didn't answer.
Deletion: Throw away those kids' entire surveys. Problem: Maybe quiet kids didn't answer, so now you only hear from loud kids.
Mean imputation: Say "We'll pretend everyone who didn't answer likes vanilla (the most common)." Problem: Now it looks like vanilla is even more popular than it really is!
KNN imputation: Find kids who are similar (same age, same grade, same hobbies) and guess they like whatever those similar kids like. This is smarter but takes more work.
Best trick: Also write down "This kid didn't answer" as extra info. Maybe kids who don't answer have something in common that matters!
What are the three types of missingness in Rubin's taxonomy?
MCAR (Missing Completely At Random), MAR (Missing At Random), MNAR (Missing Not At Random)
Why does mean imputation underestimate variance?
It replaces missing values with the mean, which adds points at the center of the distribution, reducing spread. Variance formula includes squared deviations from mean.
When is listwise deletion unbiased?
Only when data is MCAR (Missing Completely At Random). For MAR/MNAR, deletion introduces selection bias.
What problem does KNN imputation solve that mean imputation doesn't?
KNN respects local feature relationships—similar samples have similar values. Mean ignores all other features.
What is a missingness indicator feature and why use it?
A binary flag (0/1) marking whether a value was originally missing. Useful because the fact of missingness can be predictive even after imputation.
How does MICE (Multiple Imputation by Chained Equations) work?
Iteratively models each feature with missing data using other features, updating imputations in cycles until convergence. Preserves correlations.
What is the trade-off of pairwise deletion?
Preserves more data than listwise deletion, but different statistics use different subsets, potentially creating inconsistent correlation matrices.
When should you drop a feature entirely instead of imputing?
When missing % > 40-50% and missingness is MNAR, or when the feature has weak predictive power even when observed.
Why is forward fill appropriate for time series?
Assumes temporal autocorrelation—values change slowly over time. Carrying forward the last observation is more accurate than global statistics.
For [100, 105, NaN, NaN, 112], what does forward fill produce?
[100, 105, 105, 105, 112] — both NaNs are filled with the last known value (105).
What is data leakage in the context of imputation?
Fitting the imputer (e.g., computing mean or KNN neighbors) on the full dataset including test data, then imputing. Must fit only on training data.
Chalo, ise simple tarike se samajhte hain. Missing data ka matlab hai ki aapke dataset mein kuch values khali (NaN) hain, aur real-world data mein yeh bahut common problem hai. Sabse important baat jo yeh note bata raha hai woh yeh hai ki kyun data missing hai, yeh samajhna crucial hai. Rubin ne teen types diye hain: MCAR (bilkul random, jaise sensor kabhi-kabhi fail ho jaye), MAR (missingness kisi observed variable pe depend karti hai, jaise young patients income skip karte hain), aur MNAR (missingness khud us hidden value pe depend karti hai, jaise high earners apni income chupate hain). Yeh mechanism decide karta hai ki aap jo strategy use karenge woh bias laayegi ya nahi.
Ab strategies dekhein. Pehla option hai deletion — listwise mein jis bhi row mein koi NaN hai, poori row hata do. Simple hai, lekin agar 10 features mein har ek mein 5% missing ho, toh aap kareeb 40% data kho dete hain! Aur agar data MAR ya MNAR hai, toh yeh biased result deta hai. Pairwise deletion thoda smart hai — sirf specific missing values ko ignore karta hai jab statistics compute karte hain, isse zyada data bachta hai. Dusra option hai imputation — yaani missing value ko kisi estimate se bhar do, jaise mean (symmetric data ke liye), median (jab outliers ho), ya mode (categorical data ke liye).
Ab yeh why-it-matters wala part hai jo yaad rakhna zaroori hai: jab aap mean imputation karte hain, toh saari missing values ek hi number se bhar jaati hain, jisse variance artificially kam ho jaata hai (example mein 31.25 se 25 ho gaya). Iska matlab aapka model soch sakta hai ki data actual se zyada consistent hai, jo galat conclusions de sakta hai. Isliye missing values handle karna sirf "khali jagah bharna" nahi hai — yeh aapke model ki accuracy, fairness aur generalization ko directly affect karta hai. Medical diagnosis jaise real applications mein galat strategy poore model ko biased bana sakti hai, isliye pehle missingness ka type samjho, phir strategy choose karo.
