Outlier detection and treatment

2.1.3 · AI-ML › Data Preprocessing & Feature Engineering

Asli sawaal yeh hai: Kya yeh outlier kuch important bata raha hai, ya yeh hamare model ko corrupt kar raha hai?

Outliers Ko Samajhna

Outliers kyun matter karte hain?

1. Model sensitivity: Bahut saare ML algorithms (linear regression, k-means, PCA) mean-based hote hain aur extreme values ke liye highly sensitive hote hain
2. Gradient instability: Bade outliers steep gradients create karte hain, jisse optimization oscillate karne lagta hai
3. Feature scale distortion: Outliers variance inflate karte hain, jisse normalization ineffective ho jaati hai
4. Decision boundary shift: Classification mein, outliers boundaries ko apni taraf kheenchte hain

Ek point outlier kya banata hai?

• Statistical deviation (jaise, mean se >3σ door)
• Feature space mein isolation
• Low local density
• Domain-specific thresholds (jaise, human age > 150 years)

Detection Methods

1. Statistical Methods (Univariate)

$z_i=\frac{x_i-\mu }{\sigma }$

Derivation:

• Standard deviation ki definition se shuru karo: $\sigma =\sqrt{\frac{1}{n}\sum (x_i-\mu {)}^2}$
• Yeh $\mu$ ke aaspaas typical spread measure karta hai
• $(x_i-\mu )$ ko $\sigma$ se divide karne par ek dimensionless measure milta hai: "kitne typical spreads door?"
• Normal distributions ke liye 68-95-99.7 rule ke mutabik: ~99.7% data $|z|<3$ ke andar hota hai
• Yeh kyun kaam karta hai: Alag-alag scales mein comparison ko standardize karta hai

Threshold: $|z_i|>3$ (ya 2.5, tolerance par depend karta hai)

Yeh step kyun kaam karta hai: Z-score absolute distances ko relative distances mein convert karta hai, jisse yeh scale-invariant ban jaata hai.

$\text{IQR}=Q_3-Q_1$ $\text{Lower fence}=Q_1-1.5\times \text{IQR}$ $\text{Upper fence}=Q_3+1.5\times \text{IQR}$

Derivation:

• $Q_1$ (25th percentile) aur $Q_3$ (75th percentile) data ke beech ke 50% ko define karte hain
• IQR is central bulk ka spread measure karta hai
• 1.5 kyun? Tukey ka empirical rule: sensitivity aur false positives ke beech balance karta hai. Normal data ke liye, ~0.7% ko outlier pakad ta hai
• Fences se baahir ke points bulk se door hain, chahe distribution skewed ho

Yeh step kyun kaam karta hai: Percentiles (rank-based) use karta hai, isliye extreme values se immune hai jo mean/std ko distort kar deti hain.

Z-score approach:

mean = 43.6, std = 42.8
z_150 = (150 - 43.6) / 42.8 = 2.48

Yeh step kyun? Hum distance ko standardize kar rahe hain. Result: 150 flag NAHI hua (|z| < 3) kyunki outlier ne khud hi std inflate kar diya!

IQR approach:

Q1 = 25.5, Q3 = 33.5, IQR = 8
Upper fence = 33.5 + 1.5*8 = 45.5

Yeh step kyun? Hum "normal" define karne ke liye beech ke 50% use karte hain, extremes ko ignore karke. Result: 150 > 45.5, correctly flagged.

Lesson: Jab outliers mean/std ko contaminate kar dein, tab IQR robust hai.

2. Multivariate Methods

$D_M(x)=\sqrt{(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )}$

jahan $\mu$ mean vector hai, $\Sigma$ covariance matrix hai.

First principles se Derivation:

1. ${\mathbb{R}}^d$ mein Euclidean distance: $D_E=\sqrt{(x-\mu {)}^T(x-\mu )}$
   • Problem: Saari dimensions ko equally treat karta hai, correlations ignore karta hai
2. Har dimension ko standardize karo: ${\sigma }_i$ se divide karo → "units of spread" mein distance
   • Problem: Phir bhi correlations ignore karta hai (jaise, height aur weight correlated hain)
3. Covariance account karo: Covariance matrix $\Sigma$ dono variances (diagonal) aur correlations (off-diagonal) encode karta hai
4. Whitening transformation: ${\Sigma }^{-1/2}(x-\mu )$ data ko unit sphere mein transform karta hai jahan dimensions uncorrelated hain
5. Final distance: $(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$ is transformed space mein squared distance hai

Yeh kyun kaam karta hai:

• Agar features independent hain, $\Sigma$ diagonal hai → standardized Euclidean par reduce ho jaata hai
• Agar correlated hain, toh "redundant" dimensions ko down-weight karta hai
• Normality ke under ${\chi }^2$ distribution follow karta hai, ek statistical threshold deta hai

Threshold: $D_M>{\chi }_{0.975,d}^2$ ($d$ degrees of freedom ke saath chi-squared ka 97.5th percentile)

Mahalanobis kyun help karta hai:

• Euclidean distance dono ko mean [2000 sqft, 3 beds] se equally door treat karta hai
• Mahalanobis recognize karta hai ki sqft aur bedrooms correlated hain (r ≈ 0.8)
• House A is correlation ko violate karta hai → zyada $D_M$, anomaly ki tarah flag hota hai
• House B pattern follow karta hai → kam $D_M$, flag nahi hota

Step-by-step calculation (simplified 2D):

mu = [2000, 3]
Sigma = [[1e6, 0.8*1000*2],   # cov(sqft, beds) = r*σ_sqft*σ_beds
         [0.8*1000*2, 4]]
Sigma_inv = inverse(Sigma)
 
x_A = [5000, 2]
diff_A = x_A - mu = [3000, -1]
D_M_A = sqrt(diff_A^T @ Sigma_inv @ diff_A) ≈ 8.2

Yeh value kyun? Negative correlation component (bahut sqft lekin kam beds) distance mein add ho jaata hai.

3. Density-Based: Local Outlier Factor (LOF)

${\text{LOF}}_k(x)=\frac{{\sum }_{o\in N_k(x)}\frac{\text{lrd}(o)}{\text{lrd}(x)}}{|N_k(x)|}$

jahan local reachability density (lrd) hai:

${\text{lrd}}_k(x)=\frac{|N_k(x)|}{{\sum }_{o\in N_k(x)}{\text{reach-dist}}_k(x,o)}$

aur reachability distance:

${\text{reach-dist}}_k(x,o)=\max (\text{dist}(x,o),k\text{-dist}(o))$

Derivation logic:

1. k-distance: $o$ ke $k$-ve nearest neighbor tak ki distance
2. Reachability distance: "Smoothing" factor ki tarah $k$-dist use karta hai
   • Kyun? Jab points bahut paas hoon tab instability se bachata hai
   • Agar $x$ $o$ se door hai, actual distance use karo
   • Agar $x$ $o$ ki neighbourhood mein hai, stabilize karne ke liye $k$-dist(o) use karo
3. LRD: Average reachability distance ka inverse → zyada lrd = denser region
4. LOF: Neighbors ki density ka point ki density se ratio
   • LOF ≈ 1: neighbors jaisi density (normal)
   • LOF >> 1: neighbors se bahut kam density (outlier)

Yeh kyun kaam karta hai: Varying densities ke liye adapt karta hai. Ek sparse cluster mein ek point outlier nahi hai agar uske neighbors bhi sparse hain.

Dense cluster A: [points at (0,0) ± 0.5]
Dense cluster B: [points at (10,0) ± 0.5]
Sparse middle: (5, 0)
Far outlier: (20, 5)

Z-score (global): Dono (5,0) aur (20,5) flag hote hain (global mean ≈ (5,0) se door) LOF (local):

• (20,5): LOF ≈ 5 (neighbors ~11 distance par hain, lekin unke neighbors ~0.5 par hain) → outlier
• (5,0): LOF ≈ 1.2 (dono clusters ke neighbors bhi relatively isolated hain) → not outlier

Yeh kyun matter karta hai: Beech wala point expected hai (transition region), lekin door wala point truly anomalous hai.

Treatment Strategies

Outliers RAKHNE chahiye jab:

1. Domain knowledge confirm kare ki valid hai (jaise, employee data mein CEO salary)
2. Interest ka target ho (fraud detection, rare disease diagnosis)
3. Tree-based models (decision trees, random forests) outliers ko splits ke zariye handle karte hain

REMOVE karna chahiye jab:

1. Data errors (age = 200, negative prices)
2. Sample size badi hai (>10,000) aur outliers <0.1% hain
3. Linear/distance-based models (regression, KNN, SVM with RBF kernel)

TRANSFORM karna chahiye jab:

1. Information lose kiye bina impact reduce karna ho
2. Medium sample size (1,000-10,000)
3. Outliers ka business meaning hai lekin statistics distort karte hain

Treatment Methods

1. Capping (Winsorization)