Outlier detection and treatment

The key question: Is this outlier telling us something important, or is it corrupting our model?

Understanding Outliers

WHY do outliers matter?

1. Model sensitivity: Many ML algorithms (linear regression, k-means, PCA) are mean-based and highly sensitive to extreme values
2. Gradient instability: Large outliers create steep gradients, causing optimization to oscillate
3. Feature scale distortion: Outliers inflate variance, making normalization ineffective
4. Decision boundary shift: In classification, outliers pull boundaries toward them

WHAT makes a point an outlier?

• Statistical deviation (e.g., >3σ from mean)
• Isolation in feature space
• Low local density
• Domain-specific thresholds (e.g., human age > 150 years)

Detection Methods

1. Statistical Methods (Univariate)

$z_i = \frac{x_i - \mu}{\sigma}$

Derivation:

• Start with the definition of standard deviation: $\sigma = \sqrt{\frac{1}{n}\sum(x_i - \mu)^2}$
• This measures typical spread around $\mu$
• Dividing $(x_i - \mu)$ by $\sigma$ gives a dimensionless measure: "how many typical spreads away?"
• By the 68-95-99.7 rule for normal distributions: ~99.7% of data lies within $|z| < 3$
• WHY this works: Standardizes comparison across different scales

Threshold: $|z_i| > 3$ (or 2.5, depending on tolerance)

WHY this step works: The z-score converts absolute distances to relative distances, making it scale-invariant.

$\text{IQR} = Q_3 - Q_1$ $\text{Lower fence} = Q_1 - 1.5 \times \text{IQR}$ $\text{Upper fence} = Q_3 + 1.5 \times \text{IQR}$

Derivation:

• $Q_1$ (25th percentile) and $Q_3$ (75th percentile) define the middle 50% of data
• IQR measures the spread of this central bulk
• WHY 1.5? Tukey's empirical rule: balances sensitivity vs. false positives. For normal data, this catches ~0.7% as outliers
• Points beyond fences are far from the bulk, even if distribution is skewed

WHY this step works: Uses percentiles (rank-based), so immune to extreme values that would distort mean/std.

Z-score approach:

mean = 43.6, std = 42.8
z_150 = (150 - 43.6) / 42.8 = 2.48

WHY this step? We're standardizing the distance. Result: 150 is NOT flaged (|z| < 3) because the outlier itself inflated std!

IQR approach:

Q1 = 25.5, Q3 = 33.5, IQR = 8
Upper fence = 33.5 + 1.5*8 = 45.5

WHY this step? We use the middle 50% to define "normal," ignoring extremes. Result: 150 > 45.5, correctly flagged.

Lesson: IQR is robust when outliers contaminate mean/std.

2. Multivariate Methods

$D_M(x) = \sqrt{(x - \mu)^T \Sigma^{-1} (x - \mu)}$

where $\mu$ is the mean vector, $\Sigma$ is the covariance matrix.

Derivation from first principles:

1. Euclidean distance in $\mathbb{R}^d$: $D_E = \sqrt{(x - \mu)^T (x - \mu)}$
   • Problem: Treats all dimensions equally, ignores correlations
2. Standardize each dimension: Divide by $\sigma_i$ → distance in "units of spread"
   • Problem: Still ignores correlations (e.g., height & weight are correlated)
3. Account for covariance: The covariance matrix $\Sigma$ encodes both variances (diagonal) and correlations (off-diagonal)
4. Whitening transformation: $\Sigma^{-1/2}(x - \mu)$ transforms data to unit sphere where dimensions are uncorrelated
5. Final distance: $(x - \mu)^T \Sigma^{-1} (x - \mu)$ is the squared distance in this transformed space

WHY this works:

• If features are independent, $\Sigma$ is diagonal → reduces to standardized Euclidean
• If corelated, it down-weights the "redundant" dimensions
• Follows a $\chi^2$ distribution under normality, giving a statistical threshold

Threshold: $D_M > \chi^2_{0.975, d}$ (97.5th percentile of chi-squared with $d$ degrees of freedom)

WHY Mahalanobis helps:

• Euclidean distance treats both equally far from mean [2000 sqft, 3 beds]
• Mahalanobis recognizes sqft and bedrooms are correlated (r ≈ 0.8)
• House A violates this correlation → higher $D_M$, flaged as anomaly
• House B follows the pattern → lower $D_M$, not flagged

Step-by-step calculation (simplified 2D):

mu = [2000, 3]
Sigma = [[1e6, 0.8*1000*2],   # cov(sqft, beds) = r*σ_sqft*σ_beds
         [0.8*1000*2, 4]]
Sigma_inv = inverse(Sigma)
 
x_A = [5000, 2]
diff_A = x_A - mu = [3000, -1]
D_M_A = sqrt(diff_A^T @ Sigma_inv @ diff_A) ≈ 8.2

WHY this value? The negative correlation component (many sqft but few beds) adds to distance.

3. Density-Based: Local Outlier Factor (LOF)

$\text{LOF}_k(x) = \frac{\sum_{o \in N_k(x)} \frac{\text{lrd}(o)}{\text{lrd}(x)}}{|N_k(x)|}$

where local reachability density (lrd) is:

$\text{lrd}_k(x) = \frac{|N_k(x)|}{\sum_{o \in N_k(x)} \text{reach-dist}_k(x, o)}$

and reachability distance:

$\text{reach-dist}_k(x, o) = \max(\text{dist}(x, o), k\text{-dist}(o))$

Derivation logic:

1. k-distance: Distance to the $k$-th nearest neighbor of $o$
2. Reachability distance: Uses $k$-dist as "smoothing" factor
   • WHY? Prevents instability when points are very close
   • If $x$ is far from $o$, use actual distance
   • If $x$ is in $o$'s neighborhood, use $k$-dist(o) to stabilize
3. LRD: Inverse of average reachability distance → higher lrd = denser region
4. LOF: Ratio of neighbors' density to point's density
   • LOF ≈ 1: similar density to neighbors (normal)
   • LOF >> 1: much lower density than neighbors (outlier)

WHY this works: Adapts to varying densities. A point in a sparse cluster isn't an outlier if its neighbors are also sparse.

Dense cluster A: [points at (0,0) ± 0.5]
Dense cluster B: [points at (10,0) ± 0.5]
Sparse middle: (5, 0)
Far outlier: (20, 5)

Z-score (global): Both (5,0) and (20,5) are flagged (far from global mean ≈ (5,0)) LOF (local):

• (20,5): LOF ≈ 5 (neighbors are at distance ~11, but their neighbors are at ~0.5) → outlier
• (5,0): LOF ≈ 1.2 (neighbors from both clusters are also relatively isolated) → not outlier

WHY this matters: The middle point is expected (transition region), but far point is truly anomalous.

Treatment Strategies

When to KEEP outliers:

1. Domain knowledge confirms it's valid (e.g., CEO salary in employee data)
2. Target of interest (fraud detection, rare disease diagnosis)
3. Tree-based models (decision trees, random forests) handle outliers via splits

When to REMOVE:

1. Data errors (age = 200, negative prices)
2. Sample size is large (>10,000) and outliers are <0.1%
3. Linear/distance-based models (regression, KNN, SVM with RBF kernel)

When to TRANSFORM:

1. Reduce impact without losing information
2. Medium sample size (1,000-10,000)
3. Outliers have business meaning but distort statistics

Treatment Methods

1. Capping (Winsorization)