2.1.3 · D5Data Preprocessing & Feature Engineering
Question bank — Outlier detection and treatment
This is the "gotcha" bank for Outliers (2.1.3). Each line is a trap that catches people who half-understand the ideas. Read the left side, cover the right, and say your reasoning out loud before revealing. A bare "true" or "false" is not an answer here — the justification is the point.
The four rulers, before you use them
Every trap on this page leans on one of four ways to measure "abnormal". Here are all four stated in full so no symbol below is a mystery. The figure shows how differently they judge the same point.
True or false — justify
The claim is on the left; the honest verdict + reason is on the right.
A z-score above 3 always means the point is a genuine error
False — a large just means "far from the mean in units of spread". It could be a real rare event (fraud, a CEO salary) that you must keep, not delete.
The IQR fence rule flags roughly the same fraction of points on any dataset
False — the figure only holds for normal data. On skewed or heavy-tailed data the fences catch far more (or fewer), because move with the shape.
Removing all flagged outliers always makes a model better
False — removing points shifts the distribution and can delete the very signal you care about (e.g. rare-disease cases). "Flagged" means "inspect", not "delete".
Mahalanobis distance reduces to standardized Euclidean distance when features are uncorrelated
True — if is diagonal, just divides each dimension by its variance, which is exactly per-axis standardization with no cross terms.
A point with is guaranteed to be in a dense region
False — only means the point's density matches its neighbours'. In a uniformly sparse cluster everyone has , yet the region is sparse.
Winsorization (capping) removes outliers from the dataset
False — it keeps every row and only clips extreme values to a percentile. The point still exists and still nudges the model, just less violently.
The z-score method is scale-invariant, so you can skip feature scaling before using it
True for detection per feature — dividing by already makes dimensionless. But if you feed the same features to a downstream distance model you still need scaling.
Tree-based models need the same aggressive outlier removal that linear regression does
False — trees split on thresholds, so a huge value just falls on one side of a split and barely distorts the tree. Mean-based models (regression, k-means, PCA) are the sensitive ones.
If a point passes the IQR fence, it will also pass the test
False — the two use different notions of spread (rank-based vs mean-based). A single big value inflates so much that its own shrinks below 3 while it still sits outside the IQR fence.
Mahalanobis distance can flag a point that is typical on every single feature individually
True — a house with average sqft but a bedroom count that breaks the sqft-bedroom correlation is normal per-axis, yet far in the whitened space. This is the whole reason multivariate methods exist.
Spot the error
Each line states a flawed piece of reasoning. The reveal names the mistake. (All symbols come from the four definitions above.)
"Age = 43.6 mean, std = 42.8, so age 150 has z = 2.48, which is under 3, so 150 is fine."
The mistake is using contaminated statistics — the value 150 itself inflated the std to 42.8, hiding its own outlier status. Use the robust IQR fence instead.
"Euclidean distance already accounts for scale, so I don't need Mahalanobis."
Euclidean distance treats all axes equally and ignores correlations; it does not rescale by variance or decorrelate. That is precisely what adds.
"LOF > 1 for this point, so it's an outlier no matter how far above 1."
LOF is a ratio that hovers near 1 for normal points; slight values like 1.1–1.3 are noise. You need (context-chosen), so a hard "> 1" rule over-flags.
"I'll cap at the 5th and 95th percentiles, which deletes 10% of my data."
Capping deletes nothing — it clips 10% of values to the fence, keeping all rows. Confusing capping with trimming loses sample size you never meant to drop.
"Reach-dist just equals the ordinary distance between the two points."
By definition . The floor is a smoothing term that stabilizes density when points are extremely close; dropping it makes LOF jittery.
"My data isn't normal, so I'll use the z-score's 3σ rule anyway — 3 is a universal constant."
The guarantee comes from the 68-95-99.7 rule, which is a property of the normal distribution only. On non-normal data that threshold has no fixed meaning; prefer IQR.
"Mahalanobis needs no threshold — bigger is just more outlier-y."
Under normality (the squared Mahalanobis distance) follows a distribution, giving a principled cutoff (e.g. the 97.5th percentile with degrees of freedom). Without a threshold you have a score but no decision.
Why questions
The reveal must explain the mechanism, not restate the fact.
Why does the IQR method use percentiles instead of the mean and standard deviation?
Percentiles are rank-based: moving one extreme value doesn't change which point sits at the 25th or 75th rank, so the fences stay put. Mean and std, by contrast, are dragged by that same extreme value.
Why does Tukey pick the multiplier 1.5 for the IQR fences rather than 1 or 3?
It's an empirical balance: for roughly normal data catches only as outliers (few false alarms) while still reaching genuine extremes. Smaller multipliers over-flag; larger ones miss real outliers.
Why does dividing by make the z-score comparable across different features?
carries the feature's units (rupees, kg, years); dividing by — same units — cancels the units, leaving a pure count of "how many typical spreads away". Now heights and salaries live on one scale.
Why does the whitening transformation turn correlated data into a unit sphere?
simultaneously rescales each direction to unit variance and rotates away the correlations, so the elliptical cloud becomes a round ball where plain Euclidean distance is finally meaningful.
Why can LOF declare a globally-far point not an outlier?
LOF only compares a point to its own neighbours. If a point lives in an intrinsically sparse region and its neighbours are equally sparse, its density ratio stays near 1 — it fits its local context even if the global mean is distant.
Why do large outliers cause gradient-based optimization to oscillate?
A far point produces a huge squared error, hence a huge gradient; the update step overshoots, the next step corrects violently, and the loss bounces instead of settling. This links to why regularization and robust losses help.
Why is Mahalanobis distance the natural tool once features are correlated, rather than per-feature z-scores?
Per-feature z-scores treat axes as independent and double-count redundant correlated information. down-weights that redundancy, so a violation of the correlation pattern — invisible to any single z-score — becomes a large distance.
Why might you transform (e.g. log) rather than remove an outlier?
A log transform compresses the long right tail so extreme values still contribute information but stop dominating the variance — ideal when the outlier is a real, business-meaningful value you can't afford to delete.
Edge cases
Boundary and degenerate inputs — the scenarios that break naive code.
What does the z-score do when every value is identical (σ = 0)?
The formula divides by zero and is undefined. There is no spread, so "outlier" is meaningless — guard against before dividing.
What happens to the IQR fences when more than 50% of the data share one value?
and can collapse to the same number, making ; then the fences equal that value and any deviation is flagged. Rank-based methods degenerate on massively repeated values.
Can Mahalanobis distance be computed if the covariance matrix is singular?
No — doesn't exist when features are perfectly collinear or when you have fewer samples than dimensions. You must reduce dimensions or use a pseudo-inverse first.
What does LOF return if you set larger than the number of points minus one?
The -neighbourhood can't be formed, so LOF is undefined. must be small enough to have genuine neighbours, and large enough to estimate density stably.
If a dataset has two points at exactly the same extreme value, will the z-score still flag them?
Possibly not — two copies of a large value inflate and especially even more, shrinking each copy's . Duplicated extremes are a classic case where robust IQR beats z-score.
What is the "right" outlier method when the sample is tiny (say 8 points)?
With few points, and are unreliable and percentiles are coarse, so no automatic rule is trustworthy — inspect visually and lean on domain knowledge instead of a threshold.
How should you treat an outlier that is simultaneously a data-entry error and the class you care about?
You cannot both delete and study it — fix the erroneous field if recoverable, otherwise flag the row for manual review rather than blind capping. This is the boundary between cleaning and anomaly detection.
Recall Quick self-test
Name the one method that survives a heavily skewed distribution ::: The IQR fence, because it is rank-based and ignores how far the extremes reach. Name the one method that catches a correlation-breaking point that looks normal per-feature ::: Mahalanobis distance, via . Name the one method that adapts to varying local densities ::: Local Outlier Factor (LOF).