Outlier detection and treatment: Exercises — AI-ML deep dive

This is your self-testing lab for the parent topic. Every problem is graded from L1 (just recognise the idea) up to L5 (put it all together). Read the problem, try it on paper, THEN open the solution.

Before we start, one promise: every symbol is defined the moment it appears. If a formula uses $μ$ , we say " $μ$ (mu) = the average of all the numbers" right there. Nothing is assumed.

Intuition The mental model for the whole page

Picture a number line drawn in chalk. Most points cluster together in a fuzzy band — that's the "normal" crowd. An outlier is a point sitting way off on its own. Every method below is just a different ruler for answering "how far is way off?" — some rulers get fooled by the outlier itself, some don't. That tension is what these exercises train.

The figure below makes this concrete. Look at it before reading on: the blue dots are the crowd $[22..35]$ bunched together; the pink star at $150$ is the lone outlier. Now watch the yellow dashed line — that's the mean $μ$ . Notice it has been pulled to the right, away from the crowd, by that single star. Keep this picture in mind: it is the visual root of the "self-masking" idea that L2.2 and L3.1 will make you compute.

Figure — Outlier detection and treatment

Quick symbol dictionary (we re-explain each in context too):

$x_{i}$ ::: the $i$ -th data value (e.g. one person's age).
$μ$ (mu) ::: the mean — add all values, divide by how many there are.
$σ$ (sigma) ::: the standard deviation — the typical distance of a point from the mean.
$Q_{1}, Q_{3}$ ::: the 25th and 75th percentiles — cut points where 25% and 75% of sorted data lie below.
$P_{5}, P_{95}$ ::: the 5th and 95th percentiles — the values below which 5% and 95% of the sorted data fall (used for capping in L4).
$IQR$ ::: $Q_{3} - Q_{1}$ , the width of the middle half of the data.

Definition How

$Q_{1}$ and $Q_{3}$ are actually computed (small- $n$ rule) A percentile answers "below which value does this fraction of the sorted data fall?" To find $Q_{1}$ (25th percentile) of $n$ sorted values, we use the common linear-interpolation rule:

Sort the data ascending.
Compute the position $h = (n - 1) \cdot p + 1$ , where $p$ is the fraction ( $0.25$ for $Q_{1}$ , $0.75$ for $Q_{3}$ ) and positions are counted from $1$ .
If $h$ is a whole number, $Q$ is simply the value at that position. If $h$ falls between positions $⌊ h ⌋$ and $⌊ h ⌋ + 1$ , interpolate: $Q = v_{⌊ h ⌋} + (h - ⌊ h ⌋) (v_{⌈ h ⌉} - v_{⌊ h ⌋})$ . Example on $[10, 12, 14, 15, 18, 21, 60]$ ( $n = 7$ ): for $Q_{1}$ , $h = (7 - 1) (0.25) + 1 = 2.5$ , so $Q_{1}$ lies halfway between the 2nd value ( $12$ ) and 3rd value ( $14$ ): $Q_{1} = 12 + 0.5 (14 - 12) = 13$ . (Different software uses slightly different conventions; the exercises below simply give you $Q_{1}, Q_{3}$ so you can focus on the fence logic — but now you know where they come from.) The same rule with $p = 0.05$ and $p = 0.95$ gives the percentiles $P_{5}$ and $P_{95}$ we use for capping later.

Level 1 — Recognition

Recall Solution L1.1

The z-score $z_{i} = \frac{x _{i} - μ}{σ}$ answers "how many typical spreads $σ$ is this point from the mean $μ$ ?"

$∣ z ∣ = ∣ - 3.4∣ = 3.4$ . Since $3.4 > 3$ , yes, it is flagged as an outlier.
The minus sign means $x_{i} < μ$ : the point sits below the mean (to the left on the chalk number line). A positive $z$ would mean above/right. Answer: outlier, and it is a low value.

Recall Solution L1.2

Pick IQR. The IQR uses percentiles ( $Q_{1}, Q_{3}$ ) which are rank-based — they only care about the position of sorted values, not their magnitude. So a handful of huge incomes cannot inflate $Q_{1}$ or $Q_{3}$ . The z-score uses $μ$ and $σ$ , both of which get dragged upward by that same rich tail, hiding the very outliers you want to catch.

Level 2 — Application

Recall Solution L2.1

What we do: measure the width of the middle half, then push out by $1.5 \times$ that width on each side. Anything beyond is "far from the bulk." $IQR = Q_{3} - Q_{1} = 21 - 12 = 9$ $Lower fence = Q_{1} - 1.5 \times IQR = 12 - 1.5 (9) = 12 - 13.5 = - 1.5$ $Upper fence = Q_{3} + 1.5 \times IQR = 21 + 1.5 (9) = 21 + 13.5 = 34.5$ Check every point: the smallest is $10$ ( $> - 1.5$ , fine). The largest is $60$ . Since $60 > 34.5$ , $60$ is an outlier. All others fall inside the fences. Answer: IQR $= 9$ , fences $[- 1.5, 34.5]$ , outlier $= {60}$ .

Recall Solution L2.2

$z_{30} = \frac{30 - 10}{10.0995} = \frac{20}{10.0995} \approx 1.980$

At threshold $∣ z ∣ > 3$ : $1.98 < 3$ , NOT flagged.
At the looser threshold $∣ z ∣ > 2$ : $1.98 < 2$ , still NOT flagged (only just misses). This is the self-masking problem in action: $30$ is visibly extreme, yet by pulling $μ$ up to $10$ and inflating $σ$ to $\approx 10$ , it hides under the threshold.

Level 3 — Analysis

Recall Solution L3.1

(a) Z-score: $z_{150} = \frac{150 - 43.625}{40.06} \approx \frac{106.375}{40.06} \approx 2.656$ Since $2.656 < 3$ : NOT flagged. (b) IQR: $IQR = 33.5 - 26.5 = 7, Upper fence = 33.5 + 1.5 (7) = 44.0$ Since $150 > 44.0$ : flagged. (c) Why the disagreement: the single value $150$ is included in the computation of both $μ$ and $σ$ , so it inflates its own ruler — $σ$ balloons to $\approx 40$ , dividing the distance down to $z \approx 2.66$ . The IQR is built only from $Q_{1}, Q_{3}$ , which sit inside the tight cluster $[22..35]$ ; the value $150$ never touches them, so the ruler stays honest and catches it.

Recall Solution L3.2

First, the tools:

Euclidean distance $D_{E} = (x - μ)^{⊤} (x - μ)$ measures straight-line distance, treating each feature independently. Here it barely distinguishes A and B — both are $3000$ sqft above mean.
Mahalanobis distance $D_{M} = (x - μ)^{⊤} Σ^{- 1} (x - μ)$ inserts $Σ^{- 1}$ , the inverse of the covariance matrix $Σ$ (which stores each feature's spread on its diagonal and the correlation between them off-diagonal). This rotates and rescales space so the ruler runs along the natural correlation ridge. Verdict: House A should be flagged. It has huge sqft but few bedrooms — it breaks the correlation (big houses should have many rooms). Mahalanobis measures distance off the correlation ridge, so A lands far away. House B follows the ridge (big + many rooms), so even though it's far from the mean, it's a "normal-looking" big house → low $D_{M}$ .

Look at the figure: the yellow ellipse is the "normal" region Mahalanobis draws. B stays inside the tilted ellipse; A pops out sideways.

Level 4 — Synthesis

Recall Solution L4.1

Recall $P_{5}$ and $P_{95}$ are the 5th and 95th percentiles — the low and high cut values that leave $5%$ of the data below and $5%$ above respectively. Capping rule — pull anything past a percentile back to that percentile, leave the middle untouched: $x^{capped} = ⎩ ⎨ ⎧ P_{5} x P_{95} x < P_{5} P_{5} \leq x \leq P_{95} x > P_{95}$ Go value by value:

$3 < 12 \Rightarrow 12$ (capped up to $P_{5}$ )
$40$ is in $[12, 88] \Rightarrow 40$ (unchanged)
$95 > 88 \Rightarrow 88$ (capped down)
$88 = P_{95} \Rightarrow 88$ (on the boundary, kept)
$12 = P_{5} \Rightarrow 12$ (on the boundary, kept)
$200 > 88 \Rightarrow 88$ (capped down) Result: $[12, 40, 88, 88, 12, 88]$ . Why rank order survives: capping is a monotone operation — if $a \leq b$ before, then $cap (a) \leq cap (b)$ after. So "who is bigger than whom" never flips; we only compress the extremes inward. The distribution's tails get flattened (less variance) but the ordering — crucial for rank-based models — is preserved.

Recall Solution L4.2

(a) KEEP. The outliers are the signal — removing fraud from a fraud detector deletes the target. This is the domain of 3.3.02-Anomaly-detection, where rarity is the whole point. (b) REMOVE. Age $220$ is physically impossible → a data error, not variation. No transform can rescue a corrupted value. (c) TRANSFORM (e.g. log or Winsorize). Medium sample, values are real and meaningful but distort $μ / σ$ and hence any scaling. A linear model is mean-based and sensitive, so we compress rather than delete — keeping the information while calming the leverage.

Level 5 — Mastery

Recall Solution L5.1

Local Outlier Factor (LOF) compares a point's local density to the density of its neighbors: $LOF_{k} (x) = \frac{1}{∣ N _{k} ( x ) ∣} \sum_{o \in N_{k} (x)} \frac{lrd ( o )}{lrd ( x )}$ where $N_{k} (x)$ is the set of $x$ 's $k$ nearest neighbors, and $lrd$ (local reachability density) is roughly "how tightly packed is the region around this point" — high lrd = dense, low lrd = sparse.

First, what is $k$ and how do you choose it? The number $k$ is how many nearest neighbors you look at to define "local." It is a hyperparameter you pick, not something the data hands you:

Too small (e.g. $k = 1$ ): the estimate is jumpy — a single oddly-placed neighbor swings the verdict.
Too large (e.g. $k$ near the cluster size): "local" stops being local; you blur across clusters and lose the very locality LOF exists for.
Rule of thumb: many practitioners start around $k = 20$ , and choose $k$ at least as large as the smallest cluster you'd still call a group (so a genuine small cluster isn't mistaken for outliers). Often you scan a range of $k$ and check stability.

Now the two points, using $LOF \approx 1 \Rightarrow$ normal, $LOF ≫ 1 \Rightarrow$ outlier:

Point $Q = (20, 5)$ : its nearest neighbors are $\sim 11$ units away (in the clusters), but those neighbors have their own neighbors only $\sim 0.5$ away. So $Q$ is far sparser than its neighbors are → ratio $lrd (o) / lrd (Q)$ is large → $LOF_{Q} \approx 5 ≫ 1$ → flagged.
Point $P = (5, 0)$ : its neighbors are the edge points of both clusters, which are themselves relatively isolated on that side. $P$ 's density and its neighbors' densities are comparable → $LOF_{P} \approx 1.2 \approx 1$ → not flagged. $P$ is a legitimate transition point, not an anomaly. The lesson: global methods only know one "normal" band, so a sparse-but-expected gap point looks abnormal. LOF adapts to local density and rescues it.

Read the figure to lock this in. Look at the pink star $Q$ in the upper right: it stands alone, while the crowd it's nearest to is itself tightly packed — that mismatch is what makes $LOF$ big. Now look at the yellow square $P$ in the middle gap: it's isolated too, but so are the cluster-edge neighbors it points to, so the ratio stays near $1$ . The whole trick is in that comparison — notice how the arrows contrast "sparser than neighbors" ( $Q$ ) versus "as sparse as neighbors" ( $P$ ).

Recall Solution L5.2

(1) Detector — IQR (not z-score). The feature is right-skewed, so $μ / σ$ get dragged by the tail (self-masking, exactly as L2.2 and L3.1 showed). The rank-based IQR fences $Q_{3} + 1.5 IQR$ stay honest. If we later add more features, we'd escalate to Mahalanobis or dedicated 3.3.02-Anomaly-detection to catch correlation-breaking points. (2) Treatment — TRANSFORM, specifically Winsorization or a log transform. The sample is medium ( $2, 000$ rows) and the high earners are real, meaningful values, so we compress rather than delete. Capping keeps rank order (proved monotone in L4.1); a log transform pulls in the tail multiplicatively without discarding rows. (3) Downstream consequence to check — feature scaling. k-NN is distance-based: every prediction depends on how far points are apart. An untreated outlier stretches the income axis so it dominates every distance, drowning out other features. So after capping we must re-run 2.1.02-Feature-scalingand-normalization so income sits on a scale comparable to the rest. Two more consequences worth flagging: an outlier-distorted variance can wrongly inflate or sink this feature's ranking in 2.2.01-Feature-selection; and while 4.1.03-Regularization partly tames outlier leverage in parametric models, it is not a substitute for treatment in a distance-based model like k-NN.

Recall Self-test checklist (reveal to grade yourself)

Why does a big outlier hide from z-score? ::: It inflates $σ$ (the denominator), shrinking its own z-score — self-masking. What makes IQR robust? ::: It uses rank-based percentiles $Q_{1}, Q_{3}$ , untouched by extreme magnitudes. Why the multiplier $1.5$ in Tukey's fences? ::: Empirical compromise — flags only $\approx 0.7%$ of clean normal data (few false alarms) while still catching real strays. What is $k$ in LOF and how is it chosen? ::: The neighborhood size (a hyperparameter); pick it big enough to be stable and at least as large as the smallest real cluster, often around $20$ . What does $Σ^{- 1}$ add in Mahalanobis distance? ::: It rescales/rotates space along correlations, so distance is measured off the natural correlation ridge. When is a far-from-mean point NOT an outlier? ::: When it obeys the correlation pattern (Mahalanobis) or matches its neighbors' local density (LOF). Which treatment preserves rank order? ::: Winsorization (capping) — it's monotone.