Worked examples — Outlier detection and treatment

This page is the practice ground for the parent topic. We will not re-derive the formulas here — we will use them on every kind of dataset you could meet: clean data, contaminated data, skewed data, correlated data, and the trap cases where the "obvious" method quietly fails.

If any symbol below feels unfamiliar, it was built in the parent note. Prerequisite ideas about spreading and rescaling live in 2.1.02-Feature-scalingand-normalization; the anomaly-hunting cousin of this topic is 3.3.02-Anomaly-detection.

The scenario matrix

Every outlier problem falls into one of these case classes. The worked examples below each carry a tag like (C3) telling you which cell they fill.

Cell	Case class	What makes it tricky	Covered by
C1	Clean, roughly normal data	Baseline — methods should agree	Ex 1
C2	One extreme upper-tail value contaminating mean/σ	Z-score self-sabotages	Ex 2
C3	Skewed distribution, no true outlier	Method must not invent one	Ex 3
C4	Zero-spread / degenerate input	Division by $\sigma =0$	Ex 4
C5	Correlated 2-D data, "impossible combo"	Needs Mahalanobis, not Euclidean	Ex 5
C6	Varying-density clusters	Global fails, LOF succeeds	Ex 6
C7	Real-world word problem (treatment choice)	Detect and decide keep/remove/transform	Ex 7
C8	Lower-tail / negative outlier + exam boundary twist	Sign symmetry, left tail, on-the-fence	Ex 8

We define the symbols exactly as the parent did: $\mu$ = mean, $\sigma$ = standard deviation, $z_i=\frac{x_i-\mu }{\sigma }$, $Q_1,Q_3$ = 25th/75th percentiles, $\text{IQR}=Q_3-Q_1$, fences at $Q_1-1.5\text{IQR}$ and $Q_3+1.5\text{IQR}$.

---

Univariate examples

Forecast: These look tame. Guess: nothing flagged, both methods agree.

1. Compute $\mu$ and $\sigma$. $\mu =\frac{60+62+65+66+68+70+72+75}{8}=67.25$. Population $\sigma =\sqrt{\frac{1}{n}\sum (x_i-\mu {)}^2}\approx 4.68$. Why this step? The z-score asks "how many spreads from center?", so we need center ($\mu$) and spread ($\sigma$) first — and we use the population formula fixed above.
2. Largest deviation is $75$: $z=\frac{75-67.25}{4.68}\approx 1.66$. Why this step? If the most extreme point has $|z|<3$, nothing else can exceed the threshold.
3. IQR fences (linear interpolation). For these 8 sorted values numpy.percentile gives $Q_1=64.25$, $Q_3=71.5$, $\text{IQR}=7.25$. Upper fence $=71.5+1.5(7.25)=82.375$; lower fence $=64.25-10.875=53.375$. Why this step? IQR is the robust cross-check. Agreement builds trust.

Verify: All data lie in $[53.375,82.375]$ and the most extreme $|z|\approx 1.66<3$. Both methods say "no outliers." Forecast confirmed — this is what agreement looks like.

---

Forecast: Obvious outlier → surely z-score flags it? (Trap: guess yes.)

1. $\mu$ and $\sigma$ with the $150$ included. $\mu =\frac{349}{8}=43.625$, population $\sigma \approx 39.98$. Why this step? We must use the actual contaminated statistics — that is the whole point of the failure.
2. z-score of 150: $z=\frac{150-43.625}{39.98}\approx 2.66$. Why this step? Compare against the usual $|z|>3$ rule.
3. Verdict (z): $2.66<3\Rightarrow$ NOT flagged. The outlier inflated $\sigma$, hiding itself. Why this step? We must state the consequence explicitly: the strict threshold isn't met, so the obviously-bad point survives — that is the failure we came to see.
4. Now IQR. $Q_1=26.5$, $Q_3=33.75$, $\text{IQR}=7.25$, upper fence $=33.75+1.5(7.25)=44.625$. Why this step? Percentiles ignore the magnitude of extremes, so contamination can't fool them.
5. Verdict (IQR): $150>44.625\Rightarrow$ flagged. Why this step? This is the punchline: on the same data IQR succeeds where z-score failed, so we record which method to trust.

Figure 1 (below) — titled "Fig 1 (Ex 2)" — plots the eight ages. The dashed gray line is the $z=+3$ cutoff (which the red $150$ point sits below, hence missed); the solid green line is the IQR upper fence at $44.625$ (which $150$ clearly clears, hence caught). The whole story of self-masking is in the gap between those two horizontal lines.

Verify: $z=2.66<3$ (missed) but $150>44.625$ (caught). Your gut beat the z-score; IQR agreed with your gut.

---

Forecast: Skewed data has a stretched right side. Guess: the z-score over-reacts to the $12$, but IQR is calmer — and $12$ may be legitimate skew, not an error.

1. z-score path. $\mu =\frac{1+1+2+2+3+3+4+5+8+12}{10}=4.1$. Population variance $=\frac{1}{10}\sum (x_i-4.1{)}^2=10.89$, so $\sigma =\sqrt{10.89}=3.30$. Then $z_{12}=\frac{12-4.1}{3.30}\approx 2.39$ and $z_8=\frac{8-4.1}{3.30}\approx 1.18$. Why this step? We must actually compute the z-scores to compare them against IQR, not just wave at them.
2. z verdict. $2.39<3\Rightarrow$ z-score flags nothing at the strict $|z|>3$ rule; but note $12$ is already the extreme edge — on skewed data z-scores drift high because the long tail inflates $\sigma$ and pulls $\mu$ right. Why this step? We turn the raw number into a decision: the strict rule says "keep," which is the point — a statistically extreme skew tail can still pass, so we must not lean on the z-score alone for skewed features.
3. IQR (the skew-safe method). With linear interpolation numpy.percentile gives $Q_1=2$, $Q_3=4.75$, $\text{IQR}=2.75$, upper fence $=4.75+1.5(2.75)=8.875$. Why this step? IQR anchors to ranks, so the long tail cannot distort the fence the way it distorts $\mu ,\sigma$.
4. IQR verdict. $12>8.875\Rightarrow$ flagged; $8<8.875\Rightarrow$ kept. Why this step? We convert the fence into a per-point call so the disagreement with z-score is explicit: IQR flags exactly $\{12\}$, z-score flags nothing.
5. Domain sanity. A 12-second session is entirely plausible for a website. Why this step? Detection $\ne$ removal. A statistically extreme point on naturally skewed data is often a legitimate signal, not an error.

Verify: $\mu =4.1$, $\sigma \approx 3.30$, $z_{12}\approx 2.39<3$ (z flags nothing); IQR fence $=8.875$ flags $\{12\}$ only. The two methods disagree, and IQR is the trustworthy one here.

---

Forecast: All identical → no outliers. But how does the formula behave? Guess: division by zero somewhere.

1. Compute $\sigma$. Every $x_i=\mu =7$, so $\sum (x_i-\mu {)}^2=0\Rightarrow \sigma =0$. Why this step? $\sigma =0$ is the degenerate case the formula was never designed for.
2. Attempt $z_i=\frac{7-7}{0}=\frac{0}{0}$ — undefined. Why this step? It exposes that the z-score is ill-defined when spread vanishes.
3. Safe handling. Define: if $\sigma =0$, there are no outliers (all points identical). Code should guard if std == 0: return no_outliers. Why this step? Every real pipeline must cover the zero-variance column, or it crashes at runtime.

Verify: $\sigma =0$, so the numerator $x_i-\mu =0$ too and the z-score is the undefined form $0/0$; the correct policy is "no outliers, drop constant column."

---

Multivariate example

Forecast: Euclidean sees "far in sqft." Mahalanobis also knows big houses should have more beds — so A breaks the pattern. Guess: Mahalanobis distance is large.

1. Center the point. $x-\mu =[5000-2000,2-3]=[3000,-1]$. Why this step? Distance is always measured from the center of the cloud.
2. Invert the covariance. For $\Sigma$ above, $\det \Sigma =10^6\cdot 4-1600^2=4000000-2560000=1440000$. ${\Sigma }^{-1}=\frac{1}{1440000}\left[\begin{array}{cc}4& -1600\\ -1600& 10^6\end{array}\right].$ Why this step? ${\Sigma }^{-1}$ "un-stretches and un-rotates" the ellipse so we can measure distance fairly across correlated axes (the whitening idea from the parent).
3. Squared Mahalanobis distance $D_M^2=(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$. Plugging in $[3000,-1]$: $D_M^2=\frac{1}{1440000}(4\cdot 3000^2-2\cdot 1600\cdot 3000\cdot (-1)+10^6\cdot (-1{)}^2).$ Numerator $=36000000+9600000+1000000=46600000$. So $D_M^2\approx 32.36$, $D_M\approx 5.69$. Why this step? The middle cross-term is positive precisely because A violates the positive correlation (much sqft, too few beds) — that violation is what inflates the distance.
4. Threshold. With $d=2$, the $97.5\%$ cutoff of ${\chi }_2^2$ is $\approx 7.38$ for $D_M^2$. Here $32.36>7.38\Rightarrow$ flagged. Why this step? $D_M^2$ follows a ${\chi }_d^2$ under normality, giving a principled cutoff instead of a hand-picked one.

Figure 2 (below) — titled "Fig 2 (Ex 5)" — draws the green ellipse of "typical" houses (the $97.5\%$ ${\chi }^2$ boundary, stretched and tilted by $\Sigma$) with the blue cloud inside it. The orange dot is the mean; the red dot is House A, sitting outside and below the ellipse because it has the sqft of a mansion but the bedrooms of a cottage. Euclidean distance would only see the horizontal gap; the ellipse's tilt is what condemns A.

Verify: $D_M^2\approx 32.36>7.38$ → House A is a multivariate outlier even though 5000 sqft alone is unremarkable. Euclidean would have missed the pattern violation.

---

Density example

Forecast: A global z-score measures distance from the overall mean. Guess: it flags both $P$ and $Q$ — but LOF, which looks locally, should clear the in-gap point $P$ and condemn the truly isolated $Q$.

1. Global mean and z-scores. The 12 values are $\{0,1,2,3,4,20,21,22,23,24,12,60\}$; their sum is $192$, so $\mu =16$. Population variance $=\frac{1}{12}\sum (x_i-16{)}^2=261$, so $\sigma =\sqrt{261}\approx 16.16$. Then $z_P=\frac{12-16}{16.16}\approx -0.25$ and $z_Q=\frac{60-16}{16.16}\approx 2.72$. Why this step? We must compute the global statistics to see exactly what they flag, not assert it. (In a 2-D global-distance version using the mean point, both $P$ and $Q$ look "far from center"; the 1-D z-scores make the trap concrete.)
2. What z would do. At the $|z|>3$ rule z flags neither; loosen to $|z|>2.5$ and it flags only $Q$ — but crucially a distance-from-mean view treats $P$ (in an empty gap) as suspicious even though it is a legitimate transition point. Why this step? We turn the two z-numbers into a verdict so the trap is explicit: the global rule either misses everything or (loosened) catches only $Q$, and a naive far-from-center view unfairly suspects $P$ — that mismatch motivates a local method.
3. LOF setup ($k=2$). For any point, $\text{dist}(a,b)=|a-b|$ on the line. We need, for each of $P$, $Q$ and their neighbors, the $2$-nearest-neighbor distances. Why this step? LOF is built entirely from $k$-nearest-neighbor distances, so we must fix $k$ and the distance metric before computing anything.
4. Neighbors of $Q=60$. Nearest two are $24$ and $23$: distances $36$ and $37$. Its $2$-distance is $37$. But $24$'s own two nearest ($23,22$) are at distances $1,2$, so $24$ sits in a region where typical spacing is $\approx 1$. Thus $Q$ is ~$36$ away from a neighborhood whose internal scale is ~$1$. Why this step? LOF asks "is my local density far below my neighbors' density?" Here $Q$'s reachability distances ($\approx 36$–$37$) dwarf its neighbors' ($\approx 1$), giving $\text{LOF}\gg 1$ (an order of magnitude), so $Q$ is a genuine outlier.
5. Neighbors of $P=12$. Nearest two are $4$ (distance $8$) and $20$ (distance $8$). Its reachability distances are $\approx 8$. Its neighbors $4$ and $20$ live at cluster edges where their own $2$-distances are $\approx 2$. So $P$'s density is lower than its neighbors', but only by a factor of a few — $\text{LOF}\approx 8/2=4$-ish in raw ratio, which after the reachability smoothing lands near a modest value, far below $Q$'s. Why this step? $P$ is sparser than the clusters but not catastrophically so — it is a transition region, exactly the case LOF is designed to spare.
6. Verdict. Ranked by LOF, $Q\gg P$: $Q$ is the true anomaly, $P$ is tolerable. A pure distance-from-mean method inverts or blurs this. Why this step? The whole example exists to produce a ranking decision; stating it makes the contrast with the global z-score (which suspected $P$) the memorable takeaway.

Figure 3 (below) — titled "Fig 3 (Ex 6)" — places all points on the line. Blue = the two dense clusters, green = the in-gap point $P$ (LOF small, kept), red = the far point $Q$ (LOF large, flagged). The visual gap widths encode exactly the reachability distances used above.

Verify: $\mu =16$, $\sigma \approx 16.16$, $z_P\approx -0.25$, $z_Q\approx 2.72$; $Q$'s $2$-distance ($37$) is ~$37\times$ its neighbors' spacing ($\approx 1$) while $P$'s ($8$) is only ~$4\times$ its neighbors' ($\approx 2$). Hence LOF ranks $Q$ as the anomaly and spares $P$, opposite to a naive "far from mean" flag on $P$.

---

Word problem & lower-tail / exam twist

Forecast: The CEO is a real person, not an error. Guess: flagged statistically, but we should not delete a legitimate value — cap or transform instead.

1. IQR detection (linear interpolation). For $n=8$ sorted values numpy.percentile gives $Q_1=36.5$, $Q_3=45.75$, $\text{IQR}=9.25$, upper fence $=45.75+1.5(9.25)=59.625$. Why this step? Robust method, since one huge salary would wreck the mean.
2. Flag. $300>59.625\Rightarrow$ statistically an outlier. Why this step? We convert the fence into a decision — the CEO's pay is unambiguously beyond it.
3. Decision using the parent's framework. It is (a) legitimate (real CEO), (b) not a data error. So do not remove. For a linear model we cap (Winsorize) or log-transform to limit its leverage. Why this step? Detection is statistical; treatment is a business decision. Deleting a valid record biases the model.
4. Winsorize at $P_{95}$ (linear-interpolation percentile). For $n=8$ sorted values, the $95$th percentile sits at fractional rank $(n-1)\cdot 0.95=7\cdot 0.95=6.65$, i.e. $65\%$ of the way from the 7th value ($45$) to the 8th ($300$): $P_{95}=45+0.65(300-45)=45+165.75=210.75$. So $300\to 210.75$; order preserved. Why this step? We must state the interpolation scheme (numpy.percentile default) so the capped value is reproducible. Capping keeps the CEO "highest paid" (monotonicity) while shrinking the leverage. See downstream 4.1.03-Regularization which also tames extreme influence.

Verify: Fence $=59.625$ flags $300$; treatment = cap not delete, capped value $P_{95}=210.75$ under linear interpolation. The value is kept but tamed.

---

Forecast: Left-tail and right-tail should behave symmetrically. Guess: IQR's lower fence catches $-50$; the z-score again self-masks because $-50$ inflates $\sigma$. And the symmetric set gives a clean integer fence.

1. Part A — z-score. $\mu =\frac{180+182+185+188+190+192+195-50}{8}=\frac{1262}{8}=157.75$, population $\sigma \approx 78.9$. Then $z_{-50}=\frac{-50-157.75}{78.9}\approx -2.63$. Why this step? Negative outliers give negative z-scores; we check $|z|>3$, so sign does not change the rule but the magnitude still gets self-masked.
2. Part A verdict (z). $|-2.63|=2.63<3\Rightarrow$ NOT flagged — the same self-masking failure as the high outlier in Ex 2, now on the left tail. Why this step? We make the sign-symmetry explicit: the mechanism (inflated $\sigma$) is identical to Ex 2, only the tail differs.
3. Part A — IQR (linear interpolation). Sorted: $[-50,180,182,185,188,190,192,195]$. numpy.percentile gives $Q_1=181.5$, $Q_3=191.5$, $\text{IQR}=10$, lower fence $=181.5-1.5(10)=166.5$. Why this step? The lower fence is the mirror of the upper one; low outliers are caught by the lower fence, which people often forget to check.
4. Part A verdict (IQR). $-50<166.5\Rightarrow$ flagged. Symmetric to Ex 2 but on the negative side. Why this step? We record the punchline: always test both fences, or you silently miss every left-tail glitch.
5. Part B — boundary. For $[10,12,14,16,18,20,22,24]$, numpy.percentile gives $Q_1=13.5$, $Q_3=20.5$, $\text{IQR}=7$, upper fence $=20.5+1.5(7)=31.0$. Why this step? Exam questions love the "$>$ vs $\ge$" edge — a point is an outlier only when strictly beyond the fence.
6. Part B answer. $v=31.0$ sits exactly on the fence (kept, not strictly greater); $v=31.01$ is flagged. Why this step? The boundary value itself is the trap; naming both the on-fence ($31.0$, kept) and just-past ($31.01$, flagged) cases removes the ambiguity.

Verify: Part A: $z_{-50}\approx -2.63$, $|z|<3$ (missed) but $-50<166.5$ (caught) — mirror of Ex 2. Part B: upper fence $=31.0$; $v=31.0$ borderline-kept, $v>31.0$ flagged.

---

Recall

Recall Which method survives a contaminated mean?

Which detector self-masks a lone extreme value, and which one catches it? ::: The z-score self-masks (the outlier inflates $\sigma$, on either tail); IQR catches it because percentiles ignore extreme magnitudes.

Recall Why Mahalanobis over Euclidean?

What does Mahalanobis distance know that Euclidean does not? ::: It knows the correlations and scales via ${\Sigma }^{-1}$, so it flags points that violate the joint pattern (e.g., huge house, few beds) even when each coordinate alone looks normal.

Recall When does LOF beat a global z-score?

::: When density varies across the data — LOF compares a point to its local neighbors, so a point in a naturally sparse gap isn't flagged, while a truly isolated point far from any dense neighborhood is caught.

Recall Degenerate zero-variance column?

What is $z_i$ when $\sigma =0$, and what should the pipeline do? ::: $z_i=0/0$ is undefined; treat as "no outliers" and drop the constant column in 2.2.01-Feature-selection.

Recall Lower-tail outliers?

Which fence catches a negative glitch value? ::: The lower fence $Q_1-1.5\text{IQR}$; the z-score still self-masks on the left tail just as it does on the right.