Outlier detection and treatment: Visual walkthrough

We assume you know nothing beyond: a point is a pair of numbers, and you can measure a straight-line gap with a ruler. Everything else is earned below.

Step 1 — A point, a centre, and the naive ruler

WHAT. We have data with two features. Call them $f_{1}$ (say, house size in sqft) and $f_{2}$ (say, number of bedrooms). Each house is a dot on a flat picture: its horizontal position is $f_{1}$ , its vertical position is $f_{2}$ . The pile of dots has a centre — the average house — which we write $μ$ (Greek letter "mu", just means "the middle point").

WHY. To ask "is this house weird?" we first need a reference. The natural reference is the centre of the crowd. Weirdness = far from centre. So we need a way to measure "far".

PICTURE. Below, the orange dot is one house $x$ , the navy dot is the centre $μ$ . The straight magenta ruler between them is the everyday distance everyone learns first.

Figure — Outlier detection and treatment

We chose Pythagoras (the square root of a sum of squares) and not, say, just adding the gaps, because a straight ruler laid between two dots physically obeys Pythagoras. That is exactly what "straight-line distance" means.

Step 2 — Why the plain ruler lies

WHAT. We now spread the crowd out unevenly. Suppose sqft varies over thousands while bedrooms varies over a handful. The cloud of dots is a long thin stretched blob, wide along $f_{1}$ and short along $f_{2}$ .

WHY. The plain ruler counts "3000 sqft away" and "3 bedrooms away" with the same importance. But 3000 sqft is a tiny step for that feature (it ranges over huge numbers), while 3 bedrooms is a huge step for its feature (it barely ranges at all). Equal ruler-length does not mean equal weirdness.

PICTURE. Two points sit on the same magenta circle — same Euclidean distance from centre. Yet point $P$ (violet) lies inside the blob and point $Q$ (orange) pokes out of it. Equal ruler, wildly unequal weirdness. The ruler is fooled.

Step 3 — First fix: divide each direction by its spread

WHAT. For each feature separately, measure its typical spread — the standard deviation $σ$ ("sigma"), how far a typical point sits from the centre along that one axis. Then rescale: instead of raw gaps, use gaps counted in units of that feature's own spread.

WHY. If bedrooms typically wander by $σ_{2} = 1$ but sqft wanders by $σ_{1} = 1000$ , then "3 bedrooms off" is $3/1 = 3$ spreads — genuinely far — while "3000 sqft off" is $3000/1000 = 3$ spreads — the same weirdness. Dividing by $σ$ converts every axis to a common, unit-free currency: "how many typical spreads away." (This is exactly the z-score idea from the parent note, now applied per axis.)

PICTURE. We squash the wide blob and stretch the short one until the cloud becomes a round ball. Now equal ruler-length genuinely means equal weirdness — for the moment.

Step 4 — The leftover problem: tilt (correlation)

WHAT. Real blobs are not just wide-or-tall; they are usually tilted. Big houses tend to have more bedrooms, so the cloud leans diagonally — as sqft goes up, bedrooms go up too. This leaning is called correlation.

WHY. Step 3 only stretched along the horizontal and vertical axes. It cannot un-tilt a diagonal cloud. A point that lies along the tilt direction (big house, many rooms) is normal even if it's far. A point that lies across the tilt (big house, few rooms) breaks the pattern and is weird — even at the same rescaled distance. Standardizing per-axis is blind to this.

PICTURE. Two points on the same standardized circle: violet $P$ rides along the diagonal grain of the blob (normal); orange $Q$ cuts against the grain (anomalous). Same rescaled distance, different truth. We still need to account for the tilt.

Step 5 — The object that stores width AND tilt: the covariance matrix

WHAT. Bundle everything the blob's shape knows into one small table of four numbers, the covariance matrix $Σ$ ("capital sigma"): $Σ = [σ_{1}^{2} c c σ_{2}^{2}]$

WHY. We need a single object that encodes both how wide the blob is in each direction and how tilted it is. The two diagonal entries $σ_{1}^{2}, σ_{2}^{2}$ store the widths (variance = spread squared). The two off-diagonal entries $c$ store the tilt: $c = 0$ means no lean (upright blob), $c > 0$ means the "up-right" lean of our houses, $c < 0$ means a "down-right" lean. One table, the whole shape.

Definition Covariance

$c$ — spelling out "the tilt number" $c = E [(x_{1} - μ_{1}) (x_{2} - μ_{2})] = \frac{1}{n} \sum_{j = 1}^{n} (x_{1}^{(j)} - μ_{1}) (x_{2}^{(j)} - μ_{2})$

$(x_{1}^{(j)} - μ_{1})$ and $(x_{2}^{(j)} - μ_{2})$ ::: how far the $j$ -th point sits from centre along axis 1 and along axis 2.
multiplying the two gaps ::: this is the trick. If a point is big-on-both or small-on-both, the two gaps share a sign → product is positive. If it's big-on-one-small-on-the-other, the signs differ → product is negative.
$\frac{1}{n} \sum$ (average) ::: adds up those signed products. If most points move together, the positives dominate → $c > 0$ (up-right lean). If they move oppositely → $c < 0$ . If there's no pattern the positives and negatives cancel → $c = 0$ .
note $σ_{i}^{2} = E [(x_{i} - μ_{i}) (x_{i} - μ_{i})]$ ::: a feature's variance is just its covariance with itself — which is why $σ_{1}^{2}, σ_{2}^{2}$ sit on the diagonal of $Σ$ .

PICTURE. The blob with its two natural axes drawn on it: a long axis (the grain, where the blob is widest) and a short axis (across the grain). $Σ$ is precisely the recipe for these two axes and their lengths.

Step 6 — Undoing the shape: the whitening transform $Σ^{- 1}$

WHAT. We want a mathematical operation that grabs the tilted, stretched blob and squeezes it back into a perfect round ball of radius 1 — un-tilt and un-stretch in one move. That operation is applying the inverse of $Σ$ , written $Σ^{- 1}$ .

Definition What "matrix inverse

$Σ^{- 1}$ " means here The number $5$ has an inverse $\frac{1}{5}$ because $5 \times \frac{1}{5} = 1$ . For matrices, the role of " $1$ " is played by the identity matrix $I = [1001]$ (the "do-nothing" transform: it leaves every point where it is). The inverse $Σ^{- 1}$ is the unique table that satisfies $Σ^{- 1} Σ = Σ Σ^{- 1} = I,$ i.e. " $Σ^{- 1}$ then $Σ$ " changes nothing — so $Σ^{- 1}$ is exactly the transform that undoes whatever $Σ$ does. For our $2 \times 2$ shape-table there is a closed formula: $Σ^{- 1} = \frac{1}{σ _{1}^{2} σ _{2}^{2} - c ^{2}} [σ_{2}^{2} - c - c σ_{1}^{2}]$

$σ_{1}^{2} σ_{2}^{2} - c^{2}$ (the determinant) ::: measures the blob's area; dividing by it is where "undo the stretch" happens.
swapping the diagonal entries and flipping the sign of $c$ ::: this is precisely the recipe that un-tilts the lean.
it only works when the determinant is not zero — exactly the SPD requirement from Step 5.

WHY the inverse, and not $Σ$ itself? $Σ$ describes how a round ball got stretched into the data blob. To reverse a stretch you apply its inverse — exactly like undoing "multiply by 5" with "multiply by $\frac{1}{5}$ ." Where the blob is wide (large variance), $Σ^{- 1}$ shrinks that direction, so travelling there is cheap (counts little toward distance). Where the blob is thin, $Σ^{- 1}$ expands it, so travelling across the grain is expensive (counts a lot). This is the un-fooling we wanted in Steps 2 and 4, both handled at once.

PICTURE. Left: the raw tilted blob with two mismatched points. Right: after applying the whitening transform the blob is a clean unit circle, and now the anomalous point clearly sits outside while the normal one sits inside. Straight ruler on the right = honest weirdness.

Step 7 — Assembling the Mahalanobis distance

WHAT. Now measure the plain squared ruler in the whitened world. Starting from the raw gap vector $d = x - μ$ , whitening and then measuring length gives: $D_{M} (x) = (x - μ)^{T} Σ^{- 1} (x - μ)$

WHY this exact shape. Read it right-to-left as a story:

$(x - μ)$ — the raw arrow from centre to point, still in the distorted world.
$Σ^{- 1} (x - μ)$ — apply the "undo-the-blob" button: the arrow is now in the fair round world.
$(x - μ)^{T} [\dots]$ — the leading row-vector $(x - μ)^{T}$ multiplies the whitened arrow to form Pythagoras' sum of squares in that fair world (this is the matrix way of writing "square each component and add").
the outer — turn the squared length back into an ordinary length, so the units make sense.

PICTURE. The final annotated formula, each chunk coloured and tagged with the job it performs: raw arrow → whiten → measure → square-root.

Step 8 — Every special case, checked

WHAT & WHY. A ruler you trust must behave sensibly at the extremes. We walk each one.

Case A — no correlation, equal spread ( $c = 0$ , $σ_{1} = σ_{2} = σ$ ). Then $Σ = σ^{2} I$ and $Σ^{- 1} = \frac{1}{σ ^{2}} I$ , so $D_{M} = \frac{1}{σ} (x_{1} - μ_{1})^{2} + (x_{2} - μ_{2})^{2} = \frac{D _{E}}{σ} .$ Mahalanobis collapses to the plain ruler divided by spread — exactly the z-score. Sanity: passed.

Case B — no correlation, unequal spread ( $c = 0$ , $σ_{1} \neq = σ_{2}$ ). Then $Σ^{- 1} = diag (1/ σ_{1}^{2}, 1/ σ_{2}^{2})$ and $D_{M} = (\frac{x _{1} - μ _{1}}{σ _{1}})^{2} + (\frac{x _{2} - μ _{2}}{σ _{2}})^{2},$ which is exactly the standardized distance of Step 3. The tilt machinery correctly does nothing when there is no tilt. Sanity: passed.

Case C — the point IS the centre ( $x = μ$ ). Then $x - μ = 0$ , so $D_{M} = 0$ : the centre is zero-weird. As you slide $x$ toward $μ$ , $D_{M}$ shrinks smoothly to 0. No blow-up.

Case D — a perfectly redundant feature (degenerate blob). If two features are identical (perfect correlation), the blob collapses to a line, the determinant $σ_{1}^{2} σ_{2}^{2} - c^{2}$ becomes $0$ , so $Σ$ is not positive-definite and $Σ^{- 1}$ does not exist — $D_{M}$ is undefined. WHY it must fail: you cannot un-stretch a blob that has zero thickness in some direction; the "undo" button has nothing finite to undo. Fix in practice: drop or merge the redundant feature (see 2.2.01-Feature-selection) before computing $D_{M}$ .

PICTURE. Four mini-panels, one per case: (A) round ball, (B) axis-stretched ball, (C) point at centre with $D_{M} = 0$ , (D) collapsed line with a red "no inverse" cross.

Worked example Putting a number on it (the parent's house example)

Centre $μ = [2000, 3]$ , with $Σ = [1 0^{6} 1600 16004] .$ House A $= [5000, 2]$ : a big house with too few rooms — against the grain. Displacement $d = [3000, - 1]$ . Working through $D_{M} = d^{T} Σ^{- 1} d$ gives $D_{M} (A) \approx 5.69$ . House B $= [5000, 7]$ : big house with many rooms — along the grain. Displacement $d = [3000, 4]$ gives $D_{M} (B) \approx 7.93$ . Look closely: even though A is only $1$ bedroom off (versus B's $4$ ), A scores as more consistent with the grain in one sense but not the other — the numbers show that raw Euclidean size (both $\approx 3000$ sqft away) tells you almost nothing, while $D_{M}$ reweights everything through the correlation. The lesson is not "A beats B" but that plain distance and Mahalanobis distance rank points completely differently once tilt is accounted for.

The one-picture summary

Everything above compressed: raw tilted blob (plain ruler fooled) → apply $Σ^{- 1}$ to whiten → round ball where the honest ruler flags the true anomaly. The threshold circle uses the $χ^{2}$ cutoff mentioned in the parent so you can say statistically how weird is too weird.

Recall Feynman retelling — say it back in plain words

Imagine a crowd of dots that looks like a tilted, stretched pancake. You want to know if a new dot is a stranger. A plain ruler is unfair: it counts a long-but-normal direction the same as a short-but-strange direction. So you invent a magic squishing move — the whitening transform, written $Σ^{- 1}$ — that grabs the pancake and rounds it into a perfect ball. All the built-in tilt (covariance $c$ ) and stretch (variances $σ_{i}^{2}$ ) of the data got baked into one little table $Σ$ , so its inverse knows exactly how to un-bake it. In that fair round world you finally lay down your ordinary ruler, and its length is the Mahalanobis distance: cheap to travel along the crowd's grain, expensive to travel across it. Points beyond the $χ^{2}$ threshold circle are the true outliers. When there's no tilt and equal spread, the magic move does nothing and you're back to a plain z-score — which is exactly how you know the idea is trustworthy.

Recall Quick self-test

Why apply $Σ^{- 1}$ (the inverse) rather than $Σ$ itself? ::: $Σ$ describes how a round ball got stretched into the data blob; to undo that stretch we apply its inverse — the transform for which $Σ^{- 1} Σ = I$ — cheapening the wide directions and expensive-ing the thin ones. What does $c = 0$ in $Σ$ mean, and what does $D_{M}$ become? ::: No tilt/correlation; $D_{M}$ reduces to the per-axis standardized (z-score) distance. Why does $D_{M}$ break when two features are identical? ::: The blob collapses to a line, the determinant hits zero so $Σ$ is not positive-definite, $Σ^{- 1}$ does not exist, and there is nothing finite to "un-stretch."

Related vault notes: this ruler feeds anomaly systems in 3.3.02-Anomaly-detection; the per-axis rescaling echoes 2.1.02-Feature-scalingand-normalization; robustness of statistics connects to 2.1.01-Missing-data-handling and shrinking large effects to 4.1.03-Regularization. Hinglish version: 2.1.03 Outlier detection and treatment (Hinglish).