Foundations — Outlier detection and treatment

Before you can read a single formula in the parent note, you need a vocabulary of small pictures. This page builds every one of them from nothing. We never use a symbol before we draw it.

0. The raw material: data points on a line

Everything starts with numbers you collected. Ages, prices, temperatures — each measurement is one data point.

Picture it as beads dropped onto a number line. Most beads clump together; a few wander off. That wandering bead is the thing we are hunting.

• $x_i$ — one bead (look at the coral dot on the far right of the figure).
• $n$ — how many beads total.
• The number line — the stage on which "far" and "near" have meaning.

1. The middle: mean $\mu$

To say a point is "far from the crowd", you first need a single number that marks where the crowd is.

$\mu =\frac{1}{n}{\sum }_{i=1}^n{x}_i$

Let's earn every symbol in that line:

• $\sum$ (capital Greek "sigma") means add them all up. The $i=1$ below and $n$ above say "start at the first bead, stop at the last."
• $\frac{1}{n}$ then shares the total equally among all $n$ beads.

2. The typical spread: standard deviation $\sigma$

Knowing the middle isn't enough. In one dataset the beads huddle tightly; in another they scatter widely. We need a number for "typical distance from the middle."

$\sigma =\sqrt{\frac{1}{n}{\sum }_{i=1}^n(x_i-\mu {)}^2}$

Reading it piece by piece, inside-out:

• $(x_i-\mu )$ — this is the signed deviation: bead $i$'s position minus the centre. WHY subtract? Subtraction measures displacement from the reference $\mu$. But watch the sign: a bead to the right of the mean gives a positive number, a bead to the left gives a negative number. So $(x_i-\mu )$ by itself is not a distance — it is a signed gap that can point either way.
• $({)}^2$ — square it. WHY square? This is precisely where we throw away the sign: squaring turns every signed gap into a positive quantity, so left-beads and right-beads no longer cancel each other out. As a bonus it punishes far beads much harder than near ones. (Only after squaring do we have something distance-like.)
• $\sum \dots$ then $\frac{1}{n}$ — average those squared gaps.
•  ​ — undo the squaring so the answer is back in the original units (rupees, years), not squared units.

The z-score: counting rulers

Now we can name the tool the parent note leans on. Once you have $\mu$ (the centre) and $\sigma$ (the length of one ruler), the natural question is "how many rulers away is a point?"

3. Rank-based landmarks: percentiles, $Q_1$, $Q_3$, IQR

Because $\mu$ and $\sigma$ can be fooled, we build a ruler that ignores how big the wild beads are and only cares about their position in line.

4. From one dimension to many: vectors and ${\mathbb{R}}^d$

Real data has several features per row (sqft and bedrooms). One bead is no longer a point on a line — it's a point in space.

The mean is now a mean vector $\mu =({\mu }^{(1)},\dots ,{\mu }^{(d)})$: the average bead position, one coordinate per feature.

Two new pieces of notation, both easy:

• The little $T$ means transpose — flip a column of numbers on its side into a row so it can be multiplied. Because $(x-\mu )$ is the element-wise vector above, the product $(x-\mu {)}^T(x-\mu )$ is just "square each coordinate gap and add them" — the same Pythagoras you already know, extended to $d$ features. (Squaring again removes the signs, exactly as in Section 2.)

5. Correlation and the covariance matrix $\Sigma$ (a first look)

So far our beads were single numbers $x_i$. Now each bead has several features. To talk about "how two features move together," let us give the two feature-columns names: write $X_i$ for the value of the first feature at bead $i$, and $Y_i$ for the value of the second feature at the same bead $i$. So $X_i$ is exactly the first coordinate $x_i^{(1)}$ and $Y_i$ the second coordinate $x_i^{(2)}$ of the vector from Section 4 — the same data, just sliced column-by-column instead of row-by-row. Each has its own mean, ${\mu }_X$ and ${\mu }_Y$.

You don't need to hand-compute $\Sigma$ here — you only need the picture: $\Sigma$ is the recipe for the tilt and stretch of that lavender ellipse. The diagonal sets how wide the ellipse is along each axis; the off-diagonal sets its tilt. The parent note uses ${\Sigma }^{-1}$ ("Sigma inverse", the undo-operation) to squash the ellipse back into a circle so ordinary distance works again. Keep the ellipse picture; that's the payload.

6. Neighbourhoods and density (a first look)

The last family of methods (LOF) doesn't ask "far from the centre?" but "far from your neighbours?"

Prerequisite map

The diagram below is a flow map. Each box is one idea from this page; each arrow means "you need the box at the tail before the box at the head makes sense."

In words, in case the diagram does not render: Data points on a line is the root. From it, three branches grow. (1) The mean gives the centre, and the mean feeds the standard deviation (the spread), which feeds the z-score method. (2) The same data points, ranked, give percentiles and quartiles, which feed the IQR fences method. (3) Data points also become vectors in ${\mathbb{R}}^d$; vectors give Euclidean distance, which — once covariance shapes the ellipse — feeds the Mahalanobis distance; separately vectors give neighbours and density, which feed the Local Outlier Factor. All four methods (z-score, IQR, Mahalanobis, LOF) pour into the parent topic, Outlier detection and treatment.

Related vault stops

• Scaling turns these rulers comparable across features → 2.1.02-Feature-scalingand-normalization
• Missing values must be handled before you trust any mean or quartile → 2.1.01-Missing-data-handling
• Outliers as the thing you want (not noise) → 3.3.02-Anomaly-detection
• Penalising extreme model behaviour → 4.1.03-Regularization

Equipment checklist

Can you say each of these out loud before opening the parent note?