Handling missing values (deletion, imputation strategies): Foundations

Before you can read the parent note Handling missing values, you need to be fluent in a small pile of symbols and pictures. The parent note uses them freely. This page builds every one of them from nothing, in an order where each rests on the one before it.

0. The picture everything sits on: a data table

Everything in this topic lives inside a rectangular table. Rows are things you observed (patients, days, houses). Columns are properties (Age, Income, Blood Pressure). One cell = one number.

Figure 1 — a data table with rows, columns and a missing cell. The green box outlines one row (a sample); the yellow box outlines one column (a feature); the red-shaded square is a NaN. If the image fails to load, picture a 4-row grid where the Income cell of row 2 is blank.

Figure — Handling missing values (deletion, imputation strategies)

Why the topic needs this: "missing data" only means something once you have a grid. A blank square is a missing cell (just one square is empty); a "missing row" is different — it means the entire horizontal strip is unobserved or has been removed, so none of that sample's values are available. Deletion methods throw away whole rows; imputation methods fill in individual blank cells.

1. `NaN` — the symbol for "nothing is here"

NaN is not zero. Zero is a real measurement ("blood pressure of 0" — a dead patient). NaN means we simply do not know. The whole topic exists to decide what to do with these tags.

2. Indices: `i`, `j`, and the subscript $x_{ij}$

The parent note writes things like $x_{i}$ , $x_{ij}$ , $x_{i, - j}$ . These are just addresses for cells.

Figure 2 — subscripts as addresses. The red square is cell $x_{2, 3}$ ; the green outline is the whole row $x_{2}$ ; the arrow points to $x_{2, - 3}$ , that row with the red cell knocked out. If the image fails to load, picture a 3×4 grid with the (row 2, column 3) cell highlighted.

Why the topic needs this: to impute a missing cell $x_{ij}$ , methods like KNN and MICE (both explained in §6 and §10) look at the rest of that row, $x_{i, - j}$ , and at other rows $x_{i^{'}}$ . Without addresses you cannot say "use the neighbours of this cell."

3. Sum notation $\sum$ and set membership $\in$ — "add up a chosen list"

Almost every formula in the parent (mean, variance, distance, correlation) starts with a $\sum$ .

$n$ = how many items you are adding (the number of observed values).
The letter under $\sum$ (here $i$ ) is the counter; it just labels each item.

So the parent's $i \in both observed \sum$ just says: let the box be "all rows where both columns are present," and add only over those — you skip the NaN ones. That single trick is exactly what "pairwise deletion" does.

Why the topic needs this: "the mean of the observed values" is by definition a controlled sum over a chosen box of rows. You cannot average without adding, and you cannot skip blanks without a way to say "only these rows."

4. The mean $\overset{x}{ˉ}$ — the balance point

Figure 3 — the mean as a balance point. Blue squares are the observed values on a seesaw beam; the yellow triangle is the pivot at $\overset{x}{ˉ} = 17.5$ ; the red square is a mean-imputed value glued exactly at the pivot. If the image fails to load, picture four weights at 10, 15, 20, 25 balancing on a pivot at 17.5.

Why the topic needs this: the simplest way to fill a blank is to drop in $\overset{x}{ˉ}$ — "assume the missing person is average." That is mean imputation, and the seesaw picture also explains its flaw: piling extra weight exactly at the balance point makes the data look less spread out than it really is (variance shrinks).

Recall Why does the mean sit at the balance point?

Because $\overset{x}{ˉ}$ is the value $c$ that makes the total of signed distances $\sum (x_{i} - c)$ equal to zero — the pushes left and right cancel. ::: The mean is the unique balance point of the data.

5. Spread: variance $σ^{2}$ — how scattered are the points?

Why the topic needs this: it lets you see the flaw of mean imputation. Here is the tiny worked example the parent leans on, built from scratch on five values: $[10, 15, NaN, 20, 25]$ .

Before filling (using only the four observed values), the balance point is $\overset{x}{ˉ} = \frac{10 + 15 + 20 + 25}{4} = 17.5$ , and their average squared distance from it works out to $σ^{2} = 31.25$ .
After mean imputation we drop $17.5$ into the blank, giving $[10, 15, 17.5, 20, 25]$ . The new balance point is still $17.5$ (adding a point exactly at the mean cannot move it), but the average squared distance falls to $σ^{2} = 25$ .

The spread shrank from $31.25$ to $25$ purely because the filled point sits at distance $0$ from the mean — a zero-area square dragging the average down.

Recall Why does mean imputation always shrink variance?

Added points sit at distance $0$ from the mean, so they add zeros into the top of the variance sum while the bottom ( $n$ ) grows. ::: The average squared distance can only go down.

6. Distance $d (i, i^{'})$ — "how alike are two rows?"

KNN imputation needs a number for similarity between two samples. The tool is the Euclidean distance, the ordinary ruler distance.

Figure 4 — distance is Pythagoras on the feature gaps. The yellow dashed legs are the gap in Height and the gap in Age; the red line joining the two dots is their combined straight-line distance $d$ . If the image fails to load, picture a right triangle whose two legs are the per-feature gaps and whose hypotenuse is $d$ .

Why the topic needs this: KNN picks the $κ$ rows with the smallest $d$ — the nearest neighbours — and copies their known values into the blank. No distance, no notion of "nearest."

7. $ϵ$ and weights $w$ — "closer neighbours count more"

Why the topic needs this: this is the whole degenerate-case fix. Without $ϵ$ , an identical row ( $d = 0$ ) would demand division by zero. The crumb keeps the maths finite while still giving twins the heaviest vote. These weights combine the $κ$ neighbour rows from §6 into one imputed value.

8. Correlation — a $\sum$ -based measure of "do two columns move together?"

The parent's pairwise-deletion formula is a correlation, so here is what that symbol means, built on the $\sum$ from §3.

Why the topic needs this: "pairwise deletion" computes each correlation using only the rows where both $X$ and $Y$ are observed — that is the $\sum_{i \in both observed}$ trick from §3. And the reason mean imputation is dangerous is that it flattens these column-to-column relationships toward zero.

9. The taxonomy words: MCAR, MAR, MNAR

These are not equations — they are the three reasons a cell can be blank, and they decide whether deleting or imputing is safe.

These feed directly into fairness and generalisation — see bias–variance tradeoff and train/validation/test split.

10. MICE — the name of the model-based method

The parent's most advanced technique is abbreviated MICE, which appears in the map below, so define it here before you meet it.

How it all feeds the topic

Notice: mean and variance power the simplest fill-and-its-flaw; distance and weights power KNN; correlation is why deletion vs imputation matters; MICE stitches correlations together; the taxonomy decides whether any of it is even honest. Every arrow is a symbol this page just built.

Watch out for data leakage later: you must compute $\overset{x}{ˉ}$ and choose neighbours using only the training rows, never the test rows — otherwise information from the test set sneaks into your fills and your scores become dishonestly optimistic. These fills also interact with feature scaling (distance is meaningless if columns aren't on comparable scales) and outliers (which is why the median can beat the mean).

Equipment checklist

Test yourself — you are ready for the parent note if you can say each aloud:

What NaN means and why it is not 0 ::: "Not a Number" — an unknown cell, whereas 0 is a real measured value.
The difference between a missing cell and a missing row ::: A missing cell is one empty square; a missing row is an entire sample that is unobserved or removed.
What $x_{ij}$ , $x_{i, - j}$ and $x_{i^{'}}$ point to ::: The single cell in row $i$ column $j$ ; all of row $i$ except column $j$ ; and some other row (prime = another one).
How to read $\sum_{i = 1}^{n} x_{i}$ ::: Start counter $i$ at 1, stop at $n$ , add every $x_{i}$ into a running total.
What $i \in S$ and $∣ S ∣$ mean ::: " $i$ is one of the items in box $S$ "; and $∣ S ∣$ is how many items box $S$ holds.
The formula and picture for the mean $\overset{x}{ˉ}$ ::: $\overset{x}{ˉ} = \frac{1}{n} \sum x_{i}$ — the balance point of the data on a seesaw.
Why mean imputation shrinks variance ::: Filled values sit at distance 0 from the mean, adding zero-area squares that drag the average squared distance down.
What the index $k$ counts in the distance formula ::: The features (columns); it steps through each column to take one gap per feature.
How Euclidean distance handles NaN cells ::: Sum only over features observed in both rows, then rescale by total-features $F$ / usable-features $∣ O (i, i^{'}) ∣$ .
The two different meanings of "K" in this topic ::: $F$ (or " $∣ K ∣$ " in some books) = number of columns; the $κ$ in KNN = number of neighbour rows you average.
Why the weight uses $ϵ$ ::: To avoid dividing by zero when a neighbour is at distance 0 (a perfect twin).
What correlation measures and how it uses $\sum$ ::: Whether two columns move together; it sums products of paired deviations, normalised to lie in $[- 1, 1]$ .
What MICE stands for ::: Multiple Imputation by Chained Equations — predict each column's blanks from the others, looping until stable.
The difference between MCAR, MAR, and MNAR ::: Missing tied to nothing / tied to observed columns / tied to the hidden value itself.