Handling missing values (deletion, imputation strategies): Visual walkthrough

This page rebuilds one central result from the parent note from absolute zero: when you replace missing numbers with the average of what you have, the data's mean stays the same but its spread shrinks. The parent stated this ("mean imputation always underestimates variance"). Here we derive it, picture by picture, so you can never be surprised by it again.

We only need one dataset, one idea per step, and a lot of drawings.

Step 0 — What is a "number line pile" and what is missing?

WHAT. Before any formula, picture our data as dots dropped onto a horizontal ruler. Each dot is one measurement.

WHY. Every symbol below ( $\overset{x}{ˉ}$ , $σ^{2}$ ) is secretly a statement about this picture. If we anchor the picture first, the symbols are just shorthand for things we can already see.

PICTURE. Four real values $[10, 15, 20, 25]$ , and one value we never recorded — a hole where a fifth dot should be.

Figure — Handling missing values (deletion, imputation strategies)

Nothing else is defined yet. We will build the mean and the spread from scratch, on this exact picture.

Step 1 — The mean is the balance point of the dots

WHAT. We define one special position on the ruler: the point where the dots would balance if the ruler were a see-saw.

WHY THIS TOOL, NOT ANOTHER? We want a single number to stand in for the hole. Of all possible stand-ins, we will pick the one that sits "in the middle" in a very precise sense (Step 3 proves it). The balance point is the natural first guess: it is the position with equal "pull" from both sides.

PICTURE. Put a fulcrum (a triangle) under the ruler and slide it until the dots don't tip it.

For our dots: $\overset{x}{ˉ} = \frac{10 + 15 + 20 + 25}{4} = \frac{70}{4} = 17.5$

The fulcrum sits at $17.5$ . Notice the hole was not in the sum — we only average what we have.

Step 2 — The spread is the average squared distance from the balance point

WHAT. We define a second number: how wide the pile is. Not "how far is the biggest dot" — that would be fooled by one outlier — but the typical distance from the balance point.

WHY squared distance, not plain distance? Distances left of the balance point are negative, distances right are positive; if we just added them they would cancel to zero (that is literally what "balance point" means!). Squaring kills the sign — every distance becomes positive — so nothing cancels, and far dots (big distance) are punished more than near ones. That is the variance.

PICTURE. Draw a rubber band from each dot to the fulcrum; the length is the distance. Square each length; sum the squares; average.

For our four real dots: $σ^{2} = \frac{( 10 - 17.5 ) ^{2} + ( 15 - 17.5 ) ^{2} + ( 20 - 17.5 ) ^{2} + ( 25 - 17.5 ) ^{2}}{4} = \frac{56.25 + 6.25 + 6.25 + 56.25}{4} = \frac{125}{4} = 31.25$

Hold onto $31.25$ . It is the honest spread of the data we truly have.

Definition Which "divide by" —

$n$ or $n - 1$ ? (population vs. sample variance) There are two standard variance formulas, and they differ only in the denominator: $σ^{2} = \frac{1}{n} \sum (x_{i} - \overset{x}{ˉ})^{2} (population) s^{2} = \frac{1}{n - 1} \sum (x_{i} - \overset{x}{ˉ})^{2} (sample)$

The population version (divide by $n$ ) treats your dots as the whole world — it is the literal "average squared distance", which is why we use it on this page: it makes the picture exact.
The sample version (divide by $n - 1$ , called Bessel's correction) treats your dots as a sample drawn from a larger unseen population and nudges the estimate up a little, because a sample tends to look slightly tighter than the population it came from. The imputation lesson survives either choice. With population variance the shrink factor is $\frac{n}{n + m}$ (derived below). With sample variance the numerator (sum of squared distances) still gains only zeros from the fillers, while the denominator grows from $n - 1$ to $n + m - 1$ — so the estimate still shrinks, just by the factor $\frac{n - 1}{n + m - 1}$ instead. Filling holes at the mean under-reports spread no matter which formula your library uses.

Step 3 — Why the mean (and not some other value) is the natural filler

WHAT. We prove the balance point is exactly the number $c$ that sits "closest to all dots at once" in the squared-distance sense.

WHY. This justifies choosing the mean as our filler in the first place — it is the single value that makes the total squared error as small as possible. So if you must pick one number to represent the pile, this is the least-wrong one.

PICTURE. Slide a candidate filler $c$ along the ruler and watch the total squared-distance bowl. It bottoms out exactly at the fulcrum.

This is why mean imputation is even a reasonable idea. But being the best single number does not mean it leaves the picture unchanged — Step 4 shows the damage.

Step 4 — Drop the filler dot ON the balance point

WHAT. We now fill the hole. We drop a new dot at $\overset{x}{ˉ} = 17.5$ , exactly on the fulcrum, and re-count.

WHY. This is literally what mean imputation does: replace NaN with $\overset{x}{ˉ}$ . We do it, then re-measure both the balance point and the spread to see what changed.

PICTURE. A pale-yellow filler dot lands right on the triangle. Its rubber band to the fulcrum has length zero.

The filled dataset is now $[10, 15, 17.5, 20, 25]$ with $n = 5$ dots.

Does the balance point move? No. Adding a dot at the balance point cannot tip the see-saw — its pull is zero on both sides: $\overset{x}{ˉ}_{new} = \frac{10 + 15 + 17.5 + 20 + 25}{5} = \frac{87.5}{5} = 17.5.$ Same mean. This is why mean imputation feels harmless.

Step 5 — Re-measure the spread: it shrank, and here is exactly why

WHAT. Recompute the variance of the five-dot pile.

WHY. The variance is the average squared distance. We just added a dot whose squared distance is $0$ . Averaging in a zero drags the average down — always.

PICTURE. The same rubber bands as Step 3, plus one new band of length zero. The sum of squares didn't grow at all, but we now divide by 5 instead of 4.

Check: $n = 4$ , $m = 1$ , so $\frac{4}{5} \times 31.25 = 25$ . ✓ Exactly our number.

Step 6 — The median remedy for skew (defined and drawn)

WHAT. Before the edge-case grid, we build the median from zero, because Step 7 leans on it for skewed data.

WHY THIS TOOL, NOT THE MEAN? One wild far-away dot yanks the balance point toward itself (squaring punishes distance, so a distant dot has huge pull). We want a center that a single outlier cannot drag. The median is that center.

PICTURE. Line the dots up in order and walk in from both ends until you meet in the middle — that meeting position is the median. Notice the far-right outlier moves the mean but not the median.

Definition The median, and its "minimizes plain distance" property

The median is the middle dot when values are sorted: half the dots sit to its left, half to its right. For $[8, 9, 10, 11, 28]$ the sorted middle is $10$ .
Where the mean minimizes squared distances $\sum (x_{i} - c)^{2}$ , the median minimizes plain (unsquared) distances $\sum ∣ x_{i} - c ∣$ — here $∣ x_{i} - c ∣$ means "the distance from $x_{i}$ to $c$ , sign thrown away".
Why that makes it outlier-proof: plain distance grows only linearly with how far a dot is, so one runaway dot ( $28$ ) adds a moderate, not explosive, penalty — it cannot bribe the center to slide toward it the way squared distance lets it bribe the mean. That is why the parent note says: skewed data / outliers → impute with the median. See 2.1.03-Outlier-detection-and-treatment.

Step 7 — The remaining edge cases (so nothing surprises you)

WHAT. Three more boundary situations, each drawn, alongside the skew case from Step 6.

WHY. The contract: the reader must never hit a case we didn't show. Formulas are only trustworthy at their edges.

PICTURE. A 2×2 chalk grid of the corner cases.

$m = 0$ (nothing missing). Then $\frac{n}{n + 0} = 1$ : no shrink. Correct — mean imputation of nothing changes nothing.
Many holes, $m ≫ n$ . $\frac{n}{n + m} \to 0$ : the variance collapses toward zero. Fill 90 holes among 10 real dots and the data pretends to be almost a single spike. This is why mean imputation is dangerous at high missing rates.
All dots identical, $σ_{obs}^{2} = 0$ . The pile is already one spike; the filler lands on it too. $\frac{n}{n + m} \times 0 = 0$ . No shrink because there was no spread to lose — the one case where mean imputation is genuinely harmless.
Skewed pile / outliers. The balance point gets yanked toward the far outlier, so the filler lands in a sparse region where no real dot lives. Here use the median (Step 6): it minimizes $\sum ∣ x_{i} - c ∣$ , so a lone far dot can't drag it.

(And recall the fully degenerate case from Step 0: $n = 0$ has no dots at all, so no mean exists and imputation is undefined.)

Recall Quick self-checks (tap a line to reveal the answer after

:::) Mean imputation keeps the mean but shrinks the variance — true or false? ::: True; the filler sits at the mean (zero deviation) so the sum of squares is unchanged while $n$ grows. With $n = 8$ observed and $m = 2$ filled at the mean, by what factor does the population variance shrink? ::: By $\frac{8}{10} = 0.8$ , i.e. to 80% of the honest variance. Which imputation edge case leaves the variance exactly unchanged? ::: When $m = 0$ (nothing missing) or when $σ_{obs}^{2} = 0$ (all observed dots identical). Why prefer the median over the mean for skewed data? ::: The median minimizes $\sum ∣ x_{i} - c ∣$ (plain distance), which grows only linearly, so a single far outlier cannot drag the center. What happens when $n = 0$ (every value missing)? ::: The mean is undefined (divide by zero); you need at least one real observation before mean imputation is possible.

The one-picture summary

One ruler, one fulcrum, one filler dot on the fulcrum with a zero-length band — and the shrink factor $\frac{n}{n + m}$ written under the bowl. That single frame contains the whole derivation: same balance point, punctured spread.

Recall Feynman retelling (say it out loud, no symbols)

I have a bunch of dots on a ruler and one hole where a dot should be. I find the see-saw balance point of the dots I do have — that's the mean (and if there are zero dots, there's no balance point at all, so I'm stuck until I get one real value). Then I measure how wide the pile is by stretching a rubber band from each dot to the balance point, squaring each length (so left and right can't cancel), and averaging — that's the variance. Now I fill the hole by dropping a new dot right on the balance point. Its rubber band has zero length. The balance point doesn't move — a dot on the fulcrum can't tip anything — so the mean is safe. But the average rubber-band-length just got a fat zero mixed into it, so the width shrinks. Exactly: the new width is the old width times (real dots) over (real dots plus filled holes), a fraction always below one — and that shrink happens whether my library divides by $n$ or by $n - 1$ . The moral: filling holes with the average keeps the center honest but makes the data look tighter than it is, so a model trained on it grows over-confident. If the pile is lopsided, I use the median instead — the middle dot — so one wild dot can't hijack the filler.

Visual walkthrough — Handling missing values (deletion, imputation strategies)