Visual walkthrough — Feature scaling - normalization vs standardization

This page builds both scaling formulas from absolute zero. We assume you know nothing except that numbers can sit on a number line and that we can stretch and slide them. Every symbol is earned before it is used. This is the visual companion to the parent note.

We will discover that there are exactly two natural ways to "slide and stretch", and each answers a different question.

---

Step 1 — What is a feature, drawn as dots on a line

WHAT. Take one column of raw data, for example five incomes: $X=[30,45,50,55,120](\text{thousands of dollars})$ Each number is one dot on a horizontal number line.

WHY. Before we transform anything, we must see the raw object. Every operation later is just moving these dots. If you can picture the dots, you can picture the math.

PICTURE. Look at the figure. The dots are bunched near the left; the lone red dot at $120$ sits far away. That red dot is the villain of this whole story — remember it.

---

Step 2 — The only move we allow: slide then stretch

WHAT. We restrict ourselves to a linear transformation: $X_{\text{new}}=\underset{\text{stretch}}{\underbrace{a}}\cdot X+\underset{\text{slide}}{\underbrace{b}}$ Here $a$ multiplies every dot (stretch or shrink the spacing) and $b$ adds to every dot (slide the whole line left or right).

WHY this and not something fancier? Because a linear move is the only kind that never reorders the dots and never distorts their relative spacing. If house A was cheaper than house B, it must stay cheaper, and by a proportional amount. Any curved (nonlinear) transform would secretly change which points are "close", corrupting distance-based algorithms. So we demand: slide ($b$) and stretch ($a$), nothing else.

PICTURE. The figure shows the same dots twice: top row is the original, bottom row is after one $a$ (stretch) and one $b$ (slide). The pattern of gaps is identical — only the ruler underneath changed.

---

Step 3 — Wish A: "land the ends on 0 and 1" → Normalization

WHAT. Our two conditions are: the smallest value $X_{\min }$ should map to $0$, and the largest value $X_{\max }$ should map to $1$.

Condition 1 — plug in $X=X_{\min }$, demand output $0$: $0=a{X}_{\min }+b\Rightarrow \underset{\text{the slide is fixed by the min}}{\underbrace{b=-a{X}_{\min }}}$

Condition 2 — plug in $X=X_{\max }$, demand output $1$: $1=a{X}_{\max }+b=a{X}_{\max }-a{X}_{\min }=a\underset{\text{the total width}}{\underbrace{(X_{\max }-X_{\min })}}$ $\Rightarrow a=\frac{1}{X_{\max }-X_{\min }}$

WHY these two wishes? Because "fit inside a box from 0 to 1" is exactly what bounded algorithms want — pixel intensities, probabilities, or sigmoid inputs in neural networks all live in a fixed box.

PICTURE. The figure shows a rubber band pinned at $X_{\min }$ (which must land on $0$) and at $X_{\max }$ (which must land on $1$). Everything in between is dragged proportionally.

Substituting $a$ and $b$ back into the template gives the famous result:

Check the villain-free example, prices $[200,350,500,800]$ (thousands): width $=600$, so $350\to \frac{150}{600}=0.25$, $500\to 0.50$. The percentages ($0\%,25\%,50\%,100\%$) survive untouched.

---

Step 4 — Why one outlier destroys Normalization

WHAT. Feed in the villain: $X=[1,2,3,4,100]$. Now $X_{\min }=1$, $X_{\max }=100$, width $=99$. $1\to 0,2\to \frac{1}{99}\approx 0.01,3\to 0.02,4\to 0.03,100\to 1$

WHY it fails. The width is set by the single extreme value. The four honest points get crushed into the tiny interval $[0,0.03]$ — they become nearly indistinguishable. Any algorithm now "sees" four identical points and one loner.

PICTURE. The figure shows the four good dots squeezed against the left wall while the red outlier hogs the entire ruler. This is why Normalization demands no outliers — see outlier handling.

---

Step 5 — Wish B: "center at 0, spread of 1" → Standardization

WHAT. Same template $X_{\text{std}}=aX+b$, but now two statistical wishes. First we need the mean and standard deviation. Throughout, let $n$ = the number of observations (here $n=5$), and let $X_i$ be the $i$-th value.

We now introduce two shorthand words for "what happens on average" — because our two wishes are about the average value and the average spread of the transformed dots.

Condition 1 — make the new mean $0$. Using $\mathbb{E}[aX+b]=a\mathbb{E}[X]+b=a\mu +b$: $\mathbb{E}[aX+b]=a\mu +b=0\Rightarrow b=-a\mu$

Condition 2 — make the new spread $1$, i.e. variance $1$. Using $\text{Var}(aX+b)=a^2{\sigma }^2$: $\text{Var}(aX+b)=a^2{\sigma }^2=1\Rightarrow a=\frac{1}{\sigma }(\text{take }+\text{ to keep order})$

WHY these wishes? "Distance from typical, measured in spreads" is dimensionless and outlier-tolerant — perfect for regularization, PCA, and gradient descent.

PICTURE. The figure marks $\mu$ as the pivot; subtracting $\mu$ slides the pivot to $0$, dividing by $\sigma$ rescales the ruler so one standard deviation equals one tick.

---

Step 6 — Why Standardization survives the outlier

WHAT. Feed in the same villain $X=[1,2,3,4,100]$. First the mean: $\mu =\frac{1+2+3+4+100}{5}=\frac{110}{5}=22$ Now the (population) standard deviation. The squared distances from $\mu =22$ are: $(1-22{)}^2=441,(2-22{)}^2=400,(3-22{)}^2=361,(4-22{)}^2=324,(100-22{)}^2=6084$ $\sum (X_i-\mu {)}^2=441+400+361+324+6084=7610$ $\sigma =\sqrt{\frac{7610}{5}}=\sqrt{1522}\approx 39.01$ Then each z-score $\frac{X-\mu }{\sigma }$: $1\to -0.538,2\to -0.513,3\to -0.487,4\to -0.462,100\to 2.000$

WHY it survives. The denominator is $\sigma$, an average spread — a single huge value inflates it only mildly, not to the full width. So the four honest points keep visible gaps ($-0.538$ vs $-0.513$…), while the outlier is flagged at $+2.00$ rather than allowed to crush everyone.

PICTURE. The figure shows the same red villain as Step 4, but now the four good dots stay clearly separated. Compare this figure directly with Step 4 — same data, opposite outcome.

---

Step 7 — The degenerate case: all values identical

WHAT. What if every value is the same, e.g. $X=[7,7,7,7]$?

WHY it matters. Then $X_{\max }=X_{\min }$, so Normalization's denominator is $0$; and $\sigma =0$, so Standardization's denominator is $0$. Division by zero — both formulas explode.

PICTURE. The figure shows a single stacked dot: there is no spread to normalize. The lesson: a constant feature carries no information and cannot be scaled — drop it.

---

The one-picture summary

Both methods are the same linear move $X_{\text{new}}=aX+b$; they differ only in which two wishes fix $a$ and $b$. Normalization pins the ends to a box; Standardization pins the center and spread to the standard bell. The figure lays the raw dots, the normalized dots, and the standardized dots on three parallel rulers so you can watch the villain outlier shrink under standardization but dominate under normalization.

Recall Feynman retelling — say it back in plain words

We started with a bunch of dots on a line. We promised to only slide and stretch them, because that keeps their order and spacing honest. That promise is a formula with two knobs: $a$ (stretch) and $b$ (slide). Two knobs need two wishes to lock them down.

Wish A: "smallest dot lands on 0, largest lands on 1." Solving gives Normalization — every dot becomes its percentage of the way across the range. Neat, but the biggest value sets the ruler, so one freak value squashes everyone.

Wish B: "the pile balances at 0 and has a typical spread of 1." Solving gives Standardization — every dot becomes how many standard deviations from typical. Because spread is an average, one freak value only nudges the ruler, so honest points stay separated and the freak gets flagged, not obeyed. Standardization is more outlier-tolerant, but not immune — an extreme value still tugs on $\mu$ and $\sigma$ a little.

Edge case: if all dots sit on the same spot there's nothing to stretch (divide-by-zero), so we just throw that column away.

Recall Two quick self-checks

Which method would you use for pixel values known to lie in $[0,255]$? ::: Normalization — the range is bounded and outlier-free, and we often want a fixed $[0,1]$ box. A house-price column has one $10M$ mansion among $200K$ homes. Which method keeps the ordinary houses distinguishable? ::: Standardization — the mansion becomes a large z-score but the average spread barely moves.