Foundations — Feature scaling - normalization vs standardization

This page assumes you have seen nothing. Before you can read Feature scaling - normalization vs standardization, you must be able to read every squiggle it uses. We build them one at a time, each on top of the last.

0. What is a "feature" and a "dataset"?

Everything starts here, so we start here.

Look at the figure. The whole table is the dataset. The amber column is one feature — "price" — pulled out as a list of numbers. Feature scaling always operates on one column at a time: it looks down a column, and rewrites those numbers.

Why the topic needs this: scaling never mixes columns. It reads all the values in a single column to decide how to rewrite that column. (Feature types are covered in 2.1.01-Feature-types-and-data-structures.)

1. The symbol $X$ — a feature's values

Picture it: if the price column is $[200,350,500,800]$ (thousands of dollars), then $X$ is that whole list, and one specific $x$ might be $500$.

Why the topic needs it: every scaling formula is a rule of the form "take one value $x$, do arithmetic, get a new value $x_{\text{new}}$". Writing it once with the symbol saves us from rewriting it for every row.

2. The symbols $X_{\min }$ and $X_{\max }$ — the ends of the range

In the figure, the column's numbers are dots on a horizontal line. $X_{\min }$ is the leftmost dot, $X_{\max }$ the rightmost. The distance between them — the cyan bar — is called the range:

The range is literally "how wide is the spread of this column, end to end". A price column might have range $600$; a bedroom column range $4$. That mismatch is the whole problem feature scaling solves.

Why the topic needs it: normalization uses exactly these two numbers to squash any column into the window $[0,1]$.

3. Subtraction as "shifting", division as "stretching"

Before any formula, understand what the two operations do to a picture.

Top row: original dots. Middle: after subtracting a number — every dot slides left by the same amount, spacing unchanged. Bottom: after dividing — the dots pull together toward 0, spacing shrinks by the same factor.

Why the topic needs it: every scaling formula is exactly "shift, then stretch". If you see subtraction, think slide the line. If you see division, think rescale the ruler. That is all the algebra ever does.

4. The mean $\mu$ — the balance point

$\mu =\frac{x_1+x_2+\cdots +x_n}{n}$

Here $n$ is how many values there are, and $x_1,x_2,\dots$ are the individual values. The three dots $\cdots$ just mean "keep going in the same pattern".

Picture: if the dots were beads on a stiff wire, $\mu$ is the point where the wire balances on a fingertip. Big lonely values (outliers) pull the balance point toward themselves.

Why the topic needs it: standardization slides the whole column so that its balance point sits exactly at $0$. That is the "subtract $\mu$" step.

5. Variance and standard deviation $\sigma$ — the typical spread

Now we need a number for "how spread out is the column", but a smarter one than range (which only looks at the two extreme dots).

Let us earn every piece:

• $x_i-\mu$ is "how far this value sits from the balance point" — could be negative (below) or positive (above).
• We square it, $(x_i-\mu {)}^2$, for two reasons: squaring kills the minus sign (so below-average and above-average both count as positive spread), and it punishes far-away values more.
• We average the squares to get one summary number.

Why take the square root? Variance is in squared units (squared dollars — meaningless to a human). The square root brings us back to the original units, so $\sigma$ answers "a typical distance from the mean, in the same units as the data". In the figure, $\sigma$ is drawn as the cyan half-width band around $\mu$ — roughly where the "normal, non-extreme" values live.

Why the topic needs it: standardization stretches the column so that its typical spread $\sigma$ becomes exactly $1$. That is the "divide by $\sigma$" step. Outlier treatment builds on this idea in 2.1.05-Handling-outliers-detection-and-treatment.

6. Expected value $\mathbb{E}[X]$ — the mean, in disguise

Why does the parent note use both $\mu$ and $\mathbb{E}[X]$? Because when we transform data (multiply, add), the $\mathbb{E}[\cdot ]$ notation lets us write clean rules like $\mathbb{E}[aX+b]=a\mathbb{E}[X]+b$ — "scaling and shifting inside the average pops right out". You do not need probability theory; just read $\mathbb{E}[X]$ as "the average of whatever is inside the brackets".

7. A linear transformation — the master shape, and how to solve for it

Every scaling formula the parent note derives has the same skeleton. Here we not only name it, we derive the two recipes so nothing is dropped from the sky.

Recipe A — Normalization (target: send $X_{\min }\mapsto 0$ and $X_{\max }\mapsto 1$).

We have two demands, which give two equations:

Demand 1 — the smallest value must land on $0$: $a{X}_{\min }+b=0\Rightarrow b=-a{X}_{\min }.$

Demand 2 — the largest value must land on $1$. Substitute the $b$ we just found: $a{X}_{\max }+b=1\Rightarrow a{X}_{\max }-a{X}_{\min }=1\Rightarrow a(X_{\max }-X_{\min })=1.$

So the stretch is $a=\frac{1}{X_{\max }-X_{\min }}$ (one over the range — makes sense: to squeeze a width-$600$ column into width $1$, divide by $600$). Feed that back into $b=-a{X}_{\min }$ to get $b=-\frac{X_{\min }}{X_{\max }-X_{\min }}$. Putting them together: $x_{\text{new}}=\frac{x-X_{\min }}{X_{\max }-X_{\min }}.$ The subtraction slides $X_{\min }$ onto $0$; the division shrinks the range to exactly $1$.

Recipe B — Standardization (target: mean becomes $0$, spread $\sigma$ becomes $1$).

Demand 1 — the new mean must be $0$. Using $\mathbb{E}[ax+b]=a\mu +b$: $a\mu +b=0\Rightarrow b=-a\mu .$

Demand 2 — the new spread must be $1$. Stretching by $a$ multiplies the standard deviation by $a$, so we need $a\sigma =1$, giving $a=\frac{1}{\sigma }$. Then $b=-a\mu =-\frac{\mu }{\sigma }$. Together: $x_{\text{new}}=\frac{x-\mu }{\sigma }.$ Subtracting $\mu$ centres the balance point on $0$; dividing by $\sigma$ makes the typical spread exactly $1$.

Why the topic needs it: once you know it is always "solve for $a$ and $b$ to hit a target", the parent note's derivations stop looking like magic — you are just matching demands to equations.

8. The dangerous edge cases — when the denominator is zero

Both recipes divide — by the range for normalization, by $\sigma$ for standardization. Division by zero is undefined, so we must ask: can those denominators ever be zero? Yes, and in exactly one situation each.

The picture makes it obvious: if all the dots sit on one single point, there is no range and no spread to stretch — there is nothing to rescale.

Why the topic needs it: real datasets contain constant or near-constant columns all the time. Knowing why the formula breaks — zero range, zero variance — lets you spot and handle it instead of shipping silent NaNs into the model.

Equipment checklist

Cover the right side and answer aloud. If any stumps you, reread that section.