Exercises — Feature scaling - normalization vs standardization

Normalization guarantees $[0,1]$: the minimum maps to $\frac{X_{\min }-X_{\min }}{X_{\max }-X_{\min }}=0$ and the maximum to $\frac{X_{\max }-X_{\min }}{X_{\max }-X_{\min }}=1$; everything between is a fraction of the range. Standardization is unbounded: a value far from $\mu$ (an outlier) has a large $X-\mu$, so $\frac{X-\mu }{\sigma }$ can be any real number. A point 4 standard deviations out simply becomes $+4$.

Here $X_{\min }=10$, $X_{\max }=30$, and $20$ sits exactly in the middle, so normalized $=\frac{20-10}{30-10}=\frac{10}{20}=0.5$. The mean is $\mu =\frac{10+20+30}{3}=20$, and $20$ is the mean, so its z-score is $\frac{20-20}{\sigma }=0$. Middle value → $0.5$ under Min–Max, $0$ under Z-score. This is the fastest sanity check you can run.

$X_{\min }=4$, $X_{\max }=20$, range $=16$.

• $4\to \frac{4-4}{16}=0$
• $8\to \frac{8-4}{16}=\frac{4}{16}=0.25$
• $16\to \frac{16-4}{16}=\frac{12}{16}=0.75$
• $20\to \frac{20-4}{16}=1$

Result: $[0,0.25,0.75,1]$.

Where the formula comes from (build it, don't memorize it). A rescaling to a target box $[a,b]$ is just a two-point linear map: a straight line that must pass through two known dots. Call the raw value $X$ and its scaled value $Y$. We demand:

• when $X=X_{\min }$, the output hits the bottom of the box: $Y=a$;
• when $X=X_{\max }$, the output hits the top of the box: $Y=b$.

Step A — get a $0\to 1$ progress dial first. The fraction $t=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$ answers "how far along the raw range are we?" — it is $0$ at $X_{\min }$ and $1$ at $X_{\max }$. This is our familiar plain normalization. Step B — stretch the dial to width $(b-a)$ and lift it to start at $a$. Multiplying $t$ by $(b-a)$ makes it run from $0$ to $b-a$; adding $a$ slides that whole span up so it runs from $a$ to $b$. Hence $Y=a+(b-a)t=a+(b-a)\cdot \frac{X-X_{\min }}{X_{\max }-X_{\min }}.$ Check the two anchor dots: at $t=0$, $Y=a$; at $t=1$, $Y=a+(b-a)=b$. ✓ The formula is nothing more than "compute progress, then rescale-and-shift the progress."

Now the numbers. Here $a=0$, $b=100$, $X_{\min }=50$, $X_{\max }=90$, range $=40$.

• $50\to 0+100\cdot \frac{50-50}{40}=0$
• $70\to 0+100\cdot \frac{70-50}{40}=100\cdot \frac{20}{40}=50$
• $90\to 0+100\cdot \frac{90-50}{40}=100$

Result: $[0,50,100]$. The middle score sits halfway because $70$ is the midpoint of $[50,90]$.

Mean: $\mu =\frac{2+4+6+8}{4}=\frac{20}{4}=5$. Variance: $\frac{(2-5{)}^2+(4-5{)}^2+(6-5{)}^2+(8-5{)}^2}{4}=\frac{9+1+1+9}{4}=\frac{20}{4}=5$. Std dev: $\sigma =\sqrt{5}\approx 2.236$. Z-scores:

• $2\to \frac{2-5}{2.236}\approx -1.342$
• $4\to \frac{4-5}{2.236}\approx -0.447$
• $6\to \frac{6-5}{2.236}\approx 0.447$
• $8\to \frac{8-5}{2.236}\approx 1.342$

Notice the symmetry: the data is symmetric about $\mu =5$, so the z-scores mirror around $0$.

$X_{\min }=1$, $X_{\max }=100$, range $=99$.

• $1\to \frac{0}{99}=0$
• $2\to \frac{1}{99}\approx 0.0101$
• $3\to \frac{2}{99}\approx 0.0202$
• $4\to \frac{3}{99}\approx 0.0303$
• $100\to \frac{99}{99}=1$

The first four values occupy the interval $[0,0.0303]$ — barely $3\%$ of the whole $[0,1]$ box. Why: the range $99$ is set almost entirely by the outlier $100$. Dividing by a huge range shrinks every ordinary gap (which is only $1$) down to $\approx 0.01$. The outlier "eats" the scale.

Mean: $\mu =\frac{1+2+3+4+100}{5}=\frac{110}{5}=22$. Variance: $\frac{(1-22{)}^2+(2-22{)}^2+(3-22{)}^2+(4-22{)}^2+(100-22{)}^2}{5}$ $=\frac{441+400+361+324+6084}{5}=\frac{7610}{5}=1522$. Std dev: $\sigma =\sqrt{1522}\approx 39.01$. Z-scores:

• $1\to \frac{1-22}{39.01}\approx -0.538$
• $2\to \frac{2-22}{39.01}\approx -0.513$
• $3\to \frac{3-22}{39.01}\approx -0.487$
• $4\to \frac{4-22}{39.01}\approx -0.461$
• $100\to \frac{100-22}{39.01}\approx 2.000$

The first four span $[-0.538,-0.461]$ — a width of $0.077$, versus normalization's $0.0303$. More importantly the outlier lands at $\approx +2.0$ instead of hogging the top of the scale, and it is clearly labelled as extreme rather than compressing the others. This is the concrete reason outlier-heavy data prefers standardization.

Using the training min and max (never refit on test data): $\frac{70-10}{50-10}=\frac{60}{40}=\mathrm{1.5.}$ The output is $1.5$, outside the promised $[0,1]$ box. Nothing crashes — but any downstream component that assumes bounded inputs (e.g. a sigmoid expecting $[0,1]$, or an image pipeline) is now fed an out-of-range value. This is normalization's fragility to unseen extremes, and exactly why standardization (which would just give a large z-score, no bound to break) is safer for unbounded features.

Choose standardization. Reasons:

1. Outliers present — occasional huge dollar values would blow up the range and compress the bulk under Min–Max (Exercise 3.1's failure), whereas z-scores keep the bulk separated (Exercise 3.2).
2. Unbounded / unknown future max — dollars have no natural ceiling, so a fixed $[0,1]$ box will be broken by future data (Exercise 4.1).
3. Regularization fairness — L2 penalizes coefficients by size. Standardization gives every feature spread $1$, so the penalty treats them comparably; leaving raw dollar magnitudes would let the range, not the signal, decide which coefficients get shrunk.

Risk of the alternative (normalization): a single extreme value sets $X_{\max }$, squashing all normal transactions into a razor-thin sliver near $0$, destroying the model's ability to distinguish them.

Let the new data be $X^{\prime }=cX+k$ with $c>0$. Use the two facts from the definition box above. New mean: ${\mu }^{\prime }=\mathbb{E}[cX+k]=c\mathbb{E}[X]+k=c\mu +k$. New std: ${\sigma }^{\prime }=\sqrt{\text{Var}(cX+k)}=\sqrt{c^2\text{Var}(X)}=|c|\sigma =c\sigma$ (since $c>0$). New z-score: $X_{\text{std}}^{\prime }=\frac{X^{\prime }-{\mu }^{\prime }}{{\sigma }^{\prime }}=\frac{(cX+k)-(c\mu +k)}{c\sigma }=\frac{c(X-\mu )}{c\sigma }=\frac{X-\mu }{\sigma }=X_{\text{std}}.$ The $k$ cancels (both $X^{\prime }$ and ${\mu }^{\prime }$ carried it) and the $c$ cancels (numerator and denominator both scaled by it). Conclusion: converting dollars→cents or Celsius→Kelvin (a linear reshuffle) leaves standardized values identical. That "dimensionless" property is what makes z-scores comparable across features of different units.

Write $r=X_{\max }-X_{\min }$ (the range). The map is $f(X)=\frac{X-X_{\min }}{r}$, a linear function with positive slope $\frac{1}{r}$ (when $r>0$). Extremes: $f(X_{\min })=\frac{0}{r}=0$ and $f(X_{\max })=\frac{r}{r}=1$. ✓ Order preserved: slope $\frac{1}{r}>0$, so $X_a<X_b\Rightarrow f(X_a)<f(X_b)$. ✓ Relative spacing preserved: for any three points the ratio of gaps is $\frac{f(X_b)-f(X_a)}{f(X_c)-f(X_a)}=\frac{(X_b-X_a)/r}{(X_c-X_a)/r}=\frac{X_b-X_a}{X_c-X_a},$ identical to the raw ratio — dividing by the same $r$ cancels. ✓ Undefined condition: if $r=X_{\max }-X_{\min }=0$, the denominator is $0$. This happens exactly when every value is identical (a constant feature). Interpretation: a constant feature carries no information to scale — the correct fix is to drop it (or map it to a constant like $0$), not to divide by zero.

• L1–L2 correct ::: you know the mechanics of both formulas.
• L3 correct ::: you understand why outliers favour standardization.
• L4 correct ::: you can pick a scaler from data properties, not slogans.
• L5 correct ::: you can prove the invariance and degeneracy properties from scratch.