Worked examples — Feature scaling - normalization vs standardization

This page is the worked-example lab for the parent topic. We do not re-derive the formulas here — we stress-test them. We throw every kind of input at the two scaling machines and watch what comes out.

The two machines, restated so you never scroll back:

Throughout, $\mu =\frac{1}{n}{\sum }_{i=1}^n{X}_i$ and $\sigma =\sqrt{\frac{1}{n}{\sum }_{i=1}^n(X_i-\mu {)}^2}$. The symbol ${\sum }_{i=1}^n$ is a loop instruction: "let the counter $i$ run from 1 up to $n$ (the number of data points), and add up whatever follows for each value $X_i$." So $\mu$ = add all values, divide by how many; $\sigma$ = average squared distance from the mean, then square-root back to original units. We use the population version (divide by $n$), matching the parent note.

---

The scenario matrix

Every situation these two formulas can face falls into one of these cells. The examples below each carry a tag like (C3) so you can see the whole grid gets covered. The figure below is a quick visual legend of the grid.

Cell	Scenario class	What could go wrong
C1	All positive, well-spread, no outlier	Baseline — both machines behave
C2	Contains negative and zero values	Sign confusion; does $\mu$ shift correctly?
C3	One large outlier present	Normalization compresses everyone else
C4	Degenerate: all values identical	Range $=0$ and $\sigma =0$ → divide by zero
C5	Two-value / tiny dataset	Formula still legal, min=one point, max=other
C6	New (unseen) value beyond fitted min/max	Normalization escapes [0,1] (both above and below); z-score just grows
C7	Two features, different units (real-world)	Distance domination before scaling
C8	Exam twist: reverse-engineer original from scaled	Invert the transform algebraically

We hit all eight below.

---

Worked examples

Forecast: Guess before reading — what should 40 become under each method? (It's the exact center, so...)

1. Find min, max, range. $X_{\min }=20$, $X_{\max }=60$, range $=40$. Why this step? Normalization needs the two endpoints to know where "0" and "1" sit.
2. Normalize each value: $\frac{X-20}{40}$ gives $[0,0.25,0.5,0.75,1]$. Why this step? Subtracting the min shifts the smallest value to 0; dividing by the range squeezes the largest to 1, preserving each value's fraction of the way across the interval.
3. Standardize. Mean $\mu =\frac{20+30+40+50+60}{5}=40$. Deviations squared: $400,100,0,100,400$; average $=200$; $\sigma =\sqrt{200}\approx 14.14$. Why this step? $\sigma$ sets the "ruler" — one unit of z equals one $\sigma$ of real spread.
4. Z-scores: $\frac{X-40}{14.14}$ gives $[-1.414,-0.707,0,0.707,1.414]$. Why this step? Subtracting $\mu$ recenters the data so the mean sits at 0; dividing by $\sigma$ rescales so one ruler-unit of spread becomes one z-unit.

Verify: 40 is the mean, so its z-score must be exactly 0 — check. It's also the midpoint of the range, so its normalized value is exactly 0.5 — check. Both machines agree the middle value is "typical," but they label it differently (0.5 vs 0).

---

Forecast: Will $-10$°C get a z-score bigger or smaller than $+20$°C in size?

1. Mean: $\mu =\frac{-10+0+5+15+20}{5}=\frac{30}{5}=6$. Why this step? The balance point can be positive even with negatives present — subtraction handles the signs.
2. Deviations from 6: $-16,-6,-1,9,14$. Squared: $256,36,1,81,196$. Sum $=570$, average $=114$, $\sigma =\sqrt{114}\approx 10.68$. Why this step? Squaring erases sign, so negatives and positives both count as "distance."
3. Z-scores: $\frac{-10-6}{10.68}=-1.50$; $\frac{0-6}{10.68}=-0.56$; $\frac{5-6}{10.68}=-0.09$; $\frac{15-6}{10.68}=0.84$; $\frac{20-6}{10.68}=1.31$. Why this step? Recentering (subtract $\mu =6$) makes each value's sign tell us "below vs above typical"; dividing by $\sigma$ converts the raw gap into "how many standard deviations away."

Verify: Add the z-scores: $-1.50-0.56-0.09+0.84+1.31\approx 0$ — the mean of standardized data is always 0. And $-10$ (distance 16 from mean) gets a larger-magnitude z than $20$ (distance 14) — matches intuition. Negatives caused zero trouble.

---

Forecast: Under normalization, how far apart will 1, 2, 3, 4 be on the [0,1] line?

1. Normalize. $X_{\min }=1$, $X_{\max }=100$, range $=99$. Values: $\frac{0}{99}=0$, $\frac{1}{99}=0.0101$, $\frac{2}{99}=0.0202$, $\frac{3}{99}=0.0303$, $\frac{99}{99}=1$. Why this step? The single value 100 stretches the range to 99, so subtracting the min and dividing by that huge range crams everything else near 0.
2. Standardize. $\mu =\frac{1+2+3+4+100}{5}=22$. Deviations: $-21,-20,-19,-18,78$; squared $441,400,361,324,6084$; sum $7610$; average $1522$; $\sigma =\sqrt{1522}\approx 39.01$. Why this step? $\sigma$ is dragged up by the outlier, but not only by it — the small values keep their own spacing.
3. Z-scores: $\frac{1-22}{39.01}=-0.538$, $\frac{2-22}{39.01}=-0.513$, $\frac{3-22}{39.01}=-0.487$, $\frac{4-22}{39.01}=-0.461$, $\frac{100-22}{39.01}=2.000$. Why this step? Recentering on $\mu =22$ then dividing by $\sigma$ keeps the tiny 1-unit gaps between 1,2,3,4 as tiny-but-visible z-gaps instead of erasing them.

Verify: Normalized gap between 1 and 4 is $0.0303-0=0.0303$ — practically invisible. Standardized gap is $-0.461-(-0.538)=0.077$ — over 2.5× wider, and still readable. In the figure, look at the red points on the top line huddled at the left edge vs the blue points below staying legible. This is the parent note's "normalization fails with outliers" made concrete.

---

Forecast: What is $X_{\max }-X_{\min }$? What is $\sigma$? What does that do to a fraction?

1. Normalization denominator: $X_{\max }-X_{\min }=7-7=0$. Why this step? The range measures spread; identical values have zero spread.
2. Attempt the formula: $\frac{7-7}{0}=\frac{0}{0}$ — undefined. Division by zero. The machine crashes. Why this step? You cannot ask "what fraction of the way across a zero-width interval" — there's no interval.
3. Standardization $\sigma$: all deviations are 0, so $\sigma =\sqrt{0}=0$, and $\frac{7-7}{0}=\frac{0}{0}$ — also undefined. Why this step? With no spread, $\sigma =0$, and the z-score's denominator vanishes the same way the range did.

Verify: Both fail identically. This is why real libraries either drop constant columns or output all-zeros by convention (treating $\frac{0}{0}$ as 0). Rule to remember: a feature with no variation carries no information — scale-then-drop, or drop first. Relates to spotting constant features early (see 2.1.01-Feature-types-and-data-structures).

---

Forecast: With just two points, where must they land in each system?

1. Normalize. $X_{\min }=50$, $X_{\max }=90$, range $40$. $\frac{50-50}{40}=0$, $\frac{90-50}{40}=1$. Why this step? With exactly two distinct values, one is the min and the other is the max — subtracting the min then dividing by the range pins them to the endpoints 0 and 1.
2. Standardize. $\mu =\frac{50+90}{2}=70$. Deviations $-20,+20$; squared $400,400$; average $400$; $\sigma =20$. Why this step? We need the balance point and ruler before computing z-scores; two symmetric points sit exactly $\pm$ one deviation from their midpoint.
3. Z-scores: $\frac{50-70}{20}=-1$, $\frac{90-70}{20}=+1$. Why this step? Recentering on $\mu =70$ makes the two points $\mp 20$ apart; dividing by $\sigma =20$ turns those into exactly $\mp 1$.

Verify: Any two-point dataset always normalizes to exactly $\{0,1\}$ and standardizes to exactly $\{-1,+1\}$, regardless of the actual numbers. Sanity: z-scores sum to $-1+1=0$. ✓

---

Forecast: Normalization promised [0,1]. Can 1000 stay inside? Can 50?

1. Apply the frozen training parameters (never re-fit on test data!). Range $=800-200=600$. Why this step? Fitting on test data leaks information; we reuse the training min/max so test values are judged by the training scale.
2. Normalize 1000 (above max): $\frac{1000-200}{600}=\frac{800}{600}=1.333$. Why this step? The value sits beyond the training max, so the numerator exceeds the denominator → output escapes above 1.
3. Normalize 50 (below min): $\frac{50-200}{600}=\frac{-150}{600}=-0.25$. Why this step? The value sits below the training min, so the numerator goes negative → output escapes below 0. Normalization breaks its bound on both sides.
4. Standardize instead. Train $\mu =\frac{200+500+800}{3}=500$; deviations $-300,0,300$; squared $90000,0,90000$; avg $60000$; $\sigma =\sqrt{60000}\approx 244.9$. New values: $\frac{1000-500}{244.9}\approx 2.04$ and $\frac{50-500}{244.9}\approx -1.84$. Why this step? Z-scores are unbounded, so out-of-range values above or below just become large-but-valid numbers instead of breaking a promise.

Verify: $1.333>1$ and $-0.25<0$ confirm normalization's bound is not guaranteed on future data — on either side. Standardization's $2.04$ and $-1.84$ are unremarkable and interpretable ("about 2 std devs above / below train mean"). This is exactly the parent's "New data handling" comparison-table row.

---

Forecast: Before scaling, will the 2-bedroom difference matter at all next to the $10000 price gap?

1. Raw distance. $d=\sqrt{(310000-300000{)}^2+(5-3{)}^2}=\sqrt{10000^2+2^2}=\sqrt{100000000+4}\approx 10000.0002$. Why this step? Euclidean distance (see 4.1.02-Distance-metrics-euclidean-manhattan-cosine) squares each feature's gap; the price gap of $10000$ utterly swamps the bedroom gap of $2$.
2. Standardize using the training pair as the fitted $\mu ,\sigma$ per feature. Price: $\mu =305000$, $\sigma =5000$ → z's $-1,+1$. Bedrooms: $\mu =4$, $\sigma =1$ → z's $-1,+1$. Why this step? Each feature now speaks the same unit (standard deviations of its own training spread), removing the dollar-vs-room mismatch.
3. Scaled distance. $d=\sqrt{(1-(-1){)}^2+(1-(-1){)}^2}=\sqrt{2^2+2^2}=\sqrt{8}\approx 2.828$. Why this step? Both features now contribute equally — the bedroom difference is finally "heard."

Verify: Raw distance $\approx 10000$ is $\approx 99.99\%$ driven by price alone ($10000/10000.0002$). After scaling, each feature contributes half the squared distance ($4$ out of $8$). In the figure, notice the pre-scaling ellipse is stretched flat along the price axis; after scaling it's a circle. This is precisely why distance-based algorithms require scaling (links to 3.2.03-Gradient-descent-optimization convergence too).

---

Forecast: Standardization is a formula; can you run it backward algebraically?

1. Invert standardization. Start from $z=\frac{X-\mu }{\sigma }$. Multiply both sides by $\sigma$: $z\sigma =X-\mu$. Add $\mu$: $X=\mu +z\sigma$. Why this step? Every operation the machine did (subtract $\mu$, divide by $\sigma$) is undone in reverse order.
2. Plug in: $X=60+1.5\times 8=60+12=72$. Why this step? Substituting the known z, $\mu$, $\sigma$ into the inverted formula rebuilds the original value.
3. Invert normalization. From $X_{\text{norm}}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$, solve: $X=X_{\min }+X_{\text{norm}}(X_{\max }-X_{\min })$. Why this step? Same undo logic: multiply back by the range, add the min.
4. Plug in: $X=10+0.4\times (60-10)=10+0.4\times 50=10+20=30$. Why this step? Substituting the known normalized value and range rebuilds the pre-scaling number.

Verify: Re-standardize 72: $\frac{72-60}{8}=1.5$ ✓. Re-normalize 30: $\frac{30-10}{50}=0.4$ ✓. Both transforms are perfectly invertible because they are linear — a fact used when you must display predictions back in original units.

---

Recall

Recall Can Min-Max normalization ever produce a value outside [0,1], and how?

Yes ::: on unseen data — a test point above the fitted max gives $>1$, and a test point below the fitted min gives $<0$ (Example 6). Standardization has no bound at all.

Recall What happens to both formulas on a constant column?

Both divide by zero ::: range $=0$ for normalization and $\sigma =0$ for standardization → undefined (Example 4).

Recall After standardization, what must the sum of all z-scores equal?

Zero ::: because the new mean is 0 (verified in Examples 2 and 5).

Connections: outlier behavior deepens in 2.1.05-Handling-outliers-detection-and-treatment; why scaling helps penalties in 5.3.01-Regularization-L1-L2-elastic-net; scaling before 6.2.01-Principal-Component-Analysis-PCA; and the network-internal cousin 7.1.03-Batch-normalization-in-neural-networks.