2.1.4 · D5Data Preprocessing & Feature Engineering
Question bank — Feature scaling - normalization vs standardization
Parent: Feature scaling — normalization vs standardization.
Before you start, recall the two tools from the parent note in plain words:
Recall What does each transform actually
do to a number? Normalization () reports where a value sits inside the observed range, as a fraction from 0 (the smallest seen) to 1 (the largest seen). Standardization () reports how many standard deviations a value is from the average — a signed distance, unbounded.
True or false — justify
Every claim below is a full statement — decide true or false, then say why in one line.
Scaling changes which model predictions you get for k-NN and SVM.
True — these depend on distances, and scaling changes the relative weight of each feature, so the neighbours/margins change.
Scaling changes the predictions of an ordinary (unregularized) linear regression.
False — a linear model can absorb any per-feature rescaling into its coefficients, so fitted predictions are identical (only the coefficient values differ).
After standardization every feature has exactly mean 0 and standard deviation 1.
True by construction — subtracting forces mean 0 and dividing by forces the spread to 1; that is the whole point of the formula.
After min-max normalization every feature has mean 0.5.
False — the mean lands wherever the data's mean sits within its range; only the min (0) and max (1) are pinned, not the average.
Normalization guarantees outputs strictly inside for any future data point.
False — a new point below the training min or above the training max maps outside (negative or ), because min/max were frozen from training.
Standardization makes a skewed distribution symmetric / Gaussian.
False — it only shifts and rescales; the shape (skew, tails, bimodality) is untouched, since a linear map cannot bend a distribution.
Both transforms preserve the ranking (order) of the original values.
True — both are increasing linear maps with a positive slope, so if then their scaled versions keep the same order.
Standardization removes outliers from the data.
False — it only relabels them as large z-scores (e.g. ); the outlier is still present and still extreme, just expressed in std-dev units.
Feature scaling is needed for decision trees and random forests.
False — trees split on thresholds one feature at a time, so any monotonic rescaling gives identical splits; scaling is a no-op for them.
Min-max scaling to and standardization give the same result when the data is already symmetric.
False — they answer different questions (position-in-range vs distance-from-mean), so their outputs differ unless the data has a very specific spread; symmetry alone does not equate them.
Spot the error
Each line describes something a student did. Say what is wrong and what it causes.
"I computed on the full dataset, then split into train and test."
Leakage — test statistics bled into the scaler; fit the scaler on train only, then apply those frozen numbers to the test set.
"On the test set I recomputed a fresh min and max, then normalized."
Wrong — the test set must reuse the training min/max; new statistics make train and test live on inconsistent scales, breaking the model.
"My data was [1, 2, 3, 4, 100], so I normalized to spread it out nicely."
The outlier 100 defines the range, crushing 1–4 into — nearly indistinguishable; standardization (or outlier handling first) is the right move here (see 2.1.05-Handling-outliers-detection-and-treatment).
"I standardized, and now some values are negative, so I clamped them to 0."
Destroys the transform — negatives are meaningful (below-average points); clamping erases half the information the z-score encodes.
"I one-hot encoded a category into 0/1 columns and then standardized them too."
Usually a mistake — 0/1 indicators are already on a common bounded scale; standardizing them distorts their interpretability for little benefit.
"I scaled the target variable the same way I scaled the features for my regression."
Not an error per se, but you must inverse-transform predictions back to original units, or your reported numbers are meaningless z-scores.
"To make gradient descent fast I normalized only the feature with the biggest range."
Half-fix — the elongated error surface comes from features having different scales; you must scale all features so the surface becomes round (see 3.2.03-Gradient-descent-optimization).
Why questions
Answer with the mechanism, not a restatement.
Why does unscaled data make gradient descent zigzag?
Different feature ranges stretch the loss surface into a long thin valley; the gradient points across the valley more than along it, so the optimizer bounces side to side instead of heading to the minimum.
Why is standardization preferred when outliers are present?
and describe the bulk while an outlier just becomes a large z-score; min/max instead let a single extreme define the whole denominator and compress everyone else.
Why does normalization use but standardization uses as the divisor?
Both are "spread" measures, but range is set by only the two extremes (fragile), while averages every point's deviation (robust to which single value is largest).
Why does scaling matter for L1/L2 regularization but not for a plain linear fit?
The penalty compares coefficient magnitudes; if features live on different scales the coefficients do too, so the penalty punishes the small-scale feature unfairly (see 5.3.01-Regularization-L1-L2-elastic-net).
Why does PCA require scaling first?
PCA finds directions of maximum variance; an unscaled large-range feature has huge variance purely from its units and hijacks the top component regardless of true importance (see 6.2.01-Principal-Component-Analysis-PCA).
Why do distance-based algorithms care so much about scale?
Euclidean distance sums squared per-feature gaps, so a feature measured in large units dominates the sum and effectively decides the neighbours by itself (see 4.1.02-Distance-metrics-euclidean-manhattan-cosine).
Why is a standardized value called "dimensionless"?
Both numerator () and denominator () carry the same units, which cancel, leaving a pure number — "how many std devs", independent of dollars, °C, etc.
Why does batch normalization inside a neural net echo standardization?
It standardizes each layer's activations (subtract batch mean, divide by batch std) so the next layer sees stable, centered inputs — the same "center and rescale" idea applied mid-network (see 7.1.03-Batch-normalization-in-neural-networks).
Edge cases
The degenerate inputs everyone forgets.
What happens to normalization if every value in a feature is identical (constant column)?
, so you divide by zero; the feature carries no information and should be dropped, not scaled.
What happens to standardization on a constant column?
, again division by zero; same conclusion — a zero-variance feature is useless and must be removed before scaling.
If a feature is already in (like a probability), does normalization change it?
Only if its observed min/max are exactly 0 and 1; otherwise it re-stretches the observed sub-range to fill , which may or may not be what you want.
A new production data point equals the training maximum exactly. What is its normalized value?
Exactly 1 — it sits at the top of the frozen training range; anything larger would exceed 1.
Standardizing binary 0/1 data — what are the two resulting z-scores?
Two symmetric values around 0 whose exact size depends on the proportion of 1s; the two classes stay perfectly separable but lose their clean 0/1 meaning.
If the data is perfectly symmetric with mean at the range's midpoint, where does that midpoint land after normalization?
At 0.5 — the midpoint is exactly halfway between min and max, and normalization maps that fraction to 0.5.
For a single data point (n = 1), can you standardize it?
No — (no spread with one point), so the z-score is undefined; scaling needs a distribution, not a lone value.