Min-max scaling and z-score normalization

Overview

Feature scaling transforms numerical features to a common scale without distorting differences in ranges. Two fundamental methods are min-max scaling (range-based) and z-score normalization (distribution-based). Both solve the same problem: preventing features with larger magnitudes from dominating distance-based algorithms and gradient descent.

Why Feature Scaling Matters

When you MUST scale:

• Gradient descent optimization (neural networks, linear/logistic regression)
• Distance-based algorithms (k-NN, k-means, SVM with RBF kernel)
• PCA and dimensionality reduction
• Regularization (L1/L2) — unscaled features get unfairly penalized

When you DON'T scale:

• Tree-based models (decision trees, random forests, XGBoost) — they split on relative thresholds
• Naive Bayes — works with probabilities, not distances

Min-max Scaling (Normalization)

$x_{\text{scaled}} = \frac{x - x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$

Derivation from first principles:

We want a linear transformation $x_{\text{scaled}} = ax + b$ that maps:

• $x_{\text{min}} \rightarrow 0$
• $x_{\text{max}} \rightarrow 1$

From the first condition: $0 = a \cdot x_{\text{min}} + b \implies b = -a \cdot x_{\text{min}}$

From the second condition: $1 = a \cdot x_{\text{max}} + b = a \cdot x_{\text{max}} - a \cdot x_{\text{min}}$

Solving for $a$: $1 = a(x_{\text{max}} - x_{\text{min}}) \implies a = \frac{1}{x_{\text{max}} - x_{\text{min}}}$

Substituting back: $b = -\frac{x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$

Therefore: $x_{\text{scaled}} = \frac{1}{x_{\text{max}} - x_{\text{min}}} \cdot x - \frac{x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}} = \frac{x - x_{\text{min}}}{x_{\text{max}} - x_{\text{min}}}$

WHY this formula? The numerator $(x - x_{\text{min}})$ shifts the range to start at 0. The denominator $(x_{\text{max}} - x_{\text{min}})$ is the original range, so dividing by it compresses everything into [0, 1].

Derivation: Map to [0, 1] first, then stretch by $(b-a)$ and shift by $a$.

Step 1: Find range

• $x_{\text{min}} = 20$, $x_{\text{max}} = 40$
• Range $40 - 20 = 20$

Step 2: Apply formula to each value

• $x = 20$: $\frac{20-20}{20} = 0.00$
• $x = 25$: $\frac{25-20}{20} = 0.25$ — Why this step? 25 is 1/4 of the way between 20 and 40
• $x = 30$: $\frac{30-20}{20} = 0.50$ — Why? Exactly midpoint
• $x = 35$: $\frac{35-20}{20} = 0.75$
• $x = 40$: $\frac{40-20}{20} = 1.00$

Result: [0, 0.25, 0.5, 0.75, 1.0]

Step 1: Apply generalized formula with $a=-1$, $b=1$ $x_{\text{scaled}} = -1 + \frac{(x - 100)(1 - (-1))}{300 - 100} = -1 + \frac{2(x - 100)}{200}$

Step 2: Calculate

• $x = 100$: $-1 + \frac{2(0)}{200} = -1.0$
• $x = 200$: $-1 + \frac{2(100)}{200} = 0.0$ — Why? Middle value maps to middle of range
• $x = 300$: $-1 + \frac{2(200)}{200} = 1.0$

Result: [-1.0, 0.0, 1.0]

Properties of Min-max Scaling

Advantages:

• Preserves original distribution shape (just rescaled)
• Bounded output — useful for algorithms expecting specific ranges (e.g., neural networks with sigmoid)
• All features contribute equally to distance calculations
• Interpretable: 0.5 means "halfway between min and max"

Disadvantages:

• Sensitive to outliers — a single extreme value drastically changes $x_{\text{max}}$ or $x_{\text{min}}$
• New test data outside training range produces values outside [0, 1]
• Not suitable when new data might have different range

Why it feels right: Each dataset gets scaled to [0, 1], seems consistent.

Why it's wrong: The same original value maps to different scaled values in train vs test. A value of 50 might become 0.5 in training but 0.7 in testing if the test set has a smaller range. The model learned decision boundaries based on training scale, now they're misaligned.

Fix: Fit scaler on training data only, apply same transformation to test:

scaler.fit(train)  # Learn min and max from training
train_scaled = scaler.transform(train)
test_scaled = scaler.transform(test)  # Use training min/max

Z-score Normalization (Standardization)

$z = \frac{x - \mu}{\sigma}$

Derivation from first principles:

We want transformation $z = ax + b$ such that:

• $E[z] = 0$ (mean is zero)
• $\text{Var}(z) = 1$ (standard deviation is 1)

Step 1: Mean condition $E[z] = E[ax + b] = aE[x] + b = a\mu + b = 0$ $\implies b = -a\mu$

Step 2: Variance condition $\text{Var}(z) = \text{Var}(ax + b) = a^2 \text{Var}(x) = a^2 \sigma^2 = 1$ $\implies a = \pm \frac{1}{\sigma}$

We choose $a = \frac{1}{\sigma}$ (positive to preserve order).

Step 3: Combine $z = \frac{1}{\sigma} \cdot x - \frac{\mu}{\sigma} = \frac{x - \mu}{\sigma}$

WHY this formula? $(x - \mu)$ centers the distribution at 0. Dividing by $\sigma$ makes the spread exactly 1 standard deviation.

This converts absolute values to relative positions in the distribution.

Step 1: Calculate statistics

• Mean: $\mu = \frac{60+70+80+90+100}{5} = 80$
• Variance: $\sigma^2 = \frac{(60-80)^2 + (70-80)^2 + (80-80)^2 + (90-80)^2 + (100-80)^2}{5}$ $= \frac{400 + 100 + 0 + 100 + 400}{5} = 200$
• Standard deviation: $\sigma = \sqrt{200} \approx 14.14$

Step 2: Apply formula

• $x = 60$: $z = \frac{60-80}{14.14} = -1.41$ — Why? 60 is 1.41 std devs below average
• $x = 70$: $z = \frac{70-80}{14.14} = -0.71$
• $x = 80$: $z = \frac{80-80}{14.14} = 0.00$ — Why? At the mean
• $x = 90$: $z = \frac{90-80}{14.14} = 0.71$
• $x = 100$: $z = \frac{100-80}{14.14} = 1.41$

Result: [-1.41, -0.71, 0, 0.71, 1.41]

Verification: Mean of z-scores = 0, std dev = 1 ✓

Min-max scaling:

• $x_{\text{min}} = 30000$, $x_{\text{max}} = 200000$
• Range $= 200000 - 30000 = 170000$
• $x = 30000$: $\frac{30000-30000}{170000} = 0.000$
• $x = 45000$: $\frac{45000-30000}{170000} = \frac{15000}{170000} = 0.088$ — Why? All normal salaries squished near 0
• $x = 200000$: $\frac{200000-30000}{170000} = 1.000$ (the outlier)

Notice: The four normal salaries all fall in [0, 0.088] — completely compressed by the outlier.

Z-score normalization:

• Mean: $\mu = \frac{30000+35000+40000+45000+200000}{5} = \frac{350000}{5} = 70000$
• Variance: $\sigma^2 = \frac{(30000-70000)^2 + (35000-70000)^2 + (40000-70000)^2 + (45000-70000)^2 + (200000-70000)^2}{5}$ $= \frac{1.6\times10^9 + 1.225\times10^9 + 0.9\times10^9 + 0.625\times10^9 + 16.9\times10^9}{5} = \frac{21.25\times10^9}{5} = 4.25\times10^9$
• Standard deviation: $\sigma = \sqrt{4.25\times10^9} \approx 65192$
• $z_{45000} = \frac{45000-70000}{65192} = \frac{-25000}{65192} \approx -0.38$
• $z_{200000} = \frac{200000-70000}{65192} = \frac{130000}{65192} \approx 1.99$

Why z-score is better here? The outlier gets a large z-score (≈2.0) but the normal salaries still spread across roughly $[-0.61, -0.38]$ rather than being crushed into a tiny sliver near 0. However: z-score is still affected by outliers because they inflate both $\mu$ and $\sigma$ — for extreme cases, use robust scaling (median and IQR).

Properties of Z-score Normalization

Advantages:

• Less sensitive to outliers than min-max (but not immune)
• No bounded range — handles new data with any values
• Creates comparable scales when features have different units
• Works well with normally distributed data
• Required for algorithms assuming standardized inputs (PCA, LDA)

Disadvantages:

• Output not bounded (unlike [0,1])
• Harder to interpret (what does z=-0.73 mean intuitively?)
• Assumes somewhat bell-shaped distribution
• Breaks down with extreme outliers

Why it feels right: Z-score is "standardization," sounds universally applicable.

Why it's problematic: Z-score assumes distribution shape is roughly preserved. With skew, most z-scores cluster one side, and the "1 standard deviation" interpretation becomes misleading. The mean is pulled by outliers.

Fix: Use robust scaling with median and IQR: $x_{\text{robust}} = \frac{x - \text{median}(x)}{\text{IQR}(x)}$ Or apply log transformation first to reduce skew, then z-score.

When to Use Which Method

Implementation Considerations

Fitting and Transforming

ALWAYS fit on training data only:

from sklearn.preprocessing import MinMaxScaler, StandardScaler
 
# Min-max
scaler_minmax = MinMaxScaler()
scaler_minmax.fit(X_train)  # Learn min and max
X_train_scaled = scaler_minmax.transform(X_train)
X_test_scaled = scaler_minmax.transform(X_test)
 
# Z-score
scaler_std = StandardScaler()
scaler_std.fit(X_train)  # Learn mean and std
X_train_scaled = scaler_std.transform(X_train)
X_test_scaled = scaler_std.transform(X_test)

Why fit only on training? Prevents data leakage. The test set represents "unseen future data" — using its statistics leaks information about test distribution into your model.

Feature-wise vs Global Scaling

Feature-wise (standard): Each column scaled independently

• Use when features have different units/meanings
• Preserves feature relationships within their own scale

Global (rare): All features scaled using same min/max or mean/std

• Only if features represent same quantity (e.g., pixel intensities)

Inverse Transformations

Both methods are reversible:

Min-max inverse: $x = x_{\text{scaled}} \cdot (x_{\text{max}} - x_{\text{min}}) + x_{\text{min}}$

Z-score inverse: $x = z \cdot \sigma + \mu$

Use case: Convert predictions back to original scale for interpretation.

Min-max scaling is like saying "What percentage of your total did you save each week?" You both convert your money to percentages from 0% to 100%. Now you can compare!

Z-score is like asking "How far above or below your average week was each week?" If your average is ₹100/week, and one week you saved ₹150, you're +50 above average. If your friend's average is $2/week and they saved$3 one week, they're +1 above their average. Z-score tells you "how many 'typical amounts' above or below average you are" so you can compare even with different currencies.

Both methods turn "apples and oranges" into the same unit so you can fairly compare them!

Connections

• 2.1.01-Data-cleaning-techniques — scaling applied after handling missing values and outliers
• 2.1.04-One-hot-and-label-encoding — categorical encoding happens before scaling
• 2.2.01-Feature-selection-methods — scaling required before correlation-based selection
• 3.1.02-Gradient-descent-optimization — scaling dramatically speeds convergence
• 4.1.05-Principal-component-analysis — PCA requires centered/standardized data
• 5.3.02-Batch-normalization — neural network layer-wise standardization during training
• 2.1.06-Robust-scaling-and-power-transforms — advanced scaling for outliers and skewed distributions

#flashcards/ai-ml

What is feature scaling and why is it necessary? :: Feature scaling transforms numerical features to a common scale. It's necessary because features with larger magnitudes dominate distance calculations and gradient descent, causing algorithms to overweight them unfairly.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)