Feature scaling - normalization vs standardization

Overview

Feature scaling transforms numerical features to a common scale without distorting differences in ranges. Two primary methods exist: normalization (Min-Max scaling) and standardization (Z-score scaling). Choosing correctly impacts gradient descent convergence, distance-based algorithms, and regularization effectiveness.

Why Feature Scaling Matters

Three core problems unscaled features cause:

1. Gradient descent inefficiency: Features with large ranges create elongated error surfaces. The optimizer zigzags instead of taking direct paths to the minimum. With scaling, the error surface becomes spherical → faster convergence.

2. Distance metric distortion: Euclidean distance $d = \sqrt{(x_1 - y_1)^2 + (x_2 - y_2)^2}$ weighs features by their magnitude. A 1-unit difference in price ($1) vs 1-unit difference in bedrooms (1 room) shouldn't have 100,000× different impact.

3. Regularization bias: L1/L2 penalties penalize large coefficients. If feature $X_1$ has range [0, 1000] and $X_2$ has range [0, 1], the model learns small coefficient for $X_1$ and large coefficient for $X_2$ to compensate → regularization incorrectly shrinks $X_2$'s coefficient more.

Normalization (Min-Max Scaling)

$X_{\text{norm}} = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$

Derivation from first principles:

We want to map $[X_{\min}, X_{\max}] \rightarrow [0, 1]$ linearly. A linear transformation has form $X_{\text{norm}} = aX + b$.

Step 1: When $X = X_{\min}$, we need $X_{\text{norm}} = 0$: $0 = aX_{\min} + b \quad \Rightarrow \quad b = -aX_{\min}$

Step 2: When $X = X_{\max}$, we need $X_{\text{norm}} = 1$: $1 = aX_{\max} + b = aX_{\max} - aX_{\min} = a(X_{\max} - X_{\min})$ $a = \frac{1}{X_{\max} - X_{\min}}$

Step 3: Substitute back: $X_{\text{norm}} = \frac{1}{X_{\max} - X_{\min}} \cdot X - \frac{X_{\min}}{X_{\max} - X_{\min}} = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$

Why this formula works: The numerator shifts the range to start at 0, the denominator scales it to width 1.

Derivation: We want $[X_{\min}, X_{\max}] \rightarrow [a, b]$. First map to [0, 1], then scale by $(b-a)$ and shift by $a$.

Dataset: House prices = [200K, 350K, 500K, 800K]

Step 1: Find min and max

• $X_{\min} = 200{,}000$
• $X_{\max} = 800{,}000$
• Range = $600{,}000$

Step 2: Apply formula

• $200K \rightarrow \frac{200-200}{600} = 0$
• $350K \rightarrow \frac{350-200}{600} = \frac{150}{600} = 0.25$
• $500K \rightarrow \frac{500-200}{600} = \frac{300}{600} = 0.50$
• $800K \rightarrow \frac{800-200}{600} = 1.0$

Why this step? Each value's position in the original range (0%, 25%, 50%, 100%) is preserved in [0, 1].

Example 2: Test scores [40, 60, 75, 95] to [0, 10] scale

• $X_{\min} = 40$, $X_{\max} = 95$, range = 55
• $40 \rightarrow 0 + 10 \cdot \frac{40-40}{55} = 0$
• $60 \rightarrow 0 + 10 \cdot \frac{60-40}{55} = 10 \cdot \frac{20}{55} = 3.64$
• $75 \rightarrow 0 + 10 \cdot \frac{75-40}{55} = 10 \cdot \frac{35}{55} = 6.36$
• $95 \rightarrow 0 + 10 \cdot \frac{95-40}{55} = 10$

When to use normalization:

• Features have bounded ranges (e.g., pixel values 0-255, probabilities 0-1)
• No outliers present (one extreme value will compress all others)
• Algorithm needs specific bounds (e.g., sigmoid activation inputs in neural networks)
• Examples: image processing, neural networks with bounded activations, algorithms requiring specific ranges

Standardization (Z-score Scaling)

$X_{\text{std}} = \frac{X - \mu}{\sigma}$

where $\mu$ is the mean and $\sigma$ is the standard deviation.

Derivation from first principles:

We want a transformation $X_{\text{std}} = aX + b$ such that the new distribution has $\mathbb{E}[X_{\text{std}}] = 0$ and $\text{Var}(X_{\text{std}}) = 1$.

Step 1: Mean requirement: $\mathbb{E}[X_{\text{std}}] = \mathbb{E}[aX + b] = a\mathbb{E}[X] + b = a\mu + b = 0$ $b = -a\mu$

Step 2: Variance requirement: $\text{Var}(X_{\text{std}}) = \text{Var}(aX + b) = a^2 \text{Var}(X) = a^2\sigma^2 = 1$ $a = \pm\frac{1}{\sigma}$

We choose $a = +\frac{1}{\sigma}$ (positive) to preserve order.

Step 3: Combine: $X_{\text{std}} = \frac{1}{\sigma} \cdot X - \frac{\mu}{\sigma} = \frac{X - \mu}{\sigma}$

Why this formula works: Subtracting $\mu$ centers the data at 0. Dividing by $\sigma$ scales it so that 68% of data lies within [-1, 1] (assuming normality).

Dataset: Incomes = [30K, 45K, 50K, 55K, 120K]

Step 1: Calculate mean $\mu = \frac{30 + 45 + 50 + 55 + 120}{5} = \frac{300}{5} = 60K$

Step 2: Calculate standard deviation $\sigma = \sqrt{\frac{(30-60)^2 + (45-60)^2 + (50-60)^2 + (55-60)^2 + (120-60)^2}{5}}$ $= \sqrt{\frac{900 + 225 + 100 + 25 + 3600}{5}} = \sqrt{\frac{4850}{5}} = \sqrt{970} \approx 31.14K$

Step 3: Standardize each value

• $30K \rightarrow \frac{30-60}{31.14} = -0.96$ (below average)
• $45K \rightarrow \frac{45-60}{31.14} = -0.48$
• $50K \rightarrow \frac{50-60}{31.14} = -0.32$
• $55K \rightarrow \frac{55-60}{31.14} = -0.16$
• $120K \rightarrow \frac{120-60}{31.14} = 1.93$ (outlier: 1.93 std devs above mean)

Why this step? The outlier (120K) becomes 1.93 instead of dominating the scale. All values now represent "distance from typical" in standard deviation units.

Example 4: Temperature data with negative values

Dataset: Temperatures (°C) = [-10, 5, 15, 20]

• $\mu = \frac{-10+5+15+20}{5} = 6°C$
• $\sigma = \sqrt{\frac{(-10-6)^2 + (0-6)^2 + (5-6)^2 + (15-6)^2 + (20-6)^2}{5}}$
• $\sigma = \sqrt{\frac{256 + 36 + 1 + 81 + 196}{5}} = \sqrt{114} \approx 10.68°C$

Standardized:

• $-10°C \rightarrow \frac{-10-6}{10.68} = -1.50$
• $0°C \rightarrow \frac{0-6}{10.68} = -0.56$
• $5°C \rightarrow \frac{5-6}{10.68} = -0.09$
• $15°C \rightarrow \frac{15-6}{10.68} = 0.84$
• $20°C \rightarrow \frac{20-6}{10.68} = 1.31$

When to use standardization:

• Data has outliers (they become extreme z-scores but don't compress the rest)
• Features follow Gaussian distributions (z-scores have probabilistic meaning)
• Algorithm assumes normally distributed data (e.g., linear regression, LDA, Gaussian Naive Bayes)
• Unknown bounds or unbounded ranges (e.g., income, temperature)
• Examples: linear models with regularization, PCA, k-means clustering

Head-to-Head Comparison

| Aspect | Normalization | Standardization | |--------|-----------------| | Formula | $\frac{X - X_{\min}}{X_{\max} - X_{\min}}$ | $\frac{X - \mu}{\sigma}$ | | Output range | Bounded [0, 1] or [a, b] | Unbounded (typically -3 to +3) | | Outlier sensitivity | High – one outlier compresses all values | Low – outlier becomes extreme z-score | | Preserves distribution shape | Yes (relative distances unchanged) | Yes (shape preserved) | | Best for | Bounded features, neural nets, no outliers | Gaussian features, outliers present, linear models | | Statistical meaning | Position in range (percentile-like) | Distance from mean in std dev units | | New data handling | New min/max breaks bounds | New extreme values just get large z-scores |

Original data: [1, 2, 3, 4, 100]

Normalization:

• Range: $100 - 1 = 99$
• Scaled: $[0, 0.01, 0.02, 0.03, 1.0]$
• Problem: First4 values compressed into [0, 0.03] – almost indistinguishable!

Standardization:

• $\mu = 22$, $\sigma \approx 43.1$
• Scaled: $[-0.49, -0.46, -0.44, -0.42, 1.81]$
• Benefit: First 4 values remain distinguishable, outlier clearly labeled as extreme.

Common Mistakes

Why it feels right: "I need to scale everything the same way for consistency."

Why it's wrong: The test set represents future unseen data. If you use test statistics (min, max, mean, std) during training, you've leaked information about the future into your model. This causes overfitting and overoptimistic evaluation.

Correct approach:

1. Split data into train/test
2. Fit scaler on training data only (calculate $\mu$, $\sigma$, $X_{\min}$, $X_{\max from training set)
3. Transform training data
4. Transform test data using training statistics

Example:

# WRONG
scaler.fit(X_entire_dataset)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
 
# CORRECT
scaler.fit(X_train)  # Learn from training only
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)  # Apply training stats

Why it feels right: "Bounded range [0, 1] is cleaner and easier to interpret."

Why it's wrong: One outlier sets $X_{\max}$, compressing all normal values into a tiny range. Model can't distinguish between them.

Example: Heights = [150cm, 160cm, 165cm, 170cm, 250cm] (250cm is clearly an error)

• Normalized: [0, 0.10, 0.15, 0.20, 1.0]
• First 4 heights span only0.2 of the scale!

Fix: Use standardization or remove outliers first.

Why it feels right: "Linear regression doesn't require scaling."

Why it's wrong: Without regularization, you're right – the model learns appropriate coefficients. But with L1/L2 regularization, the penalty $\lambda \sum \beta_i^2$ treats all coefficients equally. A feature with range [0, 1000] needs a small coefficient (0.01) while a feature with range [0, 1] needs a large coefficient (10). Regularization penalizes the large coefficient more, even though both features might be equally important.

Example: Predicting house price from area (sq ft, range500-5000) and bedrooms (range 1-5).

• Without scaling: $\text{price} = 0.1 \cdot \text{area} + 5000 \cdot \text{bedrooms}$
• L2 penalty focuses on bedroom coefficient (5000² >> 0.1²)
• With scaling: Both coefficients have similar magnitude → fair regularization

Why it feels right: "If I scale features, I should scale the target too for consistency."

Why it's wrong: Scaling the target doesn't help gradient descent or distance metrics (they only depend on feature space). It does make interpretation harder – you must inverse-transform predictions back to original units.

When TO scale target:

• Neural networks with bounded output activations (sigmoid → normalize target to [0, 1])
• Multi-output regression where outputs have different scales

When NOT to scale target:

• Linear regression, decision trees, random forests (no benefit, only complexity)

Algorithm-Specific Requirements

| Algorithm | Scaling Required? | Preferred Method | Why | |-----------|----------------|---------------| | Linear/Logistic Regression (no regularization) | No | None | Scale-invariant; coefficients adjust | | Linear/Logistic Regression (with L1/L2) | Yes | Standardization | Fair coefficient penalties | | k-NN, SVM, k-Means | Yes | Standardization or Normalization | Distance-based; large features dominate | | Neural Networks | Yes | Normalization (for bounded activations) or Standardization | Faster convergence, stable gradients | | Decision Trees, Random Forests | None | Split thresholds are scale-invariant | | PCA | Yes | Standardization | Variance-based; large features dominate eigenvectors | | Gradient Boosting (XGBoost, etc.) | None | Tree-based splits are scale-invariant | | Naive Bayes | No | None | Uses probabilities, not distances |

Implementation Considerations

Handling new data after training:

After fitting a scaler on training data, new data must use the same transformation parameters:

# Training phase
scaler = StandardScaler()
scaler.fit(X_train)  # Stores μ, σ
X_train_scaled = scaler.transform(X_train)
 
# Later: new unseen data
X_new_scaled = scaler.transform(X_new)  # Uses stored μ, σ from training

Why? If new data has different min/max or μ/σ, the scaled values won't be comparable to training data → model fails.

What if new data has values outside training range?

• Normalization: Values outside $[X_{\min}, X_{\max}]$ will be outside [0, 1]. This is acceptable – they're genuinely extreme.
• Standardization: Values get z-scores > 3 or < -3. Also acceptable – they're outliers relative to training data.

Inverse transformation:

To convert predictions back to original scale:

$X = X_{\text{norm}} \cdot (X_{\max} - X_{\min}) + X_{\min}$ $X = X_{\text{std}} \cdot \sigma + \mu$

Active Recall Practice

Imagine you're comparing how good students are at math vs art. One student scored 95/100 in math and 8/10 in art. Is she better at math or art?

You can't compare directly because the scales are different!95 vs 8 doesn't tell you anything.

Normalization is like converting both to percentages: 95/100 = 95%, 8/10 = 80%. Now you can compare: she's slightly better at math.

Standardization is different. It asks: "How unusual is this score compared to other students?" If the math test average was 90 (and everyone scored 85-95), her 95 is only slightly above average. But if the art average was 5 (and most scored 3-7), her 8 is way above average. So she's actually more exceptional at art!

Both methods make things comparable, but they measure different things: position in the range vs distance from typical.

"OutLIER → Stand"

• Outliers? Use Standardization (they become extreme z-scores instead of breaking the scale)

Connections

• 2.1.01-Feature-types-and-data-structures – Understanding feature types informs which scaling method fits
• 2.1.05-Handling-outliers-detection-and-treatment – Outlier handling often precedes or replaces normalization
• 3.2.03-Gradient-descent-optimization – Feature scaling accelerates gradient descent convergence
• 4.1.02-Distance-metrics-euclidean-manhattan-cosine – Scaling ensures fair distance computation in k-NN, k-means
• 5.3.01-Regularization-L1-L2-elastic-net – Standardization required for fair coefficient penalties
• 6.2.01-Principal-Component-Analysis-PCA – PCA requires standardization; variance-based decomposition
• 7.1.03-Batch-normalization-in-neural-networks – Batch norm performs internal scaling layer-by-layer

#flashcards/ai-ml

What is the formula for min-max normalization to [0, 1]? :: $X_{\text{norm}} = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$

Concept Map

Hinglish (regional understanding)