Feature scaling - normalization vs standardization

2.1.4 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Feature scaling numerical features ko ek common scale par transform karta hai, bina ranges ke differences ko distort kiye. Do primary methods hain: normalization (Min-Max scaling) aur standardization (Z-score scaling). Sahi choice karna gradient descent convergence, distance-based algorithms, aur regularization effectiveness par impact karta hai.

Why Feature Scaling Matters

Teen core problems jo unscaled features cause karte hain:

1. Gradient descent inefficiency: Large ranges wale features elongated error surfaces create karte hain. Optimizer minimum tak direct path lene ki jagah zigzag karta hai. Scaling ke saath, error surface spherical ho jaata hai → faster convergence.

2. Distance metric distortion: Euclidean distance $d=\sqrt{(x_1-y_1{)}^2+(x_2-y_2{)}^2}$ features ko unke magnitude ke hisaab se weigh karta hai. Price mein 1-unit difference ($1) vs bedrooms mein 1-unit difference (1 room) ka 100,000× alag impact nahi hona chahiye.

3. Regularization bias: L1/L2 penalties large coefficients ko penalize karti hain. Agar feature $X_1$ ka range [0, 1000] hai aur $X_2$ ka range [0, 1] hai, toh model compensate karne ke liye $X_1$ ke liye small coefficient aur $X_2$ ke liye large coefficient seekhta hai → regularization galat tarike se $X_2$ ke coefficient ko zyada shrink kar deti hai.

Normalization (Min-Max Scaling)

$X_{\text{norm}}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$

First principles se derivation:

Hum chahte hain ki $[X_{\min },X_{\max }]\to [0,1]$ linearly map ho. Ek linear transformation ka form hota hai $X_{\text{norm}}=aX+b$.

Step 1: Jab $X=X_{\min }$ ho, tab hume $X_{\text{norm}}=0$ chahiye: $0=a{X}_{\min }+b\Rightarrow b=-a{X}_{\min }$

Step 2: Jab $X=X_{\max }$ ho, tab hume $X_{\text{norm}}=1$ chahiye: $1=a{X}_{\max }+b=a{X}_{\max }-a{X}_{\min }=a(X_{\max }-X_{\min })$ $a=\frac{1}{X_{\max }-X_{\min }}$

Step 3: Back substitute karo: $X_{\text{norm}}=\frac{1}{X_{\max }-X_{\min }}\cdot X-\frac{X_{\min }}{X_{\max }-X_{\min }}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$

Yeh formula kyun kaam karta hai: Numerator range ko 0 se start karne ke liye shift karta hai, denominator usse width 1 tak scale karta hai.

Derivation: Hum chahte hain $[X_{\min },X_{\max }]\to [a,b]$. Pehle [0, 1] par map karo, phir $(b-a)$ se scale karo aur $a$ se shift karo.

Dataset: House prices = [200K, 350K, 500K, 800K]

Step 1: Min aur max dhundo

• $X_{\min }=200,000$
• $X_{\max }=800,000$
• Range = $600,000$

Step 2: Formula apply karo

• $200K\to \frac{200-200}{600}=0$
• $350K\to \frac{350-200}{600}=\frac{150}{600}=0.25$
• $500K\to \frac{500-200}{600}=\frac{300}{600}=0.50$
• $800K\to \frac{800-200}{600}=1.0$

Yeh step kyun? Original range mein har value ki position (0%, 25%, 50%, 100%) [0, 1] mein preserve hoti hai.

Example 2: Test scores [40, 60, 75, 95] ko [0, 10] scale par

• $X_{\min }=40$, $X_{\max }=95$, range = 55
• $40\to 0+10\cdot \frac{40-40}{55}=0$
• $60\to 0+10\cdot \frac{60-40}{55}=10\cdot \frac{20}{55}=3.64$
• $75\to 0+10\cdot \frac{75-40}{55}=10\cdot \frac{35}{55}=6.36$
• $95\to 0+10\cdot \frac{95-40}{55}=10$

Normalization kab use karein:

• Features ke bounded ranges hain (jaise pixel values 0-255, probabilities 0-1)
• Koi outliers nahi hain (ek extreme value baaki sab ko compress kar degi)
• Algorithm ko specific bounds chahiye (jaise neural networks mein sigmoid activation inputs)
• Examples: image processing, bounded activations wale neural networks, specific ranges maangne wale algorithms

Standardization (Z-score Scaling)

$X_{\text{std}}=\frac{X-\mu }{\sigma }$

jahan $\mu$ mean hai aur $\sigma$ standard deviation hai.

First principles se derivation:

Hum ek transformation $X_{\text{std}}=aX+b$ chahte hain jaise ki nayi distribution mein $\mathbb{E}[X_{\text{std}}]=0$ aur $\text{Var}(X_{\text{std}})=1$ ho.

Step 1: Mean requirement: $\mathbb{E}[X_{\text{std}}]=\mathbb{E}[aX+b]=a\mathbb{E}[X]+b=a\mu +b=0$ $b=-a\mu$

Step 2: Variance requirement: $\text{Var}(X_{\text{std}})=\text{Var}(aX+b)=a^2\text{Var}(X)=a^2{\sigma }^2=1$ $a=\pm \frac{1}{\sigma }$

Hum $a=+\frac{1}{\sigma }$ (positive) choose karte hain taaki order preserve ho.

Step 3: Combine karo: $X_{\text{std}}=\frac{1}{\sigma }\cdot X-\frac{\mu }{\sigma }=\frac{X-\mu }{\sigma }$

Yeh formula kyun kaam karta hai: $\mu$ subtract karne se data 0 par center ho jaata hai. $\sigma$ se divide karne se data scale ho jaata hai taaki 68% data [-1, 1] ke andar aa jaaye (normality assume karte hue).

Dataset: Incomes = [30K, 45K, 50K, 55K, 120K]

Step 1: Mean calculate karo $\mu =\frac{30+45+50+55+120}{5}=\frac{300}{5}=60K$

Step 2: Standard deviation calculate karo $\sigma =\sqrt{\frac{(30-60{)}^2+(45-60{)}^2+(50-60{)}^2+(55-60{)}^2+(120-60{)}^2}{5}}$ $=\sqrt{\frac{900+225+100+25+3600}{5}}=\sqrt{\frac{4850}{5}}=\sqrt{970}\approx 31.14K$

Step 3: Har value ko standardize karo

• $30K\to \frac{30-60}{31.14}=-0.96$ (average se neeche)
• $45K\to \frac{45-60}{31.14}=-0.48$
• $50K\to \frac{50-60}{31.14}=-0.32$
• $55K\to \frac{55-60}{31.14}=-0.16$
• $120K\to \frac{120-60}{31.14}=1.93$ (outlier: mean se 1.93 std devs upar)

Yeh step kyun? Outlier (120K) scale dominate karne ki jagah 1.93 ban jaata hai. Saari values ab "typical se distance" ko standard deviation units mein represent karti hain.

Example 4: Negative values wala temperature data

Dataset: Temperatures (°C) = [-10, 5, 15, 20]

• $\mu =\frac{-10+5+15+20}{5}=6°C$
• $\sigma =\sqrt{\frac{(-10-6{)}^2+(0-6{)}^2+(5-6{)}^2+(15-6{)}^2+(20-6{)}^2}{5}}$
• $\sigma =\sqrt{\frac{256+36+1+81+196}{5}}=\sqrt{114}\approx 10.68°C$

Standardized:

• $-10°C\to \frac{-10-6}{10.68}=-1.50$
• $0°C\to \frac{0-6}{10.68}=-0.56$
• $5°C\to \frac{5-6}{10.68}=-0.09$
• $15°C\to \frac{15-6}{10.68}=0.84$
• $20°C\to \frac{20-6}{10.68}=1.31$

Standardization kab use karein:

• Data mein outliers hain (woh extreme z-scores ban jaate hain lekin baaki ko compress nahi karte)
• Features Gaussian distributions follow karte hain (z-scores ka probabilistic meaning hota hai)
• Algorithm normally distributed data assume karta hai (jaise linear regression, LDA, Gaussian Naive Bayes)
• Unknown bounds ya unbounded ranges hain (jaise income, temperature)
• Examples: regularization wale linear models, PCA, k-means clustering

Head-to-Head Comparison

| Aspect | Normalization | Standardization | |--------|-----------------| | Formula | $\frac{X-X_{\min }}{X_{\max }-X_{\min }}$ | $\frac{X-\mu }{\sigma }$ | | Output range | Bounded [0, 1] ya [a, b] | Unbounded (typically -3 se +3 tak) | | Outlier sensitivity | High – ek outlier saari values ko compress kar deta hai | Low – outlier extreme z-score ban jaata hai | | Distribution shape preserve karta hai | Haan (relative distances unchanged) | Haan (shape preserved) | | Best for | Bounded features, neural nets, koi outliers nahi | Gaussian features, outliers present, linear models | | Statistical meaning | Range mein position (percentile-jaisa) | Mean se distance std dev units mein | | New data handling | Naye min/max bounds tod dete hain | Naye extreme values sirf bade z-scores lete hain |

Original data: [1, 2, 3, 4, 100]

Normalization:

• Range: $100-1=99$
• Scaled: $[0,0.01,0.02,0.03,1.0]$
• Problem: Pehli 4 values [0, 0.03] mein compress ho gayi hain – almost indistinguishable!

Standardization:

• $\mu =22$, $\sigma \approx 43.1$
• Scaled: $[-0.49,-0.46,-0.44,-0.42,1.81]$
• Benefit: Pehli 4 values distinguishable rehti hain, outlier clearly extreme label hota hai.

Common Mistakes

Kyun sahi lagta hai: "Consistency ke liye mujhe sab kuch same tarike se scale karna chahiye."

Kyun galat hai: Test set future unseen data represent karta hai. Agar tum training ke dauran test statistics (min, max, mean, std) use karo, toh tumne future ki information apne model mein leak kar di. Isse overfitting aur overoptimistic evaluation hoti hai.

Sahi approach:

1. Data ko train/test mein split karo
2. Scaler ko sirf training data par fit karo (training set se $\mu$, $\sigma$, $X_{\min }$, $X_{\max }$ calculate karo)
3. Training data transform karo
4. Test data ko training statistics use karke transform karo

Example:

# WRONG
scaler.fit(X_entire_dataset)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
 
# CORRECT
scaler.fit(X_train)  # Learn from training only
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)  # Apply training stats

Kyun sahi lagta hai: "Bounded range [0, 1] cleaner aur interpret karna aasaan hai."

Kyun galat hai: Ek outlier $X_{\max }$ set kar deta hai, saari normal values ko ek tiny range mein compress kar deta hai. Model unhe distinguish nahi kar paata.

Example: Heights = [150cm, 160cm, 165cm, 170cm, 250cm] (250cm clearly ek error hai)

• Normalized: [0, 0.10, 0.15, 0.20, 1.0]
• Pehli 4 heights sirf scale ka 0.2 hi span karti hain!

Fix: Standardization use karo ya pehle outliers remove karo.

Kyun sahi lagta hai: "Linear regression ko scaling ki zarurat nahi hai."

Kyun galat hai: Regularization ke bina, tum sahi ho – model appropriate coefficients seekh leta hai. Lekin L1/L2 regularization ke saath, penalty $\lambda \sum {\beta }_i^2$ saare coefficients ko equally treat karti hai. Range [0, 1000] wale feature ko small coefficient (0.01) chahiye jabki range [0, 1] wale feature ko large coefficient (10) chahiye. Regularization large coefficient ko zyada penalize karti hai, even though dono features equally important ho sakte hain.

Example: Area (sq ft, range 500-5000) aur bedrooms (range 1-5) se house price predict karna.

• Scaling ke bina: $\text{price}=0.1\cdot \text{area}+5000\cdot \text{bedrooms}$
• L2 penalty bedroom coefficient par focus karti hai (5000² >> 0.1²)
• Scaling ke saath: Dono coefficients ka similar magnitude hota hai → fair regularization

Kyun sahi lagta hai: "Agar main features scale kar raha hoon, toh consistency ke liye target bhi scale karna chahiye."

Kyun galat hai: Target ko scale karne se gradient descent ya distance metrics mein koi help nahi hoti (woh sirf feature space par depend karte hain). Isse interpretation mushkil ho jaata hai – tumhe predictions ko original units mein wapas inverse-transform karna padta hai.

Target kab scale karein:

• Bounded output activations wale neural networks (sigmoid → target ko [0, 1] par normalize karo)
• Multi-output regression jahan outputs ke alag-alag scales hain

Target kab na scale karein:

• Linear regression, decision trees, random forests (koi benefit nahi, sirf complexity)

Algorithm-Specific Requirements

| Algorithm | Scaling Required? | Preferred Method | Kyun | |-----------|----------------|---------------| | Linear/Logistic Regression (no regularization) | No | None | Scale-invariant; coefficients adjust ho jaate hain | | Linear/Logistic Regression (with L1/L2) | Yes | Standardization | Fair coefficient penalties | | k-NN, SVM, k-Means | Yes | Standardization ya Normalization | Distance-based; large features dominate karte hain | | Neural Networks | Yes | Normalization (bounded activations ke liye) ya Standardization | Faster convergence, stable gradients | | Decision Trees, Random Forests | None | Split thresholds scale-invariant hain | | PCA | Yes | Standardization | Variance-based; large features eigenvectors mein dominate karte hain | | Gradient Boosting (XGBoost, etc.) | None | Tree-based splits scale-invariant hain | | Naive Bayes | No | None | Probabilities use karta hai, distances nahi |

Implementation Considerations

Training ke baad naye data ko handle karna:

Training data par scaler fit karne ke baad, naye data ko same transformation parameters use karne chahiye:

# Training phase
scaler = StandardScaler()
scaler.fit(X_train)  # Stores μ, σ
X_train_scaled = scaler.transform(X_train)
 
# Later: new unseen data
X_new_scaled = scaler.transform(X_new)  # Uses stored μ, σ from training

Kyun? Agar naye data ka alag min/max ya μ/σ hai, toh scaled values training data se comparable nahi rahenge → model fail ho jaata hai.

Agar naye data ki values training range se bahar hain toh?

• Normalization: $[X_{\min },X_{\max }]$ se bahar ki values [0, 1] se bahar hongi. Yeh acceptable hai – woh genuinely extreme hain.
• Standardization: Values ko z-scores > 3 ya < -3 milte hain. Yeh bhi acceptable hai – woh training data ke relative outliers hain.

Inverse transformation:

Predictions ko original scale par wapas convert karne ke liye:

$X=X_{\text{norm}}\cdot (X_{\max }-X_{\min })+X_{\min }$ $X=X_{\text{std}}\cdot \sigma +\mu$

Active Recall Practice

Socho tum compare kar rahe ho ki students math aur art mein kitne acche hain. Ek student ne math mein 95/100 aur art mein 8/10 score kiya. Kya woh math mein zyada acchi hai ya art mein?

Tum directly compare nahi kar sakte kyunki scales alag hain! 95 vs 8 tumhe kuch nahi bataata.

Normalization donon ko percentage mein convert karne jaisa hai: 95/100 = 95%, 8/10 = 80%. Ab compare kar sakte ho: woh math mein thodi behtar hai.

Standardization alag hai. Yeh poochhti hai: "Dusre students ke muqable mein yeh score kitna unusual hai?" Agar math test ka average 90 tha (aur sabne 85-95 score kiya), toh uska 95 sirf thoda sa average se upar hai. Lekin agar art ka average 5 tha (aur zyadatar ne 3-7 score kiya), toh uska 8 bahut zyada average se upar hai. Toh woh actually art mein zyada exceptional hai!

Dono methods cheezein comparable banate hain, lekin alag cheezein measure karte hain: range mein position vs typical se distance.

"OutLIER → Stand"

• Outliers hain? Standardization use karo (woh extreme z-scores ban jaate hain scale todne ki jagah)

Connections

• 2.1.01-Feature-types-and-data-structures – Feature types samajhna batata hai ki kaunsa scaling method fit karta hai
• 2.1.05-Handling-outliers-detection-and-treatment – Outlier handling aksar normalization se pehle hoti hai ya usse replace karti hai
• 3.2.03-Gradient-descent-optimization – Feature scaling gradient descent convergence accelerate karta hai
• 4.1.02-Distance-metrics-euclidean-manhattan-cosine – Scaling k-NN, k-means mein fair distance computation ensure karta hai
• 5.3.01-Regularization-L1-L2-elastic-net – Fair coefficient penalties ke liye standardization required hai
• 6.2.01-Principal-Component-Analysis-PCA – PCA ke liye standardization required hai; variance-based decomposition
• 7.1.03-Batch-normalization-in-neural-networks – Batch norm layer-by-layer internal scaling karta hai

#flashcards/ai-ml

Min-max normalization ka formula [0, 1] ke liye kya hai? :: $X_{\text{norm}}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$

Concept Map