Log and power transformations

2.1.9 · AI-ML › Data Preprocessing & Feature Engineering

Har Transformation Kyun Kaam Karta Hai

Goal: Large values ko compress karo, small values ko expand karo taaki right skew kam ho. Exponential relationship se shuru karo: agar $y = e^{z}$ hai, toh $z$ mein equal additive changes, $y$ mein equal multiplicative changes se correspond karte hain.

y_1 &= e^{z_1}, \quad y_2 = e^{z_2} \\ \frac{y_2}{y_1} &= e^{z_2 - z_1} \\ \log\left(\frac{y_2}{y_1}\right) &= z_2 - z_1 \end{align}$$ **Yeh kyun matter karta hai**: Income ka \$20k\rightarrow\infty$40k ho jaana \$100k\rightarrow\infty$200k hone jaisa hi feel hota hai. Log space multiplicative relationships ko **additive** bana deta hai. **Common forms**: - Natural log: $x' = \ln(x)$ — base $e$, ML mein sabse common - Log10: $x' = \log_{10}(x)$ — interpret karna aasaan hai (har unit = 10× change) - With offset: $x' = \log(x + 1)$ — $x = 0$ ko handle karta hai (log1p function)

Worked example Log Transform: Housing Prices

Problem: Ghar ki prices $50k–$5M (100× span) hain, heavily right-skewed. Original data: [5000, 12000, 25000, 85000, 5000000]

Step 1: $x^{'} = lo g_{10} (x)$ apply karo

log₁₀(50000) = 4.70
log₁₀(120000) = 5.08
log₁₀(250000) = 5.40
log₁₀(850000) = 5.93
log₁₀(5000000) = 6.70

Yeh step kyun? 100× range ko 2-unit range (4.7→6.7) mein compress karo.

Step 2: Variance change check karo

Original std dev: ~$2M (outlier ka dominance)
Log-transformed std dev: ~0.7 (balanced) Yeh step kyun? Linear regression homoscedasticity (constant variance) assume karta hai. Log transform use stabilize karta hai.

Interpretation: Log-price par 0.5 ka model coefficient matlab hai "10^0.5 ≈ 3.16× price change per unit of feature."

2. Box-Cox Transformation

Formula Box-Cox Power Transform

Goal: Optimal $λ$ dhundho jo data ko most normal banaye.

Power function $x^{λ}$ se shuru karo, lekin yeh $λ = 0$ par discontinuous hai. Gap fill karne ke liye L'Hôpital's rule use karo:

\frac{x^\lambda - 1}{\lambda} & \lambda \neq 0 \\ \log(x) & \lambda = 0 \end{cases}$$ **$-1$ aur $/\lambda$ kyun?** Transformation ko $\lambda = 0$ par **continuous** banane ke liye: $$\lim_{\lambda \to 0} \frac{x^\lambda - 1}{\lambda} = \lim_{\lambda \to 0} \frac{x^\lambda \log(x)}{1} = \log(x)$$ (L'Hôpital's use karke: numerator aur denominator dono ko $\lambda$ ke w.r.t. differentiate karo) **Key values**: - $\lambda = 1$: Koi transformation nahi (linear) - $\lambda = 0.5$: Square root - $\lambda = 0$: Log - $\lambda = -1$: Reciprocal $-1/x$ **$\lambda$ kaise dhundein**: Normality ka log-likelihood maximize karo (scipy.stats.boxcox yeh MLE se karta hai).

Worked example Box-Cox: Customer Spending

Problem: Monthly spending: [5, 15, 30, 35, 200, 450] — outliers ke saath right-skewed.

Step 1: Candidate $λ$ values test karo

λ = 1.0 → skewness = 2.1 (heavy right tail)
λ = 0.5 → skewness = 1.3
λ = 0.0 → skewness = 0.4
λ = -0.5 → skewness = -0.2 (slight left tail)

Yeh step kyun? Hum skewness ≈ 0 chahte hain (symmetric distribution).

Step 2: Optimal $λ = - 0.1$ apply karo $x^{'} = \frac{x ^{- 0.1} - 1}{- 0.1} = 10 (1 - x^{- 0.1})$ $x = 450$ ke liye: $x^{'} = 10 (1 - 45 0^{- 0.1}) = 10 (1 - 0.548) = 4.52$

Yeh step kyun? Transformation extreme values ko center ki taraf kheenchta hai bina rank order distort kiye.

Constraint: Box-Cox ke liye $x > 0$ zaroori hai. Agar zeros hain toh constant add karo: $Box-Cox (x + 1, λ)$ .

3. Yeo-Johnson Transformation

Kab Kaunsa Use Karein

Formula Decision Framework

Condition	Transformation	Reason
$x > 0$ , right-skewed	$lo g (x)$ ya $lo g (x + 1)$	Simplest, elasticity ke taur par interpretable
$x > 0$ , moderately skewed	$x$	Log se gentle, zyada structure preserve karta hai
$x > 0$ , optimal $λ$ unknown	Box-Cox	MLE se data-driven $λ$
$x$ mein zeros ya negatives hain	Yeo-Johnson	Mixed-sign data ke liye yahi option hai
Counts/frequencies	$x$ ya $lo g (x + 1)$	Poisson variance stabilize karta hai
Proportions/percentages	Logit: $lo g \frac{p}{1 - p}$	(0,1) → (-∞,∞) map karta hai

Common Mistakes

Common mistake Error 1: Zero ya Negative Values ko Log Karna

Galat approach: np.log(data) jab data mein zeros hain → -inf ya NaN.

Kyun sahi lagta hai: "Bas log apply karo, ho jayega."

Steel-man: Tum transformation par focused ho, domain constraints par nahi. Mathematically, $lo g (0)$ undefined hai.

Fix: np.log1p(data) use karo jo $lo g (x + 1)$ compute karta hai, ya ek chhota constant add karo: $lo g (x + ϵ)$ jahan $ϵ$ domain-appropriate ho (jaise counts ke liye 1, prices ke liye 0.01).

Code:

# Wrong
transformed = np.log(data)  # Breaks on zeros
 
# Right
transformed = np.log1p(data)  # log(data + 1)
# Or for negatives: Yeo-Johnson

Common mistake Error 3: Train/Test Split Ke Baad Transformation Apply Karna

Galat approach: Data split karo → train par Box-Cox $λ$ compute karo → train ke $λ$ se test par apply karo.

Kyun sahi lagta hai: "Main train par compute karke data leakage avoid kar raha hoon."

Steel-man: Tumne sahi pehchana ki $λ$ ek learned parameter hai. Lekin yeh correct hai! Galti doosre tarike se karne mein hai.

Actually galat: $λ$ ko split se pehle saare data par compute karna → test distribution ki info training mein leak ho jaati hai.

Fix:

Data split karo
Transformation fit karo ( $λ$ seekho) sirf train par
Train ko fitted $λ$ se transform karo
Test ko same $λ$ se transform karo (refit mat karo)

from sklearn.preprocessing import PowerTransformer
 
X_train, X_test = train_test_split(X)
pt = PowerTransformer(method='box-cox')
X_train_trans = pt.fit_transform(X_train)  # Learn λ
X_test_trans = pt.transform(X_test)        # Apply same λ

Mathematical Properties

Jab transformations help NAHI karte:

Tree-based models (Random Forest, XGBoost) — yeh monotonic transforms ke liye invariant hote hain
Data already normal/symmetric hai
Small sample size (< 30) — transformation artifacts introduce kar sakta hai

Practical Workflow

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
 
# 1. Diagnose skewness
data = np.array([10, 15, 18, 22, 50, 120, 500])
skew = stats.skew(data)  # > 0 → right skew
print(f"Original skewness: {skew:.2f}")
 
# 2. Try transformations
log_data = np.log1p(data)
sqrt_data = np.sqrt(data)
bc_data, best_lambda = stats.boxcox(data + 1)  # +1 if zeros
 
print(f"Log skewness: {stats.skew(log_data):.2f}")
print(f"Box-Cox λ: {best_lambda:.2f}, skewness: {stats.skew(bc_data):.2f}")
 
# 3. Visualize
fig, axes = plt.subplots(2, 2, figsize=(10, 8))
for ax, d title in zip(axes.flat,
                         [data, log_data, sqrt_data, bc_data],
                         ['Original', 'Log', 'Sqrt', 'Box-Cox']):
    ax.hist(d, bins=15, edgecolor='black', alpha=0.7)
    ax.set_title(f"{title}\nSkew: {stats.skew(d):.2f}")

Recall Aise Samjho Jaise Main 12 Saal Ka Hoon: Log Scales Kyun?

Socho tum measure kar rahe ho cheezein kitni tezi se badhti hain. Ek bacteria colony har ghante double hoti hai: 1 → 2 → 4 → 8 → 16 → 32. Agar tum ise regular paper par plot karo, toh line upar ki taraf bahut steep curve karti hai aur tumhe starting numbers dikhai nahi dete.

Lekin yahan trick hai: har doubling same type ka change hai (×2). Toh chalao ek special ruler banate hain jahan equal distances ka matlab hai "same amount se multiply karo." Ab tumhare ruler par 1→2 aur 16→32 same distance par hain. Yahi log scale hai!

Jab tum log use karte ho, $100 \to \infty $1000 (×10) bilkul same dikhta hai jaise \$ 1000 \text{ to } $10,000 (×10). Isliye hum income data ko log-transform karte hain — tumhara dimaag "kitne guna bada" ki terms mein sochta hai, na ki "kitne dollars zyada." Yeh fark hai "meri salary double ho gayi!" aur "mujhe \$ 50k raise mila" ke beech mein (jo totally alag cheezein hain agar tumne $25k se shuru kiya tha versus $200k se).

Connections

2.1.05-Feature-Scaling — Transformations vs. standardization (sirf scale nahi, shape bhi transform karo)
2.1.08-Handling-Outliers — Log naturally outliers ko compress karta hai
3.2.01-Linear-Regression-Assumptions — Homoscedasticity ke liye targets transform karna
2.2.03-Polynomial-Features — Non-linearity capture karne ka log ka alternative
4.1.02-Decision-Trees — Trees ko transformations kyun nahi chahiye (split-based hote hain)

#flashcards/ai-ml

Log transformation kaunsi problem solve karta hai? :: Right skewness reduce karta hai, large values compress karta hai, multiplicative relationships ko additive mein convert karta hai (jaise income, population), heteroscedastic data ke liye variance stabilize karta hai.

Box-Cox formula λ≠ 0 ke liye

(x^λ - 1) / λ. -1 aur λ se division ise λ=0 par continuous banata hai (jahan L'Hôpital's rule se yeh log(x) ban jaata hai).

Yeo-Johnson kab use karna zaroori hai Box-Cox ki jagah?

Jab data mein zero ya negative values hoon. Box-Cox ke liye x > 0 zaroori hai; Yeo-Johnson x≥ 0 aur x < 0 ke liye alag formulas ke saath saare real numbers handle karta hai.

log(x+1) transformation mein +1 kyun add karte hain?

x=0 ko handle karne ke liye (kyunki log(0) undefined hai). Numpy mein log1p kehte hain. Alternative: domain-appropriate constant add karo (jaise counts ke liye 1).

Log-transformed target par training ke baad original units mein predictions kaise laate hain?

Inverse apply karo: exp(ŷ_log). Agar E[y] chahiye (median nahi) toh Jensen's inequality ki wajah se bias correction exp(ŷ + σ²/2) add karo.

Box-Cox λ train/test split se pehle fit karna chahiye ya baad mein?

Baad mein. λ sirf training data par fit karo, phir same λ test data par apply karo (kisi bhi aur preprocessing parameter ki tarah). Split se pehle saare data par fit karna information leak karta hai.

Kaun se models log/power transformations se benefit nahi lete?

Tree-based models (Random Forest, XGBoost, decision trees) kyunki yeh monotonic transformations ke liye invariant hote hain — yeh distances par nahi, thresholds par split karte hain.

Box-Cox λ=0.5 kya represent karta hai?

Square root transformation. λ=1 identity hai (koi transform nahi), λ=0 log hai, λ=-1 reciprocal hai (-1/x).

Yeh Transformations Hain Kya?