Log and power transformations

What Are These Transformations?

Why Each Transformation Works

1. Logarithmic Transformation

Worked example Log Transform: Housing Prices

Problem: Home prices range $50k–$5M (100× span), heavily right-skewed. Original data: [5000, 12000, 25000, 85000, 5000000]

Step 1: Apply $x' = \log_{10}(x)$

log₁₀(50000) = 4.70
log₁₀(120000) = 5.08
log₁₀(250000) = 5.40
log₁₀(850000) = 5.93
log₁₀(5000000) = 6.70

Why this step? Compress the100× range to a 2-unit range (4.7→6.7).

Step 2: Check variance change

Original std dev: ~$2M (dominated by outlier)
Log-transformed std dev: ~0.7 (balanced) Why this step? Linear regression assumes homoscedasticity (constant variance). Log transform stabilizes it.

Interpretation: A model coefficient of 0.5 on log-price means "10^0.5 ≈ 3.16× price change per unit of feature."

2. Box-Cox Transformation

Formula Box-Cox Power Transform

Goal: Find the optimal $\lambda$ that makes data most normal.

Start with the power function $x^\lambda$ , but it's discontinuous at $\lambda = 0$ . Use L'Hôpital's rule to fill the gap:

\frac{x^\lambda - 1}{\lambda} & \lambda \neq 0 \\ \log(x) & \lambda = 0 \end{cases}$$ **Why the $-1$ and $/\lambda$?** To make the transformation **continuous** at $\lambda = 0$: $$\lim_{\lambda \to 0} \frac{x^\lambda - 1}{\lambda} = \lim_{\lambda \to 0} \frac{x^\lambda \log(x)}{1} = \log(x)$$ (Using L'Hôpital's: differentiate numerator and denominator w.r.t. $\lambda$) **Key values**: - $\lambda = 1$: No transformation (linear) - $\lambda = 0.5$: Square root - $\lambda = 0$: Log - $\lambda = -1$: Reciprocal $-1/x$ **Finding $\lambda$**: Maximize log-likelihood of normality (scipy.stats.boxcox does this via MLE).

Worked example Box-Cox: Customer Spending

Problem: Monthly spending: [5, 15, 30, 35, 200 450] — right-skewed with outliers.

Step 1: Test candidate $\lambda$ values

λ = 1.0 → skewness = 2.1 (heavy right tail)
λ = 0.5 → skewness = 1.3
λ = 0.0 → skewness = 0.4
λ = -0.5 → skewness = -0.2 (slight left tail)

Why this step? We want skewness ≈ 0 (symmetric distribution).

Step 2: Apply optimal $\lambda = -0.1$ $x' = \frac{x^{-0.1} - 1}{-0.1} = 10(1 - x^{-0.1})$ For $x = 450$ : $x' = 10(1 - 450^{-0.1}) = 10(1 - 0.548) = 4.52$

Why this step? The transformation pulls extreme values toward the center without distorting the rank order.

Constraint: Box-Cox requires $x > 0$ . Add a constant if you have zeros: $\text{Box-Cox}(x + 1, \lambda)$ .

3. Yeo-Johnson Transformation

Worked example Yeo-Johnson: Temperature Anomalies

Problem: Climate data with deviations: [-8.2, -3.1, 0.5, 2.3, 7.8, 15.4]°C (both negative and positive). Step 1: Can't use Box-Cox (negative values). Apply Yeo-Johnson with $\lambda = 0.8$ : For $x = -8.2$ (negative): $x' = -\frac{(-(-8.2) + 1)^{2-0.8} - 1}{2 - 0.8} = -\frac{9.2^{1.2} - 1}{1.2} = -\frac{13.8 - 1}{1.2} = -10.7$

Why this step? The formula ensures smooth behavior across zero without breaking.

For $x = 7.8$ (positive): $x' = \frac{(7.8 + 1)^{0.8} - 1}{0.8} = \frac{8.8^{0.8} - 1}{0.8} = \frac{6.04 - 1}{0.8} = 6.3$

Why this step? Compresses the positive tail while leaving negative values interpretable.

Result: Original range23.6°C → Transformed range ~17°C with skewness reduced from 1.8 → 0.3.

When to Use Each

Formula Decision Framework

Condition	Transformation	Reason
$x > 0$ , right-skewed	$\log(x)$ or $\log(x+1)$	Simplest, interpretable as elasticity
$x > 0$ , moderately skewed	$\sqrt{x}$	Gentler than log, preserves more structure
$x > 0$ , unknown optimal $\lambda$	Box-Cox	Data-driven $\lambda$ via MLE
$x$ includes zeros or negatives	Yeo-Johnson	Only option for mixed-sign data
Counts/frequencies	$\sqrt{x}$ or $\log(x+1)$	Stabilizes Poisson variance
Proportions/percentages	Logit: $\log\frac{p}{1-p}$	Maps (0,1) → (-∞,∞)

Common Mistakes

Common mistake Error 1: Logging Zero or Negative Values

Wrong approach: np.log(data) when data contains zeros → -inf or NaN.

Why it feels right: "Just apply log, it'll work."

Steel-man: You're focused on the transformation itself, not domain constraints. Mathematically, $\log(0)$ is undefined.

Fix: Use np.log1p(data) which computes $\log(x + 1)$ , or add a small constant: $\log(x + \epsilon)$ where $\epsilon$ is domain-appropriate (e.g., 1 for counts, 0.01 for prices).

Code:

# Wrong
transformed = np.log(data)  # Breaks on zeros
 
# Right
transformed = np.log1p(data)  # log(data + 1)
# Or for negatives: Yeo-Johnson

Common mistake Error 2: Forgetting to Inverse Transform Predictions

Wrong approach: Train on log-transformed target, predict report raw predictions.

Why it feels right: "The model outputs numbers, that's my prediction."

Steel-man: You're thinking in the transformed space where the model was trained. But users need original units.

Fix: Apply the inverse transformation:

Log: $\hat{y}_{\text{original}} = \exp(\hat{y}_{\log})$
Box-Cox: $\hat{y}_{\text{original}} = (\lambda \hat{y}' + 1)^{1/\lambda}$ Gotcha: For log-transformed targets, $E[y] \neq \exp(E[\log y])$ due to Jensen's inequality. Add a bias correction: $\exp(\hat{y} + \sigma^2/2)$ where $\sigma^2$ is residual variance.

Common mistake Error 3: Applying Transformation After Train/Test Split

Wrong approach: Split data → compute Box-Cox $\lambda$ on train → apply with train's $\lambda$ to test.

Why it feels right: "I'm avoiding data leakage by computing on train only."

Steel-man: You correctly identified that $\lambda$ is a learned parameter. But this is correct! The mistake is doing it the other way.

Actually wrong: Computing $\lambda$ on all data before splitting → leaks test distribution info into training.

Fix:

Split data
Fit transformation (learn $\lambda$ ) on train only
Transform train with fitted $\lambda$
Transform test with same $\lambda$ (don't refit)

from sklearn.preprocessing import PowerTransformer
 
X_train, X_test = train_test_split(X)
pt = PowerTransformer(method='box-cox')
X_train_trans = pt.fit_transform(X_train)  # Learn λ
X_test_trans = pt.transform(X_test)        # Apply same λ

Mathematical Properties

When transformations DON'T help:

Tree-based models (Random Forest, XGBoost) — they're invariant to monotonic transforms
Data is already normal/symmetric
Small sample size (< 30) — transformation can introduce artifacts

Practical Workflow

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
 
# 1. Diagnose skewness
data = np.array([10, 15, 18, 22, 50, 120, 500])
skew = stats.skew(data)  # > 0 → right skew
print(f"Original skewness: {skew:.2f}")
 
# 2. Try transformations
log_data = np.log1p(data)
sqrt_data = np.sqrt(data)
bc_data, best_lambda = stats.boxcox(data + 1)  # +1 if zeros
 
print(f"Log skewness: {stats.skew(log_data):.2f}")
print(f"Box-Cox λ: {best_lambda:.2f}, skewness: {stats.skew(bc_data):.2f}")
 
# 3. Visualize
fig, axes = plt.subplots(2, 2, figsize=(10, 8))
for ax, d title in zip(axes.flat,
                         [data, log_data, sqrt_data, bc_data],
                         ['Original', 'Log', 'Sqrt', 'Box-Cox']):
    ax.hist(d, bins=15, edgecolor='black', alpha=0.7)
    ax.set_title(f"{title}\nSkew: {stats.skew(d):.2f}")

Recall Explain Like I'm 12: Why Log Scales?

Imagine you're measuring how fast things grow. A bacteria colony doubles every hour: 1 → 2 → 4 → 8 → 16 → 32. If you plot this on regular paper, the line curves up super steep and you can't see the early numbers.

But here's the trick: each doubling is the same type of change (×2). So let's make a special ruler where equal distances mean "multiply by the same amount." Now1→2 is the same distance as 16→32 on your ruler. This is a log scale!

When you use log, going from $100 \to \infty $1000 (×10) looks the same as \$ 1000 \text{ to } $10,000 (×10). That's why we log-transform income data—your brain thinks in terms of "how many times bigger," not "how many dollars more." It's the difference between "my salary doubled!" and "I got a \$ 50k raise" (which means totally different things if you started at $25k vs $200k).

Connections

2.1.05-Feature-Scaling — Transformations vs. standardization (transform shape, not just scale)
2.1.08-Handling-Outliers — Log naturally compresses outliers
3.2.01-Linear-Regression-Assumptions — Transforming targets for homoscedasticity
2.2.03-Polynomial-Features — Alternative to log for capturing non-linearity
4.1.02-Decision-Trees — Why trees don't need transformations (split-based)

#flashcards/ai-ml

What problem does log transformation solve? :: Reduces right skewness, compresses large values, converts multiplicative relationships to additive (e.g., income, population), stabilizes variance for heteroscedastic data.

Box-Cox formula for λ≠ 0

(x^λ - 1) / λ. The -1 and division by λ make it continuous at λ=0 (where it becomes log(x) by L'Hôpital's rule).

When must you use Yeo-Johnson instead of Box-Cox?

When data contains zero or negative values. Box-Cox requires x > 0; Yeo-Johnson handles all real numbers with separate formulas for x≥ 0 and x < 0.

Why add +1 in log(x+1) transformation?

Handles x=0 (since log(0) is undefined). Called1p in numpy. Alternative: add domain-appropriate constant (e.g., 1 for counts).

After training on log-transformed target, how do you get predictions in original units?

Apply inverse: exp(ŷ_log). Add bias correction exp(ŷ + σ²/2) due to Jensen's inequality if you need E[y], not median.

Should you fit Box-Cox λ before or after train/test split?

After. Fit λ on training data only, then apply the same λ to test data (like any other preprocessing parameter). Fitting on all data before split leaks information.

Which models don't benefit from log/power transformations?

Tree-based models (Random Forest, XGBoost, decision trees) because they're invariant to monotonic transformations—they split on thresholds, not distances.

What does Box-Cox λ=0.5 represent?

Square root transformation. λ=1 is identity (no transform), λ=0 is log, λ=-1 is reciprocal (-1/x).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Log aur power transformations ka basic idea ye hai kiagar apka data bahut skewed hai—matlab ek taraf extreme values hain (jaise salary data mein 20 lakh wale common hain par 2 crore wale bhi hain)—to directly model banana mushkil ho jata hai. Linear regression jaise algorithms assume karte hain ki data normally distributed hai, par real-world data mein ye rarely hota hai.

Log transformation sabse simple hai: log(x) apply karo. Ye bade values ko compress kar deta hai—jaise 1 crore aur 10 crore mein original space mein 9 crore ka farak hai, but log space mein sirf 1 ka farak (because log₁₀(10^7)=7 aur log₁₀(10^8)=8). Isliye salary, population, website traffic jaise exponential growth wale features ko log transform karne se model ko samajhna aasaan ho jata hai. Caveat: agar data mein zero ya negative values hain to log(x+1) use karna padega (warna -inf aa jayega aur code crash karega).

Box-Cox transformation ek step age hai—ye automatically best power (λ) dhundh leta hai jo data ko sabse zyada normal-looking banaye. λ=0 means log, λ=0.5 means square root, λ=1 means koi transformation nahi. Scikit-learn ka PowerTransformer ye sab automatic kar deta hai. Limitation: Box-Cox sirf positive values ke liye kaam karta hai.Agar negative values hain (jaise temperature anomalies: -5°C se +12°C), to Yeo-Johnson use karna padega jo negative values handle kar sakta hai.

Practical tip: Tree-based models (Random Forest, XGBoost) ko transformations ki zarurat nahi hoti kyunki wo splits par kaam karte hain, distance par nahi. But linear models, neural networks, aur SVM ke liye ye transformations bohot helpful hain. Aur haan, ek bahut common mistake: agar apne target variable (y) ko log-transform kiya hai, to predictions ko wapas original scale mein lane ke liye exp(ŷ) karna mat bhoolna!

What Are These Transformations?

Why Each Transformation Works

1. Logarithmic Transformation

2. Box-Cox Transformation

3. Yeo-Johnson Transformation

When to Use Each

Common Mistakes

Mathematical Properties

Practical Workflow

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Data Preprocessing & Feature Engineering

Connections