Visual walkthrough — Log and power transformations

We build in pictures. Every new tool — including the two calculus ideas "limit" and "derivative" — is introduced with its own plain-words box before we use it. Let us start with what "skew" even looks like.

---

Step 1 — What "right-skewed" means, as a shape

WHAT. We have a column of positive numbers — say incomes or prices. Most values are small; a few are enormous. When we draw a histogram (bars whose height = how many values fall in that slice), the bump sits on the left and a long thin tail stretches to the right.

WHY. Every later step exists to fix this exact shape. A model like Linear Regression wants a symmetric, bell-ish spread. So first we must see the enemy.

PICTURE. Look at the burnt-orange bars below. The word "right-skewed" just means: the tail points right (toward big numbers). The plum arrow marks the tail we want to shorten.

---

Step 2 — The power function $x^{\lambda }$ is a "shape dial"

WHAT. Take each value $x$ and raise it to a power $\lambda$ (the Greek letter lambda — just a name for "the exponent we get to choose"). Different $\lambda$ bend the number line differently.

WHY. We need some knob that squashes big numbers more than small ones. Powers less than $1$ do exactly that: $\sqrt{x}=x^{0.5}$ turns $100$ into $10$ but $4$ into $2$ — the giant shrank 10×, the small one only 2×. That differential squashing is what pulls the tail in.

$$\underset{\text{new value}}{\underbrace{x^{\prime }}}=\underset{\text{old value}}{\underbrace{x}}{}^{\stackrel{\text{the dial}}{\overbrace{\lambda }}}$$

PICTURE. Each curve below is $x^{\lambda }$ for a different $\lambda$. Turn the dial down (toward $0$) and the curve flattens — the far-right tail gets crushed toward the others. Turn it up past $1$ and the tail stretches instead.

---

Step 3 — Why raw $x^{\lambda }$ has an ugly seam at $\lambda =0$

WHAT. Watch what $x^{\lambda }$ does as $\lambda$ slides down to $0$. For any positive $x$, $x^0=1$. So at the instant $\lambda =0$, every value collapses to the single number $1$.

WHY. A transformation that turns all your data into "$1$" is useless — it erased every difference. So plain $x^{\lambda }$ cannot be used at $\lambda =0$; there is a hole in our dial right where we most want a logarithm to appear.

$$x^0=1\text{for every }x>0\Rightarrow \text{all information gone}$$

PICTURE. The curves in Step 2 all funnel toward the same height $1$ at $\lambda =0$. Below we zoom on that collapse — three different starting values ($x=2,20,200$) all crash to $1$.

---

Step 4 — The repair: subtract $1$, divide by $\lambda$

WHAT. Replace $x^{\lambda }$ with

$$\text{Box-Cox}(x,\lambda )=\frac{\stackrel{\text{powered value}}{\overbrace{x^{\lambda }}}-\stackrel{\text{re-centre}}{\overbrace{1}}}{\underset{\text{re-scale}}{\underbrace{\lambda }}},\lambda \ne 0$$

WHY each piece:

• The $-1$ re-centres. Since $x^{\lambda }$ near $\lambda =0$ hovers around $1$, subtracting $1$ moves that whole cluster down to hover around $0$ instead. Now the values are differences from $1$, which are tiny — not all collapsed onto $1$.
• The $÷\lambda$ re-scales. Those tiny differences shrink as $\lambda \to 0$; dividing by the equally-tiny $\lambda$ rescues them from vanishing. Small ÷ small = something finite and useful.

Together they take the "everything crashes to $1$" seam and stretch it back open into a well-behaved curve. Crucially, this rescaling doesn't reorder your data — bigger $x$ still maps to bigger $x^{\prime }$ (the rank order is preserved, which is why tree splits are unaffected).

PICTURE. Left panel: raw $x^{\lambda }$ (the collapsing family from Step 3). Right panel: after "$-1$, then $÷\lambda$" — the curves no longer pile onto $1$; they fan out around $0$ and stay distinct even as the dial approaches $0$.

---

Step 5 — Two calculus tools we need first: "limit" and "derivative"

WHAT. Before we can find what the repaired dial does at $\lambda =0$, we need two ideas. Both are just precise names for things you already picture.

WHY. In Step 6 we hit the meaningless form $\frac{0}{0}$. The trick that resolves it (L'Hôpital's rule) compares the two curves' slopes — so we genuinely need the derivative idea in hand first. This step is that hand.

PICTURE. Left: a ball creeping toward $\lambda =0$ along $\frac{x^{\lambda }-1}{\lambda }$, homing in on a height (the limit) even though the exact point is a hollow dot. Right: the same curve with a little tangent ramp drawn at one point — its steepness is the derivative.

---

Step 6 — Why $\frac{d}{d\lambda }{x}^{\lambda }=x^{\lambda }\ln x$

WHAT. We need the slope of $x^{\lambda }$ as $\lambda$ changes (with $x$ held fixed). The answer is $x^{\lambda }\ln x$. Here is why, from the ground up — no memorising.

WHY / derivation. The clean route is to rewrite $x^{\lambda }$ using $e$, because we know how $e$ raised to something behaves.

$$x^{\lambda }=(e^{\ln x}{)}^{\lambda }=e^{\lambda \ln x}$$

• What we did: replaced $x$ by $e^{\ln x}$ (true by the very definition of $\ln$: $e^{\ln x}=x$), then used "a power of a power multiplies the exponents."
• Why: now the exponent is just $\lambda$ times the constant $\ln x$, and $e^{(\text{constant})\lambda }$ has a slope we can read off.

The single growth fact we lean on: the curve $e^{k\lambda }$ rises at a rate $k$ times its own current height. (That is what makes $e$ special — its slope equals itself when $k=1$.) Here the constant is $k=\ln x$, so

$$\frac{d}{d\lambda }{e}^{\lambda \ln x}=\underset{\text{the rate }k}{\underbrace{\ln x}}\cdot \underset{\text{current height}}{\underbrace{e^{\lambda \ln x}}}=x^{\lambda }\ln x$$

PICTURE. For a fixed $x=3$, we plot $x^{\lambda }$ against $\lambda$ and draw the tangent ramp at a point; its measured steepness matches $x^{\lambda }\ln x$ (teal dot vs. plum ramp). The $\ln x$ factor is literally why the ramp is steeper for bigger $x$ — and thus why a logarithm is about to surface.

---

Step 7 — At $\lambda =0$ the repaired dial becomes the logarithm

WHAT. We ask: what value does $\frac{x^{\lambda }-1}{\lambda }$ approach as $\lambda \to 0$? At $\lambda =0$ the fraction reads $\frac{1-1}{0}=\frac{0}{0}$ — meaningless on its own, so we take the limit (Step 5) and resolve the $\frac{0}{0}$ with L'Hôpital's rule.

Check the fine print for our case, $f(\lambda )=x^{\lambda }-1$, $g(\lambda )=\lambda$, at $a=0$:

• both $\to 0$ as $\lambda \to 0$ ✓ (it is genuinely $\frac{0}{0}$);
• $f$ is differentiable everywhere (Step 6 gave $f^{\prime }=x^{\lambda }\ln x$) and $g$ is differentiable with $g^{\prime }=1$ ✓;
• $g^{\prime }=1\ne 0$ everywhere ✓;
• the slope-ratio limit exists (we compute it next) ✓.

All hypotheses hold, so we may swap top and bottom for their derivatives:

$$\underset{\lambda \to 0}{\lim }\frac{x^{\lambda }-1}{\lambda }=\underset{\lambda \to 0}{\lim }\frac{\stackrel{f^{\prime }=\text{slope of top}}{\overbrace{x^{\lambda }\ln x}}}{\underset{g^{\prime }=\text{slope of bottom}}{\underbrace{1}}}=x^0\ln x=\ln x$$

So the hole in the dial is plugged by $\log$ itself — not bolted on arbitrarily, but demanded by continuity.

PICTURE. The teal dots trace $\frac{x^{\lambda }-1}{\lambda }$ at $\lambda =0.5,0.2,0.05$ for a fixed $x$; the plum curve is $\ln x$. Watch the dots march onto the plum curve as $\lambda \to 0$.

---

Step 8 — Reading the special settings on the dial

WHAT. Now that the dial is smooth, name the useful clicks.

WHY. In practice you rarely eyeball $\lambda$; scipy.stats.boxcox searches for the $\lambda$ that makes the histogram most symmetric. But knowing the landmarks tells you what family of squash you're getting.

$\lambda$	Formula becomes	Effect on the tail
$1$	$x-1$	none (just a shift)
$0.5$	$2(\sqrt{x}-1)$	gentle squash
$0$	$\ln x$	strong squash
$-1$	$1-\frac{1}{x}$	extreme squash / flip

PICTURE. The same skewed data from Step 1, pushed through four $\lambda$ settings. As $\lambda$ drops, the plum tail retracts; the bump slides right toward symmetry.

---

Step 9 — The degenerate cases you must not miss

WHAT & WHY. Two edges break Box-Cox; you must recognise them before they crash your pipeline.

Case A — $x=0$ or $x<0$. Box-Cox needs $x>0$ (you can't raise $0$ to a negative power, and $\log 0=-\infty$). If your column has zeros, shift it: $\text{Box-Cox}(x+c,\lambda )$ with $c$ chosen from domain sense. If it has genuine negatives (temperature anomalies, profit/loss), Box-Cox is impossible — switch to Yeo-Johnson.

Yeo-Johnson is not simply "the same $-1,÷\lambda$ on each side." For negatives it uses a different exponent and sign-flips so the two halves meet smoothly at $0$. The full four-piece rule:

$$\text{YJ}(x,\lambda )=\{\begin{array}{ll}\frac{(x+1{)}^{\lambda }-1}{\lambda }& x\ge 0,\lambda \ne 0\\ \ln (x+1)& x\ge 0,\lambda =0\\ -\frac{(-x+1{)}^{2-\lambda }-1}{2-\lambda }& x<0,\lambda \ne 2\\ -\ln (-x+1)& x<0,\lambda =2\end{array}$$

Notice the negative branch: it flips $x$ to $-x+1$ (making a positive number), raises it to the conjugate exponent $2-\lambda$ (not $\lambda$), divides by $2-\lambda$, then negates the whole thing. Those specific exponent and sign choices are exactly what make the two branches share the same value and the same slope at $x=0$ — that is what "continuous and differentiable across zero" requires. A naive "just mirror the $-1,÷\lambda$ trick" would leave a kink at $0$.

Case B — $\lambda$ known only from data. $\lambda$ is a learned parameter. Learn it on the training set only, then apply that same frozen $\lambda$ to test data — exactly like fitting a scaler in Feature Scaling. Learning $\lambda$ on all data leaks the test distribution.

PICTURE. A number line split at $0$: green region ($x>0$, Box-Cox legal), red region ($x\le 0$, Box-Cox forbidden → Yeo-Johnson), with the mirror-image arrow showing how Yeo-Johnson reflects the transform across $0$.

---

The one-picture summary

WHAT. Everything on one canvas: the dial $\lambda$ running from $1$ down through $0.5$, through $0$ (=$\log$), to $-1$; a skewed input on the left; a symmetric output on the right; the "$-1,÷\lambda$" repair labelled where it lives.

Recall Feynman retelling — say it like a story

We had lopsided data: a crowd of small numbers and a few monsters stretching a long right tail. We wanted a knob that squashes monsters harder than the crowd. Raising every number to a power $\lambda$ is that knob — pick $\lambda <1$ and big numbers shrink far more than small ones. But the raw knob jams at $\lambda =0$: everything raised to the $0$ becomes $1$, wiping out our data. So we patched the knob: subtract $1$ (slide the jam-point down to $0$) and divide by $\lambda$ (blow the now-tiny differences back up to useful size). To find what the patched knob settles on as $\lambda$ creeps to $0$ (a limit — the value we're heading toward without landing on it), we hit $\frac{0}{0}$; L'Hôpital's rule says a $\frac{0}{0}$ fraction heads toward the ratio of its two slopes (derivatives), provided both pieces are smooth and the bottom's slope isn't zero — all true here. Since $x^{\lambda }=e^{\lambda \ln x}$ grows at rate $\ln x$ times itself, its slope is $x^{\lambda }\ln x$, and the ratio spits out exactly $\ln x$. So the natural logarithm isn't a separate tool bolted on; it's the $\lambda =0$ click of one smooth dial. The "$-1$" we subtract is a harmless constant — it never changes the shape/skew, only the mean. Finally: the dial only works for positive numbers; zeros/negatives need Yeo-Johnson, whose negative branch uses the conjugate exponent $2-\lambda$ and a sign-flip so the two halves join without a kink. And always learn $\lambda$ on training data alone so no test information leaks in.

Recall Quick self-test

Why does raw $x^{\lambda }$ fail at $\lambda =0$? ::: Because $x^0=1$ for every $x$, collapsing all data onto a single value and destroying all information. What do the "$-1$" and "$÷\lambda$" each accomplish? ::: The $-1$ re-centres the cluster (which hovers near $1$) down to near $0$; dividing by the tiny $\lambda$ rescales the resulting tiny differences back to finite size. In plain words, what is a limit? ::: The single value a function's outputs home in on as the input creeps toward a target, without needing to plug the target in. In plain words, what is a derivative? ::: The slope of the curve at a point — how fast the output changes when the input is nudged a hair. Why is $\frac{d}{d\lambda }{x}^{\lambda }=x^{\lambda }\ln x$? ::: Because $x^{\lambda }=e^{\lambda \ln x}$, and $e^{k\lambda }$ grows at rate $k$ times its height, with $k=\ln x$ here. What are L'Hôpital's hypotheses? ::: Both top and bottom $\to 0$ (form $\frac{0}{0}$), both differentiable near the point, the bottom's derivative is nonzero there, and the slope-ratio limit exists. What value does $\frac{x^{\lambda }-1}{\lambda }$ approach as $\lambda \to 0$? ::: $\ln x$. Box-Cox on $x=450$ with $\lambda =-0.1$? ::: $\frac{450^{-0.1}-1}{-0.1}\approx 4.52$. Why doesn't the "$-1$" affect skewness? ::: Skewness is invariant to adding a constant or multiplying by one; the $-1$ only shifts the mean/intercept, not the shape. On the negative branch of Yeo-Johnson, what exponent is used and why? ::: The conjugate exponent $2-\lambda$ (with a sign-flip), chosen so the value and slope match the positive branch at $x=0$ — no kink. When must you abandon Box-Cox for Yeo-Johnson? ::: When the data contains zeros or negative values, since Box-Cox requires $x>0$.

---

See also: Parent: Log and power transformations · Polynomial Features (the opposite move — adding bend) · Linear Regression Assumptions (why we un-bend at all).