Visual walkthrough — One-hot encoding and label encoding
This page rebuilds the central result of One-hot encoding and label encoding from the ground up. The parent note told you that one-hot encoding puts every category at an equal distance from every other. Here we are going to watch it happen — one picture per idea — until the geometry is obvious.
We assume you know nothing except: a number is a position on a line, and two numbers can be "far apart" or "close together". Everything else we build.
Step 1 — The problem: text can't go into a formula
WHY this step: We have to be honest about the very first fork in the road. There are two ways to turn categories into numbers, and they lead to completely different worlds. This whole page exists to show why one of them is dangerous and the other is safe.
PICTURE: On the left, three word-cards Red, Blue, Green. An arrow labelled "encode" points right to the two rival worlds we are about to explore.
Step 2 — The tempting shortcut: put them on one line
WHY we look at this: We must see the failure before we appreciate the fix. The failure is not a bug in the code — it's a hidden claim the numbers make for free.
PICTURE: A single horizontal number line. Red sits at 0, Blue at 1, Green at 2. Notice the gaps: the distance from Red to Blue is , but from Red to Green is .
The numbers now insist that Green is twice as far from Red as Blue is — and that Blue sits exactly between Red and Green. Nobody asked for that. It is a lie the encoding tells, and the model believes it.
Step 3 — Why the lie hurts: order sneaks in
WHY this step: This is the payoff of Step 2 — we show the concrete damage, so the reader feels why we need something better.
PICTURE: The same number line, now with a straight prediction line drawn through it, and a dashed tree-split at . The caption: "the model reads magnitude and order that were never in the data."
So we need a representation where no category is between any other, and every pair is equally far apart. One line cannot do that with three points. We need more room.
Step 4 — Give every category its own axis
WHY this tool — vectors, not bigger integers: A single number only has one dimension to move in, so ordering is unavoidable. A vector (a list of numbers = a point in space) lets us spread categories into separate directions that don't line up. This is exactly the freedom Step 3 said we needed.
PICTURE: A 3-D box with three axes labelled "Red-axis", "Blue-axis", "Green-axis". Three coloured dots sit one step out along each axis: Red at , Blue at , Green at .
In table form for the three colours — this is literally the one-hot table from the parent note:
| Colour | Red | Blue | Green |
|---|---|---|---|
| Red | 1 | 0 | 0 |
| Blue | 0 | 1 | 0 |
| Green | 0 | 0 | 1 |
Step 5 — Measure the distance and watch the magic
WHY distance — and why this exact formula: "Are two categories treated as similar?" is the question "how far apart are their vectors?" The natural ruler in space is the Euclidean distance (straight-line length), the multi-dimensional version of the we used on the line in Step 2. We use it because it's the same notion of "far apart" the model itself uses.
PICTURE: Zoom on two of the dots, say Red and Blue . The straight segment between them is drawn and labelled . A little inset shows the same appears for every pair.
Compare with Step 2: there Red–Green was and Red–Blue was — unequal, ordered. Here every pair is — equal, orderless. The lie is gone.
Step 6 — The linear model reads it cleanly
WHY this step: Step 3 showed one column forces the model onto a single sloped line. Here we show what several columns buy us: freedom for each category to have any effect, high or low, in any order.
PICTURE: Three separate horizontal weight-bars — — at heights 600k, 400k, 250k, with no line connecting them. Caption: "independent knobs, not a slope."
Step 7 — The degenerate case: the columns are secretly dependent
WHY this is its own step: This is the edge case the contract demands. It's not a rare accident; it happens on every one-hot dataset, so it must be shown.
PICTURE: A wide table where a green box highlights that each row's three cells sum to ; a red arrow points to a mirror-image "intercept" column of all s, showing they are the same thing in disguise.
Step 8 — The cost: rooms multiply (curse of dimensionality)
WHY the final step: Honesty about limits. One-hot fixes ordering but pays in width. Knowing when the price is too high is part of understanding the tool.
PICTURE: A small tidy 3-cube on the left ("3 categories, cozy") growing into a sprawling, near-empty high-dimensional grid on the right ("thousands of categories, sparse").
The one-picture summary
Everything above compressed: the number line (order lie, unequal gaps) on the left transforms into the symmetric triangle of basis vectors on the right, all sides , all angles right angles — no category between any other.
Recall Feynman retelling — say it back in plain words
A model is a formula, and formulas eat numbers, not words. The lazy way is to line the categories up — Red 0, Blue 1, Green 2 — but a line has an order and unequal gaps, so the model wrongly decides Blue sits between Red and Green and Green is "twice" Red. To kill that lie we give every category its own direction in space: Red points along one axis, Blue along another, Green a third. Measuring the straight-line distance between any two of these points always gives — every category equally far from every other, none in the middle. A linear model then reads each category through its own weight, cleanly. Two cautions: the columns always add to , which makes the weights slippery, so we drop one column and treat it as the baseline; and each new category adds an axis, so with thousands of categories the space blows up empty and we switch tools.
Recall Quick self-check
Distance between any two one-hot vectors ::: , the same for every pair — perfect symmetry. Why label encoding is dangerous for colours ::: It puts categories on one ordered line, inventing magnitude and "betweenness" that don't exist. Why we drop one one-hot column in linear regression ::: The columns sum to , so weights are non-identifiable (multicollinearity); the dropped category becomes the reference level. The main cost of one-hot encoding ::: One new dimension per category → curse of dimensionality for high-cardinality features.