One-hot encoding and label encoding: Worked examples

This page is the practice arena for the parent topic. The parent told you what label and one-hot encoding are. Here we grind through every kind of situation a real dataset can throw at you — ordered data, unordered data, a category that never appears, a category the model has never seen, the trap of too many columns, and an exam-style twist.

Before we start, one promise: every symbol used below was defined in the parent, but we will re-anchor each one to a picture the first time it appears here so you never have to scroll back.

The scenario matrix

Think of encoding as a machine: you feed it a column of categories, it hands back numbers. The behaviour of that machine changes depending on the kind of input. This table lists every kind of input we must survive.

Cell	Scenario	Key question it tests	Right tool
A	Ordinal, clean data	Does order survive?	Label encoding
B	Nominal, few categories	Kill the fake order	One-hot
C	Nominal, high cardinality	Too many columns → Curse of Dimensionality	One-hot vs alternatives
D	Linear model, all $n$ columns kept	The dummy-variable trap	Drop-one
E	Unseen category at prediction time	Robustness to new data	handle_unknown
F	A category that appears zero times	Degenerate / empty column	Column pruning
G	Decision Trees vs Linear Regression	Does the model even care?	Tool depends on model
H	Exam twist: interpret a fitted coefficient	Reference level meaning	Drop-one + intercept

We now walk one worked example per cell (a couple share a figure). Each begins with a Forecast — pause and guess the answer before reading the steps.

Figure — One-hot encoding and label encoding

Look at the picture: on the left number line, Red–Blue–Green sit at $0, 1, 2$ — Blue is trapped between Red and Green, and Green is twice as far from Red as Blue is. On the right cube, each category is a corner; the three edges drawn are all the same length. That equal length is the $2$ the parent proved. Everything below is a consequence of choosing the left picture or the right one.

Cell A — Ordinal data, done right

Worked example Education level → integers

Statement: A survey column has values ["Bachelor's", "PhD", "High School", "Master's", "Bachelor's"]. The real-world order is High School < Bachelor's < Master's < PhD. Encode it so a model can use "more education" as a signal.

Forecast: Guess the five output numbers before reading on.

List the unique categories. {High School, Bachelor's, Master's, PhD} → $n = 4$ . Why this step? We must know how many distinct symbols exist before assigning any number; the count $n$ decides the range ${0, \dots, n - 1}$ .
Sort by the real order, not alphabetical. High School (0) < Bachelor's (1) < Master's (2) < PhD (3). Why this step? Label encoding only "works" (parent's word) when integer order matches real order. Alphabetical sort would put Bachelor's first and destroy the meaning. LabelEncoder in sklearn sorts alphabetically by default — so here you must pass the order manually (e.g. an OrdinalEncoder with categories=[...]).
Map the original list. Bachelor's→1, PhD→3, High School→0, Master's→2, Bachelor's→1.

Answer: $[1, 3, 0, 2, 1]$ .

Verify: The distinct outputs ${0, 1, 2, 3}$ are exactly ${0, \dots, n - 1}$ with $n = 4$ ✓, and the mapping is monotone: higher education ⇒ larger number ( $High School = 0 < PhD = 3$ ) ✓. See Feature Engineering for when to instead build a numeric "years of study" feature.

Cell B — Nominal, few categories

Worked example Colour → one-hot vectors

Statement: Encode ["Blue", "Green", "Red", "Green"]. Colours have no order.

Forecast: How many columns will appear, and what does row "Green" look like?

Unique set: {Blue, Green, Red}, $n = 3$ . Fix a column order alphabetically: [Blue, Green, Red]. Why this step? One-hot needs one column per category; the order of columns is arbitrary because we are deliberately refusing to encode order.
Each row → standard basis vector $e_{i}$ . Recall from the parent: $e_{i}$ is "all zeros except a single 1 in slot $i$ ." Blue= $e_{1} = [1, 0, 0]$ , Green= $e_{2} = [0, 1, 0]$ , Red= $e_{3} = [0, 0, 1]$ .
Stack the rows into a matrix: $X = 100001010010 \leftarrow Blue \leftarrow Green \leftarrow Red \leftarrow Green$

Verify — the equidistance claim. Take Blue and Red: $e_{1} - e_{3} = [1, 0, - 1]$ , and its length is $1^{2} + 0^{2} + (- 1)^{2} = 2$ . Try Blue–Green: $[1, - 1, 0]$ , length $2$ again ✓. All pairs give $2$ — nobody is "between" anybody. That is the whole point.

Cell C — High cardinality (the column explosion)

Worked example 10 000 ZIP codes

Statement: A dataset has a zip_code column with $12, 000$ distinct values. You one-hot encode it. What happens?

Forecast: Guess the number of new columns and one problem it causes.

Count columns. One column per distinct category ⇒ $12, 000$ new binary columns. Why this step? One-hot's cost is linear in the number of categories. Cheap for 3 colours, ruinous for ZIP codes.
Compute the sparsity. Each row has exactly one 1 among $12, 000$ entries, so the fraction of nonzeros is $1/12000 \approx 8.33 \times 1 0^{- 5}$ , i.e. about $0.0083%$ of the matrix is "on." Why this step? This near-empty matrix is what triggers the Curse of Dimensionality: distances become meaningless when dimensions vastly outnumber informative signal.
Pick a remedy. Options: group rare ZIPs into an "Other" bucket, or switch to Target Encoding (replace each ZIP with the mean target for that ZIP — one column instead of 12 000), paired with Regularization to stop overfitting.

Verify: $12, 000$ columns, density $= 1/12000$ . Sanity check: if instead there were 3 categories, density $= 1/3 \approx 0.333$ — far denser, no problem. The pathology is purely from large $n$ ✓.

Cell D — The dummy-variable trap in a linear model

Worked example Why

$X^{T} X$ becomes singular Statement: Using the Cell B matrix $X$ (all 3 colour columns) plus an intercept column of ones, show the columns are linearly dependent.

Forecast: Which single equation relates the columns?

Sum the three colour columns row by row. Column-Blue + Column-Green + Column-Red $= [1, 1, 1, 1]^{T}$ . Why this step? Every row is exactly one basis vector, so exactly one 1 per row ⇒ the row-sum is always 1.
Compare to the intercept column, which is also $[1, 1, 1, 1]^{T}$ . Why this step? We now have two identical columns — one is the sum of the others. That is a perfect linear dependency, the parent's $USA + India + Brazil = 1$ .
Consequence: the design matrix has rank $3$ , not $4$ , so $X^{T} X$ is not invertible and $w$ is non-identifiable.
Fix: drop one colour column (say Blue). Blue becomes the reference level, folded into the intercept.

Verify: Build $M = [1 ∣ Blue ∣ Green ∣ Red]$ . Its determinant / rank should show rank $= 3 < 4$ . After dropping Blue, the remaining $3$ columns are independent (rank $3$ ). Both checked in VERIFY. See Linear Regression.

Cell E — An unseen category at prediction time

Worked example The model meets a new country

Statement: You fit one-hot columns for {USA, India, Brazil}. At prediction time a row says Country = "Japan". What should the encoded row be?

Forecast: Should the model crash, or output something sensible?

The training vocabulary is fixed at {USA, India, Brazil} → 3 columns. "Japan" is not among them. Why this step? Encoders learn their column set from training data only; the vocabulary can't grow at inference.
Choose the policy handle_unknown="ignore" (sklearn): unknown category ⇒ all-zeros row $[0, 0, 0]$ . Why this step? All-zeros is the honest statement "none of the known categories." The linear model then predicts just the intercept (the reference-level baseline).
Contrast with the default handle_unknown="error", which raises an exception — safe for catching data bugs, unsafe for production.

Verify: The all-zeros vector $[0, 0, 0]$ has length $0$ and lies at the origin — not on any category corner. Distance from origin to any corner $e_{i}$ is $∥ e_{i} ∥ = 1 = 1$ , so "Japan" is symmetric to all known countries (equally far, $1$ , from each) ✓. Neatly consistent with "unknown = neutral."

The figure shows the origin (the unknown/all-zero point, plum) sitting equidistant from all three category corners — the geometric reason handle_unknown="ignore" is a fair default.

Cell F — A category that appears zero times

Worked example The ghost column

Statement: Your training data declares categories {S, M, L, XL} but the actual rows only ever contain S and M. What does one-hot produce, and what's the danger?

Forecast: How many columns, and which are all-zero?

Columns from the declared set: 4 columns S, M, L, XL. Why this step? If you pass an explicit category list, the encoder honours it even for categories with zero rows.
The L and XL columns are entirely zero across every row. Why this step? No row ever selects them, so their indicator never turns on.
Danger + fix: a constant (all-zero) column carries no information and adds a useless weight for the model to fit — pure Curse of Dimensionality cost with zero benefit. Prune zero-variance columns before modelling.

Verify: Variance of an all-zero column is $0$ . A feature with $0$ variance cannot correlate with anything ⇒ its regression weight is unidentifiable, same disease as Cell D. Checked numerically in VERIFY.

Cell G — Does the model even care? Trees vs linear

Worked example Same data, opposite advice

Statement: Categories {Downtown, Suburbs, Rural} encoded as label {0,1,2}. Explain why this can be fine for a decision tree but wrong for Linear Regression.

Forecast: Which model is fooled by the fake order — and which isn't?

Linear regression reads magnitude. It fits $Price = w \cdot code + b$ ; because $2 = 2 \times 1$ , it is forced to make Rural's effect exactly twice Suburbs'. That is the parent's "mathematical nonsense." Why this step? Linear models multiply the encoded number by a weight, so numeric spacing directly constrains the prediction.
A decision tree reads only order via thresholds. It splits on "code $\leq 0.5$ ?", "code $\leq 1.5$ ?". It can carve out any subset that respects the ordering, so a harmless label encoding costs it little — though it cannot freely group ${D o w n t o w n, R u r a l}$ against ${S u b u r b s}$ because the code order forbids that split. Why this step? Trees care about cut points, not distances, so the " $2$ equidistance" argument is irrelevant to them.
Rule of thumb: one-hot for linear/distance models; label (or ordinal) encoding is often acceptable — even preferred for speed — with tree ensembles.

Verify: With codes $0, 1, 2$ a linear model's constraint is $w_{Rural} = 2 w_{Suburbs}$ where $w_{c} = w \cdot c$ . Plugging: $w \cdot 2 = 2 (w \cdot 1)$ ✓ — the equality is unavoidable, confirming the linear model is straitjacketed.

Cell H — Exam twist: interpret a fitted coefficient

Worked example What does

$w_{Suburbs} = - 150, 000$ mean? Statement: After drop-one one-hot encoding (dropped category = Downtown, the reference), a house-price model fits: $Price = 600, 000 - 150, 000 \cdot Suburbs - 350, 000 \cdot Rural .$ Give the predicted price for each of the three neighbourhoods.

Forecast: What is a Downtown house predicted to cost — and why is there no Downtown term?

Downtown = reference. Set Suburbs=0, Rural=0: $Price = 600, 000$ . Why this step? The dropped category is absorbed into the intercept, so the intercept is the reference-level prediction.
Suburbs: set Suburbs=1, Rural=0: $600, 000 - 150, 000 = 450, 000$ . Why this step? Each retained coefficient is a difference from the reference, not an absolute price.
Rural: set Rural=1, Suburbs=0: $600, 000 - 350, 000 = 250, 000$ .

Answers: Downtown $=\$ 600{,}000 $, S u b u r b s$ =$450{,}000 $, R u r a l$ =$250{,}000$.

Verify: The coefficients are contrasts: Suburbs $-$ Downtown $= 450, 000 - 600, 000 = - 150, 000 = w_{Suburbs}$ ✓; Rural $-$ Downtown $= 250, 000 - 600, 000 = - 350, 000 = w_{Rural}$ ✓. All match. Regularization would shrink these contrasts toward zero if data is scarce.

Recall

Recall Why is one-hot equidistant but label encoding is not?

One-hot puts each category on a distinct axis, so every pair is $2$ apart ::: label encoding lines them on a number line, so distances (and order) are baked in unequally.

Recall What is the reference level in drop-one encoding?

The dropped category ::: its effect is absorbed into the intercept, and every remaining coefficient is a difference from it.

Recall Encoded value of an unseen category with handle_unknown="ignore"?

The all-zeros vector ::: it sits at the origin, equidistant (distance 1) from every known category corner.

Recall Why can label encoding be acceptable for trees but not linear regression?

Trees split on order via thresholds and ignore distance ::: linear regression multiplies the code by a weight, so fake numeric spacing forces spurious proportional effects.