One-hot encoding and label encoding: Exercises

This is a self-testing page. Each problem is stated cleanly; the full worked solution hides inside a collapsible callout — try first, then reveal. Problems climb five levels:

Everything here builds on the parent note. If a term feels unfamiliar, revisit it there first.

Before we start, one picture ties the whole page together — the geometric difference between the two encodings.

Figure — One-hot encoding and label encoding

Look at the left panel: label encoding drops the three colours onto a single line at positions $0, 1, 2$ . "Green" ( $2$ ) sits twice as far from "Red" ( $0$ ) as "Blue" ( $1$ ) does — an ordering the colours never asked for. On the right, one-hot encoding lifts each colour onto its own axis; the three points form a triangle where every side has the same length $2$ . That equal-distance symmetry is the heart of one-hot encoding, and several exercises below make you compute it.

Level 1 — Recognition

Recall Solution

The test question is always: "Does one category being 'more' than another make real-world sense?"

(a) Ordinal. Five stars really is better than one. Label-encode ${0, 1, 2, 3, 4}$ — the order is meaningful.
(b) Nominal. Blood type A is not "less than" O. One-hot encode → 4 columns.
(c) Ordinal. Extra Hot > Hot > Medium > Mild. Label-encode ${0, 1, 2, 3}$ .
(d) Nominal. No language is numerically "greater." One-hot encode → 3 columns.

Rule of thumb: if you can honestly write a "<" between the categories, label encoding is safe; otherwise use one-hot.

Recall Solution

LabelEncoder first collects the sorted unique labels: ['Blue','Green','Red'], giving Blue $= 0$ , Green $= 1$ , Red $= 2$ . Now map each original element: Red $\to 2$ , Blue $\to 0$ , Green $\to 1$ , Blue $\to 0$ . Output: [2, 0, 1, 0]. (Note it did not use first-seen order — that trips people up in L2.)

Level 2 — Application

Recall Solution

Apply the mapping element by element: M $\to 1$ , XL $\to 3$ , S $\to 0$ , L $\to 2$ , M $\to 1$ . Output: [1, 3, 0, 2, 1]. Because the sizes are ordinal, these numbers legitimately encode "bigger shirt = bigger number."

Recall Solution

Each row is a standard basis vector — a single $1$ in the slot matching its category:

Row	USA	India	Brazil
India	0	1	0
USA	1	0	0
Brazil	0	0	1
India	0	1	0

Sanity check: every row sums to exactly 1 (a category is present, so one indicator fires). That row-sum fact returns to bite us in L3.

Recall Solution

One-hot encoding turns a feature with $n$ categories into $n$ columns. Add them: $4 + 7 + 2 = 13 columns.$ (Compare: label encoding would keep just $3$ columns. This gap is exactly what the Curse of Dimensionality warns about.)

Level 3 — Analysis

Recall Solution

" $\geq 1.5$ " sends Brazil ( $2$ ) right and {USA $0$ , India $1$ } left. The tree is forced to group USA with India against Brazil, purely because a single threshold on the integer line can only cut contiguous codes. False claim embedded: that USA and India are "more similar" than USA and Brazil — an artifact of the arbitrary numbering, not the data. With one-hot columns the tree could split on "is Brazil?" directly, no forced grouping.

Recall Solution

Take $i \neq = j$ . The vector $e_{i} - e_{j}$ has a $+ 1$ in slot $i$ , a $- 1$ in slot $j$ , and $0$ everywhere else. Squared length = sum of squared entries: $∥ e_{i} - e_{j} ∥_{2}^{2} = (+ 1)^{2} + (- 1)^{2} = 1 + 1 = 2.$ So $∥ e_{i} - e_{j} ∥_{2} = 2 \approx 1.414$ for every pair. No category is closer to another — exactly the symmetry the right panel of the figure shows.

Recall Solution

On the integer line, distance is just $∣ a - b ∣$ :

Red–Blue $= ∣0 - 1∣ = 1$
Blue–Green $= ∣1 - 2∣ = 1$
Red–Green $= ∣0 - 2∣ = 2$

Unwanted structure: Red and Green are pushed twice as far apart as Red and Blue — the model now believes Blue is "between" Red and Green, and that Green is the "extreme" colour. None of that is true for nominal colours. This is precisely why we one-hot instead.

Level 4 — Synthesis

Recall Solution

Every row sums to 1 (from L2.2), so as columns: $USA + India + Brazil = 1,$ which is exactly the intercept column. One column is a linear combination of the others → the design matrix's columns are linearly dependent → $X^{⊤} X$ is non-invertible, so the normal equations have infinitely many solutions (weights are non-identifiable). Fix: drop one dummy (say USA) → keep $3 - 1 = 2$ columns. The dropped category becomes the reference level folded into the intercept. Alternatively, keep all columns but add Regularization, which restores a unique solution by penalising weight size.

Recall Solution

(a) $41, 000$ categories $\to 41, 000$ one-hot columns. (b) With so many columns, most are almost entirely zeros; the model has far more parameters than useful signal per column, distances become meaningless, and training/memory explode — the Curse of Dimensionality. Rare ZIPs may appear too few times to estimate a reliable weight (overfitting). (c) Use Target Encoding (replace each ZIP by the mean target for that ZIP) or group ZIPs into regions first — either collapses thousands of columns into one or a handful. Pair target encoding with Regularization/smoothing so rare ZIPs don't overfit.

Recall Solution

The intercept $600, 000$ is the reference (USA) prediction, because for a USA row both dummies are $0$ .

USA: $600, 000 - 200, 000 (0) - 350, 000 (0) = 600, 000$
India: $600, 000 - 200, 000 (1) - 350, 000 (0) = 400, 000$
Brazil: $600, 000 - 200, 000 (0) - 350, 000 (1) = 250, 000$

Each dummy's coefficient is a difference from USA, which is why dropped-category models are read "relative to the reference."

Level 5 — Mastery

Recall Solution

Condition → order-preserving label encoding ${0, 1, 2, 3}$ → 1 column. It's genuinely ordinal, so the integer order carries real meaning and stays compact.
City → one-hot, and drop one dummy (plain linear regression needs invertibility) → $8 - 1 = 7$ columns. Cities are nominal; dropping one avoids the dummy-variable trap.
ZIP → do not one-hot ( $12, 000$ columns would swamp the model). Use Target Encoding → 1 column. Nominal but far too high-cardinality for one-hot.

Total encoded columns: $1 + 7 + 1 = 9$ . Justification thread: match ordinal→label, low-cardinality nominal→one-hot(drop one), high-cardinality nominal→target encoding.

Recall Solution

City: a tree splits on thresholds and can carve out any single category with a chain of splits, so you don't need to drop a dummy (no invertibility requirement, no intercept). One-hot with all 8 columns is fine; many tree libraries even accept the raw label-encoded integers because a sequence of splits can isolate each value — order is irrelevant to a tree that can cut anywhere.
ZIP: the dimensionality problem persists — 12,000 one-hot columns still explode memory and starve rare categories of data. Keep Target Encoding (with smoothing so rare ZIPs don't overfit). Trees relax the multicollinearity concern but not the cardinality concern.

Takeaway: the multicollinearity fix (drop-one) is model-specific; the high-cardinality fix (target encoding) is universal.

Recall Solution

Full model prediction for a row is $b + w_{A} A + w_{B} B + w_{C} C$ , where exactly one dummy is 1.

A-row: $b + w_{A}$
B-row: $b + w_{B}$
C-row: $b + w_{C}$

Drop-A model: $b^{'} + w_{B}^{'} B + w_{C}^{'} C$ .

A-row: $b^{'}$
B-row: $b^{'} + w_{B}^{'}$
C-row: $b^{'} + w_{C}^{'}$

Match term-by-term by choosing $b^{'} = b + w_{A}$ , $w_{B}^{'} = w_{B} - w_{A}$ , $w_{C}^{'} = w_{C} - w_{A}$ . All three predictions coincide. So the drop-one model spans exactly the same prediction space — it removes redundancy, not capacity. $■$

Recall Self-test checklist (reveal after finishing)

Which encoding for ordinal data? ::: Order-preserving label encoding. Which encoding for low-cardinality nominal data? ::: One-hot (drop one column for plain linear models). Distance between any two distinct one-hot vectors? ::: $2$ . How many columns from $n$ categories, one-hot? ::: $n$ (or $n - 1$ after dropping the reference). Why does full one-hot break linear regression? ::: The dummy columns sum to the intercept, so $X^{⊤} X$ is singular. Fix for very high-cardinality nominal features? ::: Target encoding (with smoothing), not one-hot.

See also: Feature Engineering · Regularization · Curse of Dimensionality · Target Encoding.