One-hot encoding and label encoding: Question bank

True or false — justify

Every item is True or False, but the score is in the because. A bare T/F earns nothing.

T/F — Label encoding is always wrong for categorical data.

False. It is correct and preferred when the category has a genuine order (ordinal data like S < M < L), because the integers

0 < 1 < 2

then mean the real ranking. It is only wrong for nominal (orderless) data.

T/F — One-hot encoding never introduces a false ordering.

True. Each category becomes a standard basis vector

e_{i}

, and all pairs are equidistant (

∥ e_{i} - e_{j} ∥_{2} = 2

), so no category sits "between" two others. Order simply cannot be expressed.

T/F — For a decision tree, label encoding a nominal feature is just as safe as one-hot.

False in general. A tree splits on thresholds like "code

\geq 1.5

", which lumps together whatever categories happen to share a code range — an artifact of the arbitrary integer order, not the data. See Decision Trees.

T/F — One-hot encoding a variable with 10,000 unique values is a good default.

False. That creates 10,000 sparse columns, inflating dimensionality and inviting the Curse of Dimensionality and overfitting. High-cardinality features call for Target Encoding or hashing instead.

T/F — Keeping all

n

one-hot columns in ordinary linear regression is fine.

False. The columns sum to

1

every row, so

X^{T} X

is singular (the dummy variable trap) and the weights are non-identifiable. Drop one column or add regularization.

T/F — Dropping one one-hot column loses information.

False. The dropped reference level is recovered exactly: if a row is 0 in all

n - 1

kept columns, it must be the reference category. No information is lost, only redundancy.

T/F — Adding L2 Regularization lets you safely keep all

n

one-hot columns.

True. L2 makes

X^{T} X + λ I

invertible and picks the minimum-norm solution, so full one-hot is fine for ridge/lasso and neural nets. Dropping a column is mainly a classical OLS concern.

T/F — Label encoding and one-hot encoding produce the same result when

n = 2

Essentially true. Two categories → label encoding gives one column

{0, 1}

; one-hot-with-drop also gives one

{0, 1}

column. Order between two points is meaningless anyway, so both are equivalent.

T/F — LabelEncoder in scikit-learn is meant for feature columns.

False. LabelEncoder is documented for encoding the target

y

; using it on features silently imposes alphabetical order. Use OrdinalEncoder (features, chosen order) or OneHotEncoder instead.

Spot the error

Each line states a claim or code sketch containing a mistake. Name it.

"USA=0, India=1, Brazil=2, so I'll feed this integer column to my linear model."

The error is treating a nominal feature as ordinal. The model reads

Brazil = 2 \times India

as a real magnitude relation, which is nonsense; one-hot encode instead.

"I one-hot encoded, kept all 3 country columns, added an intercept, and ran OLS."

Multicollinearity. The three columns plus the intercept are linearly dependent (

USA + India + Brazil = 1

), so

X^{T} X

is not invertible. Drop one column.

"My encoder learned {S,M,L,XL}; at test time it saw 'XXL' and crashed."

The error is unseen categories at inference. Fit-time vocabulary is fixed; you must set handle_unknown='ignore' (one-hot) or reserve an "other" bucket, otherwise deployment breaks.

"I label-encoded colours alphabetically: Blue=0, Green=1, Red=2 — order doesn't matter since it's random."

It does matter: the model still reads Green as "between" Blue and Red. "Random/alphabetical" order is still an order, and it's a fake one. Nominal → one-hot.

"House price = 5000 − 100000 × Neighborhood, where Downtown=0, Suburbs=1, Rural=2."

The single slope forces equal-sized, monotone steps between neighborhoods — geometric nonsense for orderless regions. One-hot gives each neighborhood its own independent weight.

"I one-hot encoded, then also standardized the 0/1 columns to zero mean and unit variance."

Not strictly an error, but usually pointless and it destroys sparsity and interpretability of dummies; the binary 0/1 scale is already comparable. Standardize continuous features, leave dummies alone.

"I fit get_dummies on train and again separately on test."

Fitting separately can produce different columns if categories differ between splits, misaligning the feature matrix. Fit the encoder on train, then transform test with the same vocabulary.

Why questions

Reasoning-only. The answer is the mechanism, not a label.

Why does one-hot encoding make all categories equidistant?

Because each category is a distinct standard basis vector; the squared distance is

e_{i}^{T} e_{i} - 2 e_{i}^{T} e_{j} + e_{j}^{T} e_{j} = 1 - 0 + 1 = 2

for every pair, so no pair is "closer" than another.

Why does the dummy-variable trap only bite linear models (OLS), not trees or ridge?

OLS needs

X^{T} X

invertible; a linear dependency among columns destroys that. Trees split on one column at a time (dependency is irrelevant), and ridge adds

λ I

which restores invertibility.

Why is label encoding safe for tree models even on nominal data — with a caveat?

A deep tree can isolate any single category through repeated threshold splits, so it may recover the truth. The caveat: it wastes depth and can't cleanly separate non-adjacent codes, so one-hot or Target Encoding often still wins.

Why does dropping the first one-hot column not bias the model?

The dropped category's effect is folded into the intercept; the remaining coefficients become differences relative to that reference. The predictions are unchanged — only the interpretation of the numbers shifts.

Why can high-cardinality one-hot encoding hurt generalization?

Each rare category becomes a nearly-empty column that the model can memorize, and thousands of columns thin out the data per dimension — the Curse of Dimensionality. That raises variance and overfitting risk.

Why might Target Encoding be preferred over one-hot for a 50,000-category ID feature?

It replaces each category with a single number (e.g. the mean target for that category), collapsing 50,000 columns to one and capturing signal, at the cost of leakage risk that must be controlled with cross-fold encoding.

Why does one-hot encoding pair naturally with linear models but label encoding does not?

A linear model assigns one weight per column; one-hot gives every category its own weight (full flexibility), while a single label column forces all categories onto one shared slope, imposing a fake linear order.

Edge cases

Boundary and degenerate inputs — the ones tutorials skip.

A categorical feature with exactly one unique value in the whole dataset.

One-hot gives a constant column of all 1s; label encoding gives all 0s. Either way it carries zero information and can be dropped — a constant feature explains no variance.

A category that appears in test data but never in training.

The fitted encoder has no column/code for it. One-hot with handle_unknown='ignore' emits an all-zero row (treated as reference); otherwise you must map it to an "unknown" bucket to avoid a crash.

Encoding a feature that is ordinal but with unequal gaps (e.g. ratings 1★,2★,5★).

Label encoding assumes equal spacing between codes; if the real gaps are unequal, the linear model mis-weights them. Either encode the true numeric values or one-hot and let the model learn per-level effects.

An ordinal feature where the order is wrong or reversed in your mapping.

The model then learns the opposite monotonic trend, silently degrading fit. Unlike nominal mistakes it won't crash — it just quietly hurts, so verify the order explicitly (don't rely on alphabetical default).

A binary category ({Yes, No}) — one column or two?

One is enough. Two one-hot columns are perfectly anti-correlated (sum to 1), reintroducing the dummy trap; a single 0/1 indicator fully captures a two-value feature.

Missing values (NaN) inside a categorical column before encoding.

Neither encoder handles NaN meaningfully by default. Decide first: treat "missing" as its own category (a legitimate signal) or impute it — encoding a raw NaN either errors or silently drops the row.

A feature with hundreds of categories where only 3 are common.

Full one-hot wastes columns on rare tails that overfit. Group rare categories into an "other" bucket first, then one-hot — a form of Feature Engineering that trades a little fidelity for much better generalization.

Recall Self-test: the two-line summary

When is label encoding correct, in one sentence? ::: Only when the category has a genuine order, because the integer sizes then encode real ranking. What is the single most common one-hot bug in classical regression? ::: Keeping all $n$ columns with an intercept, causing the non-invertible dummy-variable trap; drop one column or regularize.