One-hot encoding and label encoding

Overview

When machine learning algorithms receive categorical data like "Red", "Blue", "Green" or "Dog", "Cat", "Bird", they cannot process text directly. They need numbers. But how we convert categories to numbers drastically affects model performance. Two fundamental approaches exist: label encoding (assigning integers) and one-hot encoding (creating binary indicator columns).

The critical question: Why does the conversion method matter? Because most ML algorithms interpret numbers as having magnitude and order. If we naively assign Red=0, Blue=1, Green=2, the model thinks Green > Blue > Red mathematically, creating spurious relationships.

Label Encoding

Worked example Label Encoding: T-Shirt Sizes

Dataset: Customer orders with sizes | Order ID | Size | Label Encoded | |----------|---------------| | 1 | S | 0 | | 2 | M | 1 | | 3 | L | 2 | | 4 | M | 1 | | 5 | XL | 3 |

Step-by-step:

Identify unique categories: {S, M, L, XL}
Sort by natural order: S < M < L < XL
Assign integers: S=0, M=1, L=2, XL=3 Why this step? Preserving order lets the model learn "XL customers spend more than S customers" if that pattern exists in the data.

Python:

from sklearn.preprocessing import LabelEncoder
 
sizes = ['S', 'M', 'L', 'M', 'XL']
encoder = LabelEncoder()
encoded = encoder.fit_transform(sizes)
# Result: [0, 1, 2, 1, 3]

Common mistake The Nominal Trap

Wrong: Using label encoding on nominal (no order) categories like colors or countries.

# BAD: Encoding countries
# USA=0, India=1, Brazil=2

Why it feels right: It's simple! One column, integers, done.

The problem: The model sees Brazil=2 and learns $\text{Brazil} = 2 \times \text{India}$ or $\text{Brazil} = \text{USA} + \text{India}$ . This is mathematical nonsense. Decision trees will create splits like "if country≥ 1.5", implying India and Brazil are more similar to each other than to USA, purely because of arbitrary encoding order.

The fix: Use one-hot encoding for nominal categories (see below).

One-hot Encoding

Formula Mathematical Derivation from First Principles

Goal: Represent categorical variable with $n$ categories without implying order.

Approach 1 - Failed: Single integer column creates order.

Approach 2 - Correct: Use $n$ -dimensional representation where:

Each category maps to a unique vector
No vector is a linear combination of others (linear independence)
All categories are equidistant (symmetry)

Derivation: The standard basis $\{\mathbf{e}_1, \ldots, \mathbf{e}_n\}$ satisfies all three:

Unique: $\mathbf{e}_i \neq \mathbf{e}_j$ for $i \neq j$
Independent: $\mathbf{e}_i \cdot \mathbf{e}_j = \delta_{ij}$ (Kronecker delta)
Equidistant: $\|\mathbf{e}_i - \mathbf{e}_j\|_2^2 = (\mathbf{e}_i - \mathbf{e}_j)^T(\mathbf{e}_i - \mathbf{e}_j)$ $= \mathbf{e}_i^T\mathbf{e}_i - 2\mathbf{e}_i^T\mathbf{e}_j + \mathbf{e}_j^T\mathbf{e}_j = 1- 0 + 1 = 2$ Therefore: $\|\mathbf{e}_i - \mathbf{e}_j\|_2 = \sqrt{2}$ for all $i \neq j$ .

Worked example One-hot Encoding: Country of Origin

Dataset: Products with country labels

| Product | Country | USA | India | Brazil | |------|------|-----|--------| | A | USA | 1 | 0 | 0 | | B | India | 0 | 1 | 0 | | C | Brazil | 0 | 0 | 1 | | D | India | 0 | 1 | 0 | | E | USA | 1 | 0 | 0 |

Step-by-step:

Identify unique categories: {USA, India, Brazil} → $n=3$
Create 3 new columns (one per category)
For each row, set the column matching that row's category to 1, others to 0

Why this step? Each country gets its own feature. Linear model will learn separate weights: $y = w_1 \cdot \text{USA} + w_2 \cdot \text{India} + w_3 \cdot \text{Brazil}$ . No spurious "USA < India < Brazil" relationship.

Python:

import pandas as pd
df = pd.DataFrame({'Country': ['USA', 'India', 'Brazil', 'India', 'USA']})
encoded = pd.get_dummies(df, columns=['Country'], prefix='Country')
# Result:
#    Country_Brazil  Country_India  Country_USA
# 0              1
# 1       0              1             0
# 2       1              0             0
# 3       0              1             0
# 4       0              0             1

Worked example Real-World: Predicting House Prices by Neighborhood

Problem: Predict price from neighborhood (categorical).

Data: {Downtown, Suburbs, Rural}

With label encoding (WRONG):

Downtown=0, Suburbs=1, Rural=2
Model learns: $\text{Price} = 5000 - 100000 \times \text{Neighborhood}$
Nonsense interpretation: "Each step from Downtown→Suburbs→Rural decreases price by $100k." But Suburbs and Rural aren't "more similar" than Downtown and Rural!

With one-hot encoding (CORRECT):

Three columns: Downtown, Suburbs, Rural
Model learns: $\text{Price} = 6000 \cdot \text{Downtown} + 400000 \cdot \text{Suburbs} + 25000 \cdot \text{Rural}$
Sensible interpretation: Each neighborhood has independent effect on price. Downtown houses average $600k, Suburbs$ 400k, Rural $250k.

Common mistake The Dummy Variable Trap

Wrong: Including all $n$ one-hot encoded columns in linear regression.

Why it feels right: "More features = more information for the model!"

The problem - Multicollinearity: With $n$ columns, we have a perfect linear dependency: $\text{USA} + \text{India} + \text{Brazil} = \mathbf{1}$ (all rows sum to 1) In linear regression $\mathbf{y} = X\mathbf{w}$ , the design matrix $X$ has rank $< n$ , making $X^TX$ non-invertible. The normal equations $X^TX\mathbf{w} = X^T\mathbf{y}$ have infinitely many solutions.

Mathematical proof: If $\sum_{i=1}^n x_i = 1$ , then for any coefficient vector $\mathbf{w}$ : $y = w_1 x_1 + \cdots + w_n x_n = (w_1 + c) x_1 + \cdots + (w_n + c) x_n - c(x_1 + \cdots + x_n) = (w_1 + c) x_1 + \cdots + (w_n + c) x_n - c$

For any constant $c$ , we can absorb it into the intercept. The weights are non-identifiable.

The fix: Drop one category (typically the first). Use $n-1$ dummy variables. The dropped category becomes the reference level absorbed into the intercept term.

Example: Keep India and Brazil columns, drop USA. Now coefficients mean "change relative to USA."

Comparison Table

Aspect	Label Encoding	One-hot Encoding
Output	Single integer column	$n$ binary columns (or $n-1$ with drop)
Use case	Ordinal categories	Nominal categories
Dimensionality	No increase	Increases by $n-1$
Model interpretation	Assumes order/magnitude	No assumptions, equal distance
Tree models	Works fine (trees split on thresholds anyway)	Less efficient (more splits needed)
Linear models	Dangerous for nominal data	Necessary for nominal data
Example	Education level, rating scale	Country, color, product type

When to Use Each

Implementation Notes

Recall Explain Like I'm 12

Imagine you're sorting your Pokemon cards. Label encoding is like giving them numbers based on how strong they are: Pikachu=1, Charizard=2, Mewtwo=3. The numbers mean something—higher number = stronger Pokemon.

One-hot encoding is like having separate piles: one pile for Fire type, one for Water type, one for Electric type. Each card goes in exactly one pile. The piles don't have an order—Fire isn't "bigger" than Water, they're just different types.

If you tell a computer "Pikachu=1, Charizard=2", it might think "Charizard is twice as much as Pikachu!" That's silly for things that don't have order. So for types (Fire, Water, Electric), we use the separate piles method instead. Each Pokemon gets a "yes" in one pile and "no" in all the others: [Fire: yes, Water: no, Electric: no] for Charizard.

Connections

Feature Scaling: One-hot encoded features are already scaled (binary 0/1)
Curse of Dimensionality: One-hot encoding increases feature space, can hurt some models
Regularization: L1/L2 regularization helps with one-hot encoded high-cardinality features
Target Encoding: Alternative for high-cardinality nominal variables
Decision Trees: Handle label encoding naturally through recursive partitioning
Linear Regression: Requires one-hot for nominal, avoid dummy variable trap
Feature Engineering: Encoding is the first step in categorical feature engineering

#flashcards/ai-ml

What is label encoding? :: Mapping each unique category to a distinct integer (e.g., Red=0, Blue=1, Green=2). Creates a single integer column.

What is one-hot encoding?

Creating n binary columns for n categories, where exactly one column is 1 and others are 0 for each sample. Also called dummy variable encoding.

When should you use label encoding?

For ordinal categories with natural order (e.g., education level: High School < Bachelor's < Master's), or with tree-based models even for nominal data.

When should you use one-hot encoding?

For nominal categories without natural order (e.g., country, color) especially with linear or distance-based models.

Why is label encoding dangerous for nominal data with linear models?

The model interprets integers as having magnitude/order. If USA=0, India=1, Brazil=2, the model thinks Brazil = 2×India or that Brazil and India are more similar than USA and India, which is nonsense.

What is the dummy variable trap?

Including all n one-hot encoded columns creates perfect multicollinearity (columns sum to 1), making the design matrix singular in linear regression. Drop one column as reference.

Why are one-hot encoded categories equidistant?

Each category is a standard basis vector. Distance between any two: ||e_i - e_j||₂ = √2 for all i≠j. All categories are equally "far apart" geometrically.

What is the solution for high cardinality nominal variables?

Use target encoding (mean of target per category), frequency encoding, embedings, or group categories into broader classes. One-hot encoding creates too many dimensions.

Why do tree-based models work with label encoding even for nominal data?

Trees split on thresholds (if encoded_value ≤ 2.5) through recursive partitioning. They don't assume linear relationships between encoded integers and target.

In one-hot encoding with n categories, how many columns should linear regression use?

n-1 columns. Drop one as reference level to avoid multicollinearity. The dropped category effect is absorbed into the intercept.

What is ordinal data?

Categorical data with natural order/ranking (e.g., T-shirt sizes S< M < L < XL, or ratings Poor < Fair < Good < Excellent).

What is nominal data?

Categorical data without natural order (e.g., colors, countries, product names). Categories are just different labels.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Jab machine learning models ko categorical dataena ho jaise "Red", "Blue" ya "Dog", "Cat", toh directly text nahi samajh sakte. Unko numbers chahiye. Lekin yeh conversion kaisearo, yeh bohot matter karta hai! Do main tarike hain:

Label encoding mein har category ko ek integer assign karte hain—Red=0, Blue=1, Green=2. Yeh tab thek hai jab categories mein natural order ho, jaise T-shirt size (Small < Medium < Large). Lekin agar order nahi hai, toh model samajhega ki Green > Blue > Red mathematically, jo galat hai! Country names ke liye agar USA=0, India=1, Brazil=2 karo, toh model sochega Brazil aur India "zyada similar" hain USA ke comparison mein, jo nonsense hai.

One-hot encoding mein har category ke liye alag binary column banate hain. USA = [1,0,0], India = [0,1,0], Brazil = [0,0,1]. Ab sab countries equal distance pe hain geometrically (√2 distance), koi spurious ordering nahi hai! Linear models ke liye yeh zaroori hai. Lekin ek trick hai—agar 3 categories hain toh sirf 2 columns use karo (ek drop kar do), warna "dummy variable trap" mein fas jaoge jahan matrix invert nahi hogi.

Rule simple hai: agar order hai toh label encoding, agar sirf different categories hain (nominal) toh one-hot encoding. Tree-based models (Random Forest, XGBoost) label encoding handle karlete hain kyunki woh thresholds pe split karte hain, lekin linear regression ya KNN ke sath one-hot mandatory hai nominal data ke liye. High cardinality (bohot zyada categories) mein one-hot memory kharab karega, tab target encoding ya embedings use karo.

Overview

Label Encoding

One-hot Encoding

Comparison Table

When to Use Each

Implementation Notes

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Data Preprocessing & Feature Engineering

Connections