Ordinal and target encoding

Overview

When you have categorical features with inherent ordering or high-cardinality categories, one-hot encoding becomes wasteful or even harmful. Ordinal encoding preserves order by mapping categories to integers, while target encoding replaces categories with statistics derived from the target variable—both are compact, powerful techniques when applied correctly.

Ordinal Encoding

For a feature $X$ with categories $\{c_1, c_2, \ldots, c_k\}$ where $c_1 < c_2 < \cdots < c_k$, we define: $f_{\text{ord}}(c_i) = i \quad \text{for } i \in \{1, 2, \ldots, k\}$

WHY: Linear models and tree-based models interpret $f(c_4) - f(c_2) = 2$ as "two steps apart", which matches the ordinal structure.

WHAT: You manually specify the order (or infer from domain knowledge), then replace each category with its rank.

HOW:

1. Identify features with natural order (education level, satisfaction rating, sizes S/M/L/XL)
2. Define the ordering explicitly
3. Map each category to its integer rank
4. Replace original values

$X_{\text{encoded}} = [f_{\text{ord}}(x_1), f_{\text{ord}}(x_2), \ldots, f_{\text{ord}}(x_n)]^T$

where each $x_i$ is a categorical value and $f_{\text{ord}}$ is the mapping function defined above.

Derivation from first principles:

• Goal: Represent ordinal categories numerically while preserving their relative ordering
• Constraint: The numeric representation must satisfy $a < b \implies f(a) < f(b)$
• Simplest solution: Assign consecutive integers starting from 1
• Result: $f(c_i) = i$ is the minimal bijective function that preserves order and uses the smallest range $[1, k]$

Dataset:

Education: [High School, Bachelor, Master, PhD, Bachelor, High School]

Step 1: Define order

High School < Bachelor < Master < PhD

Step 2: Create mapping

High School → 1
Bachelor    → 2
Master      → 3
PhD         → 4

Why this mapping? Because education level has clear progression—each step requires completing the previous one.

Step 3: Apply encoding

Result: [1, 2, 3, 4, 2, 1]

Why this works? A linear model learning coefficients like $\beta \cdot \text{Education}$ will correctly interpret that PhD (4) contributes more than Bachelor (2), and the difference is proportional.

Example 2: T-shirt Size

Size: [M, L, S, XL, M, S]

Step 1: Define order

S < M < L < XL

Step 2: Create mapping

S  → 1
M  → 2
L  → 3
XL → 4

Why start at 1? Starting at 0 or 1 doesn't matter for order-preservation, but 1-indexing is conventional and avoids potential issues with models that treat0 specially.

Step 3: Apply

Result: [2, 3, 1, 4, 2, 1]

Why this step? Each original category is looked up in the mapping dictionary and replaced with its integer rank.

Target Encoding

For a categorical feature $X$ and target $y$, the target encoding for category $c$ is:

$f_{\text{target}}(c) = \mathbb{E}[y \mid X = c] = \frac{\sum_{i: x_i = c} y_i}{\sum_{i: x_i = c} 1}$

WHY: High-cardinality features (many unique categories) explode with one-hot encoding. Target encoding compresses information while preserving predictive signal.

WHAT: Calculate the mean (or median, or any statistic) of the target variable for each category, then replace categories with those statistics.

HOW:

1. Split data into train/validation
2. For each category in train set, compute mean target value
3. Replace categories with their means
4. Handle unseen categories (global mean or special value)
5. Crucial: Use cross-validation or separate encoding data to avoid leakage

where $n_c$ is the count of category $c$ in the training set.

Smoothed Target Encoding (prevents overfitting on rare categories): $\text{TE}_{\text{smooth}}(c) = \frac{n_c \cdot \overline{y}_c + m \cdot \overline{y}_{\text{global}}}{n_c + m}$

where:

• $\overline{y}_c$ = mean target for category $c$
• $\overline{y}_{\text{global}}$ = global mean of target
• $n_c$ = count of category $c$
• $m$ = smoothing parameter (e.g., 10-100)

Derivation from first principles:

Goal: Create a single numerical feature that maximizes correlation with target while preventing overfitting.

Problem with raw mean: If category $c$ appears only once with $y=100$, we'd encode it as 100—but this is likely noise, not signal.

Solution: Bayesian smoothing toward the global mean.

Starting point: We want a weighted average between category mean and global mean.

$\text{TE}_{\text{smooth}}(c) = w \cdot \overline{y}_c + (1-w) \cdot \overline{y}_{\text{global}}$

What should $w$ be? It should reflect our confidence in $\overline{y}_c$. High $n_c$ → high confidence → $w \to 1$. Low $n_c$ → low confidence → $w \to 0$.

Natural choice: $w = \frac{n_c}{n_c + m}$ where $m$ controls the smoothing strength.

Substituting: $\text{TE}_{\text{smooth}}(c) = \frac{n_c}{n_c + m} \cdot \overline{y}_c + \frac{m}{n_c + m} \cdot \overline{y}_{\text{global}}$

Simplifying: $= \frac{n_c \cdot \overline{y}_c + m \cdot \overline{y}_{\text{global}}}{n_c + m}$

This is equivalent to adding $m$ "pseudo-observations" at the global mean—a classic empirical Bayes technique.

Training data:

City      | Price (k$)
-------|----------
NYC       | 500
NYC       | 600
NYC       | 550
Seattle   | 400
Seattle   | 450
Portland  | 300

Step 1: Compute mean price per city

NYC:      (500 + 600 + 550) / 3 = 550
Seattle:  (400 + 450) / 2 = 425
Portland: 300 / 1 = 300

Why this step? We're aggregating all training examples for each category to get a representative statistic.

Step 2: Replace categories with means

City      | Price | Encoded
----------|-------|--------
NYC       | 500   | 550
NYC       | 600   | 550
NYC       | 550   | 550
Seattle   | 400   | 425
Seattle   | 450   | 425
Portland  | 300   | 300

Why this works? The model now sees a numerical feature that directly correlates with the target. NYC's high encoding (550) tells the model "this city is expensive."

Step 3: Handle new city at test time

Test: [Boston, NYC, Seattle]

Boston is unseen. Use global mean = (550×3 + 425×2 + 300×1) / 6 = 475

Result: [475, 550, 425]

Why global mean for unseen? It's our best guess without data. Better alternatives: use validation set statistics or train a separate model for imputation.

Example 4: Smoothed Target Encoding

Suppose Portland appears only once. Raw encoding = 300 might be noise.

Apply smoothing with $m = 10$, global mean = 475:

$\text{TE}_{\text{smooth}}(\text{Portland}) = \frac{1 \cdot 300 + 10 \cdot 475}{1 + 10} = \frac{300 + 4750}{11} = \frac{5050}{11} \approx 459$

Why this step? We're shrinking the extreme value (300) toward the population mean (475) because we have low confidence (only 1 observation).

For NYC with 3 observations: $\text{TE}_{\text{smooth}}(\text{NYC}) = \frac{3 \cdot 550 + 10 \cdot 475}{13} = \frac{1650 + 4750}{13} \approx 492$

Why is this still close to 550? Because 3 observations give us more confidence, so $w = 3/13 \approx 0.23$ for the global mean vs0.77 for the category mean.

Wrong approach:

# Compute target encoding on ENTIRE dataset
encoding = df.groupby('city')['price'].mean()
df['city_encoded'] = df['city'].map(encoding)
# Then split train/test

Why this feels right: It's simpler—compute once, apply everywhere.

Why it's catastrophically wrong: The test set's target values leaked into the encoding. Your model sees future information. Validation accuracy will be inflated; real-world performance will crater.

The fix:

# Compute encoding ONLY on training data
encoding = train.groupby('city')['price'].mean()
train['city_encoded'] = train['city'].map(encoding)
test['city_encoded'] = test['city'].map(encoding).fillna(encoding.mean())

Why this works: The encoding is computed without seeing test targets. Unseen categories get the global mean.

Even better: Use k-fold target encoding where each fold gets encoded using statistics from the other folds.

Mistake 2: Not smoothing for rare categories

Wrong: Use raw mean even for categories with1-2 samples.

Why this feels right: "That's the true mean for that category!"

Why it's wrong: With small samples, the mean is noisy. A category with 1 sample at $y=1000$ might just be an outlier.

The fix: Apply smoothing (formula above). Rare categories get pulled toward the global mean.

Why this works: It's a bias-variance tradeoff. We accept slight bias (shrinking toward global mean) to dramatically reduce variance (wild swings from noise).

Mistake 3: Using ordinal encoding for nominal categories

Wrong: Encode [Red, Blue, Green] as [1, 2, 3] when there's no natural order.

Why this feels right: "It's just numbers, the model will figure it out."

Why it's wrong: Linear models will learn that Blue (2) is "between" Red (1) and Green (3), which is nonsense. You've injected false ordinal structure.

The fix: Use one-hot encoding, target encoding, or embedings—but never ordinal for nominal.

When to Use Which

"Let's say you're rating ice cream flavors from 'yuck' to 'yummy'. If 'yuck' is 1, 'okay' is 2, 'good' is 3, and 'yummy' is 4, you've just done ordinal encoding! The numbers match the order of how much you like them.

Now imagine you have a list of 100 different ice cream shops, and you want to predict how much people will pay. Instead of making 100 columns (one for each shop), you do something clever: replace each shop's name with the average price people paid there. Expensive shops get high numbers, cheap shops get low numbers. That's target encoding—you're using what happened before to create a useful number."

TEAPOT for target encoding pitfalls:

• Test leakage (don't encode test in training stats)
• Extreme smoothing needed for rare categories
• Avoid for ordinal features (overkill)
• Predict mode: handle unseen categories
• Overfitting risk without CV
• Tree models love it (compact, informative)

Connections

• One-hot Encoding - Alternative for low-cardinality nominal features
• Feature Scaling - May need to scale ordinal/target encoded features for distance-based models
• Cross-Validation - Essential for preventing target encoding leakage
• Bayesian Methods - Smoothed target encoding is empirical Bayes
• Curse of Dimensionality - Target encoding mitigates this for high-cardinality features
• Overfitting - Target encoding can overfit without proper CV/smoothing
• Tree-based Models - Handle ordinal encoding naturally via splits

#flashcards/ai-ml

What is ordinal encoding? :: Mapping categorical values to integers according to a specified order, preserving the ordinal relationship between categories (e.g., Poor=1, Fair=2, Good=3).

When should you use ordinal encoding? :: When categorical features have inherent ordering (education level, satisfaction ratings, sizes S/M/L/XL) and you want the model to respect that order.

Why is target encoding useful for high-cardinality features? :: It avoids the dimensionality explosion of one-hot encoding (500 cities → 500 columns) while preserving predictive signal by using target statistics (500 cities → 1 column with mean target per city).

Compare ordinal vs target encoding use cases :: Ordinal: inherent order (ratings, sizes, education). Target: high-cardinality nominal with predictive patterns (cities, user IDs, product SKUs). One-hot: low-cardinality nominal (colors, gender).

Concept Map

Hinglish (regional understanding)