Ordinal and target encoding

2.1.7 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Jab aapke paas categorical features with inherent ordering hon ya high-cardinality categories hon, tab one-hot encoding wasteful ya harmful bhi ban jaati hai. Ordinal encoding order ko preserve karta hai categories ko integers mein map karke, jabki target encoding categories ko target variable se derive kiye gaye statistics se replace karta hai—dono compact, powerful techniques hain jab sahi tarike se apply ki jayein.

Ordinal Encoding

Ek feature $X$ ke liye jisme categories $\{c_1,c_2,\dots ,c_k\}$ hain aur $c_1<c_2<\cdots <c_k$, hum define karte hain: $f_{\text{ord}}(c_i)=i\text{for }i\in \{1,2,\dots ,k\}$

WHY: Linear models aur tree-based models $f(c_4)-f(c_2)=2$ ko "do steps apart" interpret karte hain, jo ordinal structure se match karta hai.

WHAT: Aap manually order specify karte ho (ya domain knowledge se infer karte ho), phir har category ko uske rank se replace karte ho.

HOW:

1. Natural order wale features identify karo (education level, satisfaction rating, sizes S/M/L/XL)
2. Ordering explicitly define karo
3. Har category ko uske integer rank se map karo
4. Original values replace karo

$X_{\text{encoded}}=[f_{\text{ord}}(x_1),f_{\text{ord}}(x_2),\dots ,f_{\text{ord}}(x_n){]}^T$

jahan har $x_i$ ek categorical value hai aur $f_{\text{ord}}$ upar define ki gayi mapping function hai.

First principles se derivation:

• Goal: Ordinal categories ko numerically represent karo unka relative ordering preserve karte hue
• Constraint: Numeric representation yeh satisfy kare: $a<b⟹f(a)<f(b)$
• Simplest solution: 1 se consecutive integers assign karo
• Result: $f(c_i)=i$ sabse minimal bijective function hai jo order preserve karta hai aur smallest range $[1,k]$ use karta hai

Dataset:

Education: [High School, Bachelor, Master, PhD, Bachelor, High School]

Step 1: Order define karo

High School < Bachelor < Master < PhD

Step 2: Mapping create karo

High School → 1
Bachelor    → 2
Master      → 3
PhD         → 4

Yeh mapping kyun? Kyunki education level ki clear progression hai—har step ke liye previous step complete karna zaroori hai.

Step 3: Encoding apply karo

Result: [1, 2, 3, 4, 2, 1]

Yeh kaam kyun karta hai? Ek linear model jo $\beta \cdot \text{Education}$ jaisi coefficients seekhta hai, woh sahi interpret karega ki PhD (4) Bachelor (2) se zyada contribute karta hai, aur difference proportional hai.

Example 2: T-shirt Size

Size: [M, L, S, XL, M, S]

Step 1: Order define karo

S < M < L < XL

Step 2: Mapping create karo

S  → 1
M  → 2
L  → 3
XL → 4

1 se start kyun? 0 ya 1 se start karna order-preservation ke liye matter nahi karta, lekin 1-indexing conventional hai aur un models ke saath potential issues avoid karta hai jo 0 ko specially treat karte hain.

Step 3: Apply karo

Result: [2, 3, 1, 4, 2, 1]

Yeh step kyun? Har original category ko mapping dictionary mein lookup kiya jaata hai aur uske integer rank se replace kiya jaata hai.

Target Encoding

Ek categorical feature $X$ aur target $y$ ke liye, category $c$ ka target encoding hai:

$f_{\text{target}}(c)=\mathbb{E}[y\mid X=c]=\frac{{\sum }_{i:x_i=c}{y}_i}{{\sum }_{i:x_i=c}1}$

WHY: High-cardinality features (bahut saari unique categories) one-hot encoding se explode ho jaati hain. Target encoding information compress karta hai predictive signal preserve karte hue.

WHAT: Har category ke liye target variable ka mean (ya median, ya koi bhi statistic) calculate karo, phir categories ko un statistics se replace karo.

HOW:

1. Data ko train/validation mein split karo
2. Train set mein har category ke liye mean target value compute karo
3. Categories ko unke means se replace karo
4. Unseen categories handle karo (global mean ya special value)
5. Crucial: Leakage avoid karne ke liye cross-validation ya separate encoding data use karo

jahan $n_c$ training set mein category $c$ ka count hai.

Smoothed Target Encoding (rare categories par overfitting rokta hai): ${\text{TE}}_{\text{smooth}}(c)=\frac{n_c\cdot {\overline{y}}_c+m\cdot {\overline{y}}_{\text{global}}}{n_c+m}$

jahan:

• ${\overline{y}}_c$ = category $c$ ke liye mean target
• ${\overline{y}}_{\text{global}}$ = target ka global mean
• $n_c$ = category $c$ ka count
• $m$ = smoothing parameter (e.g., 10-100)

First principles se derivation:

Goal: Ek aisa single numerical feature banao jo target ke saath correlation maximize kare overfitting prevent karte hue.

Raw mean ki problem: Agar category $c$ sirf ek baar $y=100$ ke saath appear hoti hai, toh hum use 100 encode karenge—lekin yeh shayad noise hai, signal nahi.

Solution: Global mean ki taraf Bayesian smoothing.

Starting point: Hum category mean aur global mean ka weighted average chahte hain.

${\text{TE}}_{\text{smooth}}(c)=w\cdot {\overline{y}}_c+(1-w)\cdot {\overline{y}}_{\text{global}}$

$w$ kya hona chahiye? Yeh ${\overline{y}}_c$ mein hamara confidence reflect karna chahiye. High $n_c$ → high confidence → $w\to 1$. Low $n_c$ → low confidence → $w\to 0$.

Natural choice: $w=\frac{n_c}{n_c+m}$ jahan $m$ smoothing strength control karta hai.

Substituting: ${\text{TE}}_{\text{smooth}}(c)=\frac{n_c}{n_c+m}\cdot {\overline{y}}_c+\frac{m}{n_c+m}\cdot {\overline{y}}_{\text{global}}$

Simplifying: $=\frac{n_c\cdot {\overline{y}}_c+m\cdot {\overline{y}}_{\text{global}}}{n_c+m}$

Yeh global mean par $m$ "pseudo-observations" add karne ke equivalent hai—ek classic empirical Bayes technique.

Training data:

City      | Price (k$)
-------|----------
NYC       | 500
NYC       | 600
NYC       | 550
Seattle   | 400
Seattle   | 450
Portland  | 300

Step 1: Har city ka mean price compute karo

NYC:      (500 + 600 + 550) / 3 = 550
Seattle:  (400 + 450) / 2 = 425
Portland: 300 / 1 = 300

Yeh step kyun? Hum har category ke liye ek representative statistic paane ke liye saare training examples aggregate kar rahe hain.

Step 2: Categories ko means se replace karo

City      | Price | Encoded
----------|-------|--------
NYC       | 500   | 550
NYC       | 600   | 550
NYC       | 550   | 550
Seattle   | 400   | 425
Seattle   | 450   | 425
Portland  | 300   | 300

Yeh kaam kyun karta hai? Model ab ek numerical feature dekhta hai jo directly target se correlate karta hai. NYC ki high encoding (550) model ko batati hai "yeh city expensive hai."

Step 3: Test time par nayi city handle karo

Test: [Boston, NYC, Seattle]

Boston unseen hai. Global mean use karo = (550×3 + 425×2 + 300×1) / 6 = 475

Result: [475, 550, 425]

Unseen ke liye global mean kyun? Bina data ke yeh hamara best guess hai. Better alternatives: validation set statistics use karo ya imputation ke liye alag model train karo.

Example 4: Smoothed Target Encoding

Maano Portland sirf ek baar appear hoti hai. Raw encoding = 300 noise ho sakta hai.

$m=10$, global mean = 475 ke saath smoothing apply karo:

${\text{TE}}_{\text{smooth}}(\text{Portland})=\frac{1\cdot 300+10\cdot 475}{1+10}=\frac{300+4750}{11}=\frac{5050}{11}\approx 459$

Yeh step kyun? Hum extreme value (300) ko population mean (475) ki taraf shrink kar rahe hain kyunki hamara confidence low hai (sirf 1 observation).

NYC ke liye 3 observations ke saath: ${\text{TE}}_{\text{smooth}}(\text{NYC})=\frac{3\cdot 550+10\cdot 475}{13}=\frac{1650+4750}{13}\approx 492$

Yeh 550 ke itna close kyun hai? Kyunki 3 observations se zyada confidence milta hai, toh $w=3/13\approx 0.23$ global mean ke liye vs 0.77 category mean ke liye.

Galat approach:

# Compute target encoding on ENTIRE dataset
encoding = df.groupby('city')['price'].mean()
df['city_encoded'] = df['city'].map(encoding)
# Then split train/test

Yeh sahi kyun lagta hai: Yeh simple hai—ek baar compute karo, har jagah apply karo.

Yeh catastrophically galat kyun hai: Test set ki target values encoding mein leak ho gayi. Tumhara model future information dekhta hai. Validation accuracy inflated hogi; real-world performance giregi.

Fix:

# Compute encoding ONLY on training data
encoding = train.groupby('city')['price'].mean()
train['city_encoded'] = train['city'].map(encoding)
test['city_encoded'] = test['city'].map(encoding).fillna(encoding.mean())

Yeh kaam kyun karta hai: Encoding test targets dekhe bina compute ki jaati hai. Unseen categories ko global mean milta hai.

Aur bhi better: k-fold target encoding use karo jahan har fold doosre folds ke statistics se encode hota hai.

Mistake 2: Rare categories ke liye smoothing na karna

Galat: Raw mean use karo chahe category mein 1-2 samples hi hon.

Yeh sahi kyun lagta hai: "Woh us category ka true mean hai!"

Yeh galat kyun hai: Chhote samples ke saath, mean noisy hota hai. $y=1000$ ke saath 1 sample wali category sirf ek outlier ho sakta hai.

Fix: Smoothing apply karo (upar ka formula). Rare categories global mean ki taraf pull hoti hain.

Yeh kaam kyun karta hai: Yeh bias-variance tradeoff hai. Hum slight bias accept karte hain (global mean ki taraf shrinking) taaki variance dramatically reduce ho (noise se wild swings).

Mistake 3: Nominal categories ke liye ordinal encoding use karna

Galat: [Red, Blue, Green] ko [1, 2, 3] encode karo jab koi natural order na ho.

Yeh sahi kyun lagta hai: "Yeh sirf numbers hain, model figure out kar lega."

Yeh galat kyun hai: Linear models seekhenge ki Blue (2) Red (1) aur Green (3) ke "beech" hai, jo nonsense hai. Tumne false ordinal structure inject kar di hai.

Fix: One-hot encoding, target encoding, ya embeddings use karo—lekin nominal ke liye kabhi ordinal mat use karo.

When to Use Which

"Maano tum ice cream flavors ko 'yuck' se 'yummy' tak rate kar rahe ho. Agar 'yuck' 1 hai, 'okay' 2 hai, 'good' 3 hai, aur 'yummy' 4 hai, toh tumne abhi ordinal encoding kar li! Numbers tumhari pasand ke order se match karte hain.

Ab socho tumhare paas 100 different ice cream shops ki list hai, aur tum predict karna chahte ho ki log kitna pay karenge. 100 columns banane ki jagah (har shop ke liye ek), tum kuch clever karte ho: har shop ka naam us shop par logon ne average mein kitna pay kiya se replace karo. Expensive shops ko high numbers milte hain, cheap shops ko low numbers. Yahi target encoding hai—tum jo pehle hua tha use ek useful number banane ke liye use kar rahe ho."

TEAPOT for target encoding pitfalls:

• Test leakage (test ko training stats mein encode mat karo)
• Extreme smoothing rare categories ke liye zaroori hai
• Avoid for ordinal features (overkill)
• Predict mode: unseen categories handle karo
• Overfitting risk without CV
• Tree models love it (compact, informative)

Connections

• One-hot Encoding - Low-cardinality nominal features ke liye alternative
• Feature Scaling - Distance-based models ke liye ordinal/target encoded features ko scale karna pad sakta hai
• Cross-Validation - Target encoding leakage rokne ke liye essential
• Bayesian Methods - Smoothed target encoding empirical Bayes hai
• Curse of Dimensionality - Target encoding high-cardinality features ke liye isko mitigate karta hai
• Overfitting - Target encoding bina proper CV/smoothing ke overfit kar sakta hai
• Tree-based Models - Ordinal encoding ko splits ke through naturally handle karte hain

#flashcards/ai-ml

Ordinal encoding kya hai? :: Categorical values ko integers mein mapping karna ek specified order ke according, categories ke beech ordinal relationship preserve karte hue (e.g., Poor=1, Fair=2, Good=3).

Ordinal encoding kab use karni chahiye? :: Jab categorical features ki inherent ordering ho (education level, satisfaction ratings, sizes S/M/L/XL) aur tum chahte ho ki model us order ko respect kare.

Target encoding high-cardinality features ke liye kyun useful hai? :: Yeh one-hot encoding ki dimensionality explosion avoid karta hai (500 cities → 500 columns) jabki predictive signal preserve karta hai target statistics use karke (500 cities → 1 column with mean target per city).

Ordinal vs target encoding use cases compare karo :: Ordinal: inherent order (ratings, sizes, education). Target: predictive patterns wale high-cardinality nominal (cities, user IDs, product SKUs). One-hot: low-cardinality nominal (colors, gender).

Concept Map