One-hot encoding and label encoding — Hinglish notes

2.1.6 · AI-ML › Data Preprocessing & Feature Engineering

Overview

Jab machine learning algorithms ko categorical data milta hai jaise "Red", "Blue", "Green" ya "Dog", "Cat", "Bird", toh woh directly text process nahi kar sakте. Unhe numbers chahiye. Lekin categories ko numbers mein convert karne ka tarika model performance ko drastically affect karta hai. Do fundamental approaches hain: label encoding (integers assign karna) aur one-hot encoding (binary indicator columns banana).

Sabse critical sawaal: Conversion method kyun matter karta hai? Kyunki zyaadatar ML algorithms numbers ko magnitude aur order ke saath interpret karte hain. Agar hum naively Red=0, Blue=1, Green=2 assign kar dein, toh model mathematically yeh sochta hai ki Green > Blue > Red, jo ki ek jhootha relationship create karta hai.

Label Encoding

Worked example Label Encoding: T-Shirt Sizes

Dataset: Customer orders with sizes | Order ID | Size | Label Encoded | |----------|---------------| | 1 | S | 0 | | 2 | M | 1 | | 3 | L | 2 | | 4 | M | 1 | | 5 | XL | 3 |

Step-by-step:

Unique categories identify karo: {S, M, L, XL}
Natural order se sort karo: S < M < L < XL
Integers assign karo: S=0, M=1, L=2, XL=3 Yeh step kyun? Order preserve karne se model seekh sakta hai "XL customers S customers se zyaada spend karte hain" agar woh pattern data mein hai.

Python:

from sklearn.preprocessing import LabelEncoder
 
sizes = ['S', 'M', 'L', 'M', 'XL']
encoder = LabelEncoder()
encoded = encoder.fit_transform(sizes)
# Result: [0, 1, 2, 1, 3]

Common mistake The Nominal Trap

Galat: Nominal (koi order nahi) categories jaise colors ya countries par label encoding use karna.

# BAD: Encoding countries
# USA=0, India=1, Brazil=2

Kyun sahi lagta hai: Simple hai! Ek column, integers, done.

Problem: Model Brazil=2 dekhta hai aur seekhta hai $Brazil = 2 \times India$ ya $Brazil = USA + India$ . Yeh mathematical nonsense hai. Decision trees aise splits banayenge jaise "if country ≥ 1.5", matlab India aur Brazil ek doosre se USA se zyaada similar hain — sirf arbitrary encoding order ki wajah se.

Fix: Nominal categories ke liye one-hot encoding use karo (neeche dekho).

One-hot Encoding

Worked example One-hot Encoding: Country of Origin

Dataset: Products with country labels

| Product | Country | USA | India | Brazil | |------|------|-----|--------| | A | USA | 1 | 0 | 0 | | B | India | 0 | 1 | 0 | | C | Brazil | 0 | 0 | 1 | | D | India | 0 | 1 | 0 | | E | USA | 1 | 0 | 0 |

Step-by-step:

Unique categories identify karo: {USA, India, Brazil} → $n = 3$
3 nayi columns banao (ek har category ke liye)
Har row ke liye, us row ki category se match karne wali column ko 1 set karo, baaki 0

Yeh step kyun? Har country ko apna feature milta hai. Linear model alag weights seekhega: $y = w_{1} \cdot USA + w_{2} \cdot India + w_{3} \cdot Brazil$ . Koi jhootha "USA < India < Brazil" relationship nahi.

Python:

import pandas as pd
df = pd.DataFrame({'Country': ['USA', 'India', 'Brazil', 'India', 'USA']})
encoded = pd.get_dummies(df, columns=['Country'], prefix='Country')
# Result:
#    Country_Brazil  Country_India  Country_USA
# 0              1
# 1       0              1             0
# 2       1              0             0
# 3       0              1             0
# 4       0              0             1

Common mistake The Dummy Variable Trap

Galat: Linear regression mein saari $n$ one-hot encoded columns include karna.

Kyun sahi lagta hai: "Zyaada features = model ke liye zyaada information!"

Problem — Multicollinearity: $n$ columns ke saath, humein ek perfect linear dependency milti hai: $USA + India + Brazil = 1$ (saari rows ka sum 1 hota hai) Linear regression $y = X w$ mein, design matrix $X$ ka rank $< n$ hota hai, jisse $X^{T} X$ non-invertible ho jaata hai. Normal equations $X^{T} X w = X^{T} y$ ke infinitely many solutions hote hain.

Mathematical proof: Agar $\sum_{i = 1}^{n} x_{i} = 1$ , toh kisi bhi coefficient vector $w$ ke liye: $y = w_{1} x_{1} + \dots + w_{n} x_{n} = (w_{1} + c) x_{1} + \dots + (w_{n} + c) x_{n} - c (x_{1} + \dots + x_{n}) = (w_{1} + c) x_{1} + \dots + (w_{n} + c) x_{n} - c$

Kisi bhi constant $c$ ke liye, hum use intercept mein absorb kar sakte hain. Weights non-identifiable hote hain.

Fix: Ek category drop karo (typically pehli). $n - 1$ dummy variables use karo. Drop ki gayi category reference level ban jaati hai jo intercept term mein absorb ho jaati hai.

Example: India aur Brazil columns rakho, USA drop karo. Ab coefficients ka matlab hai "USA ke relative change."

Comparison Table

Aspect	Label Encoding	One-hot Encoding
Output	Single integer column	$n$ binary columns (ya $n - 1$ drop ke saath)
Use case	Ordinal categories	Nominal categories
Dimensionality	Increase nahi	$n - 1$ se increase hoti hai
Model interpretation	Order/magnitude assume karta hai	Koi assumption nahi, equal distance
Tree models	Theek kaam karta hai (trees waise bhi thresholds par split karte hain)	Kam efficient (zyaada splits chahiye)
Linear models	Nominal data ke liye dangerous	Nominal data ke liye zaroori
Example	Education level, rating scale	Country, color, product type

Kab Kise Use Karein

Implementation Notes

Recall 12 Saal Ke Bacche Ko Samjhao

Socho tum apne Pokemon cards sort kar rahe ho. Label encoding aisa hai jaise unhe kitne strong hain uske hisaab se numbers dena: Pikachu=1, Charizard=2, Mewtwo=3. Numbers ka matlab hota hai — bada number = stronger Pokemon.

One-hot encoding aisa hai jaise alag alag pile banana: ek pile Fire type ke liye, ek Water type ke liye, ek Electric type ke liye. Har card exactly ek pile mein jaata hai. Piles ka koi order nahi hota — Fire, Water se "bada" nahi hai, woh bas alag types hain.

Agar tum computer ko bolo "Pikachu=1, Charizard=2", toh woh soch sakta hai "Charizard, Pikachu ka do guna hai!" Yeh uss cheez ke liye bakwaas hai jinka koi order nahi hota. Toh types ke liye (Fire, Water, Electric), hum alag piles wali method use karte hain. Har Pokemon ko ek pile mein "haan" milta hai aur baaki mein "nahi": [Fire: yes, Water: no, Electric: no] Charizard ke liye.

Connections

Feature Scaling: One-hot encoded features already scaled hote hain (binary 0/1)
Curse of Dimensionality: One-hot encoding feature space badhata hai, kuch models ke liye hurt kar sakta hai
Regularization: L1/L2 regularization one-hot encoded high-cardinality features ke saath help karta hai
Target Encoding: High-cardinality nominal variables ke liye alternative
Decision Trees: Label encoding ko naturally recursive partitioning se handle karte hain
Linear Regression: Nominal ke liye one-hot chahiye, dummy variable trap se bachna
Feature Engineering: Encoding, categorical feature engineering ka pehla step hai

#flashcards/ai-ml

Label encoding kya hai? :: Har unique category ko ek alag integer se map karna (jaise Red=0, Blue=1, Green=2). Ek single integer column create karta hai.

One-hot encoding kya hai?

n categories ke liye n binary columns banana, jahan har sample ke liye exactly ek column 1 hota hai aur baaki 0 hote hain. Ise dummy variable encoding bhi kehte hain.

Label encoding kab use karni chahiye?

Ordinal categories ke liye jinka natural order ho (jaise education level: High School < Bachelor's < Master's), ya tree-based models ke saath nominal data ke liye bhi.

One-hot encoding kab use karni chahiye?

Nominal categories ke liye jinka natural order nahi hota (jaise country, color), khaaskar linear ya distance-based models ke saath.

Linear models mein nominal data ke liye label encoding kyun dangerous hai?

Model integers ko magnitude/order ke saath interpret karta hai. Agar USA=0, India=1, Brazil=2, toh model sochta hai Brazil = 2×India ya Brazil aur India, USA aur India se zyaada similar hain, jo bilkul bakwaas hai.

Dummy variable trap kya hai?

Saari n one-hot encoded columns include karne se perfect multicollinearity create hoti hai (columns ka sum 1 hota hai), jisse linear regression mein design matrix singular ho jaata hai. Ek column ko reference ke liye drop karo.

One-hot encoded categories equidistant kyun hoti hain?

Har category ek standard basis vector hai. Kisi bhi do ke beech distance: ||e_i - e_j||₂ = √2 sabhi i≠j ke liye. Saari categories geometrically equally "door" hoti hain.

High cardinality nominal variables ka solution kya hai?

Target encoding (har category ke liye target ka mean), frequency encoding, embeddings, ya categories ko broad classes mein group karna. One-hot encoding bahut zyaada dimensions create kar deta hai.

Tree-based models label encoding ke saath nominal data ke liye kyun kaam karte hain?

Trees thresholds par split karte hain (if encoded_value ≤ 2.5) recursive partitioning ke through. Woh encoded integers aur target ke beech linear relationships assume nahi karte.

n categories wali one-hot encoding mein linear regression ko kitne columns use karne chahiye?

n-1 columns. Reference level ke roop mein ek drop karo taaki multicollinearity na ho. Drop ki gayi category ka effect intercept mein absorb ho jaata hai.

Ordinal data kya hota hai?

Categorical data jiska natural order/ranking hota hai (jaise T-shirt sizes S < M < L < XL, ya ratings Poor < Fair < Good < Excellent).

Nominal data kya hota hai?

Categorical data jiska koi natural order nahi hota (jaise colors, countries, product names). Categories sirf alag alag labels hoti hain.