Types of data (numerical, categorical, ordinal, text)

Overview

Understanding data types is the foundation of feature engineering. Every ML algorithm makes assumptions about input data—feed it the wrong type and your model will either crash or silently produce garbage. This note explains what each data type represents, why the distinction matters for algorithms, and how to recognize and handle each type.

Core Data Types

1. Numerical Data

WHY the distinction matters:

• Continuous data → normalize/standardize for gradient-based models (prevents large-scale features from dominating)
• Discrete data → sometimes better treated as categorical if values are few (e.g., "number of bedrooms: 1, 2, 3" might work better one-hot encoded for tree models)

We want to transform data so it has mean = 0, standard deviation = 1.

Step 1: Remove the mean (centering) $x_{\text{centered}} = x - \mu$ where $\mu = \frac{1}{n}\sum_{i=1}^{n} x_i$ is the mean.

WHY? Centering shifts the distribution so its center is at zero. This removes location bias.

Step 2: Scale by standard deviation $x_{\text{scaled}} = \frac{x - \mu}{\sigma}$ where $\sigma = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2}$ is the standard deviation.

WHY? Dividing by $\sigma$ ensures the spread (variance) becomes 1. Now all features are on the same scale.

When to use: Linear regression, logistic regression, SVMs, neural networks (any algorithm using distance metrics or gradient descent).

Step 1: Calculate mean $\mu = \frac{50 + 75 + 60 + 90 + 55}{5} = \frac{330}{5} = 66$

WHY this step? We need the distribution's center to shift it to zero.

Step 2: Calculate standard deviation $\sigma = \sqrt{\frac{(50-66)^2 + (75-66)^2 + (60-66)^2 + (90-66)^2 + (55-66)^2}{5}}$ $= \sqrt{\frac{256 + 81 + 36 + 576 + 121}{5}} = \sqrt{\frac{1070}{5}} = \sqrt{214} \approx 14.63$

WHY this step? Standard deviation measures spread. We use it to normalize the scale.

Step 3: Scale each value $x_1 = \frac{50 - 66}{14.63} = \frac{-16}{14.63} \approx -1.09$ $x_2 = \frac{75 - 66}{14.63} = \frac{9}{14.63} \approx 0.62$ $x_3 = \frac{60 - 66}{14.63} \approx -0.41$ $x_4 = \frac{90 - 66}{14.63} \approx 1.64$ $x_5 = \frac{55 - 66}{14.63} \approx -0.75$

WHY this step? Now each value represents "how many standard deviations from the mean"—a universal scale.

Result: Scaled values: [-1.09, 0.62, -0.41, 1.64, -0.75] All values now have mean ≈ 0, std ≈ 1, making them comparable to other scaled features.

2. Categorical Data

KEY INSIGHT: You cannot compute "mean of colors" or "USA - India." These are labels, not numbers.

Problem: ML algorithms need numbers. How do we convert categories to numbers without implying fake order?

Solution: Create a binary column for each category.

Step 1: List all unique categories Categories: {Cat A, Cat B, Cat C}

Step 2: For each sample, create a binary vector

• If sample is Cat A: [1, 0]
• If sample is Cat B: [0, 1, 0]
• If sample is Cat C: [0, 0, 1]

WHY this works: Each category gets its own dimension. No category is "closer" to another in numeric space. Distance between any two one-hot vectors is the same (√2 in Euclidean space).

Mathematical representation: For $k$ categories, encode category $i$ as: $\mathbf{e}_i = [0, 0, \ldots, 1, \ldots, 0] \in \mathbb{R}^k$ where the 1 is in position $i$.

Step 1: Identify unique categories Categories: {UPI, Credit Card, Cash} →3 categories

WHY this step? We need to know how many binary columns to create.

Step 2: Create binary columns Create columns: is_UPI, is_CreditCard, is_Cash

Step 3: Encode each row

WHY this encoding?

• Transaction 1 is UPI → turn on UPI column, turn off others
• No category is numerically "between" others
• Distance from UPI to Cash = distance from UPI to Credit Card = √2

Alternative for tree models: Label encoding (UPI=0, Credit Card=1, Cash=2) works fine because trees split on individual values, not distances.

Why it feels right: "I converted text to numbers, the model can handle it!"

Why it's wrong: The model now thinks Cash (2) is "twice" Credit Card (1), and Credit Card is "between" UPI and Cash. It will learn weights like $w \cdot 2$ for Cash, implying mathematical relationships that don't exist.

The fix: Use one-hot encoding for algorithms that assume numeric relationships (linear models, neural nets). Use label encoding ONLY for tree-based models (which split on "is equal to X") ordinal data.

Steel-man the mistake: Label encoding saves memory (1 column vs. k columns) and can work if the model learns to ignore the fake ordering. But it's risky—why introduce a bias when one-hot is safer?

3. Ordinal Data

KEY DISTINCTION from categorical: Order matters. PhD > Master's is meaningful. But the "gap" from High School to Bachelor's ≠ gap from Master's to PhD.

Constraint: We need to preserve order: $\text{category}_1 < \text{category}_2 < \ldots < \text{category}_k$

Solution: Map categories to integers respecting order: $f: \{\text{cat}_1, \text{cat}_2, \ldots, \text{cat}_k\} \to \{0, 1, \ldots, k-1\}$ where $\text{cat}_i < \text{cat}_j \Rightarrow f(\text{cat}_i) < f(\text{cat}_j)$

WHY this works: Algorithms that use $<, >$ comparisons (tree splits: "is education≥ 2?") preserve the ordering. Distance-based algorithms (KNN, linear models) will assume uniform spacing—acceptable if the ordinal gaps are roughly similar, problematic otherwise.

Step 1: Define the ordering High School < Bachelor's < Master's < PhD

WHY this step? We must explicitly specify the order before encoding.

Step 2: Assign integer codes

• High School → 0
• Bachelor's → 1
• Master's → 2
• PhD → 3

Step 3: Apply encoding

WHY this encoding?

• A decision tree can split: "education_encoded ≥ 2" → captures Master's and PhD
• Order is preserved: 0 < 1 < 2 < 3 matches reality
• Linear model will assume equal spacing (problem if PhD is "much more" than Master's in impact)

When to worry about spacing: If predicting salary and PhD holders earn3× Bachelor's holders (not 3× the increment), consider:

• Creating multiple binary features: has_bachelors, has_masters, has_phd
• Using target encoding (replace with mean target value per category)

4. Text Data

KEY CHALLENGE: Text is high-dimensional, variable-length, and context-dependent. "Bank" (financial institution) vs. "river bank" requires understanding context.

Goal: Convert a document into a fixed-length vector of numbers.

Step 1: Build the vocabulary (set of all unique words across all documents) $V = \{w_1, w_2, \ldots, w_n\}$

WHY? We need a fixed set of dimensions. Vocabulary size = number of dimensions.

Step 2: For each document $d$, count occurrences of each word $\mathbf{x}_d = [c_1, c_2, \ldots, c_n]$ where $c_i = \text{count of word } w_i \text{ in document } d$

WHY this works: Each document becomes a point in $\mathbb{R}^n$ space. Similar documents (using similar words) are close in this space.

Mathematical form: For vocabulary $V$ of size $n$, document $d$ is: $\mathbf{x}_d \in \mathbb{R}^n, \quad (\mathbf{x}_d)_i = \#\{w_i \text{ appears in } d\}$

Limitation: Ignores word order. "Dog bites man" = "Man bites dog" (same BoW vector).

1. "The movie was good"
2. "The movie was bad"
3. "Great movie"

Step 1: Build vocabulary Extract unique words: {the, movie, was, good, bad, great} Vocabulary size = 6

WHY this step? Defines our feature space dimensions.

Step 2: Count word occurrences for each document

Document 1: "The movie was good"

• the: 1, movie: 1, was: 1, good: 1, bad: 0, great: 0
• Vector: [1, 1, 1, 0, 0]

WHY this step? Converts text to numbers by counting.

Document 2: "The movie was bad"

• the: 1, movie: 1, was: 1, good: 0, bad: 1, great: 0
• Vector: [1, 1, 0, 1, 0]

Document 3: "Great movie"

• the: 0, movie: 1, was: 0, good: 0, bad: 0, great: 1
• Vector: [0, 1, 0, 0, 1]

Result Matrix:

Interpretation:

• D1 and D2 are similar (share "the movie was") → dot product = 3
• D3 is different (different words) → dot product with D1 = 1

Next step for real use: Remove stop words ("the", "was"), apply TF-IDF to down-weight common words.

Why it feels right: "The model will just ignore unknown words."

Why it's wrong: If your test set uses "excellent" extensively to indicate positive reviews, your model sees it as all-zeros (missing signal). Performance drops because you're blind to key information.

The fix:

1. During training: Keep a <UNK> (unknown) token in vocabulary
2. During testing: Map unseen words to <UNK>
3. Better: Use subword tokenization (BPE, WordPiece) so "excellent" breaks into "excell" + "ent," reusing learned pieces
4. Best: Use pre-trained embedings (Word2Vec, BERT) that already know millions of words

Steel-man the mistake: Ignoring unknowns is simple and sometimes works if test set is similar to training. But in production, language evolves—new slang, product names, events. Handling unknowns is critical for robustness.

Advanced Considerations

80/20 Rule: 80% of performance comes from handling the 4core types correctly. Spend time here before exotic feature engineering.

Data Type Decision Tree

Is the data text in sentences?
├─ YES → Text Data (BoW, TF-IDF, embedings)
└─ NO → Is it numbers?
    ├─ YES → Can you compute meaningful average?
    │   ├─ YES → Numerical (continuous or discrete)
    │   └─ NO → Probably categorical (e.g., zip codes look numeric but aren't)
    └─ NO → Is there a meaningful order?
        ├─ YES → Ordinal (education, ratings)
        └─ NO → Categorical (colors, countries)

Connections

• Normalization and Scaling → Deep dive into min-max, z-score, robust scaling
• One-Hot Encoding vs Label Encoding → When to use which encoding strategy
• TF-IDF and Text Vectorization → Advanced text feature extraction
• Feature Engineering from Datetime → Extracting features from timestamps
• Handling High-Cardinality Categorical → Embedings, target encoding, hashing
• Curse of Dimensionality → Why one-hot encoding can explode feature space
• Feature Scaling Impact on Gradient Descent → Mathematical proof of why scaling matters

Numerical data is like counting pencils—you can say "I have 5 pencils" and do math (add3 more = 8 total). You can find the average across your class.

Categorical data is like your pencil colors—red, blue, green. You can't say "red + blue = ?" or "average color." They're just different types, no math involved.

Ordinal data is like medal rankings: bronze< silver < gold. There's an order (gold is better), but you can't say "gold is exactly twice as good as silver." The order matters, but the spacing between ranks isn't equal.

Text data is like your diary entries—full sentences and paragraphs. Computers don't understand English directly, so we have to convert it to numbers first. We might count how many times words appear: "homework" appears 3 times, "fun" appears 0 times (sadly!).

Why does this matter? If you give a calculator words, it breaks. If you treat your pencil colors like numbers ("red = 1, blue = 2"), the calculator thinks blue is "twice" red, which is nonsense. Match the data type to the tool!

#flashcards/ai-ml

What are the four fundamental data types in ML? :: Numerical (continuous/discrete), Categorical (no order), Ordinal (ordered categories), Text (natural language)

Concept Map

Hinglish (regional understanding)