2.1.1 · D5Data Preprocessing & Feature Engineering
Question bank — Types of data (numerical, categorical, ordinal, text)
Quick vocabulary refresher so nothing below is used before it is defined:
- Numerical = values where arithmetic (add, average, subtract) genuinely means something.
- Categorical = plain labels with no order (colour, city).
- Ordinal = labels that do have an order but unknown, uneven gaps (S < M < L).
- Text = free strings that must be turned into numbers before a model sees them.
- Cardinality = how many distinct values a column has.
- Interval scale = numbers with meaningful gaps but an arbitrary zero (Celsius, calendar years). You may subtract, but ratios are meaningless.
- Ratio scale = numbers with meaningful gaps and a true zero meaning "none" (Kelvin, mass, length). Here ratios like "twice as much" are valid.
True or false — justify
A ZIP/PIN code stored as an integer is numerical data
False — the digits are a label; averaging two PIN codes or subtracting them is meaningless, so it is categorical despite looking numeric.
"Number of children (0,1,2,3)" is discrete numerical data
True — the counts are countable integers where 3 really is one-more-than 2, so arithmetic is meaningful; it is discrete numerical, not categorical.
Ordinal data is just categorical data that happens to be sorted alphabetically
False — the order in ordinal data is semantic (PhD > Master's), not alphabetical; sorting {red, blue, green} gives no meaning, so it stays categorical.
For a decision tree, label-encoding an unordered category is always harmless
False (a common over-simplification) — it is safe only if the tree supports true categorical splits ("is value == X"). Many popular implementations (e.g. scikit-learn trees) only do numeric splits, so arbitrary integer codes force the tree to group whatever values happen to fall on one side of the threshold — a fake grouping that can degrade performance.
For linear regression, label-encoding an unordered category is harmless
False — a linear model multiplies the code by a weight, so encoding Cash=2, Card=1 tells it "Cash is twice Card", inventing a relationship that does not exist.
The Euclidean distance between any two distinct one-hot vectors is always
True — two different one-hot vectors differ in exactly two positions (a 1↔0 swap each), so the squared distance is , hence (see figure s01).
Ordinal encoding assumes the gap between consecutive levels is equal
True for distance-based models — KNN/linear models read 0,1,2,3 as evenly spaced, which is the exact assumption ordinal data warns you against; tree models escape this because they only use order.
Standardizing (z-score) makes the data follow a normal distribution
False — it forces mean 0 and std 1 but does not change the shape; a skewed distribution stays skewed after scaling (see figure s02).
Temperature in Celsius is fully numerical, so ratios like "40°C is twice as hot as 20°C" are valid
False — Celsius is an interval scale with an arbitrary zero, so ratios are meaningless; only Kelvin, a ratio scale with a true zero, supports "twice as hot" (see figure s03).
Text data can be fed to most ML models without any preprocessing
False — nearly all models need fixed-length numeric vectors, so text must be vectorized first (see TF-IDF and Text Vectorization).
Spot the error
A junior encodes T-shirt size as one-hot {S,M,L,XL} and feeds it to a model that must predict fit
Error: one-hot destroys the S<M<L<XL order, so the model can no longer learn "sizes near each other behave similarly"; ordinal encoding is the right choice here.
Someone one-hot encodes a "user_id" column with 200,000 unique users
Error: this explodes into 200,000 sparse columns (high cardinality) — memory blows up and the model overfits; see Handling High-Cardinality Categorical and Curse of Dimensionality.
A pipeline computes the mean and std from the entire dataset (train + test) before scaling
Error: this leaks test-set information into training statistics; and must be computed on train only, then applied to test.
An engineer treats a "star rating 1–5" as continuous numerical and predicts 3.7 stars, reporting it as a real rating
Error: the ratings are ordinal with only 5 defined levels; a fractional prediction has no defined category, so it needs rounding or an ordinal-aware model.
A model uses raw ["2024-03-14"] date strings as a single categorical feature
Error: every date is unique → maximum cardinality and no learnable pattern; instead extract day-of-week, month, is_weekend (see Feature Engineering from Datetime).
Someone standardizes a one-hot column so its mean is 0 and std is 1
Error: this destroys the interpretable 0/1 structure and the equal-distance property; binary/indicator columns are usually left as-is, only genuine numerical features are scaled.
Why questions
Why does gradient descent care whether features are on the same numeric scale?
Because a large-scale feature produces large gradients that dominate the update direction, so the model zig-zags and converges slowly; scaling equalizes step sizes (see Feature Scaling Impact on Gradient Descent).
Why can decision trees handle categorical/ordinal codes that would break a linear model?
Trees split on individual value thresholds ("is code ≥ 2?"), so they never multiply the code by a weight or measure distances — the fake spacing is unused as long as the split matches the true grouping.
Why does one-hot encoding place every category at equal distance from every other?
Each category gets its own axis with a single 1, so no two categories share direction; geometrically they sit on separate coordinate axes, all apart — no category is "in between" (see figure s01 and One-Hot Encoding vs Label Encoding).
Why is high-cardinality categorical data dangerous for one-hot encoding specifically?
Each unique value becomes a new near-empty column, so dimensionality explodes and most columns are almost all zeros, feeding the Curse of Dimensionality and starving the model of examples per column.
Why do we subtract the mean before dividing by the standard deviation in z-scoring?
Centering first puts the distribution's middle at 0 so the resulting sign tells you "above/below average"; dividing by then rescales the spread — order matters because is measured about the mean.
Why is normalizing text word counts (like TF-IDF's IDF term) useful rather than raw counts?
Raw counts over-reward long documents and common words; down-weighting words that appear everywhere lets rare, distinguishing words carry signal (see TF-IDF and Text Vectorization).
Why might discrete numerical data (e.g. "number of bedrooms: 1,2,3") sometimes be treated as categorical?
When only a few distinct values exist and the effect is non-monotonic (a 2-bedroom may be preferred over both 1 and 3), one-hot lets the model learn each level independently instead of forcing a straight-line relationship.
Edge cases
What happens to a z-score when every value in the column is identical?
The standard deviation is 0, so divides by zero — the column carries no information and should be dropped, not scaled (see Normalization and Scaling).
Is a boolean "is_subscribed" (True/False) numerical, categorical, or ordinal?
It is categorical with two levels; encoded as 0/1 it doubles as a valid numeric indicator, which is why binary columns need no one-hot expansion — one column already suffices.
How do you encode an ordinal feature when a new, unseen level appears at prediction time?
You must decide its rank in advance (e.g. map unknown to a reserved code or to the nearest known level); ordinal encoders cannot invent an order for a value they were never told about.
Where does a variable like "letter grades A,B,C,D,F" sit, and what's the encoding trap?
It is ordinal (A>B>C…), so map to ordered integers; the trap is one-hotting it and losing the order the model actually needs.
If a categorical column has exactly one unique value across all rows, what should you do?
Drop it — a constant column has zero variance and cannot help any model distinguish samples, whether one-hot or label encoded.
What is the right treatment for a "cyclical" numeric feature like hour of day (23 and 0 are adjacent)?
Plain numeric encoding wrongly makes 23 and 0 far apart; encode with sine/cosine of the hour so midnight and 11 PM sit next to each other (an idea from Feature Engineering from Datetime).
Is a calendar year (e.g. 1990, 2000) numerical, and can you take ratios of it?
It is interval data — differences like "10 years apart" are meaningful, but year 0 is arbitrary, so "year 2000 is twice year 1000" is nonsense; only true-zero (ratio) quantities allow ratios.
Recall Two-second self-test
Cover everything. Answer these three, full sentences only. A PIN code column — which type and why? ::: Categorical: its digits are a label, arithmetic on them is meaningless. Why is label-encoding an unordered category usually safe for trees but not for linear models? ::: Trees split on thresholds and never multiply the code (safe only when the split can isolate the right group), whereas linear models weight it and read the fake order as real magnitude. What breaks when you z-score a constant column? ::: causes division by zero; the column has no information and should be dropped.
Parent: Types of Data