Question bank — Ordinal and target encoding

False — both map categories to integers, but ==ordinal encoding uses a meaningful, human-chosen order== (S<M<L), while plain label encoding assigns integers in whatever order the categories appear (often alphabetical), inventing a fake order. ::: False — both map categories to integers, but ordinal encoding uses a meaningful chosen order while label encoding assigns arbitrary integers, faking an order that isn't real.

unordered feature like "City" is harmless. False — you inject a phantom ordering (City3 > City1) the model will happily exploit as if it were real, distorting distances and splits (see figure s02). Use One-hot Encoding or target encoding instead. ::: False — it fabricates an order (City3 > City1) that doesn't exist, so the model learns fake magnitude relationships between arbitrary labels.

True — target encoding produces exactly one numeric column regardless of cardinality, whereas one-hot produces one column per category, which is why it beats the Curse of Dimensionality on high-cardinality features. ::: True — it collapses any number of categories into a single numeric column, unlike one-hot which grows with cardinality.

$m=0$ equals the raw category mean. True — with $m=0$ the weight $w=n_c/(n_c+0)=1$, so the global mean gets zero weight and you recover ${\overline{y}}_c$ exactly. ::: True — $m=0$ gives weight 1 to the category mean and 0 to the global mean, so it collapses to the raw mean.

$m\to \infty$, every category's encoding approaches the global mean. True — the weight $w=n_c/(n_c+m)\to 0$, so all categories collapse toward ${\overline{y}}_{\text{global}}$; infinite smoothing erases all category signal (right edge of figure s03). ::: True — large $m$ drives $w\to 0$, so every category shrinks to the global mean and category-specific signal vanishes.

Feature Scaling before feeding it to a distance-based model. True (usually) — the integers 1..k are on an arbitrary scale, so for KNN / SVM / gradient descent you should scale them; but Tree-based Models split on thresholds and are scale-invariant, so scaling there is optional. ::: True for distance/gradient models — the raw integers are on an arbitrary scale; trees are scale-invariant so they don't need it.

Bayesian shrinkage. True — the smoothed formula adds $m$ "pseudo-observations" sitting at the global mean, which is exactly an empirical-Bayes prior pulling small-sample estimates toward the population (figure s04). ::: True — it is empirical Bayes: $m$ pseudo-observations at the global mean act as a prior that shrinks noisy small-category means.

False — the mean of any numeric target works: continuous targets (house price) give the average value per category, and multi-class targets are handled by encoding one column per class (each holding $P(\text{class}\mid c)$) or using the ordinal/regression target directly. ::: False — you take the mean of any numeric target; continuous targets give an average, and multi-class targets get one encoded column per class holding that class's probability.

df.groupby('city')['price'].mean() on the full dataset, then trained on it. Clean pipeline!" The error is target leakage: each row's own $y$ went into the mean used to encode that same row, so the model peeks at the answer. Fit the encoding on training folds only via Cross-Validation (or out-of-fold encoding). ::: Leakage — each row's own target contributed to its own encoding, letting the model see the answer. Encode using out-of-fold / cross-validation splits.

$y=100$, so I encoded it as exactly 100." That single value is almost certainly noise, not signal; trusting it causes Overfitting. Apply smoothing to shrink it toward the global mean in proportion to how few observations you have. ::: One observation is noise, not a reliable mean — encoding it verbatim overfits. Use smoothing to pull rare categories toward the global mean.

Dropping is wrong at prediction time — you can't drop real requests. Instead encode unseen categories with the global mean (or a validation-based fallback). ::: Don't drop unseen categories — at inference you can't discard live inputs. Fill them with the global mean or a sensible fallback.

The error: ordinal encoding only guarantees order, not magnitude. The gap 3→10 falsely tells a linear model XL is 7 units past L; use consecutive integers unless you have a justified interval scale. ::: Wrong — ordinal encoding encodes order only, not calibrated distances. Non-consecutive integers invent a fake magnitude gap; use 1,2,3,4.

That creates 40,000 sparse columns, triggering the Curse of Dimensionality and memory blow-up — the textbook case against one-hot. Target encoding compresses it to one informative column. ::: Wrong tool — 40k one-hot columns cause the curse of dimensionality and memory blow-up. High cardinality is exactly when target encoding wins.

Nothing's broken — Tree-based Models split on thresholds and are invariant to monotonic rescaling, so scaling a feature can't change their splits. Scaling only matters for distance/gradient models. ::: Not broken — trees are invariant to monotonic scaling, so standardizing an input can't change their splits or score.

$w=\frac{n_c}{n_c+m}$ instead of, say, $w=\frac{1}{2}$ always? Because confidence in a category mean grows with sample size; a fixed weight would trust a 1-sample and a 1000-sample category equally. $n_c/(n_c+m)$ makes big categories rely on their own mean and tiny ones lean on the global prior (figure s03). ::: Because confidence should scale with how much data a category has; $n_c/(n_c+m)$ trusts large categories and shrinks small, noisy ones — a fixed weight can't do that.

For order preservation the starting point is irrelevant (any strictly increasing integers work). Convention picks 1 to sidestep models/frameworks that treat 0 as "missing" or special; the differences between codes, not their absolute origin, carry meaning. ::: Order-preservation is unaffected by the start point; 1-indexing is just convention to avoid 0 being treated as "missing." Only the differences matter.

$m$ set too low? Low $m$ means rare categories keep their raw, noisy means, so the encoded column memorizes training targets → severe Overfitting and inflated CV scores that don't generalize. ::: Low $m$ lets rare categories keep noisy raw means, so the feature memorizes training targets — classic overfitting with optimistic scores.

Smoothing reduces the variance of the leak but doesn't remove it — each row's target still influences the mean applied to that row. Only computing the encoding on data excluding the current row/fold removes the leak. ::: Smoothing shrinks the leak's magnitude but the row's own target still enters its encoding; only out-of-fold computation truly removes it.

$m$ pseudo-rows at the global mean"? Because summing $n_c$ real rows ($n_c{\overline{y}}_c$) plus $m$ imaginary rows worth ${\overline{y}}_{\text{global}}$ and dividing by the total count $n_c+m$ is the formula — the empirical-Bayes prior is literally those pseudo-rows (figure s04). ::: Because $n_c$ real rows plus $m$ pseudo-rows at the global mean, averaged, gives exactly $(n_c{\overline{y}}_c+m{\overline{y}}_{\text{global}})/(n_c+m)$ — the prior is those pseudo-rows.

count $n_c=0$ (never seen)? It has no category mean at all, so it falls back to the global mean ${\overline{y}}_{\text{global}}$ (or a designated fallback). In the smoothed formula, plug $n_c=0$: the result is exactly the global mean. ::: It defaults to the global mean — plug $n_c=0$ into the smoothed formula and the category term vanishes, leaving ${\overline{y}}_{\text{global}}$.

$y\in \{0,1\}$: what does the target encoding of a category actually represent? The mean of 0/1 labels is the fraction of positives, i.e. the empirical probability $P(y=1\mid X=c)$ — target encoding becomes a per-category probability estimate. ::: It becomes the empirical probability $P(y=1\mid X=c)$, since the mean of 0/1 labels is just the positive rate.

multi-class target with 4 classes, how many target-encoded columns does one categorical feature produce? Four (one probability column per class, each $P({\text{class}}_j\mid c)$) — or three if you drop one to avoid redundancy since they sum to 1. Still far fewer than one-hot on a high-cardinality feature. ::: Four columns (one per class probability), or three if you drop the redundant one since they sum to 1 — still vastly fewer than one-hot.

every category has the same target mean equal to the global mean? The encoded column becomes a constant — it carries no information and the model can ignore it; target encoding only helps when categories genuinely differ in their target statistics. ::: The column is constant and useless — target encoding adds value only when categories differ in target behavior.

single category (k=1): what's the encoding? Everything maps to 1, giving a constant column with zero variance — no ordinal or predictive information, safe to drop. ::: All rows become 1, a zero-variance constant column carrying no information; it can be dropped.

1000 levels (e.g., exam rank). Ordinal or target encoding? Ordinal encoding stays one column and preserves the real ordering, so it's fine — high cardinality only forces you away from One-hot Encoding, not away from ordinal encoding, which never expands columns. ::: Ordinal is fine — it stays one column and keeps the true order; high cardinality only rules out one-hot, not ordinal encoding.

No — collisions are expected; target encoding maps to a target statistic, not to identity, so distinct categories with similar target behavior legitimately land on the same value. ::: Not a bug — encoding maps to a target statistic, so two categories with the same target mean collide by design, not by error.