Exercises — Ordinal and target encoding

• education → ordinal. It has a natural progression (each level requires the previous), so integers 1–4 encode real "distance".
• us_state → one-hot or target. 50 columns is borderline but survivable; if you're predicting a target and want compactness, target encoding works. No inherent order → never ordinal.
• blood_type → one-hot. Only 4 unordered categories — cheap, safe, no order to exploit.
• zip_code → target encoding. 30 000 categories would make one-hot explode into 30 000 sparse columns (see Curse of Dimensionality); target encoding compresses each zip into one number: the mean income there.

Rule of thumb: order present → ordinal; no order, low cardinality → one-hot; no order, high cardinality with a target → target.

Replace each value by its rank: L→3, S→1, XL→4, M→2, L→3. $[3,1,4,2,3]$

$\text{TE}(\text{NYC})=\frac{500+600+550}{3}=\frac{1650}{3}=550$ $\text{TE}(\text{Seattle})=\frac{400+450}{2}=\frac{850}{2}=425$ $\text{TE}(\text{Portland})=\frac{300}{1}=300$ Global mean over all 6 rows: ${\overline{y}}_{\text{global}}=\frac{500+600+550+400+450+300}{6}=\frac{2800}{6}\approx 466.67$ (Note: this is not the same as averaging the three city means, because NYC has more rows and pulls the row-level global mean up.)

Portland (${\overline{y}}_c=300$, $n_c=1$): ${\text{TE}}_{\text{smooth}}=\frac{1\cdot 300+10\cdot 475}{1+10}=\frac{300+4750}{11}=\frac{5050}{11}\approx 459.09$ NYC (${\overline{y}}_c=550$, $n_c=3$): ${\text{TE}}_{\text{smooth}}=\frac{3\cdot 550+10\cdot 475}{3+10}=\frac{1650+4750}{13}=\frac{6400}{13}\approx 492.31$ Portland (1 sample) is dragged far toward 475; NYC (3 samples) is dragged less. The weight on the category mean is $w=n_c/(n_c+m)$: Portland $w=1/11\approx 0.09$, NYC $w=3/13\approx 0.23$.

Boston unseen → global mean $475$. NYC → $550$. Seattle → $425$. $[475,550,425]$

Write $w=\frac{n_c}{n_c+m}$, so ${\text{TE}}_{\text{smooth}}=w{\overline{y}}_c+(1-w){\overline{y}}_{\text{global}}$. (a) $n_c\to \infty$: $w\to 1$, so ${\text{TE}}_{\text{smooth}}\to {\overline{y}}_c$. With infinite data the category mean is trustworthy — no shrinkage needed. (b) $n_c\to 0$: $w\to 0$, so ${\text{TE}}_{\text{smooth}}\to {\overline{y}}_{\text{global}}$. With no data we fall back to the population's best guess. This is exactly the empirical-Bayes behaviour: the estimate slides continuously between "trust the category" and "trust the prior."

The mean of $n_c$ real values (summing to $n_c{\overline{y}}_c$) together with $m$ fake values each ${\overline{y}}_{\text{global}}$ (summing to $m{\overline{y}}_{\text{global}}$) is: $\frac{n_c{\overline{y}}_c+m{\overline{y}}_{\text{global}}}{n_c+m}$ which is identical to the smoothing formula. So $m$ literally counts imaginary "prior" rows pinned at the global mean — larger $m$ = stronger pull toward the prior. This is why $m$ is called the smoothing strength.

Why it's a lie: each row's price (the target) helped compute the encoding for its own category. So the encoded feature secretly contains the answer for that row — the model "peeks" at $y_i$ through $x_i$'s encoding. Validation rows encoded with statistics that include themselves inflate accuracy — this is target leakage, a species of Overfitting. The fix: encode using only data the row didn't contribute to. Practically: use out-of-fold encoding (K-fold) — for each fold, compute encodings from the other folds. A validation row's encoding then never sees its own target.

Distances are dominated by income because its numeric range (~180 000) dwarfs size's range (3). The ordinal ranks, though meaningful, contribute almost nothing to any distance. Fix: apply Feature Scaling (e.g. standardisation) so both features live on comparable scales after ordinal encoding. Ordinal encoding preserves order; scaling makes that order count in a distance metric.

Magnitude gap: no. Trees split on thresholds ("income $>50\text{k}$?"), which are invariant to monotonic rescaling — so trees don't need scaling for magnitude reasons. Spacing 1,2,3,4 vs 1,2,4,8: also (almost) irrelevant for a tree, because any monotonic relabelling produces the same possible split points in the same order. A tree only uses the ordering, not the exact gaps. (Contrast: a linear model does care — it multiplies the encoded number by a coefficient, so 1,2,4,8 would let it treat XL as 8× S.) This is why ordinal encoding pairs especially naturally with trees.

• grade → ordinal A=4,B=3,C=2,D=1 (or reverse) — the rank is the credit quality signal.
• state → one-hot — 50 unordered columns is affordable and leak-proof, though smoothed target encoding is a valid compact alternative.
• employer → smoothed target encoding with out-of-fold CV — 12 000 categories make one-hot infeasible (Curse of Dimensionality), and most employers have few rows, so smoothing + OOF prevents both noise and leakage.

Encoding fold A using fold B's stats. Fold B rows: X→{0}, Y→{0,1}. Fold B global mean $=(0+0+1)/3=1/3$.

• Fold B: X mean $=0/1=0$; Y mean $=(0+1)/2=0.5$.
• Row 1 (X): $0$
• Row 2 (X): $0$
• Row 3 (Y): $0.5$

Encoding fold B using fold A's stats. Fold A rows: X→{1,0}, Y→{1}. Fold A global mean $=(1+0+1)/3=2/3\approx 0.667$.

• Fold A: X mean $=(1+0)/2=0.5$; Y mean $=1/1=1$.
• Row 4 (X): $0.5$
• Row 5 (Y): $1$
• Row 6 (Y): $1$

Final encoded column (rows 1–6): $[0,0,0.5,0.5,1,1]$ Every value came from the opposite fold, so no row saw its own target — leak-free.

We need $\frac{2(0.9)+m(0.3)}{2+m}\le \mathrm{0.5.}$ Numerator $=1.8+0.3m$. Require $1.8+0.3m\le 0.5(2+m)=1+0.5m$. $1.8+0.3m\le 1+0.5m\Rightarrow 0.8\le 0.2m\Rightarrow m\ge 4.$ Smallest integer: $m=4$. Check: $\frac{1.8+1.2}{6}=\frac{3.0}{6}=0.5$ ✓ (exactly the cap).

(a) The $m$ pseudo-rows pinned at ${\overline{y}}_{\text{global}}$ act as a prior belief that "before seeing data, every category behaves like the population"; observed rows update that belief, and the posterior is the weighted average — textbook empirical Bayes. (b) Out-of-fold encoding computes each fold's feature from the held-out remaining folds, exactly the train/validate split logic of K-fold Cross-Validation, applied to feature construction instead of scoring. (c) employer has no natural order, so assigning ranks 1,2,3,… would invent a false "employer 8 is twice employer 4" relationship, injecting a fictitious ordinal signal a linear model would happily (and wrongly) exploit.

Which encoder for a 12 000-value unordered feature with a target? ::: Smoothed target encoding computed out-of-fold. As $n_c\to \infty$, ${\text{TE}}_{\text{smooth}}\to$ ? ::: The category mean ${\overline{y}}_c$. A tree only uses the order of an ordinal encoding, not the exact spacing. ::: True The global mean is the mean over all rows, not the mean of the category means. ::: correct