Worked examples — Ordinal and target encoding

This page is a drill. The parent note told you the two mappings and their formulas. Here we throw every kind of input at those formulas — every count, every degenerate case, every "what happens at the edge" — so that when a real dataset hands you something weird, you have already seen it.

Before we start: two words we will lean on constantly.

---

The scenario matrix

Encoding looks trivial until you meet an edge. Here is every case class this topic can throw at you. Each of the worked examples below is tagged with the cell it lands in.

#	Case class	What makes it tricky	Worked ex.
A	Plain ordinal, order given	nothing — the baseline	Ex 1
B	Ordinal with a gap in the scale	is the spacing uniform?	Ex 2
C	Ordinal misused on a nominal feature	order that doesn't exist	Ex 3
D	Target encode, several counts $n_c$	mean per group	Ex 4
E	Degenerate category: $n_c=1$	one row = pure noise	Ex 5
F	Smoothing with parameter $m$	shrink toward global mean	Ex 5, 6
G	Zero-observation / unseen category at test	division by zero avoided	Ex 7
H	Limit $m\to 0$ and $m\to \infty$	what smoothing collapses to	Ex 8
I	Real-world word problem end-to-end	glue it all together	Ex 9
J	Exam twist: leakage trap	the fold-safe fix	Ex 10

The rest of the page fills in every row.

---

Ordinal cases (A, B, C)

Forecast: guess the five numbers before reading on.

1. Write the order as ranks. Poor→1, Fair→2, Good→3, Excellent→4. Why this step? Ordinal encoding's only job is $a<b\Rightarrow f(a)<f(b)$; consecutive integers are the smallest set that guarantees it.
2. Look up each row. Good=3, Poor=1, Excellent=4, Fair=2, Good=3. Why this step? Encoding is a dictionary lookup, nothing more.

$\text{Result}=[3,1,4,2,3]$

Verify: the maximum encoded value (4) sits exactly where "Excellent" is, the minimum (1) where "Poor" is. Order preserved. ✓

---

Forecast: will $2$ vs $3$ correctly say "Black is a little above Brown", or will it lie about how much?

1. Rank the categories, then look up each row. The map is White→1, Brown→2, Black→3. Now read the data [White, Black, White, Brown] left to right and substitute: White=1, Black=3, White=1, Brown=2, giving $\text{Encoded}=[1,3,1,2].$ Why this step? Same rule as Ex 1 — you rank the categories once, then do a plain dictionary lookup for each row. The order of the data rows is preserved; only the labels change to their ranks.
2. Name the danger. A tree only ever asks "is $x\le t$?", so the gaps between 1, 2, 3 are irrelevant — it just needs monotonicity. But a linear model reads $f(\text{Black})-f(\text{Brown})=1=f(\text{Brown})-f(\text{White})$ as equal jumps in skill, which is a modelling assumption you are silently making. Why this step? This is the whole reason ordinal encoding is safe for trees and risky for linear models.

Verify: the two White rows must encode to the same number — positions 1 and 3 are both $1$ ✓. Ranks are strictly increasing $1<2<3$ ✓. The "equal spacing" claim is an assumption, not a fact — flag it. See Feature Scaling if the downstream model is distance-based.

Read the figure below: the three category ranks sit on a number line at $1,2,3$. The two red double-headed arrows mark the gaps between consecutive belts. Notice they are drawn identical length — that visual equality is exactly the hidden "equal-skill-jump" assumption a linear model swallows. There is no evidence Brown→Black is the same skill leap as White→Brown; the encoding just declares it so.

---

Forecast: what false belief does the number line now force onto the model?

1. Spot the invented order. The map claims Green (3) > Blue (2) > Red (1), i.e. green is "more" than red. Color has no such ranking — it is nominal, not ordinal. Why this step? Ordinal encoding is only valid when a true order exists (Ex 1, 2). Here it manufactures a lie.
2. Fix. For a low-cardinality nominal feature use One-hot Encoding; for a high-cardinality one use target encoding (below). Why this step? Both avoid imposing a fake ranking.

Verify: sanity check — swap Red and Green labels. Ordinal encoding gives a different dataset ($[3,2,1]$ vs $[1,2,3]$) purely from relabelling; a correct nominal encoder must be label-invariant. Ordinal fails this test → confirms misuse. ✓

---

Target-encoding cases (D–H)

We reuse one running table so the numbers are traceable.

City      Price(k$)
NYC       500
NYC       600
NYC       550
Seattle   400
Seattle   450
Portland  300

Two constants we will need throughout, plus the weight that controls smoothing:

---

Forecast: which city gets the biggest number?

1. Count each category. $n_{\text{NYC}}=3,n_{\text{Sea}}=2,n_{\text{Por}}=1$. Why this step? Every mean needs its denominator; the counts also decide who is "trustworthy" later.
2. Group means. ${\overline{y}}_{\text{NYC}}=\frac{500+600+550}{3}=550,{\overline{y}}_{\text{Sea}}=\frac{400+450}{2}=425,{\overline{y}}_{\text{Por}}=\frac{300}{1}=300$ Why this step? This is $\mathbb{E}[y\mid X=c]$ — the target's signal per category, compressed to one number.
3. Replace. Encoded column = $[550,550,550,425,425,300]$.

Verify: the encoded values must lie between the min (300) and max (600) of the raw prices, and NYC (highest prices) must get the highest code. $300\le 425\le 550\le 600$ ✓, NYC largest ✓.

Read the figure below: three bars, one per city, whose heights are the encoded values. The red bar (NYC) towers highest at 550 — a visual reminder that target encoding turns "which city?" into "how expensive, on average?", so the tallest bar is literally the priciest place.

---

Forecast: raw target says 300. Will smoothing pull it up toward the global mean a little or a lot?

1. Compute the global mean carefully. ${\overline{y}}_{\text{global}}=\frac{500+600+550+400+450+300}{6}=\frac{2800}{6}=466.\overline{6}\approx 466.67$ Why this step? This is the anchor every category shrinks toward; getting it right matters because all later numbers depend on it. (Add the six prices: they total $2800$, not $2850$ — do the sum, don't eyeball it.)
2. Recall the smoothing formula. ${\text{TE}}_{\text{smooth}}(c)=\frac{n_c{\overline{y}}_c+m{\overline{y}}_{\text{global}}}{n_c+m}$ Why this step? With one observation, 300 is almost certainly noise. Smoothing = "add $m$ imaginary rows sitting at the global mean" — a Bayesian prior that says when in doubt, be average.
3. Plug in $n_c=1,{\overline{y}}_c=300,m=10,{\overline{y}}_{\text{global}}=466.67$: ${\text{TE}}_{\text{smooth}}(\text{Por})=\frac{1\cdot 300+10\cdot 466.67}{1+10}=\frac{4966.67}{11}\approx 451.5$ Why this step? The weight on the category is $w=\frac{n_c}{n_c+m}=\frac{1}{11}\approx 0.091$ — barely 9%. One row earns almost no trust, so the answer sits near the global mean.

Verify: $451.5$ must lie strictly between the raw category value 300 and the global mean 466.67, and closer to 466.67 (since $w$ is small). $300<451.5<466.67$ ✓, closer to 466.67 ✓. This shrinkage directly fights Overfitting.

---

Forecast: NYC has more data — will it shrink less than Portland did?

1. Same formula, new numbers. ${\text{TE}}_{\text{smooth}}(\text{NYC})=\frac{3\cdot 550+10\cdot 466.67}{3+10}=\frac{1650+4666.67}{13}=\frac{6316.67}{13}\approx 485.9$ Why this step? Weight on the category is $w=\frac{3}{13}\approx 0.231$ — over twice Portland's. More rows = more trust = less shrinkage.

Verify: distance from category mean, NYC vs Portland. NYC moved $550-485.9=64.1$; Portland moved $451.5-300=151.5$. The category with more data moved less ✓ — exactly the intended behaviour.

Read the figure below: each city is a vertical red arrow starting at its raw mean (black dot) and ending at its smoothed code (red square). The dashed black line is the global mean $466.67$. Every arrow points toward that dashed line — that is what "shrinkage" looks like — and Portland's arrow is far longer than NYC's, showing thin data gets dragged much harder.

---

Forecast: what do you do when the denominator's group is empty?

1. NYC, Seattle already have training means: 550 and 425 (Ex 4). Why this step? Known categories reuse the learned map — that is the point of fitting on train, applying on test.
2. Boston has no rows. Raw formula would be $\frac{0}{0}$ — undefined. Fall back to the global mean $466.67$. Why this step? With zero evidence, the smoothing formula itself already gives the right answer: set $n_c=0$ and ${\text{TE}}_{\text{smooth}}=\frac{0\cdot (\cdot )+m\cdot 466.67}{0+m}=\mathrm{466.67.}$ Smoothing automatically handles the unseen case — no special branch needed.

$\text{Result}=[466.67,550,425]$

Verify: unseen → exactly global mean ✓; the $0/0$ never occurs because $n_c+m=m>0$ ✓.

---

Forecast: at one extreme you get pure raw mean; at the other, pure global mean. Which is which?

1. $m\to 0$ (no smoothing): weight $w=\frac{n_c}{n_c+m}\to \frac{1}{1}=1$. ${\lim }_{m\to 0}{\text{TE}}_{\text{smooth}}={\overline{y}}_c=300$ Why this step? Zero imaginary rows = trust the data completely = plain target encoding (Ex 4). This is the maximum-variance / overfitting end.
2. $m\to \infty$ (infinite smoothing): weight $w\to 0$. ${\lim }_{m\to \infty }{\text{TE}}_{\text{smooth}}={\overline{y}}_{\text{global}}=466.67$ Why this step? Infinitely many prior rows drown the data = every category collapses to the same number = the feature becomes useless (maximum bias). This is the underfitting end.

Verify: plug $m=0\Rightarrow \frac{1\cdot 300+0}{1+0}=300$ ✓; plug a huge $m=10^6\Rightarrow \frac{300+10^6\cdot 466.67}{1+10^6}\approx 466.67$, arbitrarily close to the global mean ✓. So $m$ is a bias–variance dial — pick it with Cross-Validation.

Read the figure below: the red curve is the smoothed Portland code as $m$ slides from $0$ to large. It starts pinned to the lower dashed line (raw mean 300) and bends smoothly up toward the upper dashed line (global mean 466.67). Reading it left-to-right is reading the bias–variance trade-off: left edge overfits, right edge underfits.

---

Word problem + exam twist (I, J)

Forecast: Wed clearly tips best — but Tue and Fri each have thin data. Guess which of those shrinks harder.

1. Counts and group means. Mon $n=2,\overline{y}=9$; Tue $n=1,\overline{y}=9$; Wed $n=3,\overline{y}=20$; Fri $n=1,\overline{y}=15$. Why this step? Establish evidence per category before trusting any of them.
2. Global mean. ${\overline{y}}_{\text{global}}=\frac{8+10+9+20+22+18+15}{7}=\frac{102}{7}\approx 14.571$. Why this step? This is the prior everything shrinks toward, and the fallback for Thu.
3. Smooth each with $m=4$: $\text{Mon}=\frac{2\cdot 9+4\cdot 14.571}{6}=\frac{18+58.286}{6}\approx 12.714$ $\text{Tue}=\frac{1\cdot 9+4\cdot 14.571}{5}=\frac{9+58.286}{5}\approx 13.457$ $\text{Wed}=\frac{3\cdot 20+4\cdot 14.571}{7}=\frac{60+58.286}{7}\approx 16.898$ $\text{Fri}=\frac{1\cdot 15+4\cdot 14.571}{5}=\frac{15+58.286}{5}\approx 14.657$ Why this step? Notice Tue (raw 9, thin data) is pulled up to 13.46, while Wed (strong, well-populated 20) keeps most of its edge at 16.90.
4. Thu is unseen → global mean $\approx 14.571$ (Case G logic).

Verify: every smoothed value lies between its raw mean and 14.571 ✓; the ranking of smoothed codes still puts Wed on top ($16.90$ highest) ✓; Tue moved further from its raw mean than Wed did (thin vs thick data) ✓.

---

Forecast: does a row's own target sneak into its own feature?

1. Spot the leak. For Portland (the only row), ${\overline{y}}_{\text{Por}}=y_{\text{Portland}}=300$. The encoded feature for that row contains its own label. The model can "cheat" by memorising the code. Why this step? This is why the parent note screams don't fit on the full data.
2. Fold-safe fix (out-of-fold encoding). Split rows into $K$ folds; encode each fold using means computed from the other folds only, so a row is never used to build its own code. Concretely with 2 folds: put NYC-row-1 in fold A, encode it using only NYC rows in fold B. Why this step? It restores the train/apply separation within the training set — the same principle as Cross-Validation.
3. Numeric contrast. Leaky Portland code $=300$ (exact label). A fold-safe code, having no other Portland rows to average, falls back to the global mean $466.67$. The gap $466.67-300\approx 166.67$ is precisely the leaked information you just removed.

Verify: leaky code equals the raw label exactly ($=y_i$) for any $n_c=1$ category ✓; the fold-safe code for a singleton reduces to the global mean $466.67$ ✓; their difference $\approx 166.67$ is nonzero, confirming leakage was real ✓.

---

Recall Quick self-test

Portland smoothed code with $m=10$, global mean $466.67$ ::: $4966.67/11\approx 451.5$ What weight $w$ does a category with $n_c=3$ get when $m=10$? ::: $3/13\approx 0.231$ Unseen test category encodes to ::: the global mean ($466.67$ here) ${\lim }_{m\to \infty }{\text{TE}}_{\text{smooth}}$ ::: ${\overline{y}}_{\text{global}}$ Why is ordinal encoding wrong for Color? ::: it invents an order the feature does not have (nominal, not ordinal)