Visual walkthrough — Ordinal and target encoding

Ordinal encoding was the easy sibling — you just line categories up in order and hand them the numbers $1,2,3,\dots$. This page is about the harder, more interesting sibling: target encoding (we will abbreviate it TE once we have earned every piece of it).

Here is where we are headed — but read it only as a destination photo, not as something you should understand yet. Every symbol in it (the count, the two averages, the smoothing number) will be drawn from scratch in the steps below before we ever use it:

${\text{TE}}_{\text{smooth}}(c)=\frac{n_c\cdot {\overline{y}}_c+m\cdot {\overline{y}}_{\text{global}}}{n_c+m}\text{(don’t panic — we build this piece by piece)}$

We are going to assemble it from absolutely nothing — no statistics assumed, no symbol used before it is drawn. By the end you will feel why that fraction is the only sensible shape it could have.

Back to the parent for the full picture: Ordinal and Target Encoding.

---

Step 1 — The problem: a column made of words

WHAT. We have a table. One column is a category (a word like NYC), and one column is the thing we want to predict — a number we will call the target, written $y$ (think "house price"). We will refer to the rows by a counter $i=1,2,3,\dots$, so row $i$ has a category $x_i$ and a target $y_i$.

WHY. A machine-learning model multiplies and adds numbers. It cannot multiply the word NYC. So a column of words is, to the model, invisible. We need to turn each word into a number — but not just any number: one that carries information about $y$.

PICTURE. On the left, the raw table: a City column of words, a Price column of numbers. There is no arrow the model can draw from NYC to a price, because NYC is not on the number line at all.

Contrast this with One-hot Encoding, which would spawn one new column per city. With 500 cities that is 500 mostly-zero columns. That blow-up in width is one face of the Curse of Dimensionality — more precisely, it makes each row a sparse point in a 500-dimensional space where distances lose meaning and models need far more data to learn; the curse is broader than "too many columns", but wide sparse encodings are a common way to walk into it. We want one column instead.

---

Step 2 — The first honest idea: replace each word with its group's average

WHAT. For a category $c$, gather every row whose city is $c$, and take the average of their targets. Call this ${\overline{y}}_c$, the "category mean". Then everywhere the word $c$ appeared, write ${\overline{y}}_c$.

WHY. If NYC rows are usually expensive, their average target is high; if Portland rows are usually cheap, their average is low. So the single number ${\overline{y}}_c$ summarises what that word tells you about $y$. That is exactly the signal we wanted to rescue from Step 1.

PICTURE. Each group of same-coloured dots (rows of one city) collapses onto a single horizontal level — its mean. NYC lands high, Portland lands low. The words have become positions on the number line.

Term by term (recall $i$ is the row counter from Step 1):

Worked numbers (from the parent's Example 3): NYC has targets $500,600,550$, so ${\overline{y}}_{\text{NYC}}=\frac{500+600+550}{3}=\frac{1650}{3}=550.$ Likewise ${\overline{y}}_{\text{Seattle}}=425$ and ${\overline{y}}_{\text{Portland}}=300$.

---

Step 3 — Where the honest idea breaks: the group of one

WHAT. Look at Portland. It appears once, with target $300$. Its "average" is just that single number, $300$.

WHY it breaks. One data point is not a trend — it is a coin flip. If that one Portland house happened to be a mansion, its target would be huge, and we'd stamp Portland = huge forever. We are letting noise dress up as signal. A model that trusts this will overfit: it memorises a fluke instead of learning a pattern.

PICTURE. Two groups side by side. The NYC group (many dots, tightly clustered) has a mean we can trust — move any one dot and the average barely budges. The Portland group (a single lonely dot) has a mean that is entirely that one dot — move it and the mean flies with it. The error bars tell the story: wide for the lonely group, narrow for the crowded one.

---

Step 4 — The safety net: the global mean

WHAT. Compute the average target over the entire training set, ignoring which city each row is. Call it ${\overline{y}}_{\text{global}}$.

WHY. When we don't trust a group's own average, we need a fallback — a "default guess" for a house price when the city tells us nothing reliable. The overall average is the least-biased default: it is what you'd predict knowing only "this is a house in the dataset".

PICTURE. All dots, every colour, pooled together; one dashed line runs through the whole cloud at their common average. This line is the anchor we will pull unreliable groups toward.

Worked numbers. All six prices: $500,600,550,400,450,300$. ${\overline{y}}_{\text{global}}=\frac{500+600+550+400+450+300}{6}=\frac{2800}{6}\approx \mathrm{466.7.}$

(The parent's Example 3 uses a group-weighted phrasing $\frac{550\cdot 3+425\cdot 2+300\cdot 1}{6}$ — that equals the very same $2800/6$, because summing group-means-times-counts just re-adds the raw values.)

---

Step 5 — Blend the two: a slider between "own mean" and "safe mean"

WHAT. For each category, take a weighted average of its own mean ${\overline{y}}_c$ and the global mean ${\overline{y}}_{\text{global}}$:

$\text{TE}(c)=w\cdot {\overline{y}}_c+(1-w)\cdot {\overline{y}}_{\text{global}},0\le w\le 1.$

WHY a weighted average? Because it is the one shape that smoothly slides between our two known endpoints:

• $w=1$ → trust the group fully → $\text{TE}(c)={\overline{y}}_c$.
• $w=0$ → trust the group not at all → $\text{TE}(c)={\overline{y}}_{\text{global}}$.

The weights $w$ and $1-w$ sum to 1, so the result always lands between the two values — never overshooting either. That "sum to one" is what makes it an average and not just a random combination.

PICTURE. A slider. At the left end sits the global mean (safe); at the right end sits the group's own mean (risky but informative). The dot on the slider is the encoded value; $w$ is how far right it sits.

Every symbol here (note $\text{TE}$ finally spells out Target Encoding — the number that replaces the word):

---

Step 6 — Choosing the knob: let the count decide

WHAT. We must pick $w$. From Step 3 we already know the rule it must obey: big group ⇒ $w$ near 1; tiny group ⇒ $w$ near 0. The simplest formula that does exactly this is

$w=\frac{n_c}{n_c+m}.$

WHY this exact fraction? Test it against the two demands:

• Huge group, $n_c\to \infty$: the $+m$ in the bottom becomes negligible, so $w\to \frac{n_c}{n_c}=1$. ✓ Full trust.
• Empty/tiny group, $n_c\to 0$: numerator $\to 0$, so $w\to \frac{0}{m}=0$. ✓ No trust.
• And in between it rises smoothly from 0 to 1 as $n_c$ grows.

The number $m$ is the smoothing strength: it is the group size at which trust hits exactly one-half ($n_c=m\Rightarrow w=\frac{1}{2}$). Small $m$ = trust groups quickly; large $m$ = stay cautious longer.

PICTURE. Plot $w$ against $n_c$ for a couple of values of $m$. Each curve starts at 0, climbs steeply for small counts, then flattens toward 1. Mark the point where $n_c=m$ and $w=\frac{1}{2}$.

---

Step 7 — Substitute and simplify to the final formula

WHAT. Put $w=\frac{n_c}{n_c+m}$ into the blend of Step 5.

WHY. This turns two ideas — "weighted average" and "count-based trust" — into one closed formula we can compute directly.

$${\text{TE}}_{\text{smooth}}(c)=\underset{w}{\underbrace{\frac{n_c}{n_c+m}}}{\overline{y}}_c+\underset{1-w}{\underbrace{\frac{m}{n_c+m}}}{\overline{y}}_{\text{global}}$$

Both terms share the denominator $n_c+m$, so combine the numerators:

$$\boxed{{\text{TE}}_{\text{smooth}}(c)=\frac{n_c{\overline{y}}_c+m{\overline{y}}_{\text{global}}}{n_c+m}}$$

Reading the boxed formula as a story. The top is "the group's evidence ($n_c$ rows saying ${\overline{y}}_c$) plus $m$ pretend rows all saying ${\overline{y}}_{\text{global}}$." The bottom is the total headcount, real plus pretend. It is just an average over $n_c+m$ voters, where $m$ of them are planted at the safe value. That is the empirical-Bayes trick in one line.

Term by term:

PICTURE. A pan balance. On the pan sit real dots (weight $n_c$) tugging toward ${\overline{y}}_c$, and greyed pseudo-dots (weight $m$) tugging toward ${\overline{y}}_{\text{global}}$. The pointer settles at ${\text{TE}}_{\text{smooth}}$ — closer to whichever side has more mass.

---

Step 8 — Watch it work on every case

WHAT. Apply the boxed formula to all three cities, with $m=10$ and — matching the parent's Example 4 — its rounded global mean ${\overline{y}}_{\text{global}}=475$ (the exact value from Step 4 is $466.7$; we use $475$ only to line up with the parent's arithmetic, as flagged in Step 4).

WHY. A formula you have only derived is still abstract. Running it on the lonely group, the crowded group, and the in-between group lets you see the smoothing behave differently in each — this is the moment the algebra turns into intuition, and it also lets us confirm the degenerate cases (tiny group, unseen group) do what Step 6 promised.

Portland ($n_c=1$, ${\overline{y}}_c=300$) — the lonely group: $\text{TE}=\frac{1\cdot 300+10\cdot 475}{1+10}=\frac{5050}{11}\approx 459.$ It got yanked almost all the way from $300$ up to the safe $475$ — exactly right, we didn't trust it.

NYC ($n_c=3$, ${\overline{y}}_c=550$) — the crowded group: $\text{TE}=\frac{3\cdot 550+10\cdot 475}{3+10}=\frac{6400}{13}\approx 492.$ It stayed much closer to its own $550$ side than Portland stayed to its own — because $w=3/13\approx 0.23$ went to the global mean, leaving $0.77$ on the group. More data, more trust.

Seattle ($n_c=2$, ${\overline{y}}_c=425$): $\text{TE}=\frac{2\cdot 425+10\cdot 475}{2+10}=\frac{5600}{12}\approx 467.$

PICTURE. Three horizontal arrows, one per city, each starting at the raw group mean and ending at the smoothed value, all bending toward the dashed global-mean line. The length of each arrow shrinks as the group grows — a visual proof that confidence buys independence.

Note the encoded values ($459,492,467$) live on the target's scale (hundreds of thousands of dollars), so combine target encoding with Feature Scaling before feeding scale-sensitive models.

---

The one-picture summary

This last figure stacks the whole journey: words → group means → a tug-of-war against the global mean, weighted by group size → the final smoothed number line. Read it top to bottom and you have re-derived the boxed formula.

Recall Feynman retelling — say it back in plain words

We started with a column of words, which a model can't do arithmetic on. So we replaced each word with the average target of its group — expensive cities get a high number, cheap ones a low number. Great, except a group with only one row gives an average that's really just noise pretending to be a pattern. To fix that, we found a safe fallback: the average over everyone. Then we said "blend the group's own average with the safe one," using a weighted average so the answer always lands between them. The weight had to obey one rule — trust big groups, distrust tiny ones — and the simplest formula for that is $\frac{n_c}{n_c+m}$, which is 1 for huge groups and 0 for empty ones. Substituting it and cleaning up gives $\frac{n_c{\overline{y}}_c+m{\overline{y}}_{\text{global}}}{n_c+m}$: literally an average over the real rows plus $m$ imaginary rows glued to the safe value. Big groups drown out the imaginary rows and keep their own opinion; tiny groups get overruled and slide back to safe. And a brand-new, never-seen category has zero real rows, so it just returns the safe average — the formula handles it for free.

Recall

Why does $w=\frac{n_c}{n_c+m}$ approach 1 for large groups? ::: The $+m$ becomes negligible next to a huge $n_c$, so the fraction $\to n_c/n_c=1$ — full trust in the group's own mean. What is the encoding for a category unseen in training? ::: ${\overline{y}}_{\text{global}}$, because $n_c=0$ collapses the formula to $\frac{m{\overline{y}}_{\text{global}}}{m}$. In one sentence, what does $m$ represent? ::: The number of pseudo-observations pinned at the global mean — the group size at which trust is exactly one-half. How should you choose $m$ in practice? ::: Treat it as a hyperparameter — try several values and pick the best held-out score under cross-validation, starting near the typical group size.