Foundations — Ordinal and target encoding

Before you can read the parent note, you must be able to read its symbols. This page builds every one of them from nothing — no algebra assumed. We go slowly, in the order they depend on each other, so that by the last section every letter — including the smoothed encoding the parent calls ${\text{TE}}_{\text{smooth}}(c)$ — already means something to you.

---

0. What is a "category" and a "feature"?

Picture a spreadsheet. Each row is one example (one house, one student). Each column is one feature. A categorical column stores words.

Figure s01. A data table. The yellow header names each column (feature). The blue column holds words — a categorical feature. Reading down that blue column, notice NYC appears more than once: rows repeat, but distinct labels are few. That gap between "how many rows" and "how many distinct labels" is the theme of the next two sections.

Why not just number them randomly (NYC=1, Seattle=2, Portland=3)? Because the model would then believe Portland − NYC = 2 steps apart, an ordering you never intended. Encoding is only correct when the numbers mean something. That is what ordinal and target encoding each guarantee, in two different ways.

---

1. The set: $\{c_1,c_2,\dots ,c_k\}$

The three dots $\dots$ mean "and so on in the same pattern". So $\{c_1,c_2,\dots ,c_k\}$ = "the first category, the second, ..., up to the $k$-th".

The word cardinality just means "the value of $k$" — the number of distinct categories. "High-cardinality" (the parent's motivation for target encoding) simply means $k$ is large, e.g. 500 cities. Keep that word — it decides which encoding you pick. See Curse of Dimensionality for why large $k$ hurts.

---

2. The subscript index $i$ and the samples $x_1,\dots ,x_n$

$n$ and $k$ are different counts:

• $n$ = how many rows total (could be millions).
• $k$ = how many distinct labels appear (could be 4, or 500).

---

3. The mapping function $f$ and the arrow $\to$

Think of a function as a lookup dictionary: you hand it a word, it hands back a number. $f_{\text{ord}}$ is the ordinal dictionary; $f_{\text{target}}$ is the target dictionary. The subscript on $f$ just tells you which dictionary.

Figure s02. The mapping $f_{\text{ord}}$ drawn as arrows. Each ordered word on the left (blue) points to one number on the right (yellow). Crucially the words are stacked in their real-world order S, M, L, XL — so the arrows land on 1, 2, 3, 4 in the same order. If you shuffled the left column the arrows would still work as a dictionary, but the meaning (order) would be lost. That ordering choice is the subject of the next callout.

Read aloud: "the ordinal code of the $i$-th category is its position number $i$." So the smallest category gets 1, the next gets 2, and so on. That is the whole formula — the parent's fancy line is just this dictionary written compactly.

---

4. The condition line $\{i:x_i=c\}$ — "the rows where..."

This is the single scariest-looking symbol in the parent note. Let us disarm it.

Figure s03. The filter in action on a city column. Every row is listed; the pink boxes are the ones surviving the test $x_i=\text{NYC}$ — rows 1, 2, 4. Only those pink rows' targets (500, 600, 550) get added and averaged. The greyed-out Seattle and Portland rows are simply invisible to this particular category's calculation. Change $c$ to Seattle and a different set of rows lights up.

Why the topic needs it: target encoding must gather only the rows of one category to average their outcomes. This filter is how the maths says "look at just the NYC houses."

---

5. Sum $\sum$, count $n_c$, the mean, and the target $f_{\text{target}}$

Now the mean falls out naturally. A mean (average) is "total divided by count":

The bar over $y$ (read "y-bar") is standard shorthand for "the mean of". The little "${\sum }_{i=1}^n$" means "add $y_1$ through $y_n$, every row". The difference between the two formulas is which rows you include: ${\overline{y}}_c$ filters to one category, ${\overline{y}}_{\text{global}}$ takes them all.

Now we can finally fill in the second dictionary from §3 — the one we named but left blank.

---

6. The weighted average and the smoothing $w$, $m$

The last symbols appear in smoothed target encoding. They are all built from means you now understand.

${\text{TE}}_{\text{smooth}}(c)=w\cdot {\overline{y}}_c+(1-w)\cdot {\overline{y}}_{\text{global}}$

Figure s04. The slider. Blue pin = category mean (300 for lonely Portland); yellow pin = global mean (475). The pink dot is where a category with $n_c=1$ lands — flung almost onto the global pin because we barely trust one observation. The blue square shows a category with $n_c=100$ landing right next to its own mean. The dot's position is the weight $w$: near the yellow pin means $w$ near 0, near the blue pin means $w$ near 1.

Cover-all-cases table:

situation	$n_c$	$w=\frac{n_c}{n_c+m}$	encoding leans toward
unseen at test time	$n_c=0,m=10$	$0/10=0$	global mean (exactly)
lonely category	$n_c=1,m=10$	$1/11\approx 0.09$	global mean (safe)
medium category	$n_c=3,m=10$	$3/13\approx 0.23$	still mostly global
big category	$n_c=100,m=10$	$100/110\approx 0.91$	its own mean (trusted)
$m=0$ (no smoothing)	any $n_c>0$	$1$	raw mean — back to unsmoothed

---

Prerequisite map

Related vault stops once you are through here: One-hot Encoding (the wasteful alternative for high $k$), Feature Scaling (what to do with the numbers afterward), Cross-Validation (how to avoid target leakage), Tree-based Models and Overfitting (why smoothing matters).

---

Equipment checklist

Say the answer aloud before revealing.