Binning and discretization: Exercises — AI-ML deep dive

Two symbols recur, so let us pin them down once, in plain words, before anything else:

The figure below is the mental picture to keep for every exercise on this page: notice the six teal fence-posts labelled $b_{0} \dots b_{5}$ but only five orange gaps between them. Whenever a problem asks for edges vs. bins, re-run this picture in your head — the top arrow counts posts, the bottom arrow counts gaps.

Two more conventions we will lean on repeatedly, so let us fix them now:

Definition Open (infinite) edges and out-of-range values

Infinite edge $\infty$ (or $- \infty$ ): a fence-post placed "infinitely far out" so that the outermost bin has no upper (or lower) cap. In code this is not a literal number; libraries represent it as numpy.inf / -numpy.inf (or an open right side). Its meaning is simply "every value from the previous edge upward, with no ceiling." So the bin $[100, \infty)$ means " $100$ or anything larger, forever."
Out-of-range values. If your edges were fitted on training data with a finite first edge $b_{0}$ and last edge $b_{k}$ , a fresh value $x < b_{0}$ or $x > b_{k}$ (when $b_{k}$ is finite) belongs to no bin. Real libraries return a "missing" label (NaN in pandas.cut) for such points. The two standard fixes are: (a) set the outer edges to $- \infty$ and $+ \infty$ so nothing can escape, or (b) clip values to $[b_{0}, b_{k}]$ before binning. We use fix (a) whenever we want a guaranteed home for every future value.

Level 1 — Recognition

L1.1 · Which strategy?

You have website session-duration data in seconds: most sessions are 5–60 s, but a handful last hours. You want 4 bins that each hold roughly the same number of sessions. Name the binning strategy.

Recall Solution

Equal-frequency (quantile) binning. The phrase "each hold roughly the same number of sessions" is the definition of equal-frequency. Equal-width would put nearly everything in the first bin because the data is skewed by the multi-hour outliers.

L1.2 · Count the edges

You build 5 bins. How many bin edges $b_{i}$ are there?

Recall Solution

6 edges. There is always one more fence-post than gaps: edges $b_{0}, b_{1}, b_{2}, b_{3}, b_{4}, b_{5}$ create 5 gaps (bins). Formula: $k$ bins $\Rightarrow k + 1$ edges.

L1.3 · Which bin?

Edges are $[0, 15, 50, 100, \infty)$ (the final $\infty$ means "no upper cap", as defined above) with the last-bin-includes-its-left-edge convention. Which bin (1–4) does income $= 50$ fall into?

Recall Solution

Bin 3, $[50, 100)$ . The rule is $b_{i - 1} \leq x < b_{i}$ . Since $50$ is the left edge of bin 3, it belongs to bin 3, not bin 2 whose right edge $50$ is excluded. (For the record, the top bin here is $[100, \infty)$ — any income of $100$ or more, with no ceiling, so even a billionaire has a home.)

Level 2 — Application

L2.1 · Equal-width edges

Data range: $min = 20$ , $max = 200$ . Build $k = 4$ equal-width bins. Give the bin width and all edges.

Recall Solution

What we do: split the range into 4 equal pieces.

Range $R = max - min = 200 - 20 = 180$ .
Width $w = R / k = 180/4 = 45$ .
Edges $b_{i} = min + i \cdot w$ : $20, 65, 110, 155, 200$ .

So the 4 bins are $[20, 65), [65, 110), [110, 155), [155, 200]$ . Why equal-width? Because we were given nothing about density — we only partition the number line, ignoring how many points sit where.

L2.2 · Equal-frequency edges

Sorted data: $[3, 7, 7, 10, 12, 18, 25, 40]$ ( $n = 8$ ). Build $k = 4$ equal-frequency bins. Give the interior edges (the three cut-points).

Recall Solution

What we do: each bin should hold $n / k = 8/4 = 2$ points, so we cut between the 2nd-and-3rd, the 4th-and-5th, and the 6th-and-7th sorted values. The fence goes at the midpoint of each such pair.

Between value 2 ( $7$ ) and value 3 ( $7$ ): cut $= \frac{7 + 7}{2} = 7.0$ .
Between value 4 ( $10$ ) and value 5 ( $12$ ): cut $= \frac{10 + 12}{2} = 11.0$ .
Between value 6 ( $18$ ) and value 7 ( $25$ ): cut $= \frac{18 + 25}{2} = 21.5$ .

Interior edges $= [7.0, 11.0, 21.5]$ . Check the bins:

Bin 1 $[3, 7.0)$ : ${3}$ — only 1 point, because two values equal $7$ and the fence sits exactly on them, pushing both $7$ s right.
Bin 2 $[7.0, 11.0)$ : ${7, 7, 10}$ — 3 points.
Bin 3 $[11.0, 21.5)$ : ${12, 18}$ — 2 points.
Bin 4 $[21.5, 40]$ : ${25, 40}$ — 2 points.

Why not exactly 2-2-2-2? The tie at $7$ lands on a fence-post. Because bins are left-closed / right-open, both $7$ s fall into the bin on their right, so the target count cannot be met exactly. Perfect balance is a target, not a guarantee, whenever values are tied on an edge — a preview of the mastery-level problem L5.2.

L2.3 · Sturges' rule

You have $n = 1000$ samples. How many bins does Sturges' rule suggest?

Recall Solution

Sturges: $k = ⌊ 1 + lo g_{2} n ⌋$ . Here $lo g_{2} 1000 \approx 9.97$ , so $k = ⌊ 1 + 9.97 ⌋ = ⌊ 10.97 ⌋ = 10$ bins. Why $lo g_{2}$ ? Each bin edge is one binary "above/below" split; with $k$ splits you distinguish $2^{k}$ patterns, and setting $2^{k} \approx n$ gives $k \approx lo g_{2} n$ .

Level 3 — Analysis

L3.1 · Diagnose the empty bin

Data $[1, 2, 3, 4, 5, 100]$ , equal-width, $k = 3$ . Show the bins and explain why one is empty.

Recall Solution

$R = 100 - 1 = 99$ , $w = 33$ , edges $[1, 34, 67, 100]$ .

$[1, 34)$ : ${1, 2, 3, 4, 5}$ → 5 points
$[34, 67)$ : ${}$ → 0 points
$[67, 100]$ : ${100}$ → 1 point

Why empty? Equal-width places fence-posts by distance, not by data. The lone outlier $100$ stretches the range so wide that the two lower posts land in a region where no data lives. The middle bin is empty because there simply are no values between 34 and 67.

L3.2 · Same data, equal-frequency

Repeat L3.1 with equal-frequency, $k = 3$ , $n = 6$ . Give the bins.

Recall Solution

Target $n / k = 2$ per bin. Sorted: $[1, 2, 3, 4, 5, 100]$ .

Cut between 2nd ( $2$ ) and 3rd ( $3$ ) → edge $2.5$ ; cut between 4th ( $4$ ) and 5th ( $5$ ) → edge $4.5$ .
Bin 1 $[1, 2.5)$ : ${1, 2}$ ; Bin 2 $[2.5, 4.5)$ : ${3, 4}$ ; Bin 3 $[4.5, 100]$ : ${5, 100}$ .

Analysis: every bin now holds 2 points — the outlier $100$ is quietly absorbed into the top bin instead of blowing up the geometry. This is exactly the outlier robustness the parent note promised. See 2.3.01-Handling-missing-data for the sibling idea of taming bad values.

L3.3 · Bias–variance direction

As you increase $k$ (more, narrower bins), does model bias go up or down? Does variance? One sentence each.

Recall Solution

Bias goes DOWN. Narrower bins hug the true curve better — less "flatness" assumed inside each bin.
Variance goes UP. Each bin now holds fewer points, so its label is more swayed by individual noise.

This is the classic tug-of-war; details in 4.3.02-Bias-variance-tradeoff.

Level 4 — Synthesis

L4.1 · Log-then-bin

Skewed income data spans $\$ 1{,}000 $t o$ $1{,}000{,}000 $(i . e .$ $1 $k t o$ $1000 $k) . Y o u w an t 3 e q u a l - w i d t hbin s * * a f t er * * a$ \log_{10}$ transform. Give the bin edges in actual dollar amounts.

Recall Solution

What we do: equal-width in log-space, then map back to dollars. To avoid unit confusion, we carry everything in thousands of dollars internally, then multiply by $1, 000$ at the very end to report real dollars.

Work in thousands: $\$ 1 $k$ \to $v a l u e$ 1 $,$ $1000 $k$ \to $v a l u e$ 1000$.
$lo g_{10}$ range: $lo g_{10} 1000 - lo g_{10} 1 = 3 - 0 = 3$ .
Log-width $= 3/3 = 1$ . Log edges: $0, 1, 2, 3$ .
Map back with $1 0^{edge}$ (still in thousands): $1 0^{0} = 1, 1 0^{1} = 10, 1 0^{2} = 100, 1 0^{3} = 1000$ .
Multiply by $1, 000$ to get actual dollars: $\ \$ 1{,}000,\ $10{,}000,\ $100{,}000,\ $1{,}000{,}000$.

Edges in real dollars: $[\$ 1{,}000,\ $10{,}000,\ $100{,}000,\ $1{,}000{,}000] $. S o t h e t h r ee d o l l a r bin s a r e$ [$1\text{k},$10\text{k}) $,$ [$10\text{k},$100\text{k}) $,$ [$100\text{k},$1000\text{k}] $. * * W h y l o g f i r s t ? * * I n d o l l a r s p a ce t h ese bin s a r e w i l d l y u n e q u a l ($ $9 $k,$ $90 $k,$ $900$k wide) — but that is exactly what skewed data needs: wide bins where data is sparse (the rich tail), narrow where it is dense (the low end). Equal-width in log-space is a cheap way to get near-equal frequency in original space.

L4.2 · Pipeline ordering

You will do 5-fold cross-validation with equal-frequency binning. Write the correct order of the two operations (fit quantiles / split into folds) and explain in one line why.

Recall Solution

Correct order: split into folds first, then fit the quantile edges on the training fold only, and apply those same edges to the held-out fold. Why: if you compute quantiles on the whole dataset first, the test fold's values secretly influenced the edges — that is leakage and it inflates your score. Fit on train, transform test.

L4.3 · Trees don't need it (mostly)

Your teammate one-hot-encodes (2.2.03-One-hot-encoding) binned features and feeds them to a decision tree (3.1.04-Decision-trees). Why is binning-before-trees often redundant, and when is it still useful?

Recall Solution

Redundant because: a decision tree already chooses its own split thresholds on the raw continuous feature — it is essentially learning optimal bin edges automatically. Pre-binning just freezes those edges to your (possibly worse) choices. Still useful when: (a) you want the same human-interpretable buckets across many models/reports, (b) you deliberately encode domain thresholds the tree might not find with limited data, or (c) you are enabling 2.1.09-Feature-interactions between coarse categories.

Level 5 — Mastery

L5.1 · Degenerate: all values identical

Data $[5, 5, 5, 5, 5]$ . What happens with equal-width $k = 3$ ? With equal-frequency $k = 3$ ? How should code handle it?

Recall Solution

Equal-width: $R = max - min = 5 - 5 = 0$ , so width $w = 0/3 = 0$ . All edges collapse to $5$ ; the bins are empty degenerate intervals. Division is fine ( $0/3 = 0$ ) but the result is meaningless.
Equal-frequency: every quantile of a constant is $5$ , so all interior edges equal $5$ — again collapsed. You cannot make 3 non-empty bins from one distinct value.
Handling: detect $max = min$ (zero variance) and either drop the feature or emit a single bin. Robust libraries either merge duplicate edges or raise a warning; never trust silent output.

L5.2 · Duplicate quantile edges

Sorted data $[1, 1, 1, 1, 1, 2]$ , equal-frequency $k = 3$ . The lower quantiles all land on $1$ . What breaks, and what is the standard fix?

Recall Solution

Target $n / k = 6/3 = 2$ per bin ⇒ cut between the 2nd-and-3rd values and between the 4th-and-5th values. But positions 1–5 are all $1$ , so:

Cut between value 2 ( $1$ ) and value 3 ( $1$ ): midpoint $= \frac{1 + 1}{2} = 1.0$ .
Cut between value 4 ( $1$ ) and value 5 ( $1$ ): midpoint $= \frac{1 + 1}{2} = 1.0$ .

Both interior edges equal $1.0$ → duplicate edges → a zero-width, empty middle bin. Standard fix: pandas.qcut(..., duplicates="drop"), which merges the duplicate edges. You then get fewer bins than requested (here effectively 2: $[1, 1.0)$ is empty and drops, leaving a " $= 1$ " bin and a " $> 1$ " bin), which is correct — the data simply cannot support 3 equal-frequency bins. Lesson: requested $k$ is an upper bound, not a guarantee, for equal-frequency on discrete/heavy-tie data.

L5.3 · Information-loss sanity check

A feature is binarized (thresholded) into 2 equal-frequency bins on data $[1, 2, 3, 4]$ . Compute the entropy $H (f (X))$ of the bin labels (in bits) and state the information kept.

Recall Solution

Two equal bins each hold 2 of 4 points ⇒ probabilities $p = [0.5, 0.5]$ . $H = - \sum_{i} p_{i} lo g_{2} p_{i} = - (0.5 lo g_{2} 0.5 + 0.5 lo g_{2} 0.5) .$ Why $lo g_{2} 0.5 = - 1$ ? $lo g_{2}$ asks "2 to what power gives this number?" Since $2^{- 1} = \frac{1}{2} = 0.5$ , the exponent is $- 1$ , so $lo g_{2} 0.5 = - 1$ . Substituting: $H = - (0.5 \cdot (- 1) + 0.5 \cdot (- 1)) = - (- 0.5 - 0.5) = - (- 1) = 1 bit.$ The label carries exactly 1 bit — the maximum possible for 2 outcomes. Every drop of order information below the median vs. above it is preserved; everything within each half is lost.

Recall Self-test: rebuild the ladder from memory

Equal-width divides the ::: number line (range) into equal pieces Equal-frequency divides the ::: data (puts equal counts per bin) $k$ bins require how many edges ::: $k + 1$ Half-open bins mean a boundary value goes to the bin on its ::: right A value below the first edge $b_{0}$ (or above a finite $b_{k}$ ) gets ::: no bin (a "missing"/NaN label) unless you use $\pm \infty$ edges or clip Fit quantile edges on which data during CV ::: the training fold only Sturges' rule formula ::: $k = ⌊ 1 + lo g_{2} n ⌋$