Decision tree structure and terminology: Visual walkthrough

We use only one starting fact: we have a table of points, each with two numbers we can measure (call them features) and one thing we want to guess (call it the target). Everything else we build.

Step 1 — A dataset is just dots on a plane

WHAT. Picture every training example as a single dot. We measure two features per example, so we place the dot at horizontal position $x_{1}$ and vertical position $x_{2}$ . The colour of the dot is its class (the answer we eventually want to guess) — say coral dots are class $A$ , mint dots are class $B$ .

WHY. Before we can "split" anything, we need to know what we are splitting. A decision tree partitions the space of possible feature values. If we draw that space, every later idea becomes a shape we can point at.

PICTURE.

Figure — Decision tree structure and terminology

There are no questions yet — just data sitting in space. Notice the colours are mixed: coral and mint are tangled together. Our whole job is to ask questions that separate the colours.

Step 2 — One question is one straight cut

WHAT. A single internal test has the form " $x_{j} \leq t$ ?" — read as "is feature $j$ at most the threshold $t$ ?" Pick $j = 1$ (feature 1) and some number $t$ . Every dot either satisfies it (goes left) or fails it (goes right).

WHY this shape and not a diagonal? Because " $x_{1} \leq t$ " only looks at one axis, it draws a line perpendicular to that axis — a clean vertical wall at $x_{1} = t$ . A test like $x_{1} + x_{2} \leq 5$ would draw a slanted wall, which is harder to search and harder to read as a rule. Testing one feature at a time is cheap (just sort that feature and try cut points) and gives an English rule ("if income ≤ 40k…"). This is the axis-aligned choice.

PICTURE.

Step 3 — Stacking questions chops space into boxes

WHAT. Take the left region from Step 2 and ask a second question inside it — this time on the other axis, " $x_{2} \leq s$ ?". Now the left region is split into a bottom box and a top box. Do this recursively and the plane fills with rectangles.

WHY. Each question can only refine a region it already lives in — a child node only sees the dots its parent handed down. Because every wall is axis-aligned, the pieces you get can only ever be hyper-rectangles (plain rectangles in 2-D). There is no way to make a triangle or a circle with axis-aligned walls.

PICTURE.

Each numbered box in the figure corresponds to one leaf of the tree drawn beside it. The tree and the boxes are two views of the same object.

Step 4 — What should one box predict? (classification)

WHAT. Zoom into a single box holding the labels ${A, A, B, A, B}$ — three coral, two mint. We must commit to one prediction $\overset{y}{^}_{ℓ}$ for the whole box. Which colour?

WHY. Whatever single label we pick is "wrong" for every dot of the other colour in the box. So the number of mistakes we make is "count of dots not equal to our guess." To make the fewest mistakes we should pick the colour that is most common — the majority. Any dot left over is a mistake, and the smallest leftover comes from choosing the biggest group.

PICTURE.

The how-pure-is-this-box question — how we score a split before choosing it — is the job of Gini impurity and entropy. This step only decides what a box outputs once fixed.

Step 5 — What should one box predict? (regression)

WHAT. Now the target is a number, not a colour. A box holds targets ${10, 12, 14, 20}$ . We must pick one number $\overset{y}{^}_{ℓ}$ for the whole box. Which number is best?

WHY. For numbers we measure "wrongness" by squared error: the gap between each true value and our guess, squared so that being off by $+ 3$ or $- 3$ costs the same and big misses hurt more. We want the single number that makes the total squared gap as small as possible. To find the bottom of that valley we use the derivative — the tool that answers "at what value does the total stop decreasing and start increasing?" At the very bottom the slope is zero.

PICTURE.

Step 6 — Every box is flat → the staircase surface

WHAT. Put the last three steps together: each box outputs one constant (a colour, or the mean number). So across the whole plane the prediction is flat inside each box and jumps at the walls. Draw the boundary between "predict $A$ " and "predict $B$ " regions and it looks like steps.

WHY. A constant per box is by definition a piecewise-constant function. It cannot bend or slope inside a box — it can only jump at a wall, and walls are axis-aligned. That is exactly what a staircase is: flat treads (boxes) joined by vertical risers (walls). A single tree can therefore never draw a smooth diagonal boundary — only a staircase approximation of one.

PICTURE.

Step 7 — Edge cases: pure boxes, ties, and the depth trap

WHAT. Three degenerate situations, each shown in the figure:

Pure box — every dot the same colour. Majority vote is trivial; splitting further gains nothing, so this is a natural leaf.
Tie — a box with ${A, A, B, B}$ . $ar g max$ has no unique winner; the convention is to break the tie deterministically (e.g. lower class index or the class seen first).
One-dot-per-leaf — if we keep splitting until every box holds a single dot, training error hits zero, but we have carved boxes around noise.

WHY. Case 3 is the overfitting trap: perfect on training dots, terrible on new dots, because the tiny boxes memorized flukes. We stop early with limits like max_depth / min_samples_leaf, or cut back afterwards with Overfitting and pruning.

PICTURE.

The one-picture summary

This final figure fuses all seven steps: dots → axis-aligned walls → boxes → each box labelled by majority (or mean) → the staircase boundary, with a pure box, a tie box, and an over-split box flagged.

Recall Feynman: retell the whole walkthrough in plain words

Picture your data as coloured dots on a table. You are allowed to lay down only straight rulers, and every ruler must be parallel to an edge of the table (never diagonal). Each ruler you drop is one yes/no question, and it chops the table into rectangles. Inside any rectangle you must shout a single answer for everyone in it: if the answer is a colour, you shout the colour that has the most dots there (fewest people upset); if the answer is a number, you shout the average of the numbers there (smallest total squared complaint — found by rolling to the bottom of a parabola). Because every rectangle shouts just one flat answer, the border between "shout coral" and "shout mint" regions comes out as a staircase, never a smooth slope. If a rectangle already holds one colour, you're done there. If it's a perfect tie, you flip a fixed coin. And if you get greedy and shrink rectangles until each holds a single dot, you'll be "right" on today's dots and wrong on tomorrow's — so smart players stop early. That's the entire machine.

Connections

Parent: structure & terminology
Gini impurity and entropy — scoring a split before you pick it (this page only shows the outputs).
CART algorithm — the greedy recursion that actually places the walls.
Overfitting and pruning — fixing the Step-7 depth trap.
Random Forests — averaging staircases to cut variance.
Gradient Boosted Trees — stacking staircases to fix errors.
Bias-variance tradeoff — why the jagged staircase is high-variance.