Foundations — Boosting concept and intuition

This page assumes you have seen nothing. We build every symbol in the parent boosting note from the ground up.

0. The picture we are always in

Everything in boosting is about one job: given an input $x$, guess an output $y$.

• $x$ = the thing you know (a house's size, a patient's blood test). Think of it as a dot on a horizontal axis.
• $y$ = the thing you want to predict (the house price, "sick or healthy"). Think of it as height, on the vertical axis.
• A model $F$ is a machine: feed it $x$, it spits out a guess $F(x)$. On the picture, $F$ is a curve — for every horizontal position $x$ it tells you a height.

1. Subscripts and the sequence $F_0,F_1,\dots ,F_m$

Boosting does not make the curve in one shot. It makes a rough curve, then a less rough one, then better still. To talk about "the curve after $m$ improvements" we need a counter.

Read $F_{m-1}$ as "F sub m-minus-one": if we're on round $m=3$, then $F_{m-1}=F_2$, the curve built from the first two improvements.

2. The index $i$ — pointing at one data point

The curve must please many data points at once. We need a way to say "the 5th point" without writing it out.

3. Residual $r_{im}$ — the "mistake" made precise

Boosting's slogan is "fix the mistakes." What is a mistake as a number?

The double subscript $r_{im}$ just means "the residual of point $i$, measured at round $m$." Two counters, two subscripts.

4. Weak learner $h_m$ and the stump

Each improvement is done by a small, humble model.

5. The sum $\sum$, weight ${\alpha }_m$, and the additive model

Now we combine the humble models. The Greek capital sigma is just shorthand for "add these all up."

Putting it together, the parent's headline formula:

6. Loss $L(y,F)$ — scoring the badness

To say a curve is "better," we need a badness score.

7. The gradient $\partial L/\partial F$ and pseudo-residuals

The negative slope points downhill, so:

8. Learning rate $\nu$ — the tiny step

The prerequisite map

Related routes: Bagging and Bootstrap Aggregation and Random Forests average independent trees; boosting instead chains them. The engine here powers Gradient Boosting Machines (GBM) and XGBoost.

Equipment checklist