Ensemble methods (voting, stacking, blending): Foundations

Before you can read a single formula in the parent note, you need to know what every squiggle means and what picture lives behind it. This page builds each one from nothing, in an order where each idea leans on the one before it.

1. A model, and what " $f$ " means

The picture: think of $f$ as a box. An arrow goes in carrying the facts about one email (words, sender, length). An arrow comes out carrying a verdict: "Spam" or "Not Spam".

Figure — Ensemble methods (voting, stacking, blending)

Why the topic needs it: the whole point of an ensemble is to own many boxes at once. So we number them: $f_{1}, f_{2}, \dots, f_{M}$ . The little number below (the subscript) is just a name tag — "box number 3" is $f_{3}$ . It does not mean multiply or raise to a power.

2. Input $x$ , true answer $y$ , and the guess $\overset{y}{^}$

The picture: a target. The bullseye is $y$ (the truth). Where your dart lands is $\overset{y}{^}$ (your guess). The gap between them is the error.

Why the topic needs it: ensembles combine many $\overset{y}{^}$ 's to land closer to the bullseye $y$ .

3. Bold $x$ and $z$ — a whole list of numbers

The picture: a single number is a dot on a line. A list of two numbers is a dot on a flat sheet (a grid). A list of many numbers is a dot floating in a space with many directions — impossible to draw, but the idea is the same: one point, many coordinates.

Why the topic needs it: in stacking the meta-model's input $z = [p_{1}, p_{2}, \dots, p_{M}]$ is literally the list of every base model's guess. Bold reminds you: this is a list, one slot per model.

4. Probability $P (c)$ — how sure the model is

The picture: a slider bar from $0$ (left, "no") to $1$ (right, "yes"). A confident spam-detector pushes the slider near $1$ . An unsure one leaves it near $0.5$ .

Why the topic needs it: hard voting throws away the slider and keeps only "left or right". Soft voting keeps the whole slider position — richer information — which is why it often wins. You cannot understand the difference without knowing what $P_{i} (c)$ measures.

5. Summation $\sum$ — "add these all up"

Reading it slowly:

$i = 1$ under the sign = where to start counting.
$M$ on top = where to stop.
$a_{i}$ to the right = the thing being added, with $i$ changing each step.

The picture: a row of buckets, one per model. $\sum$ pours them all into one big bucket.

Why the topic needs it: the ensemble average $\overset{ˉ}{f} = \frac{1}{M} i = 1 \sum M f_{i}$ is literally "add up all $M$ guesses, then split the total into $M$ equal shares." That $\frac{1}{M}$ out front is the sharing.

6. The average, the bar $ˉ$ , and $mode$

The picture:

Average / mean: balance point of a see-saw with weights placed along it.
Mode: the tallest bar in a bar-chart of votes.

Why the topic needs it: regression ensembles use the mean (average the numbers). Classification hard-voting uses the mode (most-voted label). Same spirit — "the group's answer" — but two different tools for two different kinds of output.

7. $ar g max$ — "which choice wins?"

The picture: three horses (classes) run a race; each has a score. $max$ tells you the winning time; $ar g max$ tells you the winning horse. Soft voting wants the horse.

Why the topic needs it: soft voting's formula $\overset{y}{^} = ar g max_{c} \frac{1}{M} \sum_{i} P_{i} (c)$ says: for each class, average its probability across models, then pick the class with the top average.

8. Weights $w_{i}$ — some voices count more

The picture: a see-saw again, but now you can slide each model's weight closer to or further from the pivot. Heavier, further-out weights swing the result their way.

Why the topic needs it: the parent's [!mistake] fix uses $\overset{y}{^} = ar g max_{c} \sum_{i} w_{i} P_{i} (c)$ . If you don't know $w_{i}$ is a dial from $0$ upward, that fix is gibberish.

9. Variance $σ^{2}$ — how much guesses wobble

The picture: darts on a board. All clustered tight = low variance. Scattered wide = high variance.

Why the topic needs it: the headline result $σ_{ensemble}^{2} = σ^{2} / M$ is the whole mathematical reason ensembles work. Averaging $M$ independent wobbly guesses shrinks the wobble by a factor of $M$ . This connects straight to the bias–variance tradeoff: ensembles are a variance-crushing machine.

10. The independence catch: $Var (X + Y) = Var (X) + Var (Y)$

The picture: two dice. Rolling one tells you nothing about the other — independent. But two copies of the same model trained the same way? They stumble on the same hard cases together — not independent, so averaging them barely helps.

Why the topic needs it: this is why diversity matters. Identical models give $σ^{2} / M \to σ^{2}$ (no gain). Diverse models (different algorithms, different data via cross-validation folds) get closer to true independence and real error reduction. It's the seed idea behind Random Forests and Boosting.

11. Out-of-fold — data a model has not seen

The picture: five index cards. Cover one, study the other four, then guess the covered one. Rotate which card is covered until every card has been guessed while hidden.

Why the topic needs it: stacking feeds the meta-model these honest, unseen guesses. Skip this and the meta-model learns the base models' cheating on training data — the leakage the parent warns about. This is pure cross-validation machinery reused for a new job.

How these feed the topic

Equipment checklist

Cover the right side and see if you can answer before revealing.

What does

f_{3}

mean — the third model or

f

cubed?

The third model; the small number is a name-tag subscript, not a power.

What is the difference between

y

and

\overset{y}{^}

y

is the true answer;

\overset{y}{^}

(with the hat) is the model's guess at it.

What does the bold in

z

signal?

z

is a list of numbers (a vector), not a single number — here, one slot per base model's prediction.

What number range does a probability

P_{i} (c)

live in, and what do the ends mean?

Between

0

and

1

;

0

= impossible,

1

= certain,

0.5

= coin-flip.

Read

\sum_{i = 1}^{M} a_{i}

in plain words.

Add up

a_{1} + a_{2} + \dots + a_{M}

— start

i

at 1, stop at

M

When do we use mode vs mean to combine models?

Mode (most-frequent label) for hard-vote classification; mean (average) for regression.

What is the difference between

max_{c}

and

ar g max_{c}

max

gives the largest value;

ar g max

gives the choice $c$ that produced it.

What does a weight

w_{i}

do in weighted voting?

Scales how loudly model

i

's vote counts — bigger

w_{i}

= more influence.

In words, what does

σ^{2}

measure?

How much a model's guesses wobble / spread — its variance / instability.

Why does averaging

M

models give variance

σ^{2} / M

only sometimes?

Only when the models' errors are independent (uncorrelated); identical models wobble together and gain nothing.

What is an out-of-fold prediction and why does stacking need it?

A guess a model makes on data it was not trained on; stacking uses it so the meta-model learns honest skill, avoiding data leakage.