Ensemble methods (voting, stacking, blending): Visual walkthrough

We are going to earn every symbol. Let's start with the most basic idea: what does it even mean for a model to have "error"?

Step 1 — What "error" and "variance" actually mean

WHAT. A model looks at a house and guesses its price. The true price is a fixed number — call it $t$ (for truth). The model's guess is some number $g$ . The error is simply how far the guess is from the truth:

$ϵ = g - t$

Read this as: $ϵ$ ("epsilon", the Greek letter we use for a small leftover amount) $=$ guess minus truth. If the model guesses $260$ k and the truth is $250$ k, then $ϵ = + 10$ k. Too low? $ϵ$ is negative.

WHY variance. Retrain the same model on a slightly different sample of houses and you get a slightly different guess — so $ϵ$ changes each time. It wobbles. Across all those retrainings the errors are, on average, centred on zero — a good model overshoots as often as it undershoots — so we assume $E [ϵ] = 0$ , read "the expected (average) value of $ϵ$ is zero". Variance then measures how wide that wobble is around zero. Because the average is $0$ , the variance is simply the average of $ϵ^{2}$ :

$σ^{2} = Var (ϵ) = E [(ϵ - E [ϵ])^{2}] = E [ϵ^{2}]$

Term by term: $E [\cdot]$ is the long-run average; we subtract the mean $E [ϵ]$ to measure distance from the centre, square it so overshoots and undershoots both count as positive spread, then average. Big $σ^{2}$ = jumpy, unreliable model; small $σ^{2}$ = steady model.

PICTURE. Below, one model retrained many times: each dot is one guess, scattered around the true price (the vertical line). The spread of that scatter is $σ^{2}$ .

Figure — Ensemble methods (voting, stacking, blending)

Step 2 — Averaging many guesses: the picture of "cancelling out"

WHAT. Instead of one model, take $M$ of them and name their predictions $f_{1}, f_{2}, \dots, f_{M}$ — here $f_{i}$ just means "the guess made by model $i$ " (it plays the exact role the single guess $g$ played in Step 1, we only added a name-tag $i$ so we can tell the models apart). $M$ is how many models we have (a whole number like $3$ or $50$ ). Each model's error is then $ϵ_{i} = f_{i} - t$ , so $ϵ_{1}$ is model 1's error, $ϵ_{2}$ is model 2's, and so on. We keep the same zero-mean assumption for every model: $E [ϵ_{i}] = 0$ .

We build the ensemble guess by averaging:

$\overset{ˉ}{f} = \frac{1}{M} \sum_{i = 1}^{M} f_{i}$

Term by term: the bar over $f$ means "averaged". $\sum_{i = 1}^{M}$ is a stack-and-add instruction — "add up $f_{1}$ through $f_{M}$ ". Then $\frac{1}{M}$ divides by how many we added, turning a sum into an average.

WHY. Some models guess too high ( $ϵ_{i} > 0$ ), some too low ( $ϵ_{i} < 0$ ). If the highs and lows are unrelated, adding them lets the pluses partly cancel the minuses. The average sits closer to the truth than a typical individual.

PICTURE. Scattered individual guesses (faint) collapse toward the truth once averaged (the bold marker).

Recall

What does $f_{i}$ denote? ::: the prediction (guess) made by model number $i$ . What does the bar in $\overset{ˉ}{f}$ mean? ::: it is the average of all $M$ model guesses. Why can highs and lows cancel? ::: because independent errors point in different directions and partly offset.

Step 3 — The error of the average is the average of the errors

WHAT. Subtract the truth $t$ from the ensemble guess. Since $t$ is the same for everyone:

$\overset{ϵ}{ˉ} = \overset{ˉ}{f} - t = \frac{1}{M} \sum_{i = 1}^{M} f_{i} - t = \frac{1}{M} \sum_{i = 1}^{M} (f_{i} - t) = \frac{1}{M} \sum_{i = 1}^{M} ϵ_{i}$

The trick: $t = \frac{1}{M} \sum_{i = 1}^{M} t$ (averaging a constant gives the constant back), so we can slip $t$ inside the sum and pair each $f_{i}$ with a $t$ , producing each individual error $ϵ_{i} = f_{i} - t$ .

WHY we bother rewriting it this way. The left-hand side $\overset{ϵ}{ˉ}$ is written in terms of the ensemble guess $\overset{ˉ}{f}$ , which we cannot analyse directly — we don't have a handle on $\overset{ˉ}{f}$ 's wobble yet. But the right-hand side is written purely in terms of the individual errors $ϵ_{i}$ , whose variance $σ^{2}$ we already own from Step 1. By rewriting $\overset{ϵ}{ˉ}$ as $\frac{1}{M} \sum_{i} ϵ_{i}$ we convert an unknown into a combination of knowns, so the question "how much does $\overset{ϵ}{ˉ}$ wobble?" becomes the answerable "how much does an average of wobbly numbers wobble?" — which the variance rules of Step 4 can crack open.

PICTURE. A number line of individual errors $ϵ_{i}$ (orange ticks) and their average $\overset{ϵ}{ˉ}$ (teal), sitting near zero.

Step 4 — Why the tool we need is variance of a sum (and its one rule)

WHAT. We want $Var (\overset{ϵ}{ˉ})$ — the wobble-width of the averaged error. $Var (\cdot)$ just means "the variance of whatever is inside".

WHY this tool and not another? We are not asking where the errors land (that's the mean, which we already fixed at $0$ ). We are asking how spread out the ensemble error is — spread is exactly what variance measures. So variance is the right microscope.

Variance obeys two rules we will lean on:

Pull-out rule: $Var (a X) = a^{2} Var (X)$ . Scaling a quantity by $a$ scales its squared spread by $a^{2}$ (squared because variance is built from squared distances).
Add rule (independence only): if $X$ and $Y$ are independent — meaning knowing one tells you nothing about the other — then $Var (X + Y) = Var (X) + Var (Y)$ .

PICTURE. Two independent wobble-clouds combine into one wider cloud whose spreads add; two correlated clouds (drifting together) do not add cleanly — a preview of Step 6.

Step 5 — Turning the crank: the $σ^{2} / M$ result

WHAT. Assume for now the errors are independent and each has the same variance $σ^{2}$ . Apply the two rules to $\overset{ϵ}{ˉ} = \frac{1}{M} \sum_{i} ϵ_{i}$ :

\overset{\text{pull-out}}{=}\frac{1}{M^2}\,\text{Var}\!\left(\sum_{i=1}^M \epsilon_i\right) \overset{\text{add}}{=}\frac{1}{M^2}\sum_{i=1}^M \text{Var}(\epsilon_i)=\frac{1}{M^2}\cdot M\sigma^2=\frac{\sigma^2}{M}$$ Reading the chain left to right: the $\frac{1}{M}$ comes out squared as $\frac{1}{M^2}$; independence lets the variance of the sum become a sum of variances; each of the $M$ terms equals $\sigma^2$, so the sum is $M\sigma^2$; the $M$ cancels one factor, leaving $\frac{\sigma^2}{M}$. **WHY it matters.** Ten independent models cut the wobble to a **tenth**. This is *pure variance reduction with zero extra bias* — the mathematical heart of voting and of [[2.7.2-Random-Forest|Random Forests]]. It connects straight to the [[2.6.8-Bias-variance-tradeoff|bias–variance tradeoff]]: averaging attacks the variance term while leaving bias untouched. **PICTURE.** The curve $\sigma^2/M$ plunging as $M$ grows — steep at first, then flattening. ![[deepdives/dd-ai-ml-2.6.16-d2-s05.png]] > [!formula] The central result > $$\boxed{\;\sigma^2_{\text{ensemble}}=\dfrac{\sigma^2}{M}\;}\qquad(\text{errors independent, equal variance})$$ --- ## Step 6 — The edge case: what if the errors are *correlated*? **WHAT.** Independence was a big "if". In reality models trained on the same data make *similar* mistakes — their errors move together. We measure "moving together" with a new tool, the **covariance** $\text{Cov}(\epsilon_i,\epsilon_j)$: it is the average of the product $\epsilon_i\,\epsilon_j$ (positive when the two errors tend to share a sign, negative when they oppose). We repackage it as a **correlation** $\rho$ (rho) via $\text{Cov}(\epsilon_i,\epsilon_j)=\rho\,\sigma^2$, where $\rho$ is a dimensionless number in $[-1,+1]$: $\rho=0$ means unrelated, $\rho=1$ means identical errors, $\rho=-1$ means perfectly opposite. **WHY covariance enters now.** In Step 4 the add rule held *only* under independence. When errors are dependent the honest rule for a sum picks up cross-terms: $$\text{Var}\!\left(\sum_{i=1}^M \epsilon_i\right)=\underbrace{\sum_{i=1}^M \text{Var}(\epsilon_i)}_{M\text{ diagonal terms}}+\underbrace{\sum_{i\neq j}\text{Cov}(\epsilon_i,\epsilon_j)}_{M(M-1)\text{ off-diagonal pairs}}$$ Reading it: the first block is the $M$ "self" terms, each $\sigma^2$; the second block collects every *pair* of distinct models, and there are $M(M-1)$ ordered pairs, each contributing $\rho\sigma^2$. Substituting: $$\text{Var}\!\left(\sum_i \epsilon_i\right)=M\sigma^2 + M(M-1)\rho\sigma^2$$ Now divide by $M^2$ (the pull-out rule again, since $\bar\epsilon=\frac{1}{M}\sum_i\epsilon_i$): $$\text{Var}(\bar\epsilon)=\frac{M\sigma^2 + M(M-1)\rho\sigma^2}{M^2}=\frac{\sigma^2}{M}+\frac{(M-1)}{M}\rho\sigma^2=\rho\,\sigma^2+\frac{1-\rho}{M}\,\sigma^2$$ The last equality is just algebra ($\frac{1}{M}+\frac{M-1}{M}\rho = \rho + \frac{1-\rho}{M}$). Term by term: the first piece $\rho\sigma^2$ is the **shared** wobble that *never* averages away; the second piece $\frac{1-\rho}{M}\sigma^2$ is the **independent leftover** that shrinks with $M$ just like Step 5. **WHY it matters.** Push $M\to\infty$ and the second term vanishes, but the floor $\rho\sigma^2$ remains. So adding more copies of *the same kind of model* eventually helps nothing. This is exactly why ensembles demand **diversity** — from different [[2.5.6-Feature-engineering|features]], different algorithms, or the [[2.6.12-Cross-validation|out-of-fold]] resampling behind bagging. **PICTURE.** Three curves: $\rho=0$ (dives to zero), $\rho=0.3$ (levels off partway), $\rho=1$ (flat — no benefit at all). ![[deepdives/dd-ai-ml-2.6.16-d2-s06.png]] > [!mistake] "Just add more models forever" > **Feels right:** Step 5 says more $M$ = less variance. > **Wrong when errors correlate:** the floor $\rho\sigma^2$ caps you. Twenty near-identical trees ($\rho\approx 1$) are barely better than one. **Fix:** buy diversity, not quantity — vary the algorithm, the features, or the training subsample. --- ## Step 7 — Degenerate corners: $M=1$, $\rho=0$, and negative $\rho$ **WHAT & WHY.** Always test a formula at its extremes to be sure it's sane. - **$M=1$ (one model):** the general formula gives $\rho\sigma^2 + \frac{1-\rho}{1}\sigma^2 = \rho\sigma^2 + (1-\rho)\sigma^2 = \sigma^2$. Good — a "crowd of one" is just the original model, unchanged. - **$\rho=0$ (fully independent):** the general formula collapses to $\frac{\sigma^2}{M}$ — exactly Step 5. So Step 5 is just the special case $\rho=0$ of Step 6. Nothing contradicts. - **$\rho=1$ (identical models):** gives $\sigma^2$ for *any* $M$ — copying a model never helps. - **$\rho<0$ (anti-correlated models):** here the errors tend to *oppose* — when one overshoots the other undershoots — so they cancel even harder than random. The floor $\rho\sigma^2$ is now **negative-leaning**, pushing the ensemble variance *below* $\frac{\sigma^2}{M}$. For example with $M=2,\ \rho=-1$ the variance is $(-1)\sigma^2+\frac{1-(-1)}{2}\sigma^2=-\sigma^2+\sigma^2=0$: two perfectly opposite errors average to exactly zero wobble. Negative correlation is the *dream* case — it is why deliberately diverse (even oppositely-biased) learners can beat naive averaging. (Caveat: $\rho$ cannot be pushed arbitrarily negative for many models at once; for $M$ errors the smallest possible shared $\rho$ is $-\frac{1}{M-1}$.) **PICTURE.** A tiny consistency map showing the corners — including a negative-$\rho$ bar dipping below the independent line. ![[deepdives/dd-ai-ml-2.6.16-d2-s07.png]] --- ## The one-picture summary ![[deepdives/dd-ai-ml-2.6.16-d2-s08.png]] The single figure fuses the whole journey: scattered individual errors (Step 1–3) → averaging pulls them toward zero (Step 4–5) → and the correlation floor $\rho\sigma^2$ marks where averaging stops paying off (Step 6–7). ```mermaid flowchart LR A["One model wobbles by sigma^2"] --> B["Average M models"] B --> C["Error of average = average of errors"] C --> D["Variance rules: pull out and add"] D --> E["If independent: sigma^2 over M"] E --> F["If correlated: floor rho sigma^2 remains"] F --> G["So chase diversity not quantity"] ``` > [!recall]- Feynman retelling — say it like a story > One model is like one wobbly dart-thrower: sometimes high, sometimes low, but on average dead-on (that's the zero-mean assumption), and the width of that wobble we call $\sigma^2$. Get a whole team throwing at the same target and *average* where their darts land — the too-highs and too-lows fight and partly cancel, so the average sits steadier. If every thrower's mistakes are truly their own (independent), the team's wobble shrinks to $\sigma^2$ divided by how many people threw, $M$: ten throwers, one-tenth the wobble. But if they all lean the same way — say a crooked wall pushes everyone right — that shared lean, measured by the correlation $\rho$ through the covariance cross-terms, never averages out and leaves a floor $\rho\sigma^2$. And if two throwers actively *oppose* each other ($\rho<0$), their errors cancel even better than chance. So the golden rule of ensembles isn't "hire more throwers", it's "hire *different* throwers". That single sentence powers voting, stacking, and random forests. > [!mnemonic] Remember the two-piece formula > **"Shared stays, private shrinks."** $\underbrace{\rho\sigma^2}_{\text{shared, stays}} + \underbrace{\tfrac{1-\rho}{M}\sigma^2}_{\text{private, shrinks with }M}$. **See also:** [[2.6.8-Bias-variance-tradeoff]] · [[2.7.2-Random-Forest]] · [[2.7.3-Boosting-methods]] · [[2.6.12-Cross-validation]] · [[2.8.5-Regularization]] · [[Ensemble methods (voting, stacking, blending)]]