Worked examples — Ensemble methods (voting, stacking, blending)

This is a worked-examples companion to the parent note. There we built the why; here we grind through every kind of number an ensemble question can throw at you. The rule of this page: you forecast the answer first, then we compute, then we verify by plugging back.

Before we compute anything, one reminder of the words we will use (each is a promise the parent note made — here we cash it in with numbers):

---

The scenario matrix

Every ensemble arithmetic problem is one of these cells. The worked examples below are tagged with the cell they cover so you can see nothing is skipped.

Cell	What makes it tricky	Covered by
A. Clear majority	odd number of voters, one class wins outright	Example 1
B. Tie / even voters	2-vs-2 hard vote → no winner, tie-break needed	Example 2
C. Hard vs soft disagree	majority says one thing, confidence says another	Example 3
D. Weighted vote	models are not equally trusted	Example 4
E. Variance formula	how much does averaging $M$ models actually help?	Example 5
F. Correlated models (degenerate)	errors not independent — the formula's limit	Example 6
G. Multi-class $\arg \max$	3+ classes, soft vote picks the top average	Example 7
H. Stacking / meta-weights	out-of-fold predictions feed a weighted meta-model	Example 8
I. Real-world word problem	medical triage, cost of a wrong call	Example 9
J. Exam twist	"how many models to beat 95%?" — solve for $M$	Example 10

---

Example 1 — Cell A: a clear majority

Forecast: Count the hands before reading on — three of one, two of the other. Which wins?

1. List the votes and tally them. Spam appears 3 times, Not-Spam appears 2 times. Why this step? Hard voting is literally the mode (most frequent value) — no probabilities involved, so counting is the entire computation.
2. Take the majority. $3>2$, so the ensemble says Spam. Why this step? With an odd number of voters and two classes, a tie is impossible — the mode is always unique.

Verify: $3+2=5$ voters accounted for, and $3/5=60\%$ backed Spam. Majority means share $>50\%$; $60\%>50\%$ ✓.

---

Example 2 — Cell B: an even vote → a tie

Forecast: Two each. Can hard voting even answer?

1. Tally. Spam = 2, Not-Spam = 2. Why this step? Even voter counts allow exact ties; you must check for this before declaring a winner.
2. Recognise the failure. $\text{mode}$ is undefined — there is no single most-frequent class. Why this step? The naive "majority wins" rule silently breaks here, exactly like the $\arctan$ formula breaking across quadrants — you cannot let the reader hit this unwarned.
3. Apply a tie-break rule — but choose it by what your models expose.
   • If every model outputs a probability, fall back to soft voting (average the probabilities — a real number rarely ties exactly). Preferred whenever available, because it uses real information.
   • If some models are label-only (e.g. a plain SVM, a rule-based classifier, or any model without calibrated probabilities), soft voting is impossible — you have no numbers to average. Then break the tie by prior class frequency: predict whichever class is more common in the training data, since that is the safer base rate.
   • If even priors are unavailable or equal, use a fixed deterministic rule — lowest label index / alphabetical order (this is scikit-learn's default). It is arbitrary but reproducible, which matters for testing. Why this ordering? Prefer the rule that uses the most information you actually have; only drop to arbitrary label order as a last resort.

Verify: $2=2$ confirms the tie. Note the cure: whichever applicable rule you pick produces a deterministic answer, so the ensemble never returns "no prediction." Sanity check on the soft-vote fallback — if the four $P(\text{Spam})$ values were $0.6,0.4,0.55,0.35$, then $\bar{P}=1.9/4=0.475<0.5$ → Not-Spam, tie broken.

---

Example 3 — Cell C: hard and soft disagree

Forecast: Hard voting counts who crossed 0.5. Soft voting averages the raw numbers. Guess whether they agree.

1. Hard vote: threshold each at 0.5. $0.90\to$ Spam, $0.45\to$ Not-Spam, $0.48\to$ Not-Spam. Tally: Spam 1, Not-Spam 2 → Not-Spam. Why this step? Hard voting throws away magnitude — it only asks "which side of 0.5?" So the confident $0.90$ counts the same as a barely-over $0.51$ would.
2. Soft vote: average the probabilities into $\bar{P}$. $\bar{P}=\frac{0.90+0.45+0.48}{3}=\frac{1.83}{3}=\mathrm{0.61.}$ Since $0.61>0.5$ → Spam. Why this step? Soft voting keeps the magnitude; one very confident correct model can outvote two lukewarm dissenters. Look at the figure below — the tall lavender bar drags the average across the line.

Figure 1 — Three probability bars (lavender, coral, mint) for one email. The dashed line is the 0.5 threshold; the solid butter line is the soft-vote average $\bar{P}=0.61$. Even though two of the three bars sit below 0.5 (so hard voting says Not-Spam), the tall 0.90 bar pulls the average above the line, so soft voting says Spam.

Verify: Recompute the mean: $0.90+0.45+0.48=1.83$, $1.83/3=0.61$ ✓. The disagreement is real and instructive: hard voting = 2-vs-1 for Not-Spam; soft voting = Spam. This is exactly why soft voting usually wins when probabilities are well-calibrated — it uses the extra information hard voting discards. See 2.6.8-Bias-variance-tradeoff for why that extra information reduces variance.

---

Example 4 — Cell D: weighted voting

Forecast: The strong model (0.95) says a mild Spam; two weak models say Not-Spam. Whom does the weighting favour?

1. Compute the weighted numerator. ${\sum }_i{w}_i{P}_i=0.95(0.55)+0.70(0.20)+0.60(0.10)=0.5225+0.14+0.06=\mathrm{0.7225.}$ Why this step? Equal weights would treat the 60%-accurate model as trustworthy as the 95%-accurate one — the parent note's [!mistake] callout warns against exactly that. Weighting by accuracy fixes it.
2. Divide by the total weight to get a proper probability (a weighted $\bar{P}$). $\bar{P}=\frac{0.7225}{0.95+0.70+0.60}=\frac{0.7225}{2.25}=\mathrm{0.3211.}$ Why this step? Weights must be normalised so the result stays in $[0,1]$; otherwise it isn't a probability.
3. Threshold. $0.3211<0.5$ → Not-Spam.

Verify: Try unweighted for contrast: $\frac{0.55+0.20+0.10}{3}=0.283<0.5$, also Not-Spam. Weighting changed the value ($0.321$ vs $0.283$) but not the decision here — a good reminder that weights matter most near the boundary. Numeric check: $0.7225/2.25=0.3211$ ✓.

---

Example 5 — Cell E: how much does averaging help?

Forecast: The parent note derived ${\sigma }_{\text{ens}}^2={\sigma }^2/M$. Guess the number before dividing.

1. Apply the variance-reduction formula. ${\sigma }_{\text{ens}}^2=\frac{{\sigma }^2}{M}=\frac{0.16}{4}=\mathrm{0.04.}$ Why this step? Averaging independent errors — this is the whole mathematical reason ensembles exist. Standard deviation is the real "spread," so also note ${\sigma }_{\text{ens}}=\sqrt{0.04}=0.20$ vs original $\sqrt{0.16}=0.40$.
2. State the improvement factor. Variance shrank by a factor of $M=4$; standard deviation shrank by $\sqrt{M}=2$. Why this step? People quote variance, but you feel error in units of standard deviation — halving it is the headline. This is the same $1/\sqrt{M}$ law behind 2.7.2-Random-Forest.

Verify: $0.04\times 4=0.16$ recovers the single-model variance ✓, and $0.20\times 2=0.40$ recovers the single-model standard deviation ✓.

---

Example 6 — Cell F: the degenerate case — correlated models

Forecast: If four models are clones, does averaging help at all? Guess before we build the formula.

1. Write the ensemble error and expand its variance. The ensemble error is the average error $\overline{ϵ}=\frac{1}{M}{\sum }_i{ϵ}_i$. Its variance expands into two kinds of terms: $\text{Var}(\overline{ϵ})=\frac{1}{M^2}\left(\underset{\text{diagonal}}{\underbrace{\sum _i\text{Var}({ϵ}_i)}}+\underset{\text{off-diagonal}}{\underbrace{\sum _{i\ne j}\text{Cov}({ϵ}_i,{ϵ}_j)}}\right).$ Why this step? When errors are not independent, the cross-terms $\text{Cov}({ϵ}_i,{ϵ}_j)$ no longer vanish — that is the whole difference from the parent note's derivation. We must keep them.
2. Count and substitute each kind of term. There are $M$ diagonal terms, each ${\sigma }^2$. There are $M(M-1)$ off-diagonal pairs, each $\text{Cov}({ϵ}_i,{ϵ}_j)=\rho {\sigma }^2$, where $\rho$ (read "rho") is the correlation — a number from 0 (unrelated errors) to 1 (identical errors). So: $\text{Var}(\overline{ϵ})=\frac{1}{M^2}(M{\sigma }^2+M(M-1)\rho {\sigma }^2)=\frac{{\sigma }^2}{M}+\frac{M-1}{M}\rho {\sigma }^2.$ Why this step? The first term is the old "averaging helps" piece; the new second term is the penalty for models copying each other's mistakes. It is exactly what the parent note's independence assumption hid.
3. Plug in $\rho =0$ (independent — the parent note's case). ${\sigma }_{\text{ens}}^2=\frac{0.16}{4}+\frac{3}{4}(0)(0.16)=\mathrm{0.04.}$ Why this step? It must reproduce Example 5 — a formula that doesn't collapse to the known case would be wrong.
4. Plug in $\rho =1$ (perfectly correlated — the degenerate limit). ${\sigma }_{\text{ens}}^2=\frac{0.16}{4}+\frac{3}{4}(1)(0.16)=0.04+0.12=\mathrm{0.16.}$ Why this step? This is the crucial limiting behaviour: identical models give ${\sigma }_{\text{ens}}^2={\sigma }^2$ — zero improvement. Averaging clones changes nothing.
5. Read the moral. Ensembles buy accuracy from diversity, not from headcount. This is why we use different algorithms, different feature subsets, and 2.6.12-Cross-validation folds — see also 2.5.6-Feature-engineering for creating diverse inputs.

Figure 2 — Ensemble variance (lavender curve) as the error correlation $\rho$ slides from 0 to 1, for $M=4$ and ${\sigma }^2=0.16$. At $\rho =0$ (independent) the variance is at its lowest, $0.04$; at $\rho =1$ (clones) it climbs back to the single-model dotted line at $0.16$ — no improvement. The mint shaded gap is the accuracy you earn by keeping models diverse.

Verify: At $\rho =0$ we got $0.04$ = Example 5 ✓. At $\rho =1$ we got $0.16={\sigma }^2$ ✓ (no gain). A halfway case $\rho =0.5$: $0.04+0.75(0.5)(0.16)=0.04+0.06=0.10$, which lies between $0.04$ and $0.16$ ✓ — monotone in $\rho$, as physics demands.

---

Example 7 — Cell G: multi-class $\arg \max$

Forecast: No single class dominates every row. Average each column and eyeball the winner.

1. Average each class column into its own $\bar{P}$. $\bar{P}(\text{Cat})=\frac{0.6+0.2+0.1}{3}=\frac{0.9}{3}=0.30,\bar{P}(\text{Dog})=\frac{0.3+0.5+0.4}{3}=\frac{1.2}{3}=0.40,\bar{P}(\text{Fox})=\frac{0.1+0.3+0.5}{3}=\frac{0.9}{3}=\mathrm{0.30.}$ Why this step? With $C$ classes, soft voting averages each class independently — think of $C$ parallel binary averages.
2. Take $\arg \max$ — the class with the largest average. $\bar{P}(\text{Dog})=0.40$ is largest → predict Dog. Why this step? $\arg \max$ means "the label where the max occurs," not the max value itself. Note Dog won even though no single model ranked it first — the ensemble found a consensus middle ground.
3. Handle the edge case: what if two classes tie for the top average? Here Cat and Fox both average $0.30$ — but they lost to Dog, so no problem. Suppose instead two classes had both topped the table at, say, $0.40$ each. Then $\arg \max$ is ambiguous and you need a rule, exactly as in Example 2:
   • break by highest single-model confidence for either class (a proxy for who is more sure);
   • or by prior class frequency in training;
   • or, as a deterministic last resort, by lowest label index (most libraries return the first max encountered). Why this step? $\arg \max$ is silently undefined on ties; a reader who never saw this rule would hit an unhandled scenario.

Verify: The three averages must sum to 1 (they are averages of rows that each sum to 1): $0.30+0.40+0.30=1.00$ ✓. Winner Dog at $0.40$ ✓.

---

Example 8 — Cell H: stacking with a weighted meta-model

Forecast: The meta-model trusts RF most (0.5). Expect an answer pulled toward 270k, plus a 5k nudge.

1. Form the meta-feature vector $\mathbf{z}$. $\mathbf{z}=[250,270,260]$ (in thousands). Why this step? In stacking the base predictions are the features the meta-model reads — this is the whole idea.
2. Compute the weighted sum plus intercept. $\hat{y}=b+{\sum }_i{w}_i{z}_i=5+0.2(250)+0.5(270)+0.3(260).$ $=5+50+135+78=268\text{k}.$ Why this step? A linear meta-model is just a dot product plus bias — it learns how much to trust each base model, unlike plain averaging which forces equal weight.
3. Sanity-check the weights. $0.2+0.5+0.3=1.0$, so ignoring the intercept the output is a weighted average that must land between the smallest (250) and largest (270) base prediction.

Verify: Without the intercept: $50+135+78=263$k, which lies in $[250,270]$ ✓. Adding $b=5$k gives $268$k ✓. Contrast plain averaging: $(250+270+260)/3=260$k — stacking's $268$k differs precisely because it weights and shifts. So stacking here nudges the estimate up by $8$k relative to naive averaging, reflecting the meta-model's learned trust in RF plus its bias correction. Remember (parent note): those OOF predictions must come from 2.6.12-Cross-validation to avoid leakage.

---

Example 9 — Cell I: a real-world word problem

Forecast: The average is around $0.55$. But the threshold moved down to catch more urgent cases. Guess the call.

1. Average the probabilities into $\bar{P}$ (soft vote). $\bar{P}=\frac{0.80+0.30+0.55}{3}=\frac{1.65}{3}=\mathrm{0.55.}$ Why this step? Same soft-vote machinery as Example 3 — the medical dressing changes only the threshold, not the averaging.
2. Compare to the policy threshold $0.35$, not $0.5$. $0.55>0.35$ → flag as URGENT. Why this step? Because a missed emergency is 10× costlier, we lower the bar for raising the alarm — we deliberately accept more false alarms to catch true urgents. This is threshold tuning, closely related to 2.8.5-Regularization's idea of trading one error for another.

Verify: $1.65/3=0.55$ ✓. Notice the answer is robust here: at the strict standard threshold $0.5$ the patient is still urgent ($0.55>0.5$), and only a very cautious threshold above $0.55$ would flip it. To see the cost logic pay off, imagine a borderline patient with $\bar{P}=0.40$: at the standard $0.5$ they would be sent home ($0.40<0.5$), but the cost-aware threshold $0.35$ catches them ($0.40>0.35$) — exactly the missed-emergency we were trying to prevent. Units check: probabilities and thresholds are both dimensionless, so the comparison is valid ✓.

---

Example 10 — Cell J: the exam twist ("solve for $M$")

Forecast: More voters → higher accuracy (if independent). The parent note claims $M=3$ gives $\approx 78.4\%$. Guess how many it takes to reach 95%.

1. Set up "ensemble correct" as "more than half correct." For odd $M$, majority means at least $k^{\star }=\frac{M+1}{2}$ of the models are right. Probability of exactly $k$ correct is the binomial $\left(\genfrac{}{}{0ex}{}{M}{k}\right)p^k(1-p{)}^{M-k}$ with $p=0.7$. Why this step? Independent correct/wrong outcomes across models is exactly the binomial setting — the same distribution behind Example 5's variance law.
2. Compute $M=3$ (the warm-up).
   • Exactly 2 correct: $\left(\genfrac{}{}{0ex}{}{3}{2}\right)(0.7{)}^2(0.3)=3(0.49)(0.3)=\mathrm{0.441.}$
   • Exactly 3 correct: $(0.7{)}^3=\mathrm{0.343.}$ $P(\text{correct},M=3)=0.441+0.343=0.784=78.4\%.$ Why this step? Reproduces the parent note's figure and confirms our binomial machinery before scaling up.
3. March $M$ upward until we cross 95%. Summing $P(\ge k^{\star }\text{ correct})$ for each odd $M$:
   • $M=3:0.784$
   • $M=5:0.8369$
   • $M=7:0.8740$
   • $M=9:0.9012$
   • $M=11:0.9218$
   • $M=13:0.9376$
   • $M=15:0.9500$ (just at the line)
   • $M=17:0.9597$ → first value strictly above 95%. Why this step? We only need to find the smallest $M$ that clears the bar, so we walk up in steps of 2 (keeping $M$ odd to avoid ties) and stop at the first crossing.

Verify: The warm-up $M=3$ gives $0.784$, matching Example 10's forecast and the parent note ✓. The accuracies are monotonically increasing in $M$ ($0.784<0.837<\cdots <0.960$) ✓, exactly as the wisdom-of-crowds intuition demands. Complement check for $M=3$: wrong = $\le 1$ correct $=(0.3{)}^3+3(0.7)(0.3{)}^2=0.027+0.189=0.216$, and $1-0.216=0.784$ ✓. So the answer to the twist is $M=17$ models for accuracy to strictly exceed 95% (with $M=15$ landing exactly on 95.0%).

---

Recall Quick self-test

Even number of hard voters splitting 2-2 gives what? ::: An undefined mode (a tie) — fall back to soft voting if probabilities exist, else prior frequency, else label order. Four identical (perfectly correlated) models reduce variance by what factor? ::: None — ${\sigma }_{\text{ens}}^2={\sigma }^2$; averaging clones does nothing. In stacking, what plays the role of "features" for the meta-model? ::: The base models' out-of-fold predictions. Why can soft voting disagree with hard voting? ::: Soft voting keeps confidence magnitudes; one very confident model can outweigh several lukewarm dissenters. What does $\bar{P}$ stand for on this page? ::: The average of the models' probabilities for one class. How many independent 70%-accurate models to push majority-vote accuracy above 95%? ::: $M=17$ (odd); $M=15$ sits exactly at 95.0%.