Ensemble methods (voting, stacking, blending): Exercises

Before we start, here is the one picture the whole page leans on: the difference between hard voting (count labels) and soft voting (average probabilities).

Figure — Ensemble methods (voting, stacking, blending)

The figure has two panels. The left panel ("Hard voting") turns each model into a 0/1 label and stacks the counts; the right panel ("Soft voting") keeps each model's probability and averages them. In both panels the red bar labelled ENS is the ensemble's final answer, and the dashed line is the $0.5$ threshold. Notice the two panels can end up on opposite sides of that line — that clash is the heart of Exercise 3.1.

Level 1 — Recognition

Exercise 1.1

Three classifiers vote on one email: Spam, Spam, Not-Spam. Under hard voting, what is the ensemble label?

Recall Solution

Hard voting = majority of the labels (the mode). We have two "Spam" and one "Not-Spam". $\overset{y}{^} = mode (Spam, Spam, Not-Spam) = Spam (2/3) .$ WHAT we did: counted votes. WHY: hard voting ignores confidence and only tallies discrete labels.

Exercise 1.2

Match each word to its one-line meaning: Voting, Stacking, Blending.

Recall Solution

Voting ::: combine base predictions by a fixed rule (mode or average) — no learning on top.
Stacking ::: train a meta-model on out-of-fold base predictions to learn how to combine them.
Blending ::: like stacking, but the meta-model trains on a single held-out validation set instead of k-fold out-of-fold predictions. The key contrast: voting has no trainable combiner; stacking/blending learn the combiner. See 2.6.12-Cross-validation for the out-of-fold idea.

Level 2 — Application

Exercise 2.1

Three classifiers output these $P (Spam)$ values for one email: $0.51, 0.55, 0.49$ . Give (a) the soft-vote average and (b) the soft-vote label (threshold $0.5$ ).

Recall Solution

$\overset{ˉ}{P} = \frac{0.51 + 0.55 + 0.49}{3} = \frac{1.55}{3} = 0.51 \overline{6} \approx 0.517 .$ Since $0.517 > 0.5$ , the label is Spam. WHY average, not mode? Soft voting keeps the magnitude of confidence; here two models barely lean Spam and one barely leans Not-Spam, and the average tips just over the line.

Exercise 2.2

Same email, but now use hard voting on those same three probabilities (each model labels Spam iff $P > 0.5$ ). Do hard and soft agree?

Recall Solution

Labels: $0.51 \to$ Spam, $0.55 \to$ Spam, $0.49 \to$ Not-Spam. Majority $=$ Spam ( $2/3$ ). Both hard and soft say Spam — they agree here. WHY they agree: the two Spam probabilities ( $0.51, 0.55$ ) sit above $0.5$ and the single Not-Spam probability ( $0.49$ ) sits just below it, all by a whisker. Because every model is near the threshold, no single one is confident enough to drag the average across, so counting labels (majority) and averaging probabilities (soft) land on the same side. Disagreement only appears when one model is far more confident than the others — that is exactly what Exercise 3.1 constructs.

Exercise 2.3

Weighted soft voting. Weights $w = (0.5, 0.3, 0.2)$ ; $P (Spam) = (0.9, 0.45, 0.70)$ . Compute the weighted score $\sum_{i} w_{i} P_{i} (Spam)$ and give the label (threshold $0.5$ , weights sum to $1$ ).

Recall Solution

$\sum_{i} w_{i} P_{i} = 0.5 (0.9) + 0.3 (0.45) + 0.2 (0.70) = 0.45 + 0.135 + 0.14 = 0.725 .$ $0.725 > 0.5 \Rightarrow$ Spam. The high-weight, high-confidence model ( $w = 0.5$ , $P = 0.9$ ) dominates.

Level 3 — Analysis

Exercise 3.1

Build a case where hard and soft voting disagree. Three models, $P (Spam) = (0.9, 0.45, 0.45)$ . Give both decisions and explain the disagreement.

Recall Solution

Hard: labels $=$ (Spam, Not-Spam, Not-Spam) $\Rightarrow$ majority Not-Spam ( $2/3$ ). Soft: $\overset{ˉ}{P} = \frac{0.9 + 0.45 + 0.45}{3} = \frac{1.8}{3} = 0.6 \Rightarrow$ Spam. WHY they split: one very confident model ( $0.9$ ) is outvoted by two barely-doubtful models ( $0.45$ ). Hard voting throws away that " $0.9$ vs $0.45$ " gap; soft voting keeps it, so the confident model overrides. This is exactly what the figure below shows.

Exercise 3.2

Independent-error accuracy. Three classifiers, each 70% accurate, errors independent. Compute the majority-vote ensemble accuracy.

Recall Solution

Ensemble is correct when at least 2 of 3 are correct. With $p = 0.7$ : $P (exactly 2) = (2 3) p^{2} (1 - p) = 3 (0.49) (0.3) = 0.441,$ $P (exactly 3) = p^{3} = 0.343.$ $P (correct) = 0.441 + 0.343 = 0.784 = 78.4%.$ WHAT this shows: three 70% models beat any single one (70%) — the wisdom-of-crowds bump, powered by variance cancellation.

Exercise 3.3

Variance reduction. $M = 5$ models, each error variance $σ^{2} = 0.20$ , errors uncorrelated. Find the ensemble error variance. Then say what happens if the errors are perfectly correlated instead.

Recall Solution

Uncorrelated: $σ_{ens}^{2} = \frac{σ ^{2}}{M} = \frac{0.20}{5} = 0.04$ . WHY the $1/ M$ : the average of $M$ independent errors has variance $\frac{1}{M ^{2}} \cdot M σ^{2} = σ^{2} / M$ . Perfectly correlated ( $ρ = 1$ ): averaging identical errors changes nothing, so $σ_{ens}^{2} = σ^{2} = 0.20$ . Lesson: ensembles pay off only through diversity; identical models give zero reduction. This is why 2.7.2-Random-Forest deliberately decorrelates trees.

Level 4 — Synthesis

Exercise 4.1

General correlated-variance formula. $M$ models, each variance $σ^{2}$ , every pair correlated by $ρ$ . Show the ensemble variance is $σ_{ens}^{2} = ρ σ^{2} + \frac{1 - ρ}{M} σ^{2},$ then evaluate it for $M = 10, σ^{2} = 0.20, ρ = 0.3$ .

Recall Solution

First, what is $ρ$ ? The correlation coefficient $ρ$ between two errors is defined as their covariance divided by the product of their standard deviations: $ρ = \frac{Cov ( ϵ _{i} , ϵ _{j} )}{Var ( ϵ _{i} ) Var ( ϵ _{j} )} = \frac{Cov ( ϵ _{i} , ϵ _{j} )}{σ \cdot σ} .$ Here every model shares the same variance $σ^{2}$ (so each standard deviation is $σ$ ). Rearranging that definition gives the link we need: $Cov (ϵ_{i}, ϵ_{j}) = ρ σ^{2} .$ WHY this matters: $ρ$ is just covariance measured in "units of $σ^{2}$ ", so it is dimensionless and always between $- 1$ and $1$ . That is the exact quantity the derivation below multiplies each cross-term by.

Ensemble error $\overset{ϵ}{ˉ} = \frac{1}{M} \sum_{i} ϵ_{i}$ . Its variance: $Var (\overset{ϵ}{ˉ}) = \frac{1}{M ^{2}} (\sum_{i} Var (ϵ_{i}) + \sum_{i \neq = j} Cov (ϵ_{i}, ϵ_{j})) .$ There are $M$ variance terms ( $= M σ^{2}$ ) and $M (M - 1)$ covariance terms (each $= ρ σ^{2}$ from the definition above): $= \frac{1}{M ^{2}} (M σ^{2} + M (M - 1) ρ σ^{2}) = \frac{σ ^{2}}{M} + \frac{( M - 1 )}{M} ρ σ^{2} = ρ σ^{2} + \frac{1 - ρ}{M} σ^{2} . ✓$ Numbers: $0.3 (0.20) + \frac{1 - 0.3}{10} (0.20) = 0.06 + \frac{0.7}{10} (0.20) = 0.06 + 0.014 = 0.074 .$ Even with $M = 10$ , correlation pins the variance at $0.074$ (vs the wishful $0.02$ from $σ^{2} / M$ ).

Exercise 4.2

Design a leakage-free stacking split. You have 10,000 rows and want 5-fold out-of-fold predictions. (a) How many rows of meta-training data do you produce? (b) In fold 3, how many rows does each base model train on vs predict on?

Recall Solution

(a) Every original row is a hold-out in exactly one fold, so it gets exactly one out-of-fold prediction $\Rightarrow$ 10,000 meta-training rows (no more, no less). (b) Each fold has $10000/5 = 2000$ rows. In fold 3, base models train on the other 4 folds $= 8000$ rows and predict on 2000 rows. WHY: the meta-model must only ever see predictions the base model made on data it did not train on — otherwise the meta-model learns the base models' memorised training quirks, not their real skill. That is the whole point of out-of-fold construction.

Level 5 — Mastery

Exercise 5.1

Full mixed decision. Four models give $P (Spam)$ : $(0.80, 0.40, 0.65, 0.30)$ with reliability weights $w = (0.4, 0.1, 0.4, 0.1)$ (sum $= 1$ ). We adopt this explicit tie-break policy for hard voting: if the label counts tie, fall back to the class with the higher plain soft-vote average probability. Compute: (a) hard-vote label (using the stated tie-break), (b) plain soft-vote label, (c) weighted soft-vote score and label. Then say which decision you'd trust and why.

Recall Solution

Tie-break policy (stated in the problem): a $k$ – $k$ label tie is broken by the higher plain soft-vote average — this is a deliberate, declared rule, not a hidden convention, so a marker can reproduce it exactly.

(a) Hard: labels (thresh $0.5$ ) $=$ (Spam, Not, Spam, Not) $\Rightarrow$ tie $2$ – $2$ . Applying the declared tie-break, compare average probabilities: the Spam-leaning models average $\frac{0.80 + 0.65}{2} = 0.725$ while the Not-leaning ones average $\frac{0.40 + 0.30}{2} = 0.35$ ; the plain overall average $0.5375 > 0.5$ favours Spam $\Rightarrow$ Spam. (b) Plain soft: $\overset{ˉ}{P} = \frac{0.80 + 0.40 + 0.65 + 0.30}{4} = \frac{2.15}{4} = 0.5375 \Rightarrow$ Spam. (c) Weighted soft: $0.4 (0.80) + 0.1 (0.40) + 0.4 (0.65) + 0.1 (0.30) = 0.32 + 0.04 + 0.26 + 0.03 = 0.65 \Rightarrow$ Spam. Trust: the weighted soft score ( $0.65$ ) — it uses both confidence and each model's reliability, and it lands well clear of $0.5$ (unlike plain soft's razor-thin $0.5375$ ). The two reliable models ( $w = 0.4$ ) both lean Spam, which is reassuring.

Exercise 5.2

Break-even reliability. A single model is 82% accurate. You add two more independent models each of accuracy $p$ and take a 3-way majority vote. Find the smallest $p$ so the ensemble is at least as accurate as the single 82% model. (Use majority-correct $= p^{3} + 3 p^{2} (1 - p)$ assuming all three share accuracy $p$ ; solve $\geq 0.82$ .)

Recall Solution

Majority-correct with common accuracy $p$ : $A (p) = p^{3} + 3 p^{2} (1 - p) = 3 p^{2} - 2 p^{3} .$ Set $A (p) = 0.82$ : $3 p^{2} - 2 p^{3} = 0.82$ . Solving numerically, the relevant root in $(0, 1)$ is $p \approx 0.729 .$ So each added model only needs about 72.9% accuracy for a 3-way vote to match a lone 82% model — because the vote's variance-cancellation buys extra accuracy. Below $\approx 72.9%$ the crowd would drag it down. WHY this is the mastery point: individual weakness is fine if errors are independent — diversity converts three mediocre models into one strong committee. Compare 2.7.3-Boosting-methods, which instead makes models dependent on purpose.

Recall Quick self-check

Soft vs hard: which uses probabilities? ::: Soft. Ensemble variance for $M$ independent models each $σ^{2}$ ? ::: $σ^{2} / M$ . Why out-of-fold predictions in stacking? ::: To prevent data leakage from base models' own training rows. Variance floor when errors correlate by $ρ$ ? ::: $ρ σ^{2}$ .