2.6.16 · D5Model Evaluation & Selection
Question bank — Ensemble methods (voting, stacking, blending)
Before you start, one word you must own: two models are correlated when they tend to be wrong on the same examples. When errors line up like that, averaging them cancels almost nothing — this idea is the hinge that most of the traps below turn on. Keep bias and variance in mind too: averaging attacks variance, diversity attacks bias.
True or false — justify
Ensembling always improves accuracy over the best single model.
False. If the base models are highly correlated (make the same mistakes), averaging cancels almost nothing; and a weak, biased meta-model can even undo a strong base learner. Ensembles help most when members are accurate AND diverse.
The variance formula holds for any ensemble of models.
False. It holds only when errors are independent. With correlation the true variance is , which floors at — you cannot average below the shared-error component no matter how large is.
Soft voting is always at least as good as hard voting.
False. Soft voting is only better when the probabilities are well-calibrated. An overconfident, badly-calibrated model can drag the probability average toward its wrong answer, whereas hard voting only counts its one vote.
Adding more models to a voting ensemble monotonically increases accuracy.
False. Adding redundant (correlated) or weak-below-50% models can hurt. Beyond a point, extra correlated members add cost without cutting the shared error.
In stacking, the meta-model should be more powerful than the base models.
False. The meta-model usually should be simple (Ridge, Logistic Regression). It only needs to learn a weighting over a handful of base predictions; a complex meta-model overfits those few meta-features fast.
Weighted voting with weights equal to validation accuracy is guaranteed optimal.
False. Accuracy weights ignore correlation between models. Two nearly-identical strong models get double weight and drown out a weaker but independent model that actually adds new information.
If every base model is 60% accurate, a majority vote of many of them approaches 100% accuracy.
True — but only under independence. Condorcet's jury theorem says majority accuracy as when errors are independent and each model beats 50%. Real models share errors, so it plateaus well below 100%.
Blending and stacking are the same technique.
False. Both train a meta-model, but blending uses one fixed hold-out set to make meta-features, while stacking uses out-of-fold predictions from k-fold cross-validation. Blending is simpler but wastes data and has higher-variance meta-features.
A random forest is an example of a voting ensemble.
True (essentially). A random forest averages/majority-votes many decorrelated trees. It is a bagging ensemble — a specialized voting scheme where diversity comes from bootstrap samples and random feature subsets.
Spot the error
"We trained all base models on the full training set, then trained the meta-model on their predictions for that same set."
Data leakage. The base models have already seen those rows, so their predictions look artificially perfect. The meta-model learns their training-fit, not their generalization, and collapses at test time. Fix: use out-of-fold predictions.
"Our three models are Logistic Regression, Ridge Regression, and Linear SVM, so the ensemble is very diverse."
Not diverse. All three are essentially linear decision boundaries — they'll be wrong on the same non-linear cases, so their errors are correlated. True diversity needs different model families (e.g. add a tree-based or boosting model).
"We tuned each base model to the highest possible accuracy, then ensembled them."
Chasing accuracy can kill diversity. Over-tuned models often converge to the same optimum and same mistakes. A slightly-less-accurate but differently-wrong model frequently improves the ensemble more than a marginal accuracy gain.
"For soft voting we averaged raw SVM decision-function scores with logistic-regression probabilities."
Scale mismatch. SVM decision scores are unbounded margins, not probabilities in . Averaging them with true probabilities lets the large-magnitude model dominate. Calibrate the SVM (e.g. Platt scaling) so both are on the same probability scale first.
"To save time we retrained the base models only on the out-of-fold folds and used those for the final test predictions."
Wrong final models. Out-of-fold models are for building meta-features. For the actual test set you must retrain each base model on ALL training data so it uses every available example.
"Our meta-model got 99% training accuracy, so stacking clearly worked."
Meaningless signal. High training accuracy on the meta-features often just means the meta-model memorized the (leaky or in-sample) base predictions. Judge stacking only on a held-out/CV score.
"We added the same strong model twice with different random seeds to strengthen the ensemble."
Near-redundant members. Two seeds of the same model are highly correlated, so they don't cancel each other's errors — you mostly double one voter's influence, skewing the vote without adding real information.
Why questions
Why does averaging reduce variance but not bias?
Averaging cancels the random, model-to-model fluctuations (variance) because they point in different directions. But every model's systematic mistake (bias) points the same way, so the average keeps it. Killing bias requires diverse model types, not more copies.
Why must base models be diverse and not just accurate?
If all models are wrong on the same examples, combining them just reproduces that shared mistake — the correlated-error floor . Diversity means their errors point in different directions and partly cancel.
Why does stacking use a separate meta-model instead of fixed averaging?
Fixed averaging gives every model the same weight everywhere. A meta-model learns context-dependent weighting — "trust the tree on this region, trust the linear model on that region" — which fixed weights cannot express.
Why is soft voting often better than hard voting?
It keeps the confidence information. A model that is 90% sure and right can overrule two models that are barely 51% sure and wrong — information hard voting throws away by collapsing each vote to a single label.
Why can a weak learner (barely above 50%) still help an ensemble?
Each such model still nudges the majority slightly toward correct. Under independence, many weak-but-better-than-random voters compound (Condorcet), so the collective far exceeds any individual — provided their errors aren't correlated.
Why do we combine ensembling with cross-validation in stacking?
Cross-validation gives predictions on data each base model did not train on, so the meta-features reflect true generalization. Without it the meta-model trains on inflated, leaked base predictions.
Why doesn't more regularization on the meta-model always help?
A little regularization stops the meta-model overfitting the few meta-features. Too much shrinks all weights toward equal — turning your learned stacking back into plain averaging and discarding the whole point.
Edge cases
What happens in hard voting with an even number of binary classifiers that split 50/50?
A tie with no majority. Implementations break it by a default rule (often the lowest class index or the class ordering), which can silently bias predictions. Use an odd count, or switch to soft voting.
What if one base model is far stronger than all the others combined?
Equal-weight voting can hurt — the weak majority outvotes the strong model. Here weighted voting or stacking (which can learn to lean almost entirely on the strong model) is safer than plain democracy.
What happens if every base model outputs the identical prediction?
(perfectly correlated). Variance reduction is zero — the ensemble equals a single model. This is the degenerate floor of the variance formula: no diversity, no benefit.
What if the base models each output well-calibrated probabilities but the classes are heavily imbalanced?
Simple probability averaging can drift toward the majority class. The ensemble may need threshold tuning or class-weighted meta-features, because "average probability > 0.5" is not the right cutoff under imbalance.
What if you have almost no data — should you still stack?
Usually no. Stacking spends data on out-of-fold meta-features and a meta-model; with tiny data the meta-model overfits and the folds are too small. Simple (soft) voting or a single well-regularized model is more robust.
What happens to the variance floor as with fixed correlation ?
The term vanishes, so variance . Infinite models cannot beat the shared-error floor — the only way lower is to reduce (more diversity), not add copies.
Recall One-line summaries to self-test
Averaging kills ::: variance, not bias — bias needs diverse model families. The variance formula assumes ::: independent (uncorrelated) errors. Stacking's #1 danger is ::: data leakage — fixed by out-of-fold predictions. Blending vs stacking ::: blending uses one hold-out set, stacking uses k-fold OOF. Soft voting needs ::: well-calibrated probabilities to beat hard voting.