Ensemble methods (voting, stacking, blending)

Why Ensembles Work: The Mathematical Foundation

The Bias-Variance Perspective

WHY do ensembles reduce error? Consider the prediction error decomposition:

$\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}$

WHAT each term means:

• Bias: Systematic error from wrong assumptions (underfitting)
• Variance: Sensitivity to training data fluctuations (overfitting)
• Irreducible: Noise in the data itself

HOW ensembles help:

1. Averaging reduces variance: If models make independent errors, averaging cancels randomness
2. Diversity reduces bias: Different model types capture different patterns
3. The ensemble can be simpler per-model: Each weak learner can specialize

$\sigma^2_{\text{ensemble}} = \frac{\sigma^2}{M}$

Derivation from scratch:

Let's say each model $f_i$ makes prediction with error $\epsilon_i$ where $\text{Var}(\epsilon_i) = \sigma^2$ and errors are uncorrelated.

Ensemble prediction: $\bar{f} = \frac{1}{M}\sum_{i=1}^M f_i$

Ensemble error: $\bar{\epsilon} = \frac{1}{M}\sum_{i=1}^M \epsilon_i$

Variance of ensemble: $\text{Var}(\bar{\epsilon}) = \text{Var}\left(\frac{1}{M}\sum_{i=1}^M \epsilon_i\right)$

Why this step? Variance is linear in independent sums.

$= \frac{1}{M^2}\sum_{i=1}^M \text{Var}(\epsilon_i)$

Why this step? When errors are independent, $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$.

$= \frac{1}{M^2} \cdot M\sigma^2 = \frac{\sigma^2}{M}$

This shows variance decreases with ensemble size—but only if errors are independent!

Method 1: Voting Classifiers

Hard Voting

WHAT it does: Each classifier outputs a class label. The ensemble predicts the most frequent class.

For $M$ classifiers predicting class labels $\hat{y}_1, \hat{y}_2, ..., \hat{y}_M$:

$\hat{y}_{\text{ensemble}} = \text{mode}(\hat{y}_1, \hat{y}_2, ..., \hat{y}_M)$

Why this works: If each classifier is better than random (accuracy > 50%), the probability of majority being wrong decreases exponentially with ensemble size.

Mathematical intuition: With 3 models each70% accurate (independent errors):

• Probability all 3 wrong: $0.3^3 = 0.027$ (2.7%)
• Probability at least 2 wrong: $0.3^2 \times 0.7 \times 3 + 0.3^3 = 0.216$ (21.6%)
• Ensemble accuracy≈ 78.4% (when at least 2 are correct)

Soft Voting

WHAT it does: Average the predicted probabilities from each classifier, then choose the class with highest average probability.

$\hat{y}_{\text{ensemble}} = \arg\max_c \frac{1}{M}\sum_{i=1}^M P_i(c)$

where $P_i(c)$ is the probability that model $i$ assigns to class $c$.

Why soft voting often wins: It uses richer information (confidence levels) rather than just binary decisions. A highly confident correct prediction can override less confident wrong predictions.

When to use which:

• Hard voting: When models don't output probabilities (e.g., SVM without probability calibration)
• Soft voting: When models output well-calibrated probabilities—typically gives1-2% better accuracy

Why it's wrong: Not all models are equally reliable! A95% accurate model should carry more weight than a 60% accurate one.

The fix: Use weighted voting: $\hat{y} = \arg\max_c \sum_{i=1}^M w_i \cdot P_i(c)$ where $w_i$ is the weight for model $i$ (often set to validation accuracy or inverse log-loss).

Method 2: Stacking (Stacked Generalization)

The Stacking Architecture

HOW it works (2-layer approach):

Layer 1 (Base Models):

• Train diverse models on the training data: $f_1, f_2, ..., f_M$
• Generate predictions on validation/out-of-fold data

Layer 2 (Meta-Model):

• Use base model predictions as features
• Train a meta-learner (often Logistic Regression, Ridge, or Gradient Boosting)
• The meta-model learns: "When should I trust model1 vs model 2?"

1. Base predictions: $p_1 = f_1(\mathbf{x}), p_2 = f_2(\mathbf{x}), ..., p_M = f_M(\mathbf{x})$

2. Meta-features: $\mathbf{z} = [p_1, p_2, ..., p_M]$ (sometimes also include original features)

3. Final prediction: $\hat{y} = g(\mathbf{z})$ where $g$ is the meta-model

Why this step? The meta-model $g$ learns context-dependent weighting—it might learn "trust model 1 for high-dimensional data, trust model 2 for sparse data."

Preventing Data Leakage in Stacking

Why it's VERY wrong: Data leakage! If base models see the same data the meta-model trains on, the meta-model learns to overfit to the base models' training-set biases, not their true generalization ability.

The fix: Use out-of-fold predictions via k-fold cross-validation:

STEP-BY-STEP (5-fold example):

1. Split training data into 5 folds
2. For fold $i$:
   • Train each base model on folds $\{1,2,3,4,5\} \setminus \{i\}$
   • Predict on fold $i$ (these predictions are "unseen" by the base models)
3. Concatenate all out-of-fold predictions → training data for meta-model
4. Retrain base models on all training data for final test predictions

Training (on 10,000 houses):

Fold 1 (2000 houses):
  - Train LinearReg on folds 2-5 → predict fold 1
  - Train RF on folds 2-5 → predict fold 1
  - Train GB on folds 2-5 → predict fold 1
Repeat for folds 2-5...

Meta-training data (10,000 rows):
  house_id | LinearReg_pred | RF_pred | GB_pred | actual_price
  ------|-------------
  1        | 250k           | 270k    | 265k    | 260k
  2        | 180k           | 175k    | 182k    | 178k
  ...

Meta-model (Ridge Regression) learns: $\text{price} = 0.3 \times \text{LinearReg} + 0.4 \times \text{RF} + 0.3 \times \text{GB}$

Why these weights? Meta-model discovered RF slightly outperforms on validation—it gets 40% weight.

Test time:

• Retrain all base models on full10,000 houses
• For new house: get3 predictions, feed to meta-model
• Meta-model outputs final price

Typical improvement: Stacking often gives 2-5% error reduction over best single model.

Method 3: Blending

Blending vs Stacking

| Aspect | Stacking | Blending | |--------|----------| | Meta-training data | Out-of-fold predictions (all training data used) | Hold-out set predictions (typically 20-30% of data) | | Base model training | K times per fold | Once on training set | | Computational cost | High (K × M model traings) | Lower (M model trainings) | | Data efficiency | Better (uses all data for meta-training) | Worse (hold-out not used for base models) |

HOW blending works:

1. Split data: 70% train, 30% hold-out
2. Train base models on 70% train
3. Generate predictions on 30% hold-out
4. Train meta-model on hold-out predictions
5. For test: base models → meta-model

Split:

• Training set: 35,000 images (70%)
• Hold-out (blending set): 15,000 images (30%)

Base models trained on 35k:

• ResNet-50 (validation acc: 92%)
• EfficientNet (validation acc: 93%)
• Vision Transformer (validation acc: 91%)

Blending set predictions (15k images):

# Each row is an image, columns are model probabilities for each class
blend_features = np.column_stack([
    resnet_preds,      # (15000, 10) for 10 classes
    efficientnet_preds,
    vit_preds
])  # Shape: (15000, 30)
 
# Meta-model: Logistic Regression
meta_model.fit(blend_features, blend_labels)

Why blending for Kaggle? Competitions have strict time limits—blending is faster to iterate than full stacking.

Trade-off: Lost 15k training images for base models, but meta-model learns good combination.

Choosing Between Methods

Use Voting when:

• You want simplicity and interpretability
• You have 3-10 diverse models
• Models have similar performance (within 2-3% accuracy)
• You don't have much validation data

Use Stacking when:

• You want maximum performance
• You have enough data (10k+ samples)
• You can afford 5-10× training time
• Base models have varying strengths on different data subsets

Use Blending when:

• Stacking is too expensive
• You have a natural validation split (e.g., time-series: train on old data, blend on recent)
• Quick iteration is needed (competitions, protyping)

Diversity is Key

$\sigma^2_{\text{ensemble}} = \frac{\sigma^2}{M} + \rho\sigma^2\left(1 - \frac{1}{M}\right)$

Derivation insight:

• When $\rho = 0$ (independent): reduces to $\sigma^2/M$ (perfect scaling)
• When $\rho = 1$ (identical models): $\sigma^2_{\text{ensemble}} = \sigma^2$ (no benefit!)

PRACTICAL implication: Diversity matters more than individual accuracy! A 90% accurate model that makes different mistakes is more valuable than a 92% model that makes the same mistakes as others.

HOW to ensure diversity:

1. Different algorithms: Combine linear, tree-based, neural networks
2. Different features: Train on different feature subsets
3. Different training data: Bagging, bootstrapping, different time windows
4. Different hyperparameters: Vary depth, regularization, learning rate

Diverse ensemble:

• Logistic Regression on transaction features (amount, location)

  • Strength: Fast, interpretable, good for obvious patterns
  • Weakness: Mises complex interactions

• Random Forest on enginered features (rolling statistics, velocity)

  • Strength: Captures non-linear patterns, robust to outliers
  • Weakness: Can overfit rare fraud patterns

• Neural Network on raw transaction sequences

  • Strength: Learns temporal patterns
  • Weakness: Needs lots of data, black-box

Stacked meta-model (Logistic Regression) learns:

• Trust LR for high-amount foreign transactions (clear signal)
• Trust RF for moderate-amount domestic with unusual timing
• Trust NN for low-amount but suspicious sequence patterns

Result: Ensemble catches 15% more fraud than best single model (RF) while maintaining low false positive rate.

Implementation Considerations

Computational Cost

Training time for stacking with $M$ base models, $K$ folds, dataset size $N$:

$T_{\text{stacking}} = K \times \sum_{i=1}^M T_i(N \times \frac{K-1}{K}) + T_{\text{meta}}(N)$

where $T_i$ is training time for model $i$.

Why this step? Each fold trains on $(K-1)/K$ of the data, repeated $K$ times per model.

Typical: 5-fold stacking with 5 models = 25× base training time + meta-training.

Why it can be wrong: Overfitting at the meta-level! If:

• Base models are too similar (low diversity)
• Validation set is too small (meta-model overfits)
• Base models are already overfitting

The fix:

1. Ensure base model diversity (different algorithms, features)
2. Use regularized meta-models (Ridge, Lasso, small trees)
3. Monitor meta-model validation performance separately
4. Sometimes simple voting beats complex stacking on small datasets

Hyperparameter Tuning

Question: Should you tune base models before or after ensemble construction?

Answer: Before, but with a caveat:

1. Tune each base model independently to maximize individual performance
2. Then check correlation between predictions—if two models give nearly identical predictions, drop one
3. Build ensemble with diverse, well-tuned models
4. Optionally: tune meta-model hyperparameters (regularization strength)

When to Use Each Meta-Model

Diversity = "Don't Invite Very Educated Robots Simultaneously Into Teams Yielding (similar predictions)"

Now, how do you combine their guesses?

Voting is like: everyone writes their guess, and you pick the middle one (or the most common if they all guess differently).

Stacking is like: you watch which friend is usually right in different situations. Maybe Friend 1 is better when the jar is round, Friend 2 is better when it's square, and Friend 3 is best for small jars. You learn a pattern: "When the jar is round, trust Friend 1 more." That's what the meta-model does!

Blending is a simpler version: you give them a practice jar first, see how close each friend gets, then create a formula like "50% Friend 1's guess + 30% Friend 2's + 20% Friend 3's" based on who did best in practice.

The magic: even if each friend is sometimes wrong, when you combine them smartly, the group guess is usually closer than any single guess! That's because they make different types of mistakes, and averaging cancels out the errors.

Connections

• 2.6.1-Train-test-split: Foundation for creating validation sets for blending
• 2.6.12-Cross-validation: Used in stacking for out-of-fold predictions
• 2.6.8-Bias-variance-tradeoff: Theoretical foundation for why ensembles reduce error
• 2.7.1-Bagging-(Bootstrap-Aggregating): Special case of voting with bootstrapped data
• 2.7.2-Random-Forest: Ensemble of decision trees using bagging
• 2.7.3-Boosting-methods: Sequential ensembles (different from parallel voting/stacking)
• 2.8.5-Regularization: Important for meta-models to prevent overfitting
• 2.5.6-Feature-engineering: Different feature sets create diverse base models

#flashcards/ai-ml

What is the key principle behind ensemble learning? :: Combining predictions from multiple diverse models to achieve better performance than any single model, because different models make different types of errors that cancel out when aggregated.

Concept Map

Hinglish (regional understanding)