Ensemble methods (voting, stacking, blending)

2.6.16 · AI-ML › Model Evaluation & Selection

Ensembles Kyon Kaam Karte Hain: Mathematical Foundation

Bias-Variance Perspective

KYUN ensembles error reduce karte hain? Prediction error decomposition consider karo:

$\text{Error}={\text{Bias}}^2+\text{Variance}+\text{Irreducible Error}$

KYA matlab hai har term ka:

• Bias: Wrong assumptions se systematic error (underfitting)
• Variance: Training data fluctuations ke liye sensitivity (overfitting)
• Irreducible: Data mein khud ka noise

KAISE ensembles help karte hain:

1. Averaging variance reduce karta hai: Agar models independent errors karte hain, to averaging randomness cancel kar deta hai
2. Diversity bias reduce karti hai: Alag-alag model types alag-alag patterns capture karte hain
3. Ensemble mein har model simpler ho sakta hai: Har weak learner specialize kar sakta hai

${\sigma }_{\text{ensemble}}^2=\frac{{\sigma }^2}{M}$

Scratch se derivation:

Maano har model $f_i$ prediction karta hai error ${ϵ}_i$ ke saath jahan $\text{Var}({ϵ}_i)={\sigma }^2$ aur errors uncorrelated hain.

Ensemble prediction: $\bar{f}=\frac{1}{M}{\sum }_{i=1}^M{f}_i$

Ensemble error: $\overline{ϵ}=\frac{1}{M}{\sum }_{i=1}^M{ϵ}_i$

Ensemble ka variance: $\text{Var}(\overline{ϵ})=\text{Var}\left(\frac{1}{M}{\sum }_{i=1}^M{ϵ}_i\right)$

Yeh step kyun? Variance independent sums mein linear hota hai.

$=\frac{1}{M^2}{\sum }_{i=1}^M\text{Var}({ϵ}_i)$

Yeh step kyun? Jab errors independent hon, $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$.

$=\frac{1}{M^2}\cdot M{\sigma }^2=\frac{{\sigma }^2}{M}$

Yeh dikhata hai ki variance ensemble size ke saath decrease hota hai—lekin sirf tab jab errors independent hon!

Method 1: Voting Classifiers

Hard Voting

YEH KYA KARTA HAI: Har classifier ek class label output karta hai. Ensemble sabse zyada baar aane wali class predict karta hai.

$M$ classifiers jo class labels ${\hat{y}}_1,{\hat{y}}_2,...,{\hat{y}}_M$ predict karte hain, unke liye:

${\hat{y}}_{\text{ensemble}}=\text{mode}({\hat{y}}_1,{\hat{y}}_2,...,{\hat{y}}_M)$

Yeh kyon kaam karta hai: Agar har classifier random se better hai (accuracy > 50%), to majority ke galat hone ki probability ensemble size ke saath exponentially decrease hoti hai.

Mathematical intuition: 3 models ke saath, har ek 70% accurate (independent errors):

• Probability ki teeno galat hain: $0.3^3=0.027$ (2.7%)
• Probability ki kam se kam 2 galat hain: $0.3^2\times 0.7\times 3+0.3^3=0.216$ (21.6%)
• Ensemble accuracy ≈ 78.4% (jab kam se kam 2 sahi hon)

Soft Voting

YEH KYA KARTA HAI: Har classifier se predicted probabilities ka average nikalo, phir sabse zyada average probability wali class choose karo.

${\hat{y}}_{\text{ensemble}}=\arg {\max }_c\frac{1}{M}{\sum }_{i=1}^M{P}_i(c)$

jahan $P_i(c)$ woh probability hai jo model $i$ class $c$ ko assign karta hai.

Soft voting aksar kyun jeetta hai: Yeh richer information (confidence levels) use karta hai, na ki sirf binary decisions. Ek highly confident sahi prediction kam confident galat predictions ko override kar sakti hai.

Kab kaunsa use karo:

• Hard voting: Jab models probabilities output nahi karte (e.g., SVM bina probability calibration ke)
• Soft voting: Jab models well-calibrated probabilities output karte hain—typically 1-2% better accuracy deta hai

Yeh galat kyun hai: Sabhi models equally reliable nahi hote! Ek 95% accurate model ko 60% accurate model se zyada weight milna chahiye.

Fix: Weighted voting use karo: $\hat{y}=\arg {\max }_c{\sum }_{i=1}^M{w}_i\cdot P_i(c)$ jahan $w_i$ model $i$ ka weight hai (aksar validation accuracy ya inverse log-loss se set hota hai).

Method 2: Stacking (Stacked Generalization)

Stacking Architecture

KAISE KAAM KARTA HAI (2-layer approach):

Layer 1 (Base Models):

• Training data par diverse models train karo: $f_1,f_2,...,f_M$
• Validation/out-of-fold data par predictions generate karo

Layer 2 (Meta-Model):

• Base model predictions ko features ke roop mein use karo
• Ek meta-learner train karo (aksar Logistic Regression, Ridge, ya Gradient Boosting)
• Meta-model seekhta hai: "Mujhe model 1 par kab bharosa karna chahiye vs model 2 par?"

1. Base predictions: $p_1=f_1(\mathbf{x}),p_2=f_2(\mathbf{x}),...,p_M=f_M(\mathbf{x})$

2. Meta-features: $\mathbf{z}=[p_1,p_2,...,p_M]$ (kabhi kabhi original features bhi include hote hain)

3. Final prediction: $\hat{y}=g(\mathbf{z})$ jahan $g$ meta-model hai

Yeh step kyun? Meta-model $g$ context-dependent weighting seekhta hai—yeh seekh sakta hai ki "high-dimensional data ke liye model 1 par bharosa karo, sparse data ke liye model 2 par."

Stacking mein Data Leakage Rokna

Yeh BAHUT galat kyun hai: Data leakage! Agar base models wahi data dekhte hain jis par meta-model train hota hai, to meta-model base models ke training-set biases par overfit karna seekhta hai, na ki unki true generalization ability par.

Fix: K-fold cross-validation ke zariye out-of-fold predictions use karo:

STEP-BY-STEP (5-fold example):

1. Training data ko 5 folds mein split karo
2. Fold $i$ ke liye:
   • Har base model ko folds $\{1,2,3,4,5\}\setminus \{i\}$ par train karo
   • Fold $i$ par predict karo (yeh predictions base models ke liye "unseen" hain)
3. Saare out-of-fold predictions concatenate karo → meta-model ke liye training data
4. Final test predictions ke liye base models ko saare training data par retrain karo

Training (10,000 ghar par):

Fold 1 (2000 ghar):
  - LinearReg ko folds 2-5 par train karo → fold 1 predict karo
  - RF ko folds 2-5 par train karo → fold 1 predict karo
  - GB ko folds 2-5 par train karo → fold 1 predict karo
Folds 2-5 ke liye repeat karo...

Meta-training data (10,000 rows):
  house_id | LinearReg_pred | RF_pred | GB_pred | actual_price
  ------|-------------
  1        | 250k           | 270k    | 265k    | 260k
  2        | 180k           | 175k    | 182k    | 178k
  ...

Meta-model (Ridge Regression) seekhta hai: $\text{price}=0.3\times \text{LinearReg}+0.4\times \text{RF}+0.3\times \text{GB}$

Yeh weights kyun? Meta-model ne discover kiya ki RF validation par thoda better perform karta hai—ise 40% weight milta hai.

Test time par:

• Saare base models ko full 10,000 ghar par retrain karo
• Naye ghar ke liye: 3 predictions lo, meta-model ko do
• Meta-model final price output karta hai

Typical improvement: Stacking aksar best single model se 2-5% error reduction deta hai.

Method 3: Blending

Blending vs Stacking

| Aspect | Stacking | Blending | |--------|----------| | Meta-training data | Out-of-fold predictions (saara training data use hota hai) | Hold-out set predictions (typically data ka 20-30%) | | Base model training | Har fold ke liye K baar | Training set par ek baar | | Computational cost | Zyada (K × M model trainings) | Kam (M model trainings) | | Data efficiency | Better (meta-training ke liye saara data use hota hai) | Worse (hold-out base models ke liye use nahi hota) |

KAISE kaam karta hai blending:

1. Data split karo: 70% train, 30% hold-out
2. Base models ko 70% train par train karo
3. 30% hold-out par predictions generate karo
4. Hold-out predictions par meta-model train karo
5. Test ke liye: base models → meta-model

Split:

• Training set: 35,000 images (70%)
• Hold-out (blending set): 15,000 images (30%)

Base models 35k par train kiye:

• ResNet-50 (validation acc: 92%)
• EfficientNet (validation acc: 93%)
• Vision Transformer (validation acc: 91%)

Blending set predictions (15k images):

# Har row ek image hai, columns har class ke liye model probabilities hain
blend_features = np.column_stack([
    resnet_preds,      # (15000, 10) for 10 classes
    efficientnet_preds,
    vit_preds
])  # Shape: (15000, 30)
 
# Meta-model: Logistic Regression
meta_model.fit(blend_features, blend_labels)

Kaggle ke liye blending kyun? Competitions mein strict time limits hote hain—blending full stacking se faster iterate karna deta hai.

Trade-off: Base models ke liye 15k training images lost ho gayin, lekin meta-model achha combination seekh leta hai.

Methods mein se Kaise Choose Karo

Voting tab use karo jab:

• Tum simplicity aur interpretability chahte ho
• Tumhare paas 3-10 diverse models hain
• Models ka performance similar hai (2-3% accuracy ke andar)
• Tumhare paas zyada validation data nahi hai

Stacking tab use karo jab:

• Tum maximum performance chahte ho
• Tumhare paas enough data hai (10k+ samples)
• Tum 5-10× training time afford kar sako
• Base models ki alag-alag data subsets par varying strengths hain

Blending tab use karo jab:

• Stacking bahut expensive ho
• Tumhare paas natural validation split ho (e.g., time-series: old data par train karo, recent par blend karo)
• Quick iteration chahiye (competitions, prototyping)

Diversity Hai Key

${\sigma }_{\text{ensemble}}^2=\frac{{\sigma }^2}{M}+\rho {\sigma }^2\left(1-\frac{1}{M}\right)$

Derivation insight:

• Jab $\rho =0$ (independent): ${\sigma }^2/M$ tak reduce hota hai (perfect scaling)
• Jab $\rho =1$ (identical models): ${\sigma }_{\text{ensemble}}^2={\sigma }^2$ (koi benefit nahi!)

PRACTICAL implication: Diversity, individual accuracy se zyada matter karti hai! Ek 90% accurate model jo alag galtiyan karta hai woh ek 92% model se zyada valuable hai jo baaki models jaise hi galtiyan karta ho.

KAISE diversity ensure karo:

1. Alag algorithms: Linear, tree-based, neural networks combine karo
2. Alag features: Alag feature subsets par train karo
3. Alag training data: Bagging, bootstrapping, alag time windows
4. Alag hyperparameters: Depth, regularization, learning rate vary karo

Diverse ensemble:

• Logistic Regression on transaction features (amount, location)

  • Strength: Fast, interpretable, obvious patterns ke liye achha
  • Weakness: Complex interactions miss kar leta hai

• Random Forest on engineered features (rolling statistics, velocity)

  • Strength: Non-linear patterns capture karta hai, outliers ke liye robust
  • Weakness: Rare fraud patterns par overfit kar sakta hai

• Neural Network on raw transaction sequences

  • Strength: Temporal patterns seekhta hai
  • Weakness: Bahut data chahiye, black-box hai

Stacked meta-model (Logistic Regression) seekhta hai:

• High-amount foreign transactions ke liye LR par bharosa karo (clear signal)
• Unusual timing wale moderate-amount domestic ke liye RF par bharosa karo
• Low-amount lekin suspicious sequence patterns ke liye NN par bharosa karo

Result: Ensemble best single model (RF) se 15% zyada fraud pakad leta hai, saath mein low false positive rate bhi maintain karta hai.

Implementation Considerations

Computational Cost

Training time stacking ke liye, $M$ base models, $K$ folds, dataset size $N$ ke saath:

$T_{\text{stacking}}=K\times {\sum }_{i=1}^M{T}_i(N\times \frac{K-1}{K})+T_{\text{meta}}(N)$

jahan $T_i$ model $i$ ke liye training time hai.

Yeh step kyun? Har fold $(K-1)/K$ data par train hota hai, har model ke liye $K$ baar repeat hota hai.

Typically: 5 models ke saath 5-fold stacking = 25× base training time + meta-training.

Yeh galat kyun ho sakta hai: Meta-level par overfitting! Agar:

• Base models bahut similar hain (low diversity)
• Validation set bahut chhota hai (meta-model overfit karta hai)
• Base models pehle se hi overfit kar rahe hain

Fix:

1. Base model diversity ensure karo (alag algorithms, features)
2. Regularized meta-models use karo (Ridge, Lasso, small trees)
3. Meta-model validation performance alag se monitor karo
4. Kabhi kabhi small datasets par simple voting complex stacking se better hota hai

Hyperparameter Tuning

Sawaal: Kya base models ko ensemble construction se pehle ya baad mein tune karna chahiye?

Jawaab: Pehle, lekin ek caveat ke saath:

1. Har base model ko independently tune karo individual performance maximize karne ke liye
2. Phir predictions ke beech correlation check karo—agar do models almost identical predictions dete hain, to ek drop kar do
3. Diverse, well-tuned models ke saath ensemble banao
4. Optionally: meta-model hyperparameters tune karo (regularization strength)

Kab Kaunsa Meta-Model Use Karo

Diversity = "Don't Invite Very Educated Robots Simultaneously Into Teams Yielding (similar predictions)"

Ab, tum unke guesses kaise combine karte ho?

Voting aisa hai jaise: sabhi apna guess likhte hain, aur tum beech wala pick karte ho (ya jo sabse common ho agar sab alag guess karein).

Stacking aisa hai jaise: tum dekhte ho ki kaunsa dost alag-alag situations mein usually sahi hota hai. Shayad Dost 1 tab better hota hai jab jar round ho, Dost 2 tab better hota hai jab square ho, aur Dost 3 chhote jars ke liye best hai. Tum ek pattern seekhte ho: "Jab jar round ho, Dost 1 par zyada bharosa karo." Yahi meta-model karta hai!

Blending ek simpler version hai: tum unhe pehle ek practice jar dete ho, dekhte ho har dost kitna close aata hai, phir ek formula banate ho jaise "50% Dost 1 ka guess + 30% Dost 2 ka + 20% Dost 3 ka" based on ki practice mein kaun best tha.

Magic yeh hai ki: bhale hi har dost kabhi kabhi galat ho, jab tum unhe smartly combine karte ho, group guess usually kisi bhi single guess se zyada close hota hai! Kyunki woh alag-alag types ki galtiyan karte hain, aur averaging errors cancel kar deta hai.

Connections

• 2.6.1-Train-test-split: Blending ke liye validation sets banane ka foundation
• 2.6.12-Cross-validation: Stacking mein out-of-fold predictions ke liye use hota hai
• 2.6.8-Bias-variance-tradeoff: Theoretical foundation ki ensembles error kyun reduce karte hain
• 2.7.1-Bagging-(Bootstrap-Aggregating): Bootstrapped data ke saath voting ka special case
• 2.7.2-Random-Forest: Bagging use karte hue decision trees ka ensemble
• 2.7.3-Boosting-methods: Sequential ensembles (parallel voting/stacking se alag)
• 2.8.5-Regularization: Meta-models ke liye overfitting rokne hetu important
• 2.5.6-Feature-engineering: Alag feature sets diverse base models banate hain

#flashcards/ai-ml

Ensemble learning ka key principle kya hai? :: Kai diverse models ki predictions combine karna taaki kisi bhi single model se better performance mile, kyunki alag-alag models alag-alag types ki galtiyan karte hain jo aggregate karne par cancel ho jaati hain.

Concept Map