Bagging and bootstrap aggregating — Hinglish notes

2.3.6 · AI-ML › Tree-Based & Instance Methods

Bagging KYA hai?

Yeh word literally tod ke dekho: Bootstrap aggregating → Bagging.

Averaging kyun help karta hai? (Derivation scratch se)

Maano har base model $b$ ek prediction $\hat{f}_{b} (x)$ produce karta hai jise hum ek random variable treat karte hain (randomness resampled training data se aata hai). Uska mean aur variance likho:

$E [\hat{f}_{b} (x)] = μ, Var (\hat{f}_{b} (x)) = σ^{2} .$

Bagged predictor average hai:

$\overset{ˉ}{f} (x) = \frac{1}{B} \sum_{b = 1}^{B} \hat{f}_{b} (x) .$

Step 1 — mean unchanged rehta hai. Kyun? Expectation linear hoti hai: $E [\overset{ˉ}{f} (x)] = \frac{1}{B} \sum_{b} E [\hat{f}_{b} (x)] = \frac{1}{B} \cdot B μ = μ .$ Toh bagging bias nahi badalta. (Yeh crucial hai: yeh sirf variance ko touch kar sakta hai.)

Step 2 — variance shrink hota hai. Maano har model ka variance $σ^{2}$ hai aur kisi bhi do models ke beech pairwise correlation $ρ$ hai. $Var (\sum a_{i} X_{i}) = \sum_{i} a_{i}^{2} Var (X_{i}) + \sum_{i \neq = j} a_{i} a_{j} Cov (X_{i}, X_{j})$ use karte hain jahan $a_{i} = 1/ B$ :

$Var (\overset{ˉ}{f}) = \frac{1}{B ^{2}} [diagonal B σ^{2} + off-diagonal B (B - 1) ρ σ^{2}] .$

Step 3 — simplify karo. $Var (\overset{ˉ}{f}) = ρ σ^{2} + \frac{1 - ρ}{B} σ^{2}$

Yeh kaise karte hain (algorithm)

Figure — Bagging and bootstrap aggregating

Bootstrap: Resampling kyun kaam karta hai

Har bootstrap sample ka size $n$ hota hai lekin yeh with replacement draw kiya jaata hai, isliye kuch points kai baar aate hain aur kuch miss ho jaate hain.

Kitne points left out hote hain? Probability ki ek specific point ek draw mein choose nahi hoti hai woh hai $1 - \frac{1}{n}$ . $n$ independent draws ke baad:

$P (point i absent) = (1 - \frac{1}{n})^{n} n \to \infty e^{- 1} \approx 0.368.$

Worked examples

Common mistakes (steel-manned)

Recall Feynman: 12-saal ke bachche ko explain karo

Socho ek weather-guesser jo random tarike se aksar galat hota hai — kabhi bahut zyada, kabhi bahut kam. Ab bahut saare doston se wohi sawaal poocho, lekin har dost ko kal ke weather facts ka thoda-sa shuffled copy do. Har dost phir bhi randomly-galat guess karta hai, lekin jab aap average karo unke saare guesses, toh zyada-waale aur kam-waale cancel ho jaate hain aur aap truth ke paas pahunch jaate ho. Trick: facts shuffle karo ek deck se cards pick karke aur har card wapas rakh ke agli pick se pehle — toh har dost ko alag deck milta hai. Kuch cards kabhi pick nahi hote (lagbhag ek-tihaayi) — tum unhi ka use karte ho har dost ko free mein secretly grade karne ke liye.

Flashcards

"Bagging" ka matlab kya hai?

Bootstrap AGGregating — bootstrap resamples par train karo, phir aggregate karo.

Size n ka bootstrap sample kaise draw kiya jaata hai?

n-point dataset se with replacement n points sample kiye jaate hain.

Bagging bias reduce karta hai ya variance?

Sirf variance; bias unchanged rehta hai kyunki

E [\overset{ˉ}{f}] = μ

Bagged ensemble ka variance formula kya hai?

Var (\overset{ˉ}{f}) = ρ σ^{2} + \frac{1 - ρ}{B} σ^{2}

B→∞ hone par ensemble variance kiske paas jaata hai?

ρ σ^{2}

— inter-model correlation se set hone wala floor.

Har model ke liye data ka kitna fraction out-of-bag hota hai, aur kyun?

~37%, kyunki

(1 - 1/ n)^{n} \to e^{- 1}

OOB error kya hai aur yeh useful kyun hai?

Har point ko sirf un trees se predict karo jinhone usse train nahi kiya — held-out set ke bina free validation.

Deep trees jaise unstable learners kyun bag karte hain?

Woh resamples ke across bahut badlate hain → low correlation ρ → bada variance reduction.

ρ σ^{2}

floor se aage jaane ke liye kaun si ek quantity lower karni chahiye?

Pairwise correlation ρ (Random Forests ko motivate karta hai).

Kya B (trees ki sankhya) badhane se overfitting ho sakti hai?

Nahi — bada B sirf averaging limit ki taraf converge karta hai; yeh akele kabhi overfit nahi karta.

Connections

Decision Trees — canonical high-variance base learner jise bagging fix karta hai.
Random Forests — bagging plus feature subsampling to reduce $ρ$ .
Bias-Variance Tradeoff — bagging variance term par attack karta hai.
Bootstrap (statistics) — bagging ke peeche ka resampling engine.
Boosting — contrast: sequentially bias reduce karta hai, overfit kar sakta hai.
Out-of-Bag Error — free validation estimate.
Ensemble Methods — umbrella family.