Bagging and bootstrap aggregating: Question bank

Before you start, meet every symbol this page leans on — nothing here is used before it is named and pictured.

Definition The four symbols (and one picture each)

$\hat{f}_{b} (x)$ = the prediction of base model number $b$ at input $x$ . Because each model is trained on a different random resample, this number wobbles — think of $B$ archers all aiming at the same target, each arrow landing a little differently.
$\overset{ˉ}{f} (x) = \frac{1}{B} \sum_{b = 1}^{B} \hat{f}_{b} (x)$ = the ensemble average, the single prediction bagging actually outputs. Picture the centre of mass of all $B$ arrow-holes.
$μ = E [\hat{f}_{b} (x)]$ = the true expected prediction, the spot the arrows scatter around on average. If the target's bull's-eye sits away from $μ$ , that gap is bias.
$σ^{2} = Var (\hat{f}_{b} (x))$ = the variance of one model — the spread of the arrows around $μ$ , i.e. how much a single model wobbles when the data is reshuffled.
$ρ$ = the pairwise correlation between any two models — how much they wobble together. $ρ = 0$ = independent, $ρ = 1$ = carbon copies, and (edge case!) $ρ < 0$ = they lean opposite ways.

The figure below draws these ideas literally on two rows of dots.

Figure — Bagging and bootstrap aggregating

Intuition What the figure shows (read this before continuing)

The dashed navy vertical line marks $μ$ , the true expected prediction — the value the models scatter around.
The top row (orange dots) are $B$ independent models ( $ρ \approx 0$ ): their predictions are spread wide (that width is $σ$ , one model's wobble), yet their violet star average $\overset{ˉ}{f}$ lands right on the dashed $μ$ line — the independent wobble cancelled.
The bottom row (magenta dots) are correlated models ( $ρ$ high): they clump tightly but are all shifted the same way off $μ$ ; their violet star average stays shifted, because the shared lean cannot be averaged out — that leftover offset is the floor $ρ σ^{2}$ .
Take-away in one line: averaging squeezes the independent spread but is powerless against the common lean. That is exactly the two-term formula below.

Definition The one identity we need: variance of a weighted sum

Before deriving, we need one fact about how variances add. For a sum $S = \sum_{i} a_{i} X_{i}$ (fixed weights $a_{i}$ , random $X_{i}$ ): $Var (\sum_{i} a_{i} X_{i}) = \sum_{i} a_{i}^{2} Var (X_{i}) + \sum_{i \neq = j} a_{i} a_{j} Cov (X_{i}, X_{j}) .$ Why is this true? (a one-line sketch). Variance means "average squared distance from the mean". Write each $X_{i}$ 's deviation as $d_{i} = X_{i} - E [X_{i}]$ ; then $S - E [S] = \sum_{i} a_{i} d_{i}$ . Squaring a sum expands into every pairwise product: $(\sum_{i} a_{i} d_{i})^{2} = \sum_{i} a_{i}^{2} d_{i}^{2} + \sum_{i \neq = j} a_{i} a_{j} d_{i} d_{j} .$ Now take the average of both sides: the average of $d_{i}^{2}$ is $Var (X_{i})$ (the diagonal, a term's wobble with itself), and the average of $d_{i} d_{j}$ is $Cov (X_{i}, X_{j})$ (the off-diagonal, how two terms wobble together). That is the identity — no magic, just "square the sum, then average term by term."

True or false — justify

Bagging lowers a model's bias.

False.

E [\overset{ˉ}{f}] = μ

exactly (the arrows' scatter-centre is unchanged by averaging), so the ensemble sits at the same

μ

as one model — only the wobble

σ^{2}

shrinks, never the bull's-eye-to-

μ

gap.

Bagging a shallow decision stump gives a big accuracy boost.

False. A stump is high-bias/low-variance; there is little variance to remove, and the bias survives untouched. You get a slightly-less-noisy but still-wrong model.

With independent models (

ρ = 0

), averaging

B

of them cuts variance to

σ^{2} / B

True. Plug

ρ = 0

: the floor term

ρ σ^{2}

vanishes and you keep only

\frac{1}{B} σ^{2}

, the classic

1/ B

shrink.

No matter how many trees you add, ensemble variance can never drop below

ρ σ^{2}

True (for $ρ \geq 0$ ). As

B \to \infty

the

\frac{1 - ρ}{B} σ^{2}

term

\to 0

, leaving the correlation floor

ρ σ^{2}

that only diversity (lower

ρ

) can beat.

A bootstrap sample of size

n

contains all

n

distinct data points.

False. Drawing with replacement forces duplicates, so on average only

\approx 63%

of distinct points appear; the missing

\approx 37%

are out-of-bag.

Increasing

B

(number of bagged trees) can eventually cause overfitting.

False. More trees only push

\overset{ˉ}{f}

toward its

B \to \infty

averaging limit — a monotone convergence, not a divergence. Overfitting-with-rounds is a Boosting property, not a bagging one.

Bagging is a form of ensemble learning that trains models in sequence.

False. Bagging trains all

B

models in parallel and independently; sequential dependence is the signature of boosting.

OOB error is a biased, over-optimistic estimate of test error.

False. Each point is scored only by trees that never saw it, so the estimate is honest — comparable to cross-validation, and it comes free of a held-out set. See Out-of-Bag Error.

If all base models were identical (

ρ = 1

), bagging still reduces variance.

False. Set

ρ = 1

Var (\overset{ˉ}{f}) = σ^{2}

for any

B

. Averaging clones of the same wobble changes nothing.

Majority voting of independent classifiers can be more accurate than any single voter.

True. If each is right with

p > 0.5

and errors are independent, the binomial majority concentrates around the correct answer — the "wisdom of crowds" effect.

Spot the error

"Bagging works because it uses more data than a single model."

Error. Each bootstrap sample has the same size

n

and even fewer distinct points (

\approx 63%

). The gain is from averaging away variance, not from extra data.

"Because ensembles beat single models, bagging improves every error component."

Error. It improves only the variance component of the bias–variance decomposition; bias and irreducible noise are unchanged, per

E [\overset{ˉ}{f}] = μ

"To make bagging stronger, bag a stable learner like linear regression."

Error. Stable learners barely change across resamples, so

ρ \approx 1

and the floor

ρ σ^{2} \approx σ^{2}

— essentially no reduction. Bag unstable learners (deep trees) instead.

"Random Forests are just bagging with more trees."

Error. Random Forests add feature subsampling at each split to actively lower $ρ$ , attacking the floor itself — something more trees alone can never do.

"OOB samples are the duplicated points inside a bootstrap sample."

Error. OOB points are the ones left out of a sample (never drawn), not the duplicates inside it.

"Averaging predictions always cancels the errors, so more models is always better."

Error. Cancellation needs independence. If errors are correlated (

ρ

high), the shared mistakes survive averaging and the floor

ρ σ^{2}

blocks further gain.

"The master formula holds for any collection of base models."

Error. It assumes equal variance

σ^{2}

across models and a single common pairwise correlation

ρ

. With unequal variances or mixed correlations you must use the full covariance sum instead.

Why questions

Why does bagging touch variance but not bias?

Because averaging is a linear operation: linearity of expectation keeps the mean at

μ

(no bias change), while variance of an average genuinely shrinks with the count and correlation.

Why does the master formula split into exactly a floor term and a shrinking term?

The diagonal of the covariance sum (

B

own-variances) becomes the

\frac{1 - ρ}{B} σ^{2}

shrinker, while the

B (B - 1)

shared-covariance terms fuse into the constant floor

ρ σ^{2}

— the shared wobble that averaging cannot remove.

Why do we sample with replacement rather than without?

With replacement lets each resample differ randomly from the others, injecting the diversity that makes the models' errors partially cancel; sampling without replacement of size

n

would just return the original set every time.

Why does lowering

ρ

matter more than raising

B

once

B

is large?

The

\frac{1 - ρ}{B} σ^{2}

term is already tiny for large

B

, so you're stuck on the floor

ρ σ^{2}

; only diversity moves that floor down.

Why is OOB error "free" cross-validation?

Every point is naturally held out by

\approx 37%

of the trees, so those trees form a ready-made validation ensemble for that point — no separate split or re-training needed.

Why must the base learner be unstable for bagging to shine?

Instability means predictions swing across resamples, giving low

ρ

and high

σ^{2}

— exactly the two conditions the formula rewards with a large variance drop.

Why does majority voting need roughly independent voters to help?

The binomial concentration that pushes accuracy above

p

assumes votes are independent Bernoulli trials; correlated errors make many voters wrong together, collapsing the crowd advantage.

Edge cases

What happens to

Var (\overset{ˉ}{f})

B \to \infty

with

ρ = 0

\to 0

— perfectly independent averaging drives the wobble all the way out. This is the idealized best case, rarely reached in practice.

What happens to

Var (\overset{ˉ}{f})

B \to \infty

with

ρ = 1

It stays at

σ^{2}

— identical models never help, the floor equals a single model's variance.

Can

ρ

be negative, and what would that do?

Yes — if models systematically lean opposite ways,

ρ < 0

makes

ρ σ^{2} < 0

, pushing

Var (\overset{ˉ}{f})

below the independent case (better than

ρ = 0

!). But it is bounded by

ρ \geq - \frac{1}{B - 1}

, so this super-cancellation is only reachable with few, deliberately anti-correlated models.

What is the most extreme variance reduction the formula allows?

Set

ρ = - \frac{1}{B - 1}

(its legal floor): then

ρ σ^{2} + \frac{1 - ρ}{B} σ^{2} = 0

. Perfectly anti-correlated equicorrelated models can, in theory, cancel all variance — the mirror image of the

ρ = 1

dead case.

For very small

n

(say

n = 1

), what fraction is out-of-bag?

Zero — with one point, the single draw always picks it, so nothing is left out; the

\approx 37%

figure is a large-

n

limit of

(1 - \frac{1}{n})^{n}

If a base model already has variance

σ^{2} = 0

(a constant predictor), what does bagging achieve?

Nothing — there is no wobble to average away, and its bias (however large) is untouched. Bagging only pays off when

σ^{2} > 0

What if the base learner has huge bias but tiny variance?

Bagging is nearly useless; it can barely shave the already-small variance while the dominant bias remains. Prefer a lower-bias learner (or a different method) first.

If two "different" trees are trained on almost-identical bootstrap samples, what is

ρ

and what does it mean for the ensemble?

ρ \approx 1

, so the variance floor

ρ σ^{2} \approx σ^{2}

— the ensemble behaves like one tree despite the extra count.

Recall One-sentence self-test

If someone claims "adding more bagged trees fixed my high-bias underfitting model," what's your reply? ::: It can't — bagging leaves the ensemble mean at $μ$ ; they need a lower-bias learner, not more copies of the same biased one.