Noesis
2.3.7 · D5Tree-Based & Instance Methods

Question bank — Random forest algorithm

1,381 words6 min read↑ Back to topic

Before we start, one shared vocabulary reminder so nothing below uses an unearned symbol:

The one law that most of these traps orbit around:

True or false — justify

Adding more trees to a random forest keeps lowering the error toward zero.
False. Only the term shrinks with ; the variance bottoms out at the floor , which more trees cannot cross.
A random forest is just bagging applied to decision trees.
False. Bagging alone leaves all trees splitting on the same strong feature at the top, so stays high. Random forest also restricts each split to a random feature subset, which de-correlates the trees.
Pruning each tree hard makes the whole forest more accurate.
False. Averaging already removes the variance, so we want deep low-bias trees. Pruning raises each tree's bias and usually hurts the ensemble.
Random forests cannot overfit.
False. They resist overfitting from adding trees, but very noisy data or an close to keeps trees correlated and can still overfit. Randomness reduces, not eliminates, overfitting.
If two trees were perfectly independent (), the forest variance would fall to zero as .
True. With the floor is zero, so . This is exactly why we chase low .
Out-of-bag error requires a separate held-out validation split.
False. The OOB rows (~ per tree) are already left out by bootstrap sampling, giving a free validation set — no extra split or cross-validation loop needed.
Using (all features at every split) turns a random forest back into plain bagging.
True. With no feature restriction every tree sees all features and picks the same top split, so the de-correlation vanishes and only the row-bootstrap (bagging) randomness remains.
A bootstrap sample of size contains distinct rows.
False. Rows are drawn with replacement, so duplicates appear and about of the original rows are missing from each sample.

Spot the error

"I'll set trees to keep training fast; the variance formula says the floor is regardless of , so few trees is fine."
The floor is , but with tiny the term is still large. You need enough trees (hundreds) for that term to become negligible; only then is the floor reached.
"To lower correlation , I'll make as small as possible — set ."
Too small does lower , but each tree becomes too weak (high bias) because it's forced to split on near-useless features. The sweet spot is for classification, balancing de-correlation against per-tree strength.
"My forest overfits, so I'll add more trees to fix it."
More trees never reduce overfitting once you're past the plateau — they only shave the term. To fight overfitting, lower , cap tree depth, or clean noisy data instead.
"Feature A has the highest Mean-Decrease-in-Impurity, so it's definitely the most predictive feature."
MDI is biased toward high-cardinality and correlated features; a split can look impure-reducing without being genuinely predictive. Cross-check with permutation importance on OOB data. See Feature Importance.
"Since averaging reduces variance, I should also reduce each tree's variance by pruning."
Backwards. The forest's job is to remove variance for you; individual trees should stay high-variance/low-bias. Pruning trades away the low bias you actually want. See Bias-Variance Tradeoff.
"Random forest reduces bias like Gradient Boosting does."
No. Random forest is a variance-reducing ensemble (parallel, averaging). Gradient boosting is bias-reducing (sequential, each tree fixes the previous residuals). Different jobs entirely.

Why questions

Why does forcing each split to a random feature subset reduce the variance floor?
It lowers . Trees that are pushed onto different features make correlated different mistakes, and the floor of the variance law is — shrinking shrinks the irreducible floor.
Why grow the trees deep and unpruned instead of shallow?
Deep trees are low-bias, high-variance individuals. The ensemble average cancels their variance, so the only thing left to minimize per tree is bias — which deep growth achieves.
Why is the OOB fraction about and not something like ?
Each of the draws misses a given row with probability ; over independent draws that's .
Why does more trees "never hurt accuracy" yet still cost you something?
Adding trees only helps or plateaus accuracy (it monotonically reduces the term), but past the plateau you pay compute and memory for essentially no gain.
Why can't we just make one giant deep tree instead of a forest?
A single deep tree has huge variance — it memorizes quirks of its exact training set. The forest exists precisely to average that variance away across de-correlated trees.
Why is lowering described as "more important" than raising ?
Raising only removes the term, which you can drive small anyway; but sets the permanent floor that no amount of can beat.

Edge cases

What happens to the variance formula when (all trees identical)?
— averaging identical trees gains nothing. The forest is no better than a single tree.
What happens when and ?
. Perfectly independent trees average to a zero-variance estimate — the ideal, unreachable in practice because real trees on the same data are never fully independent.
If is very small (say ), is the " OOB" figure accurate?
No. only approaches as grows; for it's , so the OOB fraction is noticeably smaller.
For a regression forest, is the final prediction a vote?
No — it's the average of the tree outputs. Voting (majority) is the classification rule; averaging is the regression rule.
If every feature except one is pure noise, will small still help?
It hurts. With one useful feature, a small often excludes it at a split, forcing weak splits on noise — trees become too biased. Here a larger (or the boosting family) may work better.
What does the forest predict for a data point identical to a training row it memorized?
Not necessarily that row's label — each tree only saw ~ of rows, and the split structure generalizes, so the vote/average reflects the region, not a lookup. This is why forests still generalize.

Connections