XGBoost fundamentals and tuning — Hinglish notes

2.3.13 · AI-ML › Tree-Based & Instance Methods

XGBoost exist kyun karta hai?

Random Forest se compare karo (jo independent trees ko parallel mein average karta hai), boosting trees ko sequentially aur dependently banata hai — har ek is baat par depend karta hai ki pehle kya aaya tha.

MODEL kya hai? (Additive form)

Hum trees greedily add karte hain: round $t$ par prediction hai $\overset{y}{^}_{i}^{(t)} = \overset{y}{^}_{i}^{(t - 1)} + f_{t} (x_{i}) .$ Hume $f_{t}$ (naya tree) chunna hai jo objective ko sabse zyada reduce kare.

Objective — regularization andar kyun built-in hai

Kyun: ordinary gradient boosting sirf loss minimize karta hai; XGBoost $Ω$ ko optimization ke andar add karta hai taaki trees prune hon aur leaf weights shrink hon — automatically, afterthought ki tarah nahi.

KAISE: Leaf weights ko first principles se derive karna

Additive step ko round $t$ par objective mein daalo (pehle ke trees ke constants drop karo): $L^{(t)} = \sum_{i = 1}^{n} l (y_{i}, \overset{y}{^}_{i}^{(t - 1)} + f_{t} (x_{i})) + Ω (f_{t}) .$

Step 1 — Loss ko $\overset{y}{^}^{(t - 1)}$ ke around 2nd order tak Taylor-expand karo. Yeh step kyun? Hum ek arbitrary loss ko seedha tree structures par optimize nahi kar sakte, isliye hum ise ek quadratic se approximate karte hain — jise hum closed form mein solve kar sakte hain.

Har example ke liye gradient aur Hessian define karo: $g_{i} = \partial_{\overset{y}{^}} l (y_{i}, \overset{y}{^})_{\overset{y}{^}^{(t - 1)}}, h_{i} = \partial_{\overset{y}{^}}^{2} l (y_{i}, \overset{y}{^})_{\overset{y}{^}^{(t - 1)}} .$ Tab $L^{(t)} \approx \sum_{i} [g_{i} f_{t} (x_{i}) + \frac{1}{2} h_{i} f_{t} (x_{i})^{2}] + γ T + \frac{λ}{2} \sum_{j} w_{j}^{2} + const .$

Step 2 — Examples ko leaf ke anusaar group karo. Ek tree har $x_{i}$ ko exactly ek leaf $j$ mein bhejta hai. Maano $I_{j} = {i : x_{i} leaf j mein jaata hai}$ aur wo leaf constant $w_{j}$ output karta hai. Kyunki $f_{t} (x_{i}) = w_{q (i)}$ : $L^{(t)} = \sum_{j = 1}^{T} [(\sum_{i \in I_{j}} g_{i}) w_{j} + \frac{1}{2} (\sum_{i \in I_{j}} h_{i} + λ) w_{j}^{2}] + γ T .$ Maano $G_{j} = \sum_{i \in I_{j}} g_{i}$ aur $H_{j} = \sum_{i \in I_{j}} h_{i}$ .

Yeh step kyun? Ab objective har $w_{j}$ mein independent quadratics ka sum hai — minimize karna trivial hai.

Step 3 — $w_{j}$ ke upar minimize karo. Derivative zero set karo: $\frac{\partial}{\partial w _{j}} (G_{j} w_{j} + \frac{1}{2} (H_{j} + λ) w_{j}^{2}) = G_{j} + (H_{j} + λ) w_{j} = 0.$

Step 4 — Wapas substitute karo aur ek fixed tree structure ke liye best achievable objective nikalo:

Split kaise chuna jaata hai — Gain formula

Hum har tree try nahi kar sakte. Isliye greedy tarike se grow karte hain: ek node par, uske data ko Left ( $L$ ) aur Right ( $R$ ) mein split karne ki koshish karo. Improvement hai (score pehle) $-$ (score baad mein):

$- γ$ kyun? Split karne se ek leaf add hoti hai, jiska cost $γ$ hai. Agar Gain $\leq 0$ ho, split mat karo — yahi built-in pruning hai.

Figure — XGBoost fundamentals and tuning

Worked Example 1 — Leaf weight (regression, squared error)

Squared error $l = \frac{1}{2} (y - \overset{y}{^})^{2}$ ke liye: $g_{i} = \overset{y}{^}_{i}^{(t - 1)} - y_{i}$ (negative residual), $h_{i} = 1$ .

Maano ek leaf mein 3 examples hain jinka residual $(y - \overset{y}{^}) = + 2, + 3, + 5$ hai, toh $g_{i} = - 2, - 3, - 5$ . $λ = 1$ lo.

$G = - 10$ , $H = 3$ .
$w^{*} = - G / (H + λ) = - (- 10) / (3 + 1) = 2.5$ .

Yeh step kyun? Leaf $2.5$ output karta hai — mean residual $3.33$ ke paas hai lekin $λ$ ki wajah se 0 ki taraf shrunk hai. Bada $λ$ ⇒ zyada shrinkage ⇒ kam overfit.

Worked Example 2 — Kya split karna chahiye?

Node ka $G = - 10, H = 3$ hai. Proposed split: Left $(G_{L} = - 8, H_{L} = 2)$ , Right $(G_{R} = - 2, H_{R} = 1)$ . $λ = 1, γ = 2$ . $Gain = \frac{1}{2} [\frac{64}{3} + \frac{4}{2} - \frac{100}{4}] - 2 = \frac{1}{2} [21.33 + 2 - 25] - 2 = \frac{1}{2} (- 1.67) - 2 = - 2.83.$ Gain $< 0$ ⇒ split reject karo. Kyun: loss mein reduction $γ$ complexity cost ko beat nahi karti.

Worked Example 3 — Log-loss gradients

Binary classification, $p = σ (\overset{y}{^})$ , loss $= - [y ln p + (1 - y) ln (1 - p)]$ . Tab $g_{i} = p_{i} - y_{i}$ aur $h_{i} = p_{i} (1 - p_{i})$ . Agar $y = 1$ aur current $p = 0.2$ hai: $g = - 0.8$ (bada push upar), $h = 0.16$ . Kyun: confident-wrong points ko bade gradients milte hain ⇒ agla tree unpar focus karta hai.

Key Hyperparameters (jo 80/20 matter karta hai)

Param	Controls	Direction
`eta` / learning_rate	har tree ka contribution shrink karta hai: $\overset{y}{^}^{(t)} = \overset{y}{^}^{(t - 1)} + η f_{t}$	↓ = zyada robust, zyada trees chahiye
`n_estimators`	trees ki sankhya $K$	`eta` ke saath pair karo; early stopping use karo
`max_depth`	tree complexity	↓ overfit se ladne ke liye
`min_child_weight`	leaf mein min $\sum h_{i}$	↑ overfit se ladne ke liye
`gamma`	split ke liye min gain ( $γ$ )	↑ = zyada pruning
`lambda`, `alpha`	leaf weights par L2 / L1	↑ = zyada regularization
`subsample`	tree ke liye row sampling	<1 randomness add karta hai
`colsample_bytree`	feature sampling	<1 trees ko decorrelate karta hai

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum ek tasveer bana rahe ho aur ek dost tumhari galtiyan dikhata hai. Tum thoda theek karte ho. Phir ek aur dost batata hai ki kya abhi bhi galat hai, aur tum woh thoda theek karte ho. Har dost sirf ek chhoti si correction karta hai (yahi learning rate hai), aur tum ek dost ki nahi sunते agar unka "fix" matter karne ke liye bahut chhota ho (yahi gamma pruning hai). Bahut saare doston ke baad, tumhari tasveer shandar hoti hai — lekin agar tum hamesha ke liye doston ko bulate raho, toh woh un chezon ko "fix" karne lagte hain jo pehle se theek theen aur kharab kar dete hain. Isliye tum sahi waqt par ruk jaate ho (early stopping). XGBoost aaise vinammra doston ki team hai jo gradient (which way fix karna hai) aur Hessian (how carefully fix karna hai) dono use karke jawab ko paas la dete hain.

Recall — khud ko test karo

Flashcards

XGBoost ka additive model form kya hai?

\overset{y}{^}_{i} = \sum_{k = 1}^{K} f_{k} (x_{i})

, regression trees ka sum jo sequentially add hote hain, har ek residual errors fit karta hai.

XGBoost loss ka 2nd-order Taylor expansion kyun use karta hai?

Kisi bhi differentiable loss ko tree output mein ek quadratic mein convert karne ke liye, jise har leaf ke liye closed form mein minimize kiya ja sake (Newton-style), gradient

g_{i}

aur Hessian

h_{i}

use karke.

Optimal leaf weight ka formula?

w_{j}^{*} = - G_{j} / (H_{j} + λ)

jahan

G_{j} = \sum_{i \in I_{j}} g_{i}

H_{j} = \sum_{i \in I_{j}} h_{i}

Split Gain ka formula?

\frac{1}{2} [G_{L}^{2} / (H_{L} + λ) + G_{R}^{2} / (H_{R} + λ) - (G_{L} + G_{R})^{2} / (H_{L} + H_{R} + λ)] - γ

γ

kya karta hai?

Split karne ke liye minimum loss reduction required; har leaf ke liye complexity penalty ki tarah kaam karta hai, automatic pruning enable karta hai (split reject karo agar Gain ≤ 0).

λ

kya karta hai?

Leaf weights par L2 regularization;

w_{j}^{*}

ko 0 ki taraf shrink karta hai, overfit kam karta hai.

Squared error ke liye

g_{i}

aur

h_{i}

kya hain?

g_{i} = \overset{y}{^}_{i}^{(t - 1)} - y_{i}

(negative residual),

h_{i} = 1

Log-loss ke liye

g_{i}

aur

h_{i}

kya hain?

g_{i} = p_{i} - y_{i}

h_{i} = p_{i} (1 - p_{i})

, jahan

p_{i} = σ (\overset{y}{^}_{i})

XGBoost mein Random Forest ki tarah unlimited trees kyun nahi add kar sakte?

Trees dependent hain aur residuals fit karte hain, isliye extra trees noise overfit karte hain; averaging ki jagah early stopping use karo.

Learning rate (eta) ka kya role hai?

Har tree ka contribution scale karta hai:

\overset{y}{^}^{(t)} = \overset{y}{^}^{(t - 1)} + η f_{t}

; chhota eta = slower, zyada robust learning jisme zyada trees chahiye.

Sabse high-leverage tuning pair kaunse do hyperparameters hain?

Chhota learning_rate + n_estimators jo validation set par early stopping se control hota hai.

XGBoost structurally Random Forest se kaise alag hai?

RF independent parallel trees average karta hai (variance reduction); XGBoost dependent sequential trees ka sum karta hai jo residuals fit karte hain (boosting se bias reduction).

Connections

Gradient Boosting — XGBoost ek regularized, 2nd-order implementation hai.
Decision Trees — base learner ( $f_{k}$ ).
Random Forests — contrast: bagging/parallel vs boosting/sequential.
Regularization (L1 L2) — $λ, α$ leaf weights par kaam karte hain.
Bias-Variance Tradeoff — boosting bias reduce karta hai; regularization variance control karta hai.
Taylor Series — gradient+Hessian approximation ka basis.
Newton's Method — $w^{*} = - G / (H + λ)$ ek damped Newton step hai.