AdaBoost algorithm

AdaBoost ka core idea simple hai: ek hi bada strong model banane ke bajaye, hum bahut saare weak learners (aksar depth-1 decision stumps) ko ek-ek karke banate hain. Har stump bas coin-flip se thoda accha hona chahiye. Magic yeh hai ki har round ke baad hum un data points ka weight badha dete hain jinhe pichle learner ne galat classify kiya — yaani "in par zyada dhyan do". Isliye agla learner un mushkil examples par focus karta hai. Yeh "adaptive" behaviour hi naam mein hai.

Do knobs important hain. Pehla: sample weight $w_i$ — galat point ka weight $e^{+\alpha}$ se multiply hokar badhta hai, sahi point ka $e^{-\alpha}$ se ghatta hai. Dusra: har learner ka vote power $\alpha_m = \tfrac12\ln\frac{1-\varepsilon_m}{\varepsilon_m}$. Jitna accurate learner (chhota $\varepsilon$), utna loud vote. Agar $\varepsilon=0.5$ (random), toh $\alpha=0$ — us learner ko ignore kar dete hain. Yeh formula aasman se nahi aaya: yeh exponential loss $e^{-yf(x)}$ ko greedily minimize karne se derive hota hai, isliye $y \in \{-1,+1\}$ rakhte hain taaki $yh(x)$ correct pe $+1$ aur wrong pe $-1$ ho.

Final prediction ek weighted vote hai: $H(x)=\operatorname{sign}(\sum_m \alpha_m h_m(x))$. Yaad rakho: yeh bagging se alag hai — bagging parallel aur independent hota hai (variance kam), boosting sequential aur dependent hota hai (bias kam). Ek warning: noisy data / outliers pe AdaBoost overfit kar sakta hai, kyunki galat labels ka weight exponentially bada ho jaata hai. Exam aur interviews mein loop yaad rakhna: WEIGH → ERR → VOTE → RE-WEIGH.