Bayesian hyperparameter optimization

2.6.14 · AI-ML › Model Evaluation & Selection

The Core Problem

YEH APPROACH KYUN? Models train karna costly hai—$f(\lambda )$ ka har evaluation ghanton le sakta hai. Jab 10+ hyperparameters hon toh hum grid search afford nahi kar sakte. Bayesian optimization har evaluation ko meaningful banata hai yeh balance karke:

• Exploitation: jaane-maane achhe configurations ke paas search karo
• Exploration: unexplored regions try karo jo better ho sakte hain

Derivation from First Principles

Step 1: The Surrogate Model (Gaussian Process)

KYA HAI: Hum unknown function $f(\lambda )$ ko Gaussian Process (GP) ki tarah model karte hain.

KYUN: GP sirf prediction nahi deta, balki uncertainty bhi deta hai—jo exploration/exploitation balance karne ke liye zaroori hai.

Ek Gaussian Process define hota hai:

• Mean function $\mu (\lambda )$: $f(\lambda )$ ka humara best guess
• Covariance (kernel) function $k(\lambda ,{\lambda }^{\prime })$: $f(\lambda )$ aur $f({\lambda }^{\prime })$ kitne correlated hain

Bilkul shuruaat se: $n$ evaluations ${\mathcal{D}}_n=\{({\lambda }_i,y_i){\}}_{i=1}^n$ observe karne ke baad jahan $y_i=f({\lambda }_i)+ϵ$ (noise), new point $\lambda$ pe GP posterior hai:

${\mu }_n(\lambda )=\mu (\lambda )+k^T{K}^{-1}(y-\mu )$ ${\sigma }_n^2(\lambda )=k(\lambda ,\lambda )-k(\lambda {)}^T{K}^{-1}k(\lambda )$

Derivation:

1. Prior: $f\sim \mathcal{G}\mathcal{P}(\mu ,k)$ ka matlab hai ki kisi bhi set of points ke liye, $f$ multivariate Gaussian hai
2. Likelihood: $y|f\sim \mathcal{N}(f,{\sigma }_{noise}^{2}I)$
3. Posterior (Bayes' rule): Observations diye hone pe, posterior bhi ek GP hai updated mean aur covariance ke saath

Yahan:

• $k(\lambda )=[k(\lambda ,{\lambda }_1),\dots ,k(\lambda ,{\lambda }_n){]}^T$ (vector)
• $K$ hai $n\times n$ covariance matrix $K_{ij}=k({\lambda }_i,{\lambda }_j)$
• $y=[y_1,\dots ,y_n{]}^T$, $\mu =[\mu ({\lambda }_1),\dots ,\mu ({\lambda }_n){]}^T$

YEH FORMULAS KYUN? GP posterior ek joint Gaussian ki conditional distribution hai. Mean ${\mu }_n(\lambda )$ humari prediction hai, aur ${\sigma }_n^2(\lambda )$ uncertainty quantify karta hai—jahan hum ne sample nahi liya wahan zyada hoga.

YEH KERNEL KYUN? Yeh smoothness encode karta hai: nearby hyperparameters ki performance similar honi chahiye. Chhhota $\ell$ → wiggly function, bada $\ell$ → smooth.

Step 2: The Acquisition Function

KYA HAI: Ek acquisition function $\alpha (\lambda )$ score karta hai ki $f$ ko $\lambda$ pe evaluate karna kitna "valuable" hai.

KYUN: GP humein ${\mu }_n(\lambda )$ aur ${\sigma }_n(\lambda )$ har jagah deta hai. Hum inhe use karte hain ${\lambda }_{n+1}$ pick karne ke liye jo exploration aur exploitation best balance kare.

Expected Improvement (EI)

Intuition: Hum expect karte hain ki $f(\lambda )$ current best observation $f^{+}={\max }_{i=1}^n{y}_i$ se kitna improve karega?

Bilkul shuruaat se derivation: $\lambda$ pe improvement hai $I(\lambda )=\max (0,f(\lambda )-f^{+})$. Kyunki $f(\lambda )$ ek random variable hai (GP posterior), hum compute karte hain: $\text{EI}(\lambda )=\mathbb{E}[I(\lambda )]=\mathbb{E}[\max (0,f(\lambda )-f^{+})]$

$f(\lambda )\sim \mathcal{N}({\mu }_n(\lambda ),{\sigma }_n^2(\lambda ))$ ke saath, hum derive kar sakte hain:

(\mu_n(\lambda) - f^+) \Phi(Z) + \sigma_n(\lambda) \phi(Z) & \text{if } \sigma_n(\lambda) > 0 \\ 0 & \text{if } \sigma_n(\lambda) = 0 \end{cases}$$ jahan: $$Z = \frac{\mu_n(\lambda) - f^+}{\sigma_n(\lambda)}$$ aur $\Phi, \phi$ standard normal CDF aur PDF hain. **Derivation steps:** 1. Maan lo $u = f(\lambda) - f^+$, toh $u \sim \mathcal{N}(\mu_n(\lambda) - f^+, \sigma_n^2(\lambda))$ 2. $\mathbb{E}[\max(0, u)] = \int_0^\infty u \cdot \frac{1}{\sigma_n} \phi\left(\frac{u -(\mu_n - f^+)}{\sigma_n}\right) du$ 3. Substitute karo $z = \frac{u - (\mu_n - f^+)}{\sigma_n}$, $u = \sigma_n z + (\mu_n - f^+)$ 4. Integration ke baad ($\int z\phi(z)dz = -\phi(z)$ use karke), upar wala formula milta hai **YEH KYUN KAAM KARTA HAI:** - High $\mu_n(\lambda)$ (exploitation): achha hone ki probability zyada - High $\sigma_n(\lambda)$ (exploration): uncertain hai, ho sakta hai bahut achha ho - Formula $\Phi(Z)$ aur $\phi(Z)$ terms ke zariye **automatically dono balance** karta hai #### Alternative: Upper Confidence Bound (UCB) $$\text{UCB}(\lambda) = \mu_n(\lambda) + \kappa \sigma_n(\lambda)$$ **KYUN:** Mean (exploitation) aur standard deviation (exploration) ko directly combine karta hai. $\kappa$ tradeoff control karta hai (typically $\kappa \approx 2$). ### Step 3: The Algorithm **Bayesian Optimization Loop:** ``` 1. Initialize with random samples: D_0 = {(λ_i, y_i)} for i=1..n_init 2. For iteration n = n_init, n_init+1, ..., n_max: a. Fit GP to D_n → get μ_n(λ), σ_n(λ) b. Find λ_next = argmax α(λ) (optimize acquisition function) c. Evaluate y_next = f(λ_next) d. Update D_{n+1} = D_n ∪ {(λ_next, y_next)} 3. Return λ* = argmax_{λ ∈ D} f(λ) ``` **HAR STEP KYUN:** - **a.** GP saare past evaluations se seekhta hai - **b.** Acquisition function sabse informative next point propose karta hai - **c.** Hum ek expensive evaluation ki cost pay karte hain - **d.** GP naye data se update hota hai, explored regions ke aas-paas uncertainty kam hoti hai ## Worked Examples > [!example] > **Example 1: Neural Network Learning Rate Tuning** **Setup:** Learning rate $\lambda \in [10^{-5}, 10^{-1}]$ (log scale) ke respect mein ek neural net ki validation accuracy optimize karo. **Iteration 1:** Random initialization evaluate karta hai: - $\lambda_1 = 0.01$, $y_1 = 0.85$ - $\lambda_2 = 0.001$, $y_2 = 0.70$ - $\lambda_3 = 0.05$, $y_3 = 0.80$ Current best: $f^+ = 0.85$ **Iteration 2:** RBF kernel ke saath GP fit karo. $\lambda = 0.005$ pe: - $\mu_2(0.005) = 0.83$ (0.001 aur 0.01 ke beech interpolate karta hai) - $\sigma_2(0.005) = 0.06$ (thodi uncertainty) - $Z = \frac{0.83 - 0.85}{0.06} = -0.33$ - $\text{EI}(0.005) = (0.83 - 0.85) \Phi(-0.33) + 0.06 \phi(-0.33)$ - $= -0.02 \cdot 0.37 + 0.06 \cdot 0.38 = -0.0074 + 0.0228 = 0.0154$ $\lambda = 0.08$ pe (unexplored): - $\mu_2(0.08) = 0.78$ (extrapolating, lekin uncertain) - $\sigma_2(0.08) = 0.12$ (high uncertainty) - $Z = \frac{0.78 - 0.85}{0.12} = -0.58$ - $\text{EI}(0.08) = -0.07 \cdot 0.28 + 0.12 \cdot 0.33 = -0.0196 + 0.0396 = 0.02$ **YEH STEP KYUN?** $\text{EI}(0.08) > \text{EI}(0.005)$ kam mean ke bawajood kyunki uncertainty zyada hai—exploration jeet jaata hai. **Action:** $\lambda_4 = 0.08$ evaluate karo, $y_4 = 0.87$ milta hai → naya best! **Iteration 3:** GP ko ab pata hai ki 0.08 achha hai, wahan ke aas-paas search focus karta hai kam uncertainty ke saath. > [!example] > **Example 2: Multi-dimensional — SVM with C and γ** **Setup:** SVM optimize karo: - $C \in [10^{-2}, 10^3]$ (regularization) - $\gamma \in [10^{-4}, 10^1]$ (RBF kernel width) 5 random points ke baad, GP seekhta hai: - High $C$, low $\gamma$: overfit karta hai (accuracy 0.75) - Low $C$, high $\gamma$: underfit karta hai (accuracy 0.70) - Mid-range: ab tak best $C=10, \gamma=0.1$, accuracy 0.88 **Acquisition function** suggest karta hai $C=15, \gamma=0.08$ (best ke paas lekin unexplored): - $\mu(\text{this point}) = 0.87$ (0.88 ke kaafi paas) - $\sigma(\text{this point}) = 0.08$ (moderate uncertainty) - High EI kyunki yeh promising region mein hai thodi uncertainty ke saath **Evaluate karo:** 0.89 milta hai → naya best. **YEH KYUN KAAM KARTA HAI:** $50 \times 50 = 2500$ points grid-search karne ki jagah, Bayesian optimization ~20-30 evaluations mein near-optimal dhundh leta hai **landscape ko seekhkar**. ## Common Mistakes > [!mistake] > **Mistake 1: Default GP hyperparameters tuning ke bina use karna** **Kyun sahi lagta hai:** "GP data se khud figure out kar lega." **Problem:** Kharab kernel hyperparameters (jaise galat length scale) → kharab surrogate model → kharab acquisition decisions. Agar $\ell$ bahut bada hai, GP over-smooth hai aur important variations miss karta hai. Bahut chhhota ho toh over-wiggly hai aur generalize nahi karta. **Steel-man:** Tum chahte ho ki algorithm automatic ho aur meta-tuning ki zaroorat na ho. **Fix:** ==Optimize kernel hyperparameters== **marginal likelihood** (evidence) maximize karke: $$\log p(y | \lambda_1, \ldots, \lambda_n, \theta) = -\frac{1}{2} y^T K^{-1} y - \frac{1}{2} \log |K| - \frac{n}{2} \log 2\pi$$ jahan $\theta = \{\sigma_f, \ell, \sigma_{noise}\}$. Zyaadatar libraries (jaise scikit-optimize, GPyOpt) yeh automatically karti hain, lekin verify karo ki yeh enabled hai. > [!mistake] > **Mistake 2: Hyperparameters ko normalize/scale na karna** **Kyun sahi lagta hai:** "Main raw ranges [0.001, 100] use kar lunga." **Problem:** GP kernels distances $\|\lambda - \lambda'\|$ compute karte hain. Agar ek hyperparameter [0, 1] pe span karta hai aur doosra [1, 10000] pe, toh doosra distance calculations mein dominate karta hai, jisse GP sochta hai ki woh hamesha door hain. **Steel-man:** Math kisi bhi scale ko handle kar lena chahiye. **Fix:** Optimization se pehle ==transform to uniform scale==: - Learning rates, regularization ke liye log-scale: $\tilde{\lambda} = \log_{10} \lambda$ - Mixed types ke liye standardization: $\tilde{\lambda} = (\lambda - \mu) / \sigma$ Libraries mein aksar `Real(..., prior='log-uniform')` options hote hain. > [!mistake] > **Mistake 3: Insufficient initialization** **Kyun sahi lagta hai:** "Bayesian optimization smart hai, main 2-3 points se start karunga." **Problem:** GP ko reasonable model banane ke liye kaafi data chahiye. Bahut kam points ke saath, surrogate unreliable hoti hai aur acquisition function kharab points propose kar sakta hai. Rule of thumb: **kam se kam $2d$ se $5d$** random initializations jahan $d$ dimensionality hai. **Steel-man:** Tum total evaluations minimize karna chahte ho, toh chhota start karo. **Fix:** Initialization aur optimization phases balance karo: - **Low-D** ($d \leq 5$): 5-10 random points - **High-D** ($d > 5$): $3d$ se $5d$ random points ya better coverage ke liye **Sobol sequences** (quasi-random) use karo ## Connections to Other Methods **Connects to:** - [[2.6.11-Grid-Search]] — BO evaluations dramatically reduce karta hai (30 vs. 10,000) - [[2.6.12-Random-Search]] — BO random initialization use karta hai lekin results se seekhta hai - [[3.4.2-Gaussian-Processes]] — GP woh probabilistic model hai jo BO ko power deta hai - [[4.3.1-Multi-Armed-Bandits]] — Acquisition functions exploration-exploitation solve karte hain (bandits mein UCB ki tarah) - [[2.6.15-Neural-Architecture-Search]] — BO discrete choices (architectures) optimize kar sakta hai appropriate kernels ke saath **Contrast with:** - **Evolutionary algorithms** (genetic algorithms): population-based, koi probabilistic model nahi - **Gradient-based meta-optimization** (MAML): differentiable hyperparameters chahiye - **Hyperband**: adaptive resource allocation (early stopping), BO ka complementary ## Key Formulas Summary > [!formula] > **Gaussian Process Posterior:** > $$\mu_n(\lambda) = k(\lambda)^T K^{-1} y, \quad \sigma_n^2(\lambda) = k(\lambda, \lambda) - k(\lambda)^T K^{-1} k(\lambda)$$ **Expected Improvement:** $$\text{EI}(\lambda) = (\mu_n(\lambda) - f^+) \Phi(Z) + \sigma_n(\lambda) \phi(Z), \quad Z = \frac{\mu_n(\lambda) - f^+}{\sigma_n(\lambda)}$$ **RBF Kernel:** $$k(\lambda, \lambda') = \sigma_f^2 \exp\left(-\frac{\|\lambda - \lambda'\|^2}{2\ell^2}\right)$$ > [!recall]- > **Ek 12-saal ke bachche ko explain karo:** Imagine karo tum sabse yummy cookie recipe dhundh rahe ho. Tum butter ki matra, sugar ki matra, aur baking time change kar sakte ho. Lekin cookies banana time leta hai! Tum har combination try nahi kar sakte. Toh trick yeh hai: kuch batches bake karke taste karne ke baad, tum yaad karne lagte ho ki kaun sa combination achha tha aur kaun sa nahi. Phir, sirf randomly guess karne ki jagah, tum sochte ho: "Hmm, zyada butter waale tasty the, lekin maine BAHUT zyada butter abhi try nahi kiya—woh try karte hain." Ya: "Medium sugar theek tha, lekin low sugar ke baare mein mujhe pata nahi, toh woh check karte hain." Bayesian optimization ek smart notebook ki tarah hai jo: 1. Tere saare past cookie experiments yaad rakhta hai 2. Guess karta hai ki koi bhi naya combination kitna achha ho sakta hai (yahi GP hai) 3. Bataata hai ki aage kaun sa experiment karo sabse zyada seekhne ke liye (yahi acquisition function hai) Yeh kam se kam cookie batches mein best recipe dhundhne mein help karta hai! > [!mnemonic] > **BO = "Best Optimized"** > - **B**ayes: Unknown function model karne ke liye probability use karo > - **O**ptimize: Acquisition function next point pick karta hai > - **GP** (Gaussian Process): "**G**ood **P**redictions" uncertainty ke saath > - **EI** (Expected Improvement): "**E**xplore **I**ntelligently" Yaad rakho: **"Build model, Optimize acquisition, Get next point"** → BO-loop ## Practical Implementation ```python from skopt import gp_minimize from skopt.space import Real, Integer def objective(params): lr, batch_size = params # Train model, return validation accuracy accuracy = train_and_evaluate(lr, batch_size) return -accuracy # Minimize negative accuracy space = [ Real(1e-5, 1e-1, prior='log-uniform', name='lr'), Integer(16, 256, name='batch_size') ] result = gp_minimize( objective, space, n_calls=50, # Total evaluations n_initial_points=10, # Random initialization acq_func='EI', # Expected Improvement random_state=42 ) best_params = result.x best_score = -result.fun ``` **YEH CODE STRUCTURE KYUN:** - `prior='log-uniform'` wide range learning rates handle karta hai - `n_initial_points` ensure karta hai ki GP ki achhi initialization ho - **Negative** accuracy minimize karo (libraries convention se minimize karti hain) #flashcards/ai-ml Bayesian hyperparameter optimization ka core idea kya hai? :: Ek probabilistic surrogate model (Gaussian Process) banao objective function ka, phir ek acquisition function use karo jo intelligently next hyperparameter configuration select kare evaluate karne ke liye, exploration aur exploitation balance karte hue. Ek Gaussian Process ke do components kya hain? ::: Mean function $\mu(\lambda)$ jo objective value predict karta hai, aur covariance/kernel function $k(\lambda, \lambda')$ jo points ke beech similarity model karta hai aur uncertainty quantify karta hai. Write the Expected Improvement formula :: EI(λ) = (μ_n(λ) - f^+) Φ(Z) + σ_n(λ) φ(Z), jahan Z = (μ_n(λ) - f^+)/σ_n(λ), f^+ current best hai, Φ standard normal CDF hai, φ PDF hai. Acquisition function kya balance karta hai? ::: Exploitation (jaane-maane achhe configurations ke paas search karna high $\mu$ ke saath) aur exploration (uncertain regions try karna high $\sigma$ ke saath). Learning rate jaise hyperparameters ke liye log-scale kyun use karte hain? ::: Learning rates orders of magnitude span karti hain (1e-5 se 1e-1 tak). Log scale pe distances meaningful hote hain—0.01 se 0.001 jaana utna hi significant hai jitna 0.001 se 0.0001. Isse GP kernel ko chhoti learning rates ko "equally distant" samajhne se bachata hai. RBF kernel formula kya hai aur length scale ℓ kya control karta hai? ::: $k(\lambda, \lambda') = \sigma_f^2 \exp(-\|\lambda - \lambda'\|^2/(2\ell^2))$. Length scale $\ell$ smoothness control karta hai: chhhota $\ell$ matlab function jaldi vary karta hai (nearby points alag ho sakte hain), bada $\ell$ matlab slow variation (door ke points similar hain). d hyperparameters ke liye kitne random initialization points use karne chahiye? ::: Kam se kam $2d$ se $5d$ points, ya low dimensions ke liye 5-10. Isse ensure hota hai ki GP ke paas optimization shuru hone se pehle reliable surrogate model banane ke liye kaafi data ho. UCB (Upper Confidence Bound) acquisition kaise kaam karta hai? ::: $\text{UCB}(\lambda) = \mu_n(\lambda) + \kappa\sigma_n(\lambda)$, exploitation (mean) aur exploration (std dev) directly combine karta hai. $\kappa \approx 2$ tradeoff control karta hai—zyada $\kappa$ matlab zyada exploration. Bayesian optimization ke dauran GP kernel hyperparameters optimize kyun karte hain? ::: Kharab kernel parameters (galat length scale) kharab surrogate models le jaate hain. Hum marginal likelihood $\log p(y|\lambda_1\ldots\lambda_n, \theta)$ maximize karte hain kernel hyperparameters $\theta$ dhundhne ke liye jo observed data best explain karein, GP predictions improve karte hue. Grid search ke comparison mein Bayesian optimization ka kya advantage hai? ::: BO evaluations se seekhta hai aur promising regions pe search focus karta hai, typically 20-50 evaluations mein near-optimal configurations dhundh leta hai grid search ki zaroorat ke hazar evaluations ki jagah, jo bahut zaroori hai jab training expensive ho. ## 🖼️ Concept Map ```mermaid flowchart TD Problem[Expensive black-box f lambda] -->|motivates| BayesOpt[Bayesian Optimization] GridRandom[Grid and Random search] -->|inefficient baseline| BayesOpt BayesOpt -->|goal| Argmax[Find lambda star maximizing f] BayesOpt -->|builds| Surrogate[Surrogate model] Surrogate -->|implemented as| GP[Gaussian Process] GP -->|defined by| Mean[Mean function mu] GP -->|defined by| Kernel[Covariance kernel k] Data[Observations D_n] -->|Bayes rule| GP GP -->|yields| Posterior[Posterior mean and variance] Posterior -->|feeds| Acq[Acquisition function] Acq -->|balances| Explore[Exploration vs Exploitation] Acq -->|selects next| NextEval[Next lambda to evaluate] NextEval -->|updates| Data ```