2.6.14 · D5Model Evaluation & Selection
Question bank — Bayesian hyperparameter optimization
Before starting, we make every symbol used below concrete. Read this once; then the questions are self-contained.
The figure below shows all three at once — this single picture powers most of the True/False items.
Recall
The three moving parts ::: A surrogate (a Gaussian Process giving and everywhere), an acquisition function (a cheap score like EI or UCB saying "sample here next"), and the loop (fit → propose → evaluate → repeat).
True or false — justify
Bayesian optimization guarantees it finds the global optimum.
False — it is a smart heuristic; the GP is only a belief about , and if the true function violates the smoothness the kernel assumes, the search can miss a narrow peak entirely.
Every iteration of Bayesian optimization is cheaper than one grid-search evaluation.
False — one loop step includes fitting the GP and optimizing the acquisition function, which cost compute; the win is fewer expensive training runs, not cheaper bookkeeping.
The surrogate model must eventually equal the true function everywhere.
False — we only need it accurate enough near promising regions; far-away accuracy is wasted effort, and the acquisition function deliberately never forces us to nail down bad regions.
Expected Improvement can be negative.
False — is an expectation of a quantity that is by the , so EI is always even when the mean sits below the current best.
If the GP posterior standard deviation is zero at a point, EI there is zero.
True — zero uncertainty means we already know the value exactly, so there is nothing left to gain by re-sampling it; the piecewise EI formula above returns for .
Increasing in UCB makes the search more greedy.
False — a larger weights uncertainty more, so it explores more; small makes it greedy (exploit the high mean).
Bayesian optimization is pointless when each training run is fast.
True — when evaluations are cheap, random search or grid search usually win because they parallelize trivially and skip the sequential GP-fitting overhead, so the extra machinery no longer pays off.
The RBF (squared-exponential) kernel assumes the objective is a smooth function of the hyperparameters.
True — its correlation decays smoothly with distance, so it bakes in "nearby give similar scores"; a jagged, discontinuous landscape will be modeled poorly.
A larger length scale means the GP treats far-apart points as more correlated.
True — decays slower for big , so distant points share information and the fitted function is smoother/flatter.
Spot the error
"We pick the next point as ."
Error — that maximizes the predicted mean only, which is pure exploitation and gets stuck; you must maximize the acquisition function (EI or UCB) so uncertainty also drives the choice.
"Since higher accuracy is better, we minimize the acquisition function."
Error — the acquisition function scores value of sampling, and we maximize it; the max/min is about itself, not .
"The current best in EI is the best value the GP predicts."
Error — is the best observed value among points we actually evaluated, not a GP prediction; using a predicted optimum would make EI over-optimistic.
"Random initialization is optional; just start the loop with one point."
Error — with too few initial points the GP has almost no shape to fit and the first acquisition step is nearly blind, so a handful of random samples (random search-style) seeds the model.
"We fit a fresh GP only to the newest data point each iteration."
Error — the GP is refit to all observations each round; that is exactly how past experiments keep informing future proposals.
"Because , when the term is positive."
Error — that first term is negative (a below-best mean), and it is the second term (always positive) that can keep EI positive via exploration.
"UCB and EI will always propose the same next point."
Error — they weight mean and uncertainty differently, so they can disagree; EI's tradeoff is nonlinear via , while UCB's is a fixed linear blend.
Why questions
Why does the GP give us uncertainty and not just a prediction , and why do we care?
Because it returns a full posterior distribution (mean and spread), and the uncertainty is what powers exploration — without it we'd never justify sampling regions we haven't seen.
Why does EI automatically balance exploration and exploitation without a hand-tuned knob?
The term rewards high mean (exploit) and rewards high uncertainty (explore), so both effects are baked into one closed-form number.
Why do we optimize the acquisition function instead of directly?
Because is the expensive black box we can only query a few times, while is a cheap function of the already-fitted GP that we can evaluate and optimize thousands of times for free.
Why is the exploration/exploitation dilemma here the same tension studied in multi-armed bandits?
Both must decide between pulling a known-good option and gathering info about uncertain ones; UCB in fact comes straight from the bandit literature.
Why can Bayesian optimization struggle as the number of hyperparameters grows large?
The GP's distance-based kernel weakens in high dimensions (everything looks far apart), so its predictions become uncertain everywhere and the search loses focus — a reason approaches like NAS use other surrogates.
Why do we usually optimize hyperparameters like learning rate on a log scale?
Because performance changes over orders of magnitude, so equal ratios matter more than equal differences; log-spacing makes the smooth-kernel assumption reasonable across the range.
Why does the per-iteration cost of standard Bayesian optimization grow badly as evaluations pile up?
Fitting the GP inverts an covariance matrix, an operation, so after hundreds of evaluations the bookkeeping itself can rival a training run — motivating sparse GPs or other surrogates.
Edge cases
What does the acquisition function propose if noise makes two evaluations of the same give different ?
The GP treats with noise variance, so it won't force at observed points, meaning even a "seen" point keeps a little uncertainty and can be re-sampled.
What happens right at the start when the GP has zero observations?
It collapses to the prior — a flat mean with constant uncertainty everywhere — so every point looks equally worth trying, which is why we seed with random initial samples first.
If the true optimum lies on the boundary of the hyperparameter space , can Bayesian optimization find it?
Yes, but the acquisition optimizer must be allowed to propose boundary points; the GP has larger uncertainty near edges (fewer neighbors), which actually nudges exploration toward them.
What if two hyperparameters barely affect performance while a third dominates?
A single global length scale misjudges them all; using per-dimension length scales (ARD) lets the GP learn to ignore the flat directions and focus on the influential one.
What happens to as you sample the same region over and over?
It shrinks toward the noise floor there, so EI in that region drops toward zero and the acquisition function is pushed to explore elsewhere — the built-in escape from over-exploitation.
Degenerate case: the objective is actually constant (all give the same score). What does the loop do?
The GP flattens to that constant with shrinking uncertainty, EI decays toward everywhere, and the search harmlessly wanders since no point is better than .
What breaks if a hyperparameter is discrete or categorical (e.g. optimizer choice, layer type)?
A Euclidean-distance RBF kernel is meaningless across unordered categories, so plain GP-BO misbehaves; you need a kernel built for that structure (e.g. Hamming/one-hot) or a tree-based surrogate instead.
Why can't you just run Bayesian-optimization iterations to be safe?
Because the GP fit makes late iterations slow and returns diminishing acquisition gains once uncertainty is low everywhere, so budgets are usually kept to tens–hundreds of evaluations.