Question bank — Hyperparameter tuning for deep nets

False — it is a hyperparameter set before training; gradient descent tunes weights and biases, not the knob that multiplies its own step.

False — the test set is touched once, at the very end; using it to make choices leaks its information and biases your reported accuracy upward.

Roughly true — more capacity can memorise the training set better, so training loss usually falls; but this says nothing about generalisation, which is why we judge capacity on validation loss.

False — beyond a point extra capacity overfits, so validation loss rises even while training loss falls (see Bias-Variance Tradeoff).

False — doubling batch halves gradient noise, so the effective step character changes; the linear scaling rule says you should roughly double the learning rate too.

False — grid repeats identical values along unimportant axes; random search gives every run a distinct value of the important knob, so it explores that dimension far more thoroughly.

False — ~90% of samples land in $[10^{-2},10^{-1}]$; a multiplicative knob needs log-uniform sampling so each decade gets equal probability.

False — too large a ${\lambda }_{L2}$ pulls weights toward $0$ so hard the model underfits; regularisation strength is itself a hyperparameter to balance (see Regularization - L2 and Dropout).

False — the schedule controls how $\eta$ changes over training; warmup avoids early instability and decay lets you settle into the minimum (see Learning Rate Schedules and Warmup).

False — that sanity check tests whether your code can learn at all; a failure usually means a bug in the model or loss, not the hyperparameter values.

Error — training loss rewards memorisation, so it picks zero regularisation; hyperparameters that control overfitting must be chosen on validation loss.

Error — the explosion point sits right at instability; pick roughly one order below it (e.g. $\eta \approx 3\times 10^{-3}$) for a safety margin.

Error — grid cost multiplies: $5^5=3125$ trainings, not $5\times 5$; the combinatorial product is exactly why grid search is infeasible for deep nets.

Error — the occasional near-duplicate is far cheaper than grid's guaranteed duplicates along unimportant axes; random still tests the important knob at many distinct values.

Error — it shrinks the weights: the update becomes $(1-\eta \lambda )\theta -\eta g$, so $\theta$ decays by the factor $(1-\eta \lambda )$ each step, independent of the gradient term.

Error — Batch Normalization introduces choices (whether to use it, its interaction with LR and batch size) and changes the loss landscape, which shifts the good learning-rate range.

Because we need a held-out signal for generalisation; on training data the best hyperparameters are whatever memorises it, which tells us nothing about unseen data.

It multiplies every gradient step, so it alone decides whether training converges, oscillates, or diverges — all other knobs only refine an already-reasonable run.

Sampling the exponent uniformly makes the density $p(\eta )\propto 1/\eta$, so the integrated probability over $[10^k,10^{k+1}]$ is the same for every $k$.

Matching expected displacement over $k$ examples assumes gradients stay roughly constant; for very large $k$, curvature makes one big step overshoot, which is why warmup is added.

Every run draws a fresh random value on each axis, so no matter which axis matters, that axis is explored at as many distinct values as there are runs.

Validation loss depends on ${\theta }^{\ast }(\lambda )$, the result of a whole training run, and many hyperparameters (depth, layer count) are discrete — so no usable gradient w.r.t. $\lambda$ exists.

The meaningful quantity is the averaging window $\approx 1/(1-\beta )$; small changes near $\beta =0.999$ swing that window hugely, so spacing evenly in $(1-\beta )$ is the multiplicative-correct scale (see Adam and Momentum Optimizers).

The update $\theta \leftarrow \theta -0\cdot g$ never moves; loss stays flat forever — the degenerate "too small" limit taken to its extreme.

It is flat, because $\eta$ is so tiny that each step barely changes the weights — negligible but not divergent progress.

The decay term flips the sign of the weights each step, causing oscillation/divergence — a sign that $\eta$ or $\lambda$ is set far too large.

Prefer keeping enough training data to learn at all, and use cross-validation to reuse data as validation across folds instead of sacrificing a big fixed chunk.

You still report the test number from the config chosen on validation, and you do not re-pick using test loss — otherwise you overfit the test set and bias the result.

Zero regularisation with high capacity — the classic overfitting corner where training loss goes near zero but validation loss climbs.