5.6.9 · D5Machine Learning (Aerospace Applications)
Question bank — Optimization — SGD, momentum, Adam — derivations
True or false — justify
Gradient descent always decreases the loss on every step.
False. It only decreases if is small enough that the first-order Taylor Expansion stays accurate; too large a step can overshoot up the far wall of a valley.
The negative gradient points straight at the nearest minimum.
False. It points along steepest local descent only; the path to a minimum through a curved or ravine-shaped loss landscape can zig-zag far from the straight line to the minimum.
The SGD mini-batch gradient equals the true gradient plus a fixed constant.
False. It equals the true gradient plus zero-mean noise — it is unbiased (), so the error averages out, it is not a constant offset.
Momentum with multiplies your learning rate by 10 in every direction.
False. The amplification only applies to directions where the gradient sign is consistent; in oscillating directions the alternating terms cancel instead of adding.
Adam's per-coordinate scaling means it never needs a learning rate .
False. Adam still multiplies by ; the term only normalizes the step to roughly , it does not set the overall scale.
Bias correction in Adam matters just as much at step 10000 as at step 1.
False. The factor as grows, so correction is huge early (at , ) and negligible late.
Because SGD is noisy, it can never converge to a minimum.
False. With a decaying schedule (see Learning Rate Scheduling) the noise term shrinks over time, letting SGD settle into a minimum.
Full-batch gradient descent is strictly more accurate and therefore always the better choice.
False. It is exact but deterministic, so it stalls at saddle points, and it is per step — SGD's noise escapes saddles and buys many more updates per second.
Spot the error
"Momentum: , and since , velocity always shrinks toward zero."
The error: velocity shrinks only if new gradients stop arriving; while keeps feeding in with the same sign, grows toward , not zero.
"Adam divides by to normalize the step."
It divides by , not . The square root is essential: , so and the ratio — a unit step. Dividing by would give the wrong scale .
"Since is used in Adam, it must be over the batch."
The error: is element-wise squaring of the gradient vector, not the squared sum. Each coordinate gets its own second-moment estimate.
"The update rule is to go downhill."
Sign error: downhill is . Adding the gradient climbs toward steepest increase, so the loss would rise.
" in Adam is the learning rate for tiny gradients."
The error: is a numerical guard preventing division by zero when ; it is not a learning rate.
"Bias correction divides by , a fixed number."
The error: it divides by (note the exponent ), which changes every step and only equals at .
" immediately, so Adam needs no correction."
The error: unrolling gives , which is below early on — exactly why correction exists.
Why questions
Why does momentum use an exponentially weighted sum rather than a plain average of past gradients?
The weights let recent gradients dominate while old ones fade, so the velocity adapts to a changing landscape instead of being dragged by stale directions — it is a form of Exponential Moving Average.
Why does dividing by make noise-only directions take ~zero step?
In pure noise (mean cancels) but , so the ratio — Adam refuses to move where there is no consistent signal.
Why do we absorb the gradient's norm into instead of normalizing every step?
Normalizing would give equal-length steps everywhere, ignoring that near a minimum the gradient shrinks and we want smaller steps; keeping un-normalized auto-slows near flat regions.
Why is the mini-batch estimate's variance a feature and not just a bug?
The variance perturbs the parameters off exact saddle points, where the true gradient is zero and deterministic Gradient Descent would freeze forever.
Why does Adam help most when different parameters have wildly different gradient scales?
The per-coordinate rescales each parameter's step to ~, so a huge-gradient weight does not explode while a tiny-gradient weight does not stall — a common situation across early vs late layers of an aircraft-sensor net.
Why can't we just solve directly for a neural net?
With millions of nonlinear parameters the equation has no closed form and countless solutions (minima, maxima, saddles); iterative descent is the only tractable route.
Edge cases
What does momentum do on the very first step, when ?
With , the first momentum step equals a plain SGD step — momentum only diverges from SGD once history accumulates.
What is Adam's step when a coordinate's gradient is exactly zero for all time?
Both and stay 0, so the update is — that parameter never moves, which is correct since it has no signal.
What happens to momentum if ?
The velocity recursion collapses to , and the update becomes plain SGD — momentum with zero memory is just SGD.
What happens to momentum as ?
The amplification and old gradients almost never fade, so the ball barely feels new terrain — it overshoots badly and can become unstable.
What is Adam's effective step direction if a gradient is steady in sign?
, , so the step is — a clean unit-scaled step in the correct downhill direction, independent of .
At a saddle point where , what does deterministic gradient descent do versus SGD?
Deterministic GD has so it stops dead; SGD's noisy nudges it off the saddle and lets descent resume.
What happens if the learning rate is set larger than the region where the Taylor approximation holds?
The step overshoots the local linear model, so can increase and the iterates may oscillate or diverge — the whole "descent" guarantee relies on small .
Recall One-line self-test
If you can answer this without peeking, you own the chapter. Why does Adam's first step have magnitude regardless of how big is? ::: Bias correction gives and , so , leaving the step at .