Question bank — Learning rate effects on convergence

False — a smaller $\eta$ converges more slowly, but given enough iterations both small and moderate $\eta$ reach the same minimum; if you stop early, the small one can actually have higher loss because it hasn't arrived yet.

True for an $L$-smooth loss — the Descent Lemma above bounds the step's effect by $-\eta \Vert g{\Vert }^2+\frac{L{\eta }^2}{2}\Vert g{\Vert }^2$, which is negative precisely when $\eta <2/L$, so each step is a guaranteed descent.

False — the clean $1/L$ rate is the worst-case safe choice that comes with the extra assumption of convexity (usually $L$-smoothness plus strong convexity); on a merely $L$-smooth non-convex loss it only guarantees you don't diverge, and the actually-fastest rate depends on the whole spectrum of the Hessian Matrix, sometimes pushing closer to $2/L$.

True — the gradient $2w$ with $\eta =0.5$ gives $w_1=w_0-0.5\cdot 2w_0=0$; this coincides with a Newton step because the Hessian ($\lambda =2$) is constant.

False — the speed-up is roughly true only while $\eta$ stays well below the stability limit; near or beyond $2/L$ doubling makes it oscillate or diverge, so iterations go to infinity, not down.

False — a too-large $\eta$ is the most common cause, but divergence can also come from exploding gradients, bad data scaling, or numerical blow-up in Backpropagation; check the gradient norms before blaming $\eta$.

False — on the test quadratic $\mathcal{L}(w)=w^2$ (so $L=2$, edge $\eta =1$) the iterate flips sign with the same magnitude forever (a perfect oscillation), so it neither converges nor diverges; you need strict inequality.

False — the gradient points uphill (steepest ascent); the minus sign is what turns it downhill. Removing the minus would send you climbing.

A rise after stable training rarely means $\eta$ is too big; it more often signals overfitting or an escape into a worse region — lowering $\eta$ is a band-aid, so diagnose the cause (regularization, initialization) first. Note a plateau is a different symptom: it makes loss stagnate flat, not rise, so don't lump it in with rising-loss causes.

The whole point of $\eta$ is to convert the gradient's [loss/parameter] units into a [parameter] step; in practice $\eta$ carries whatever scaling makes $\Delta \theta$ dimensionally a parameter, so it is not a bare pure number in the units sense even when written dimensionless.

Stability needs $\eta <2/\lambda =0.5$, not $\eta =\lambda$; using $\eta =4$ is eight times over the limit and will diverge exponentially.

Different parameters have different curvatures; a single $\eta$ is too big for steep directions and too small for flat ones, which is exactly why the Adam Optimizer uses a per-parameter effective rate.

Far from the minimum a large step is efficient, but near the minimum the same step overshoots and jitters; Learning Rate Schedules shrink $\eta$ late in training to settle precisely.

With $\lambda =2$ the limit is $\eta <1$; at $\eta =1.5$ the iterate flips sign and doubles each step ($|w_t|=2^t$), so the loss explodes, it does not decrease.

No — Momentum accumulates past gradients and can make an already-too-large step more explosive; momentum cures slow crawling in ravines and escapes saddles, not the stability ceiling on $\eta$.

The raw gradient only encodes direction and a magnitude tied to the loss's steepness; $\eta$ is the knob that sets the physical distance moved, decoupling step size from how steep the slope happens to be.

On a quadratic each step multiplies the distance-to-minimum by the constant $|1-\eta \lambda |$, and repeatedly multiplying by the same factor is exactly geometric (exponential) decay.

$L$ is the largest eigenvalue of the Hessian Matrix over the whole landscape, which changes at every point and is astronomically expensive to compute, so we grid-search $\eta$ instead of solving for it.

A high condition number means the largest and smallest Hessian eigenvalues differ wildly (steep vs flat directions); $\eta$ must be small enough not to diverge along the steep axis, which then crawls painfully along the flat axis — a single rate can't serve both.

Too-small $\eta$ leaves you far from the minimum (high loss), too-large $\eta$ oscillates or diverges (high loss), and only the middle band both moves fast and stays stable (low loss) — exactly the curve drawn in the first figure.

Dividing by the recent gradient size $\sqrt{v_t}$ gives large effective steps where gradients are small (flat regions) and small steps where gradients are large (steep regions), automatically balancing the per-direction curvature.

The update becomes ${\theta }_{t+1}={\theta }_t$ — no movement at all; this is a stationary point (minimum, maximum, or saddle) and $\eta$ cannot rescue it without a perturbation or Momentum.

Nothing — every update is ${\theta }_t-0\cdot \nabla \mathcal{L}={\theta }_t$, so the parameters freeze and no learning occurs.

It flips the sign of the step to ${\theta }_t+|\eta |\nabla \mathcal{L}$, which moves uphill — gradient ascent; you climb toward maxima and away from minima, so $\eta <0$ is never used for minimisation (only deliberately for maximising an objective).

No — with zero gradient the step is zero for every $\eta$; you are stuck until curvature or noise appears, which is why plateaus are dangerous even with a well-tuned rate.

The iterate maps $w_0\to w_0-1\cdot 2w_0=-w_0$, so it bounces $w_0,-w_0,w_0,\dots$ forever with unchanged magnitude — a knife-edge oscillation, neither converging nor diverging.

No — the gradient there is zero, so any $\eta$ yields a zero step and you simply stay put; the rate only matters while there is a nonzero slope to follow.

Along the flat direction even a large $\eta$ barely moves you (small gradient), yet that same $\eta$ overshoots the steep direction — this mismatch is the core reason plain gradient descent struggles on ill-conditioned problems.