This page is a trap course. Each line is a question ::: answer reveal. Cover the right side, commit to an answer out loud with a reason, then check. Bare "true"/"false" earns you nothing — the reasoning is the point.
Everything you need to answer the traps is defined on this page — you should not have to click away. The cross-links point to deeper topics, not to missing definitions.
The picture below is the whole story: stand at θt, read the uphill arrow ∇L, and walk the scaled distance η∥∇L∥ in the opposite direction to reach θt+1.
Two ideas the traps lean on repeatedly, both drawn out for you:
(a) The Taylor "flat plane" (used in WY3, WY4, SE4). Let Δθ be the ==increment in θ== — the small displacement vector you'd add to your current position, so the new position is θ+Δθ. Zoom close enough to any smooth surface and it looks like a tilted flat plane. That plane is the first-order Taylor approximation
L(θ+Δθ)≈current heightL(θ)+linear tilt∇L(θ)⊤Δθ.
On a flat tilt the fastest-down direction is obvious — straight against the gradient. The approximation is only trustworthy for smallΔθ; step too far and the real curved surface bends away from the plane.
(b) The angle α (used in WY1, SE5). For any candidate step direction Δθ, let α be the angle between the gradient arrow ∇L and the step Δθ. Then the loss change is
∇L⊤Δθ=∥∇L∥∥Δθ∥cosα.
This is most negative (biggest drop) when cosα=−1, i.e. α=180∘ — pointing exactly backwards along the gradient.
(c) The quadratic bowl and its stability bound (used in SE1, WY8). The simplest curved loss is a bowl L(θ)=21θ⊤Aθ, where A is the symmetric Hessian matrix — it stores the curvature of the bowl in every direction. For this to be a genuine upward bowl (a convex valley with a bottom) we require A to be positive definite: every eigenvalue λ>0, so the surface curves up in every direction. Those eigenvalues are the curvatures along the bowl's principal axes, and λmax is the steepest-curved one.
Substituting ∇L=Aθ into the update gives θt+1=θt−ηAθt=(I−ηA)θt, where I is the identity matrix — the "do-nothing" matrix that leaves any vector unchanged (Iθ=θ), here representing the part of θt that is kept before the −ηAθt correction. Along each principal axis with curvature λ, the error is multiplied by the number 1−ηλ every step, so it shrinks only if ∣1−ηλ∣<1. Unpacking that inequality:
−1<1−ηλ<1⟹0<ηλ<2⟹0<η<λ2.
The tightest constraint comes from the largestλ, giving the safe ceiling
η<λmax(A)2.
False. It points toward the steepest increase (uphill). We negate it to head downhill; only for special symmetric bowls does the negated arrow happen to aim straight at the minimum.
TF2. A larger gradient magnitude always means we are far from the minimum.
False. Magnitude measures local steepness, not distance. A steep cliff can sit right next to the bottom, and a near-flat plateau can be far away.
TF3. If ∇L(θ)=0, then θ is a minimum.
False. A zero gradient marks any stationary point — minimum, maximum, or saddle. See Local Minima and Saddle Points for the ones that aren't minima.
TF4. Gradient descent always finds the lowest possible loss.
False. It finds a local minimum reachable from the start point. Only when L is convex is every local minimum the global one — that guarantee is exactly what Convex Optimization gives you.
TF5. Multiplying every loss value by 10 leaves the optimal θ∗ unchanged.
True. Scaling L scales its gradient by the same factor, so the locationθ∗ of the minimum is identical — but the effective step size changes, so you'd want a smaller η.
TF6. The minus sign in the update is just a convention; you could drop it and flip η's sign.
True in effect, dangerous in practice. Using +η∇L with η>0 climbs uphill (gradient ascent); equivalently, a negative learning rate η<0 in the standard formula also climbs. You must flip something, and keeping the minus with η>0 is the convention that avoids sign confusion.
TF7. If the loss goes up on one step, the learning rate is definitely too big.
Not definitely. With the full gradient on a smooth loss, a strict up-step means the step left the Taylor-valid zone — a formal sufficient condition for a guaranteed decrease is η<2/λmax (see the bowl above), so violating it causes the rise. But with mini-batch noise (Stochastic Gradient Descent) the gradient is only an estimate, so a single up-step can be sampling noise while the average still descends; you diagnose true divergence from a rising trend, not one step.
TF8. Gradient descent on a straight-line (linear) loss L(θ)=c⊤θ converges.
False. A tilted plane has no bottom; the gradient is a constant nonzero vector, so you march off to −∞ forever.
TF9. Two runs from different start points must reach the same minimum.
False in general (non-convex landscapes have many basins). True only for convex L, where all paths funnel into the one global basin.
SE1. "Set η as large as possible so we reach the valley in the fewest steps."
Error: past a threshold the step overshoots the minimum and the loss oscillates or diverges. For the positive-definite quadratic bowl L=21θ⊤Aθ defined above, the safe ceiling is η<2/λmax(A), where λmax is the largest eigenvalue (steepest curvature) of the Hessian A — see Learning Rate Schedules.
SE2. Update code: w[0]-=lr*g[0]; w[1]-=lr*g[0].
Error: the second line reuses g[0] instead of g[1], so parameter 1 is nudged by the wrong slope. Vectorize as w -= lr*g to make this impossible.
SE3. "The gradient is a single number telling us how steep the loss is."
Error: with many parameters ∇L is a vector, one partial derivative per parameter. It has both a direction (in parameter space) and a magnitude; collapsing it to one number loses the direction.
SE4. "We move opposite the gradient because −∇L is the direction where the loss is guaranteed to be lowest."
Error: it's the direction of fastest initial decrease, valid only for an infinitesimal step (from the first-order Taylor term). Over a finite step curvature can make another direction better.
SE5. "Because cosα=−1 minimizes ∇L⊤Δθ, we should pick the angle α=−1."
Error: as defined in part (b) above, α is the angle between the gradient and the step. cosα=−1 occurs at α=180∘, not α=−1; the reader confused the cosine value with the angle itself.
SE6. "η changes the direction of the step so we can steer around obstacles."
Error: η is a positive scalar; it only scales length. Direction is fixed by −∇L. Steering different directions requires methods like Momentum and Adam or Second-Order Methods.
SE7. "For L(θ)=θ2 with θ0=4, since ∇L=8>0 we add the step."
Error: positive gradient means uphill to the right, so we subtract to go left: θ1=4−η(8). Adding would push θ larger and raise the loss.
SE8. "Adam removes the need for gradients."
Error: Momentum and Adam still uses ∇L every step; it only reweights those gradients with running averages. No gradient, no descent.
WY1. Why is the negative gradient the steepest-descent direction, out of all possible unit directions?
Because (part b) the loss change is ∥∇L∥∥Δθ∥cosα with α the angle to the gradient, and among fixed-length steps this is most negative when cosα=−1, i.e. Δθ points exactly opposite ∇L.
WY2. Why do we even need a learning rate — why not step by the raw gradient?
The gradient gives direction and a steepness-scaled length, but that length is in arbitrary units of loss; a raw step can wildly overshoot. η decouples "how far" from "how steep."
WY3. Why does the first-order Taylor approximation justify gradient descent at all?
As in part (a), near θ the surface flattens to the tilted plane L(θ)+∇L⊤Δθ. On a plane the fastest-down direction is obvious — straight against the gradient — and that is what we follow.
WY4. Why can a too-large step increase the loss even though we chose the downhill direction?
Downhill is only guaranteed inside the small region where the tilted plane of part (a) still hugs the real surface. A big step leaves that region and can land on the far uphill wall of the valley.
WY5. Why does gradient descent slow to a crawl near a minimum?
Near the bottom the surface flattens, so ∥∇L∥→0, making each step −η∇L tiny. Progress decays — often geometrically, like θt=4(0.8)t in the parent's quadratic example.
WY6. Why do saddle points trap naive gradient descent even though they aren't minima?
At a saddle ∇L=0, so the update stalls with zero step even though the surface descends in some direction — the plain gradient can't feel that. See Local Minima and Saddle Points.
WY7. Why compute ∇L with backpropagation rather than by hand in neural nets?
Networks have millions of parameters; Backpropagation reuses the chain rule to get all partials in roughly one backward pass, instead of differentiating each weight separately.
WY8. Why does the stability bound η<2/λmax(A) involve the largest eigenvalue?
On the positive-definite bowl L=21θ⊤Aθ (part c) the update multiplies error by (I−ηA); the steepest-curved direction (largest eigenvalue λmax of the Hessian A) is the first to make ∣1−ηλ∣ reach 1 and diverge, so it sets the ceiling for all directions.
EC1. What happens if you start exactly at the minimum, ∇L=0?
The step is −η⋅0=0: you stay put forever, which is the correct behaviour — you've already arrived.
EC2. What if η=0?
No parameter ever moves; θt+1=θt for all t. You "descend" nowhere — a common silent bug when a schedule decays η to zero too early.
EC3. What if η<0 (a negative learning rate)?
Then −η∇L points along the gradient, so you climb uphill toward maxima — gradient ascent in disguise. It is almost never wanted for minimization; the sign of η silently flips your objective. (Complex-valued η is meaningless here since θ and L are real.)
EC4. What if two parameters have very different gradient scales (elongated valley)?
A single η overshoots the steep direction while creeping in the shallow one, causing zig-zag. Per-parameter scaling (Adam) or preconditioning (Second-Order Methods) fixes this.
EC5. What if the loss is convex but flat along a whole line (many equal minima)?
The gradient is zero along that valley floor, so descent stops at whichever point on the line it first touches — every one is a valid global minimum.
EC6. What about a non-differentiable kink in the loss (e.g. absolute-value ∣θ∣ or ReLU)?
At the kink the slope jumps, so no single derivative exists. We use a subgradient: any slope of a straight line that touches the graph at the kink and stays below it everywhere else. For L=∣θ∣ at θ=0, every value in [−1,1] is a valid subgradient (see the figure); descent picks one and proceeds, though steps near the kink can jitter.
EC7. What if the gradient magnitude is enormous (exploding gradients)?
The step −η∇L becomes huge and flings θ to nonsense values. Practitioners clip the gradient's norm before updating so direction is kept but length is capped.
EC8. What if data is so large that computing the full gradient every step is infeasible?
Use a mini-batch estimate — that's Stochastic Gradient Descent. The gradient becomes noisy but each step is cheap, trading exactness for speed.
EC9. What if L has a razor-thin global minimum next to a wide shallow local one?
Small η may settle in the wide basin and never find the thin global dip. Momentum or warm restarts (Momentum and Adam, Learning Rate Schedules) help escape the shallow trap.
The whole update rule as a loop — direction from the gradient, distance from η, repeat:
Recall One-line self-test
The update is θt+1=θt−η∇L: name what each of the three ingredients contributes.
Answer ::: θt = where you stand; −∇L = the downhill direction; η = the distance you walk. Miss any one and descent breaks.
