Learning rate effects on convergence

Definition & Core Concept

Derivation: Why η Appears in Gradient Descent

From Taylor Expansion (First Principles)

Goal: Minimize loss function $\mathcal{L}(\theta)$ .

Step 1: Approximate $\mathcal{L}$ near current point $\theta_t$ using 1st-order Taylor: $\mathcal{L}(\theta_t + \Delta\theta) \approx \mathcal{L}(\theta_t) + \nabla_\theta\mathcal{L}(\theta_t)^T \Delta\theta$

Why this step? Taylor series lets us predict loss change for small moves. We keep only the linear term (gradient) because we assume small steps.

Step 2: To decrease $\mathcal{L}$ , choose $\Delta\theta$ in direction opposite to gradient: $\Delta\theta = -\eta \nabla_\theta\mathcal{L}(\theta_t)$

Why opposite? Gradient points uphill (direction of stepest increase). Negative gradient points downhill.

Why η? The gradient $\nabla\mathcal{L}$ has units [loss/parameter]. To get a parameter update [parameter units], we multiply by a scalar η. The magnitude of η controls step size.

Step 3: Update rule becomes: $\theta_{t+1} = \theta_t - \eta \nabla_\theta\mathcal{L}(\theta_t)$

Three Convergence Regimes

Figure — Learning rate effects on convergence

1. Learning Rate Too Small (η≪ optimal)

Worked example Small η Example: Linear Regression

Setup: Minimize $\mathcal{L}(w) = (wx - y)^2$ where true $w^* = 2$ , start $w_0 = 0$ , data point $(x,y) = (1, 2)$ .

Gradient: $\nabla_w\mathcal{L} = 2(wx - y) \cdot x = 2(w - 2)$

Update with η = 0.01:

Iter 0: $w_0 = 0$ , gradient $= 2(0-2) = -4$ , update $w_1 = 0 - 0.01(-4) = 0.04$
Iter 1: $w_1 = 0.04$ , gradient $= 2(0.04-2) = -3.92$ , update $w_2 = 0.04 + 0.0392 = 0.0792$
Iter 10: $w_{10} \approx 0.33$ (still far from $w^* = 2$ ) Why this step? Each update moves only 1% of the gradient magnitude. After 10 steps, traveled ~0.33 units toward goal2 units away.

Symptom: Slow convergence. Loss decreases monotonically but glacially. Training takes thousands/millions of iterations.

Mathematics: For strongly convex $\mathcal{L}$ with condition number $\kappa$ , small η → convergence rate $(1 - \eta\mu)^t$ where $\mu$ is strong convexity constant. Small η makes $(1-\eta\mu) \approx 1$ , so exponential decay is slow.

2. Learning Rate Too Large (η ≫ optimal)

Worked example Large η Example: Quadratic Bowl

Setup: $\mathcal{L}(w) = w^2$ , true minimum $w^* = 0$ , start $w_0 = 1$ .

Gradient: $\nabla_w\mathcal{L} = 2w$

Update with η = 1.5 (too large for this problem):

Iter 0: $w_0 = 1$ , gradient $= 2$ , update $w_1 = 1 - 1.5(2) = 1 - 3 = -2$
Iter 1: $w_1 = -2$ , gradient $= -4$ , update $w_2 = -2 - 1.5(-4) = -2 + 6 = 4$
Iter 2: $w_2 = 4$ , gradient $= 8$ , update $w_3 = 4 - 1.5(8) = 4 - 12 = -8$ Why this step? Each update overshoots the minimum at 0, landing on the opposite side farther away. The magnitude doubles each iteration: $|w_t| = 2^{t}$ .

Symptom: Divergence or oscillation. Loss increases or ping-pongs. Parameters explode: $w_t \to \pm\infty$ .

Critical threshold: For $\mathcal{L}(w) = w^2$ , eigenvalue of Hessian is $\lambda = 2$ . Stability requires $\eta< \frac{2}{\lambda} = 1$ . Our η=1.5 exceds this, causing exponential divergence.

3. Optimal Learning Rate (η in "sweet spot")

Worked example Good η Example: Same Quadratic

Setup: $\mathcal{L}(w) = w^2$ , $w_0 = 1$ , now use η = 0.5.

Update:

Iter 0: $w_0 = 1$ , gradient $= 2$ , update $w_1 = 1 - 0.5(2) = 0$ Why this step? Reached minimum in one step! For this simple quadratic, η = 0.5 =1/λ is the exact optimal rate (Newton's method with constant Hessian). More generally with η = 0.3:
Iter 0: $w_0 = 1 \to w_1 = 1 - 0.3(2) = 0.4$
Iter 1: $w_1 = 0.4 \to w_2 = 0.4 - 0.3(0.8) = 0.16$
Iter 2: $w_2 = 0.16 \to w_3 = 0.16 - 0.3(0.32) = 0.064$

Symptom: Exponential convergence. Each iteration reduces distance by constant factor: $|w_t| = 0.4^t$ (geometric decay).

Theory: For strongly convex $\mathcal{L}$ with smoothness $L$ , optimal η ≈ 1/L. This balances curvature (second derivative) with gradient magnitude.

Quantitative Stability Analysis

Formula Convergence Condition (Smooth Functions)

For $L$ -smooth loss (Lipschitz gradient: $\|\nabla\mathcal{L}(x) - \nabla\mathcal{L}(y)\| \leq L\|x-y\|$ ), gradient descent converges if: $\boxed{\eta < \frac{2}{L}}$

Derivation (1D case for intuition):

Taylor: $\mathcal{L}(w - \eta g) \approx \mathcal{L}(w) - \eta g^2 + \frac{L\eta^2 g^2}{2}$ where $g = \nabla\mathcal{L}$
Loss decreases if $-\eta g^2 + \frac{L\eta^2 g^2}{2} < 0$
Simplify: $\eta(L\eta - 2) < 0$
Since $\eta > 0$ , need $L\eta < 2$ , thus $\eta < 2/L$ Practical rule: Use $\eta \approx \frac{1}{L}$ for safety margin.

For neural networks, $L$ is hard to compute (depends on Hessian eigenvalues), so we:

Start with small η (e.g., 0.001)
Grid search: try {0.1, 0.01, 0.001, 0.0001}
Monitor training loss curve for divergence/stagnation

Common Mistakes

Adaptive Learning Rates (Modern Solution)

Practical Guidelines

Scenario	Recommended η	Reasoning
Convex (linear/logistic regression)	0.01 - 0.1	Can use larger; loss surface is smooth
Small neural network (MNIST)	0.001 - 0.01	Moderate non-linearity
Deep network (ResNet)	0.001 - 0.001	Many layers amplify gradient issues
Adam optimizer	0.001 (default)	Adaptive normalization reduces sensitivity
Fine-tuning pretrained model	0.001 - 0.001	Already near good solution

Tuning protocol:

Start at 0.001 (safe default)
Train 1-2 epochs, check loss curve:
- Loss explodes → decrease η by10×
- Loss barely moves → increase η by 3-10×
Once stable, tune by2× increments

Recall Explain to a 12-Year-Old

Imagine you're blindfolded in a hilly park, trying to find the lowest point. Your friend shouts directions: "Walk 5 steps north!" That's the gradient (direction).

But how big should your steps be? That's the learning rate.

Tiny steps (like baby steps): Safe! You won't fall into a ditch. But it takes FOREVER. You might be out there all day.
Giant steps (like leaping 10 feet): Super fast! But you might jump OVER the lowest spot, land on the other side of the hill, then jump back over it, back and forth, never actually stopping at the bottom.
Medium steps: Just right. You walk steadily downhill and reach the bottom in reasonable time.

The learning rate is like choosing your step size. Too small = slow. Too big = you overshoot and bounce around. Goldilocks = fast and steady.

Connections

Gradient Descent - η is the step size parameter in basic GD
Momentum - Adds velocity term; reduces sensitivity to η choice
Adam Optimizer - Per-parameter adaptive η; auto-tunes step sizes
Learning Rate Schedules - Time-varying η(t) for better convergence
Lipschitz Continuity - Smoothness constant L bounds safe η
Condition Number - High κ → need smaller η for stability
Hessian Matrix - Second derivatives determine optimal η
Backpropagation - Computes gradients that η scales

#flashcards/ai-ml

What does learning rate η control in gradient descent? :: The magnitude (step size) of parameter updates. It scales the gradient vector to determine how far to move in parameter space each iteration.

Write the gradient descent update rule with learning rate :: $\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t)$

What happens if learning rate is too small?

Convergence is very slow. Loss decreases monotonically but takes many iterations to reach minimum (glacier-like progress).

What happens if learning rate is too large?

Overshooting and divergence. Parameters oscillate around minimum with increasing magnitude, or loss explodes to infinity.

For L-smooth functions, what is the stability condition for η?

\eta < \frac{2}{L}

where L is the Lipschitz constant of the gradient. Practical choice:

\eta \approx \frac{1}{L}

for safety.

Why does adaptive learning rate (like Adam) help?

It uses per-parameter step sizes based on gradient history. Parameters with large typical gradients get smaller effective η, normalizing updates across dimensions with different scales.

If training loss increases from iteration1, what's likely wrong?

Learning rate is too large. The first update already overshot, indicating η exceds the stability threshold for this loss landscape.

Give the optimal η for minimizing

\mathcal{L}(w) = w^2

starting from w=1 in one step

η = 0.5. Since gradient is2w and Hessian eigenvalue is 2, optimal rate is 1/λ = 1/2.

What's the relationship between learning rate and number of iterations to converge?

Non-monotonic. Very small η → many iterations. Optimal η → fewest iterations. Too large η → infinite iterations (doesn't converge). There's a sweet spot.

Why can't we always use very large η to converge faster?

Large η causes overshooting. Updates jump past the minimum to the opposite side, often with increasing magnitude, leading to oscillation or divergence rather than convergence.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho bhai, gradient descent mein learning rate (η) sabse important hyperparameter hai. Yeh control karta hai ki har iteration mein parameters ko kitna update karein. Socho aise: tum ek pahadi pe ho, aur sabse niche jana hai (loss minimize karna). Gradient tumhe direction bata hai—kidhar jana hai. Par kitne bade kadam maroge? Wahi learning rate decide karta hai.

Agar learning rate bohot chhota hai (jaise 0.001), toh tum bahut aahista chaloge. Har step mein thoda-thoda age badhoge. Loss kam to hoga, par itni slowly ki training khatam hi nahi hogi—ghanton lag jayenge! Dosri taraf, agar learning rate bohot bada hai (jaise 1.5), toh tum itne bade jumps maroge ki minimum ko cross kar jaoge, dosri side land karoge, phir wapas jump, aur aise hi oscillate karte rahoge. Kabhi converge hi nahi hoga, ulta loss badh jayega! Isliye sweet spot chahiye—na zyada chhota, na zyada bada.

Practical tip: Deep learning mein usually 0.001 se start karo (Adam optimizer ke sath). Agar loss pehle iteration mein hi explode ho raha, matlab η bohot bada hai—10x kam karo. Agar loss bilkul nahi ghat raha, matlab bohot chhota hai—3-5x badha do. Modern optimizers jaise Adam apne ap per-parameter rates adjust karte hain, isliye thoda zyada forgiving hain. Par samajh lena zaroori hai—learning rate tuning hi decide karti hai ki model converge karega ya nahi!

Definition & Core Concept

Derivation: Why η Appears in Gradient Descent

From Taylor Expansion (First Principles)

Three Convergence Regimes

1. Learning Rate Too Small (η≪ optimal)

2. Learning Rate Too Large (η ≫ optimal)

3. Optimal Learning Rate (η in "sweet spot")

Quantitative Stability Analysis

Common Mistakes

Adaptive Learning Rates (Modern Solution)

Practical Guidelines

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Calculus & Optimization Basics

Connections