Learning rate effects on convergence

1.2.13 · AI-ML › Calculus & Optimization Basics

Definition & Core Concept

Derivation: η Gradient Descent Mein Kyun Aata Hai

Taylor Expansion Se (First Principles)

Goal: Loss function $\mathcal{L}(\theta )$ minimize karna.

Step 1: Current point ${\theta }_t$ ke paas $\mathcal{L}$ ko 1st-order Taylor se approximate karo: $\mathcal{L}({\theta }_t+\Delta \theta )\approx \mathcal{L}({\theta }_t)+{\nabla }_{\theta }\mathcal{L}({\theta }_t{)}^T\Delta \theta$

Yeh step kyun? Taylor series hume predict karne deti hai ki chhote moves ke liye loss kaise change hoga. Hum sirf linear term (gradient) rakhte hain kyunki hum assume karte hain ki steps chhote hain.

Step 2: $\mathcal{L}$ decrease karne ke liye, $\Delta \theta$ ko gradient ke opposite direction mein choose karo: $\Delta \theta =-\eta {\nabla }_{\theta }\mathcal{L}({\theta }_t)$

Opposite kyun? Gradient uphill point karta hai (steepest increase ki direction). Negative gradient downhill point karta hai.

η kyun? Gradient $\nabla \mathcal{L}$ ki units [loss/parameter] hoti hain. Parameter update [parameter units] paane ke liye, hum ise scalar η se multiply karte hain. η ki magnitude step size control karti hai.

Step 3: Update rule ban jaata hai: ${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }\mathcal{L}({\theta }_t)$

Teen Convergence Regimes

1. Learning Rate Bahut Chhota (η ≪ optimal)

Mathematics: Strongly convex $\mathcal{L}$ ke liye condition number $\kappa$ ke saath, chhota η → convergence rate $(1-\eta \mu {)}^t$ jahan $\mu$ strong convexity constant hai. Chhota η $(1-\eta \mu )\approx 1$ banata hai, isliye exponential decay slow hoti hai.

2. Learning Rate Bahut Bada (η ≫ optimal)

Critical threshold: $\mathcal{L}(w)=w^2$ ke liye, Hessian ki eigenvalue $\lambda =2$ hai. Stability ke liye $\eta <\frac{2}{\lambda }=1$ chahiye. Hamaara η=1.5 yeh exceed karta hai, isliye exponential divergence hoti hai.

3. Optimal Learning Rate (η "sweet spot" mein)

Theory: Strongly convex $\mathcal{L}$ ke liye smoothness $L$ ke saath, optimal η ≈ 1/L. Yeh curvature (second derivative) ko gradient magnitude ke saath balance karta hai.

Quantitative Stability Analysis

Neural networks mein, $L$ compute karna mushkil hota hai (Hessian eigenvalues par depend karta hai), isliye hum:

1. Chhote η se shuru karo (jaise 0.001)
2. Grid search karo: {0.1, 0.01, 0.001, 0.0001} try karo
3. Divergence/stagnation ke liye training loss curve monitor karo

Common Mistakes

Adaptive Learning Rates (Modern Solution)

Practical Guidelines

Tuning protocol:

1. 0.001 se shuru karo (safe default)
2. 1-2 epochs train karo, loss curve check karo:
   • Loss explode ho → η 10× decrease karo
   • Loss barely move kare → η 3-10× increase karo
3. Stable hone ke baad, 2× increments mein tune karo

Connections

• Gradient Descent - η basic GD mein step size parameter hai
• Momentum - Velocity term add karta hai; η choice ke liye sensitivity reduce karta hai
• Adam Optimizer - Per-parameter adaptive η; step sizes auto-tune karta hai
• Learning Rate Schedules - Better convergence ke liye time-varying η(t)
• Lipschitz Continuity - Smoothness constant L safe η bound karta hai
• Condition Number - High κ → stability ke liye chhota η chahiye
• Hessian Matrix - Second derivatives optimal η determine karte hain
• Backpropagation - Gradients compute karta hai jo η scale karta hai

#flashcards/ai-ml

Gradient descent mein learning rate η kya control karta hai? :: Parameter updates ki magnitude (step size). Yeh gradient vector ko scale karta hai ye determine karne ke liye ki har iteration mein parameter space mein kitna door move karna hai.

Learning rate ke saath gradient descent update rule likhо :: ${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }\mathcal{L}({\theta }_t)$

Concept Map