Momentum and Nesterov momentum

3.2.3 · AI-ML › Training Deep Networks

KIYA hai wo problem jo hum fix kar rahe hain?

Plain (vanilla) gradient descent update: ${\theta }_{t+1}={\theta }_t-\eta {\nabla }_{\theta }L({\theta }_t)$

Step sirf current gradient par depend karta hai. Agar gradients ek valley ke across sign flip karte rehte hain, toh progress cancel ho jaati hai. Hum chahte hain ki past ko yaad rakha jaaye taaki yeh smooth ho sake.

Momentum — pehle principles se derive karna

Step 1 — Ek velocity $v$ define karo past gradients ke exponentially weighted average ke roop mein. $\beta \in [0,1)$ ko momentum coefficient hone do (kitna past hum rakhte hain): $v_t=\beta v_{t-1}+{\nabla }_{\theta }L({\theta }_t)$

Yeh step kyun? Unrolling se milta hai $v_t={\sum }_{k=0}^t{\beta }^k\nabla L({\theta }_{t-k})$ — ek weighted sum jahan recent gradients sabse zyada count karte hain aur purane geometrically decay karte hain. Woh ek smoothed gradient hai.

Step 2 — Raw gradient ki jagah velocity ke saath step lo: ${\theta }_{t+1}={\theta }_t-\eta v_t$

Nesterov momentum — ek smarter look-ahead

Step 1 — Look-ahead point compute karo (jahan sirf momentum humein le jaata): ${\tilde{\theta }}_t={\theta }_t-\eta \beta v_{t-1}$

Yeh step kyun? Yeh anticipated position hai. Gradient ko yahan evaluate karne se zyada up-to-date curvature information milti hai.

Step 2 — Look-ahead gradient use karke velocity update karo: $v_t=\beta v_{t-1}+{\nabla }_{\theta }L({\tilde{\theta }}_t)$

Step 3 — Step lo: ${\theta }_{t+1}={\theta }_t-\eta v_t$

Worked examples

Common mistakes (steel-manned)

80/20 — essentials

Flashcards

Connections

• Gradient Descent — woh baseline jise momentum improve karta hai.
• Stochastic Gradient Descent (SGD) — momentum usually mini-batch SGD par apply hota hai.
• Adam Optimizer — momentum (1st moment) ko adaptive scaling (2nd moment) ke saath combine karta hai.
• RMSProp — adaptive learning rates; aksar momentum se compare kiya jaata hai.
• Hessian and Condition Number — kyun ill-conditioning momentum ko zaroori banata hai.
• Exponentially Weighted Moving Average — woh math jo velocity ke underlying hai.
• Learning Rate Schedules — stability ke liye $\beta$ ke saath interact karta hai.

Concept Map