Gradient descent intuition and update rule — Hinglish notes

1.2.12 · AI-ML › Calculus & Optimization Basics

Core Idea

Yeh kyun kaam karta hai: Gradient $\nabla L (θ)$ steepest increase ki direction mein point karta hai. Isliye $- \nabla L (θ)$ steepest decrease ki taraf point karta hai. Gradient ke opposite direction mein baar baar move karke, hum minimum ki taraf "neeche roll" karte hain.

The Mathematics: Update Rule Derive Karna

Step 1: First-Order Taylor Approximation

Shuru karo is baat se ki hum gradients use kyun kar sakte hain. Ek point $θ$ ke aas paas, loss function ko approximate kiya ja sakta hai:

L (θ + Δ θ) \approx L (θ) + \nabla L (\hat{T} Δ θ

Yeh step kyun? Taylor's theorem kehta hai ki koi bhi smooth function locally linear hoti hai. Gradient $\nabla L (θ)$ humein batata hai ki chote $Δ θ$ ke liye $L$ kaise change hoti hai.

Step 2: Sabse Acha Direction Chuno

Hum woh $Δ θ$ dhundhna chahte hain jo $L$ ko sabse zyada decrease kare. Loss mein change hai:

L (θ + Δ θ) - L (θ) \approx \nabla L (θ)^{T} Δ θ

Iske liye negative hona zaroori hai (loss decrease ho), humein chahiye $\nabla L (θ)^{T} Δ θ < 0$ .

Key insight: Dot product $\nabla L (θ)^{T} Δ θ = ∣\nabla L ∣ \cdot ∣Δ θ ∣ cos (α)$ , jahan $α$ gradient aur humare step direction ke beech ka angle hai.

Yeh sabse zyada negative tab hota hai jab $cos (α) = - 1$ , matlab $α = 180°$ — hum gradient ke bilkul opposite move karte hain!

Step 3: Step Size Control Karo

Agar hum $Δ θ = - \nabla L (θ)$ set karein, toh hum overshoot kar sakte hain. Step ko scale karne ke liye learning rate $η > 0$ introduce karo:

Δ θ = - η \nabla L (θ)

Yeh step kyun? Gradient direction deta hai, lekin optimal magnitude nahi. $η$ humein yeh tune karne deta hai ki hum kitni aggressively descend karein.

Figure — Gradient descent intuition and update rule

Worked Examples

Worked example Example 2: Linear Regression (2D Parameter Space)

Problem: Data $(x_{1}, y_{1}), (x_{2}, y_{2})$ par squared error loss use karke $y = m x + b$ fit karo:

L (m, b) = \frac{1}{2} i = 1 \sum 2 (y_{i} - (m x_{i} + b))^{2}

Diya gaya data: $(1, 3), (2, 5)$ . $m_{0} = 0, b_{0} = 0$ , $η = 0.01$ se shuru karo.

Gradients compute karo:

\frac{\partial L}{\partial m} = i \sum (m x_{i} + b - y_{i}) \cdot x_{i}

\frac{\partial L}{\partial b} = i \sum (m x_{i} + b - y_{i})

Yeh formulas kyun? Chain rule: $\frac{\partial}{\partial m} [\frac{1}{2} (y_{i} - \overset{y}{^}_{i})^{2}] = - (y_{i} - \overset{y}{^}_{i}) \cdot \frac{\partial y ^ _{i}}{\partial m} = - (y_{i} - \overset{y}{^}_{i}) \cdot x_{i}$ .

Iteration 0: $m = 0, b = 0$

Predictions: $\overset{y}{^}_{1} = 0, \overset{y}{^}_{2} = 0$
Errors: $e_{1} = 0 - 3 = - 3, e_{2} = 0 - 5 = - 5$
$\frac{\partial L}{\partial m} = (- 3) (1) + (- 5) (2) = - 3 - 10 = - 13$
$\frac{\partial L}{\partial b} = - 3 + (- 5) = - 8$
Updates:
$m_{1} = 0 - 0.01 (- 13) = 013$
$b_{1} = 0 - 0.01 (- 8) = 0.08$ Yeh step kyun? Gradient negative hai, isliye hum positive direction mein move karte hain — dono parameters error reduce karne ke liye increase hote hain.

Common Mistakes

Common mistake Mistake 3: SAARE Parameters Update Karna Bhool Jaana

Galat (multivariable case mein):

theta[0] = theta[0] - eta * grad[0]
theta[1] = theta[1] - eta * grad[0]  # Copied wrong gradient!

Kyun hota hai: Copy-paste errors, khaaskar high-dimensional $θ$ mein.

Fix: Vectorized updates use karo: theta = theta - eta * grad jahaan dono NumPy arrays hain.

Deep Dive: Yeh Update Rule Locally Optimal Kyun Hai

Sawaal: Kya gradient direction provably sabse best local direction hai?

Jawaab: Haan, infinitesimal steps ke liye! Yeh raha rigorous argument:

Hum solve karna chahte hain:

∥Δ θ ∥ = ϵ min L (θ + Δ θ)

First-order Taylor use karke:

∥Δ θ ∥ = ϵ min \nabla L (θ)^{T} Δ θ

Yeh tab minimize hota hai jab $Δ θ$ , $\nabla L (θ)$ ke antiparallel ho:

Δ θ^{*} = - ϵ \frac{\nabla L ( θ )}{∥\nabla L ( θ ) ∥}

$η$ se scale karke (jo $ϵ /∥\nabla L ∥$ absorb karta hai), hum recover karte hain:

Δ θ = - η \nabla L (θ)

Conclusion: Gradient descent local neighborhood mein steepest descent method hai.

Learning Rate: The Critical Hyperparameter

Rule of thumb: Quadratic losses $L (θ) = \frac{1}{2} θ^{T} A θ$ ke liye, stability ke liye zaroori hai:

η < \frac{2}{λ _{m a x} ( A )}

jahaan $λ_{m a x}$ , $A$ ki largest eigenvalue hai.

Kyun? Update ban jaata hai $θ_{t + 1} = (I - η A) θ_{t}$ . Yeh tabhi converge hoga jab $(I - η A)$ ki eigenvalues $(- 1, 1)$ mein hon.

Doosre Concepts se Connections

Loss Functions — Gradient descent koi bhi differentiable loss minimize karta hai
Backpropagation — Neural networks mein $\nabla L$ efficiently compute karta hai
Learning Rate Schedules — $η$ ke liye adaptive strategies (decay, warm-up)
Stochastic Gradient Descent — Full dataset ki jagah mini-batches use karta hai
Momentum and Adam — Velocity/adaptive rates ke saath enhanced GD
Convex Optimization — Convex $L$ ke liye GD global minimum guarantee karta hai
Local Minima and Saddle Points — Non-convex landscapes mein challenges
Second-Order Methods — Newton's method sirf gradient nahi, Hessian bhi use karta hai

Recall Ek 12-Saal ke Bacche ko Samjhao

Socho tum ek video game khel rahe ho jahan tum ek pahaad par maze mein faase ho, aur tumhe neeche khazaana dhundhna hai. Lekin bahut zyaada fog hai — tum sirf apne paon dekh sakte ho!

Tumhari strategy yeh hai:

Zameen feel karo. Kaun sa taraf neeche jhukta hai? Yahi tumhara hint hai.
Neeche ki taraf ek chota sa step lo.
Ruko aur phir feel karo. Repeat karo.

Gradient descent exactly yehi hai! Tumhara "zameen feel karna" gradient (dhaalan) calculate karna hai. "Neeche step lena" tumhari position ko is rule se update karna hai:

new position = old position - (step size) \times (slope)

"Step size" ko learning rate kehte hain. Bahut bada ho toh tum trip karke khazaane se aage nikal jaoge. Bahut chota ho toh tumhe wahan pahunchne mein zamaana lag jaega.

Machine learning mein, "khazaana" woh best model parameters hain jo tumhari predictions ko super accurate banate hain!

Flashcards

What is the gradient descent update rule? :: $θ_{t + 1} = θ_{t} - η \nabla L (θ_{t})$ , where $η$ is the learning rate and $\nabla L (θ_{t})$ is the gradient at $θ_{t}$ .

Why do we use the NEGATIVE gradient in gradient descent?

The gradient

\nabla L

points in the direction of steepest INCREASE of the loss. We want to DECREASE loss, so we move in the opposite direction:

- \nabla L

What happens if the learning rate $η$ is too large? :: The updates overshoot the minimum, causing the loss to oscillate or diverge instead of converging. The linear approximation (Taylor expansion) breaks down.

What happens if the learning rate $η$ is too small? :: Convergence is extremely slow — the algorithm takes tiny steps and requires many iterations to reach the minimum.

In the hill analogy, what does the gradient represent?

The gradient represents the direction and stepness of the slope. It points UPHILL (stepest ascent), so we go the opposite way to descend.

Derive the gradient descent direction from first principles.

Using Taylor expansion:

L (θ + Δ θ) \approx L (θ) + \nabla L (θ)^{T} Δ θ

. To minimize the change, we need

\nabla L (θ)^{T} Δ θ < 0

. This is most negative when

Δ θ

is antiparallel to

\nabla L

, giving

Δ θ = - η \nabla L (θ)

What is the "stepest descent" property of gradient descent?

Among all directions with the same step magnitude, the negative gradient direction produces the largest decrease in the loss function (locally, under first-order approximation).

In Example 1 (

L (θ) = θ^{2}

), why does

θ_{t} = 4 \cdot (0.8)^{t}

Each update multiplies the current value by

(1 - 2 η) = 1 - 0.2 = 0.8

(since

\nabla L = 2 θ

and

η = 0.1

), forming a geometric sequence converging to 0.