Gradient descent intuition and update rule

Core Idea

Why this works: The gradient $\nabla L(\theta)$ points in the direction of steepest increase. So $-\nabla L(\theta)$ points toward stepest decrease. By iteratively moving opposite to the gradient, we "roll downhill" toward a minimum.

The Mathematics: Deriving the Update Rule

Step 1: First-Order Taylor Approximation

Start with why we can use gradients at all. Around a point $\theta$ , the loss function can be approximated:

L(\theta + \Delta\theta) \approx L(\theta) + \nabla L(\^T \Delta\theta

Why this step? Taylor's theorem says any smooth function is locally linear. The gradient $\nabla L(\theta)$ tells us how $L$ changes for small $\Delta\theta$ .

Step 2: Choose the Best Direction

We want to find $\Delta\theta$ that decreases $L$ the most. The change in loss is:

L(\theta + \Delta\theta) - L(\theta) \approx \nabla L(\theta)^T \Delta\theta

For this to be negative (loss decreases), we need $\nabla L(\theta)^T \Delta\theta < 0$ .

Key insight: The dot product $\nabla L(\theta)^T \Delta\theta = |\nabla L| \cdot |\Delta\theta| \cos(\alpha)$ , where $\alpha$ is the angle between the gradient and our step direction.

This is most negative when $\cos(\alpha) = -1$ , meaning $\alpha = 180°$ — we move exactly opposite to the gradient!

Step 3: Control Step Size

If we set $\Delta\theta = -\nabla L(\theta)$ , we might overshoot. Introduce a learning rate $\eta > 0$ to scale the step:

\Delta\theta = -\eta \nabla L(\theta)

Why this step? The gradient gives direction, but not the optimal magnitude. $\eta$ lets us tune how agressively we descend.

Figure — Gradient descent intuition and update rule

Worked Examples

Worked example Example 1: Minimizing a Quadratic Function

Problem: Minimize $L(\theta) = \theta^2$ starting from $\theta_0 = 4$ with $\eta = 0.1$ .

Step-by-step:

Compute gradient: $\nabla L(\theta) = \frac{d}{d\theta}(\theta^2) = 2\theta$

Why? Power rule of differentiation.

Iteration 0: $\theta_0 = 4$

Gradient: $\nabla L(4) = 2(4) = 8$
Update: $\theta_1 = 4 - 0.1(8) = 4 - 0.8 = 3.2$
Why this step? We move opposite to gradient (which is positive8), so we go left.

Iteration 1: $\theta_1 = 3.2$

Gradient: $\nabla L(3.2) = 6.4$
Update: $\theta_2 = 3.2 - 0.1(6.4) = 2.56$

Iteration 2: $\theta_2 = 2.56$

Gradient: $\nabla L(2.56) = 5.12$
Update: $\theta_3 = 2.56 - 0.512 = 2.048$

Pattern: $\theta_t = 4 \cdot (0.8)^t$ — exponentially approaching $\theta^* = 0$ .

Worked example Example 2: Linear Regression (2D Parameter Space)

Problem: Fit $y = mx + b$ to data $(x_1, y_1), (x_2, y_2)$ using squared error loss:

L(m, b) = \frac{1}{2}\sum_{i=1}^{2} (y_i - (mx_i + b))^2

Given data: $(1, 3), (2, 5)$ . Start with $m_0=0, b_0=0$ , $\eta=0.01$ .

Compute gradients:

\frac{\partial L}{\partial m} = \sum_{i} (mx_i + b - y_i) \cdot x_i

\frac{\partial L}{\partial b} = \sum_{i} (mx_i + b - y_i)

Why these formulas? Chain rule: $\frac{\partial}{\partial m}[\frac{1}{2}(y_i - \hat{y}_i)^2] = -(y_i - \hat{y}_i) \cdot \frac{\partial \hat{y}_i}{\partial m} = -(y_i - \hat{y}_i) \cdot x_i$ .

Iteration 0: $m=0, b=0$

Predictions: $\hat{y}_1 = 0, \hat{y}_2 = 0$
Errors: $e_1 = 0-3=-3, e_2=0-5=-5$
$\frac{\partial L}{\partial m} = (-3)(1) + (-5)(2) = -3 - 10 = -13$
$\frac{\partial L}{\partial b} = -3 + (-5) = -8$
Updates:
$m_1 = 0 - 0.01(-13) = 013$
$b_1 = 0 - 0.01(-8) = 0.08$ Why this step? Gradient is negative, so we move in positive direction — both parameters increase to reduce error.

Common Mistakes

Common mistake Mistake 3: Forgetting to Update ALL Parameters

Wrong (in multivariable case):

theta[0] = theta[0] - eta * grad[0]
theta[1] = theta[1] - eta * grad[0]  # Copied wrong gradient!

Why it happens: Copy-paste errors, especially in high-dimensional $\theta$ .

Fix: Use vectorized updates: theta = theta - eta * grad where both are NumPy arrays.

Deep Dive: Why This Update Rule is Optimal (Locally)

Question: Is the gradient direction provably the best local direction?

Answer: Yes, for infinitesimal steps! Here's the rigorous argument:

We want to solve:

\min_{\|\Delta\theta\| = \epsilon} L(\theta + \Delta\theta)

Using first-order Taylor:

\min_{\|\Delta\theta\| = \epsilon} \nabla L(\theta)^T \Delta\theta

This is minimized when $\Delta\theta$ is antiparallel to $\nabla L(\theta)$ :

\Delta\theta^* = -\epsilon \frac{\nabla L(\theta)}{\|\nabla L(\theta)\|}

Scaling by $\eta$ (which absorbs $\epsilon / \|\nabla L\|$ ), we recover:

\Delta\theta = -\eta \nabla L(\theta)

Conclusion: Gradient descent is the stepest descent method in the local neighborhood.

Learning Rate: The Critical Hyperparameter

Rule of thumb: For quadratic losses $L(\theta) = \frac{1}{2}\theta^T A \theta$ , stability requires:

\eta < \frac{2}{\lambda_{\max}(A)}

where $\lambda_{\max}$ is the largest eigenvalue of $A$ .

Why? The update becomes $\theta_{t+1} = (I - \eta A)\theta_t$ . This converges only if eigenvalues of $(I - \eta A)$ are in $(-1, 1)$ .

Connections to Other Concepts

Loss Functions — Gradient descent minimizes any differentiable loss
Backpropagation — Efficiently computes $\nabla L$ in neural networks
Learning Rate Schedules — Adaptive strategies for $\eta$ (decay, warm-up)
Stochastic Gradient Descent — Uses mini-batches instead of full dataset
Momentum and Adam — Enhanced GD with velocity/adaptive rates
Convex Optimization — GD guarantees global minimum for convex $L$
Local Minima and Saddle Points — Challenges in non-convex landscapes
Second-Order Methods — Newton's method uses Hessian, not just gradient

Recall Explain to a 12-Year-Old

Imagine you're playing a video game where you're stuck in a maze on a hillside, and you need to find the treasure at the bottom. But it's super fogy — you can only see your feet!

Here's your strategy:

Feel the ground. Which way tilts downward? That's your hint.
Take a small step downhill.
Stop and feel again. Repeat.

Gradient descent is exactly this! Your "feeling the ground" is calculating the gradient (the slope). The "step downhill" is updating your position using the rule:

\text{new position} = \text{old position} - \text{(step size)} \times \text{(slope)}

The "step size" is called the learning rate. Too big, and you'll trip and overshoot the treasure. Too small, and you'll take forever to get there.

In machine learning, the "treasure" is the best model parameters that make your predictions super accurate!

Flashcards

What is the gradient descent update rule? :: $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$ , where $\eta$ is the learning rate and $\nabla L(\theta_t)$ is the gradient at $\theta_t$ .

Why do we use the NEGATIVE gradient in gradient descent?

The gradient

\nabla L

points in the direction of steepest INCREASE of the loss. We want to DECREASE loss, so we move in the opposite direction:

-\nabla L

What happens if the learning rate $\eta$ is too large? :: The updates overshoot the minimum, causing the loss to oscillate or diverge instead of converging. The linear approximation (Taylor expansion) breaks down.

What happens if the learning rate $\eta$ is too small? :: Convergence is extremely slow — the algorithm takes tiny steps and requires many iterations to reach the minimum.

In the hill analogy, what does the gradient represent?

The gradient represents the direction and stepness of the slope. It points UPHILL (stepest ascent), so we go the opposite way to descend.

Derive the gradient descent direction from first principles.

Using Taylor expansion:

L(\theta + \Delta\theta) \approx L(\theta) + \nabla L(\theta)^T \Delta\theta

. To minimize the change, we need

\nabla L(\theta)^T \Delta\theta < 0

. This is most negative when

\Delta\theta

is antiparallel to

\nabla L

, giving

\Delta\theta = -\eta \nabla L(\theta)

What is the "stepest descent" property of gradient descent?

Among all directions with the same step magnitude, the negative gradient direction produces the largest decrease in the loss function (locally, under first-order approximation).

In Example 1 (

L(\theta) = \theta^2

), why does

\theta_t = 4 \cdot (0.8)^t

Each update multiplies the current value by

(1 - 2\eta) = 1 - 0.2 = 0.8

(since

\nabla L = 2\theta

and

\eta=0.1

), forming a geometric sequence converging to 0.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Gradient descent ek simple lekin powerful optimization technique hai jo machine learning ka backbone hai. Socho ki tumhe ek loss function minimize karni hai — basically, tumhare model ki errors ko kam karna hai. Loss function ko imagine karoek pahad ki tarah, jahan sabse niche ki point (valley) tumhara target hai — wahi pe model sabse accurate hota hai.

Gradient descent ka logic simple hai: agar tum blindfolded ho ek pahad pe, toh tum kaise niche jaoge? Tum feel karoge ki konsi direction mein slope sabse steep downward hai, aur wahan ek chhota step loge. Phir dobara feel karoge, aur phir se step. Repeat karte raho jab tak tum valley mein nahi pahunch jate. Mathematically, yeh gradient ka negative direction hota hai kyunki gradient uphill direction point karta hai — hume opposite jana hai. Update rule simple hai: theta_new = theta_old - (learning_rate × gradient). Yahan learning rate (eta) controls karo ki kitna bada step lena hai — bahut bada toh tum overshoot karoge aur oscillate karoge, bahut chhota toh bahut time lagega converge hone mein.

Yeh technique isliye powerful hai kyunki yeh differentiable functions ke liye kaam karta hai, aur backpropagation ke through neural networks mein efficiently compute ho sakta hai. Lekin dhyan rakho: agar loss surface non-convex hai (multiple valleys), toh tum local minimumein fas sakte ho instead of global minimum. Real-world mein hum variants use karte hain jaise Stochastic Gradient Descent (mini-batches pe), Momentum (inertia add karta hai), aur Adam (adaptive learning rates). Core concept yahi hai: gradient tumhe direction bata hai, aur tum systematically parameters update karte raho until loss minimize ho jaye.

Practical terms mein, jab tum koi neural network train karte ho — images classify karna ho, language generate karna ho, ya game khelna ho — gradient descent hi hai jo weights ko iteratively adjust karta hai taki predictions better hoti jayein. Starting point random hota hai, phir lakhs of iterations mein model "seekhta" hai by following the gradient. Yeh optimization ka workhorse hai AI mein!