Noesis
1.2.4Calculus & Optimization Basics

Gradients and directional derivatives

1,650 words8 min readdifficulty · medium6 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is a partial derivative (building block)

WHY treat others as constant? Because a partial derivative measures change along one coordinate axis. Walking east tells you nothing about the north slope, so we freeze north.

WHAT is the gradient

HOW to derive the directional derivative

Goal: rate of change of ff if we move from point a\mathbf{a} in unit direction u\mathbf{u} (with ∥u∥=1\|\mathbf{u}\|=1).

Define a 1-D slice along that direction: g(t)=f(a+tu)g(t) = f(\mathbf{a} + t\mathbf{u}) The directional derivative is just g′(0)g'(0) — the ordinary 1-D derivative of that slice.

Apply the chain rule. Let x(t)=a+tu\mathbf{x}(t) = \mathbf{a}+t\mathbf{u}, so xi(t)=ai+tuix_i(t)=a_i+t u_i and dxidt=ui\dfrac{dx_i}{dt}=u_i: g′(t)=∑i=1n∂f∂xidxidt=∑i=1n∂f∂xi uig'(t) = \sum_{i=1}^n \frac{\partial f}{\partial x_i}\frac{dx_i}{dt} = \sum_{i=1}^n \frac{\partial f}{\partial x_i}\,u_i

That sum is exactly a dot product:

WHY the gradient points steepest uphill

Write the dot product using the geometric form: Duf=∇f⋅u=∥∇f∥ ∥u∥cos⁡θ=∥∇f∥cos⁡θD_{\mathbf{u}}f = \nabla f \cdot \mathbf{u} = \|\nabla f\|\,\|\mathbf{u}\|\cos\theta = \|\nabla f\|\cos\theta where θ\theta is the angle between u\mathbf{u} and ∇f\nabla f (and ∥u∥=1\|\mathbf{u}\|=1).

  • This is largest when cos⁡θ=1\cos\theta=1, i.e. θ=0\theta=0 ⇒ move along ∇f\nabla f. → steepest ascent, value =∥∇f∥=\|\nabla f\|.
  • Smallest (most negative) when θ=π\theta=\pi ⇒ move opposite ∇f\nabla f. → steepest descent, this is why gradient descent uses −∇f-\nabla f.
  • Zero when θ=π/2\theta=\pi/2 ⇒ move perpendicular to ∇f\nabla f. → you're walking along a level set / contour, height unchanged. So the gradient is perpendicular to contour lines.
Figure — Gradients and directional derivatives

Worked examples

Common mistakes (steel-manned)

Forecast-then-verify

Flashcards

What is the directional derivative DufD_{\mathbf u}f in dot-product form?
Duf=∇f⋅uD_{\mathbf u}f = \nabla f \cdot \mathbf u with ∥u∥=1\|\mathbf u\|=1.
Why must u\mathbf u be a unit vector?
So the result measures slope per unit distance, not scaled by the vector's length.
In which direction does ∇f\nabla f point?
Direction of steepest ascent; its magnitude is the steepest slope.
What is the relationship between ∇f\nabla f and contour lines?
It is perpendicular (orthogonal) to level sets, since Duf=0D_\mathbf{u}f=0 along a contour.
Why does gradient descent use −∇f-\nabla f?
−∇f-\nabla f is the direction of steepest descent (cos⁡θ=−1\cos\theta=-1), fastest decrease of ff.
Is the gradient a scalar or a vector?
A vector with one component per input variable.
Derivation: how do we get Duf=∇f⋅uD_\mathbf{u}f=\nabla f\cdot\mathbf u?
Set g(t)=f(a+tu)g(t)=f(\mathbf a+t\mathbf u), apply chain rule; g′(0)=∑i∂f∂xiui=∇f⋅ug'(0)=\sum_i \frac{\partial f}{\partial x_i}u_i=\nabla f\cdot\mathbf u.
Max value of DufD_\mathbf{u}f over all unit u\mathbf u?
∥∇f∥\|\nabla f\|, achieved when u\mathbf u aligns with ∇f\nabla f.
Recall Feynman: explain to a 12-year-old

You're on a bumpy hill. Wherever you stand, one direction goes up the fastest — like the steepest part of a slide, but upward. The gradient is an arrow pointing exactly that steepest-up way, and how long the arrow is tells how steep it is. If you want to reach the bottom (train the model), just always walk the opposite way of the arrow. And if you walk sideways to the arrow, you stay at the same height — that's walking along a flat ring on the hill.

Connections

  • Partial derivatives — the components that build ∇f\nabla f.
  • Chain rule — used to derive the directional derivative.
  • Gradient descent — walks along −∇f-\nabla f.
  • Level sets and contours — gradient is orthogonal to them.
  • Dot product and projections — geometry behind ∇f⋅u\nabla f\cdot\mathbf u.
  • Jacobian and Hessian — higher-order generalizations.

Concept Map

slope along one axis

vector of all partials

1-D slice g of t

chain rule

is a dot product

dot product with u

geometric form

theta equals 0 max

theta equals pi min

points toward

minus nabla f

Partial derivative

Gradient nabla f

g of t equals f of a plus t u

Move in unit direction u

Sum of partials times u_i

Directional derivative D_u f

nabla f times cos theta

Steepest ascent

Steepest descent

Gradient descent in ML

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Socho tum ek pahaadi (hill) par khade ho jiski height ek function f(x,y)f(x,y) se milti hai. Har direction mein chalne par slope alag hoga — koi taraf zyada chadhai, koi taraf utraai. Directional derivative batata hai ki agar tum kisi ek unit direction u\mathbf u mein step lo to ff kitni tezi se change hoga. Yeh nikalta hai simple dot product se: Duf=∇f⋅uD_\mathbf{u}f=\nabla f\cdot\mathbf u.

Gradient ∇f\nabla f ek vector hai jismein saare partial derivatives hote hain. Iski khaas baat: yeh hamesha sabse tez chadhai (steepest ascent) ki taraf point karta hai, aur iski length hi maximum slope hoti hai. Isiliye machine learning mein loss kam karne ke liye hum ulti direction yaani −∇f-\nabla f mein chalte hain — isko gradient descent kehte hain.

Ek important intuition: gradient hamesha contour lines (same height wali lines) ke perpendicular hota hai. Kyunki contour ke along chalne se height change nahi hoti, matlab Duf=0D_\mathbf u f=0, aur dot product zero tabhi hota hai jab do vector 90° par ho. Isliye jab bhi confuse ho ki gradient contour ke saath jaata hai ya across — yaad rakho: across, perpendicular, uphill.

Ek chhoti galti se bacho: dot product lene se pehle direction vector ko normalize zaroor karo (length 1 banao), warna slope galat, bada aa jaayega. Bas itna dhyaan rakho aur tum har optimization problem confidently attack kar sakte ho.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,496 wordsD2Visual walkthroughThe derivation in pictures · 2,509 wordsD3Worked examplesEvery scenario covered · 2,437 wordsD4ExercisesGraded, full solutions · 2,115 wordsD5Question bankConcept traps, quick reveals · 1,498 words

Test yourself — Calculus & Optimization Basics

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Partial derivativesAI-ML · 1.2.3Chain rule — proof, composite function derivativesMaths · 4.1.16Gradient descent and variants — convergence analysisMaths · 4.10.20Dot product — formula, cosine formula, Cauchy-Schwarz inequality proofMaths · 4.5.2Functions, limits, and continuityAI-ML · 1.2.1Taylor series approximationAI-ML · 1.2.7Functions of several variables — graphs, level curves, level surfacesMaths · 4.4.1Tangent planes and linear approximations to surfacesMaths · 4.4.5Differentiability in multiple variablesMaths · 4.4.6Critical points — finding, classifyingMaths · 4.4.12