Gradient Descent Optimization

The unifying principle

Let a state be a vector $x \in \mathbb{R}^n$ and a scalar objective $f(x)$ we want to minimize. The gradient

$\nabla f(x) = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right)$

points in the direction of steepest ascent. By first-order Taylor expansion,

$f(x + \delta) \approx f(x) + \nabla f(x)^\top \delta.$

To decrease $f$ maximally for a fixed step size $\|\delta\|$, Cauchy–Schwarz says choose $\delta \parallel -\nabla f(x)$. This gives the update

$\boxed{x_{t+1} = x_t - \eta\, \nabla f(x_t)}$

with learning rate / step size $\eta > 0$. A minimum is a fixed point where $\nabla f(x^\star) = 0$ and the Hessian $\nabla^2 f(x^\star) \succeq 0$ (positive semidefinite). Convergence is guaranteed for convex $f$ with $L$-Lipschitz gradient when $\eta \le 1/L$. That's the whole engine—everything below is the same equation with $f$ renamed.

How it shows up in each field

Maths — the pure optimization problem

Here $f$ is any differentiable objective and descent is a numerical root-finder for $\nabla f = 0$. Notation is the canonical one above. Continuous-time it becomes the gradient flow ODE: $\dot{x}(t) = -\nabla f(x(t)), \qquad \frac{d}{dt}f(x(t)) = -\|\nabla f\|^2 \le 0,$ so $f$ is a Lyapunov function—it only ever decreases. Example: minimize $f(x) = \tfrac{1}{2}x^\top A x - b^\top x$ (convex, $A \succ 0$). Then $\nabla f = Ax - b$, and descent iteratively solves the linear system $Ax = b$.

AI-ML — minimizing loss over parameters

$x$ becomes model weights $\theta$, and $f$ is the empirical loss $\mathcal{L}(\theta) = \frac{1}{N}\sum_i \ell(y_i, h_\theta(x_i))$. Because $N$ is huge, we estimate the gradient on a minibatch (stochastic gradient descent): $\theta_{t+1} = \theta_t - \eta \,\nabla_\theta \mathcal{L}_{\text{batch}}(\theta_t).$ Same idea: the landscape is non-convex, so we settle into some good local basin. Backpropagation is just the chain rule computing $\nabla_\theta$. Example: logistic regression with $\ell = -[y\log p + (1-y)\log(1-p)]$, $p = \sigma(\theta^\top x)$, gives the clean update $\theta \leftarrow \theta - \eta (p - y)x$.

Physics — energy minimization / relaxation to equilibrium

$f$ is potential energy $U(x)$; the force is $F = -\nabla U(x)$. Overdamped (friction-dominated) motion is literally gradient flow: $\gamma\,\dot{x} = -\nabla U(x).$ A system relaxes to the state of least energy exactly as descent finds a minimum; stable equilibria are the minima. Adding noise gives Langevin dynamics $\dot{x} = -\nabla U + \sqrt{2T}\,\xi$, the physical analog of stochastic descent—and the thermal cousin of momentum/SGD noise. Example: a spring, $U(x) = \tfrac{1}{2}kx^2$, $-\nabla U = -kx$; the mass slides to $x^\star = 0$, the potential's minimum.

Stock-Market — portfolio optimization

$x$ becomes portfolio weights $w$ (fractions per asset). Markowitz mean–variance minimizes risk penalized against return: $f(w) = \tfrac{1}{2}\,w^\top \Sigma\, w - \lambda\, \mu^\top w, \qquad \sum_i w_i = 1,$ with covariance $\Sigma$, expected returns $\mu$, risk-aversion $\lambda$. Projected gradient descent: $w_{t+1} = \Pi_{\{\mathbf{1}^\top w = 1\}}\!\left(w_t - \eta(\Sigma w_t - \lambda\mu)\right).$ The "steepest slope" is now steepest reduction in risk-adjusted loss; equilibrium is the efficient-frontier allocation. Example: two anticorrelated assets—descent naturally diversifies, pushing weight into the pair that lowers $w^\top\Sigma w$, the quadratic risk bowl.

Why this bridge matters

• Intuition transfer (Physics → ML): thinking of loss as energy explains why momentum helps—it's inertia rolling a ball past shallow ridges: $v_{t+1} = \beta v_t - \eta\nabla f$, $x_{t+1}=x_t+v_{t+1}$. Simulated annealing and Langevin sampling come straight from thermodynamics.
• Convexity transfer (Maths → Finance): the Markowitz quadratic is convex, so math guarantees a unique global optimum—no bad local minima, unlike deep nets.
• Landscape transfer (ML → Physics): non-convex loss surfaces, saddle points, and "wide vs sharp minima" import directly into understanding glassy energy landscapes.
• Shared failure modes: vanishing gradients (flat regions), ill-conditioning (large Hessian eigenvalue ratio → zig-zagging), and step-size instability ($\eta > 2/L$ diverges) are the same pathology whether you're training a net, relaxing a crystal, or rebalancing a book.

The unifying takeaway: an objective + a gradient = a dynamical system that flows to equilibrium. Learn it once, wield it four ways.

Connections

• 03-Convex-Optimization
• 07-Gradient-Flow-and-Lyapunov-Functions
• 12-Stochastic-Gradient-Descent
• 14-Backpropagation
• 21-Potential-Energy-and-Equilibrium
• 23-Langevin-Dynamics
• 31-Markowitz-Mean-Variance
• 34-Risk-Adjusted-Returns

#bridge