Lagrange multipliers — one and two constraints

WHY does this work? (Derivation from scratch)

WHAT we want: maximize/minimize $f(\mathbf{x})$ where $\mathbf{x}$ is forced to lie on the surface $g(\mathbf{x})=0$.

Step 1 — Think about moving along the constraint. Suppose $\mathbf{r}(t)$ is any smooth curve that lies inside the constraint surface, so $g(\mathbf{r}(t))=0$ for all $t$. At a constrained extremum point $\mathbf{r}(t_0)=\mathbf{x}^*$, the value $f(\mathbf{r}(t))$ must have an ordinary extremum, so

$\frac{d}{dt}f(\mathbf{r}(t))\Big|_{t_0}=0.$

Why this step? On the constraint surface we are no longer free — we can only travel along curves staying in the surface. A constrained max is just an ordinary max of the restricted function.

Step 2 — Apply the chain rule.

$\nabla f(\mathbf{x}^*)\cdot \mathbf{r}'(t_0)=0.$

This holds for every tangent direction $\mathbf{r}'(t_0)$ of the surface. So $\nabla f$ is orthogonal to all tangent directions — i.e. $\nabla f$ is normal to the constraint surface.

Step 3 — But $\nabla g$ is also normal to the surface. Differentiate $g(\mathbf{r}(t))=0$: $\nabla g\cdot\mathbf{r}'=0$, so $\nabla g$ is normal too. Two vectors normal to the same surface (one-dim normal space) must be parallel:

Two constraints

Now the point must satisfy both $g(\mathbf{x})=0$ and $h(\mathbf{x})=0$. Geometrically the feasible set is the intersection curve of two surfaces.

WHY two multipliers? Along that intersection curve, the allowed tangent direction $\mathbf{T}$ must be perpendicular to both $\nabla g$ and $\nabla h$. At an extremum $\nabla f\cdot\mathbf{T}=0$, so $\nabla f$ has no component along $\mathbf{T}$ — meaning $\nabla f$ lies in the plane spanned by $\nabla g$ and $\nabla h$:

Worked Example 1 — One constraint (a classic)

Maximize $f(x,y)=xy$ subject to $g=x^2+y^2-1=0$ (point on unit circle).

$\nabla f=(y,x)$, $\nabla g=(2x,2y)$. Set $\nabla f=\lambda\nabla g$:

• $y=2\lambda x$ (why: first component)
• $x=2\lambda y$ (why: second component)

Multiply: $xy=4\lambda^2 xy$. If $xy\ne 0$, then $4\lambda^2=1$, $\lambda=\pm\tfrac12$. From $y=2\lambda x$ with $\lambda=\tfrac12$: $y=x$. Plug into constraint: $2x^2=1$, $x=\pm\tfrac{1}{\sqrt2}$.

Max value $f=xy=\tfrac12$ at $(\tfrac1{\sqrt2},\tfrac1{\sqrt2})$. Why accept it: check $\lambda=-\tfrac12$ gives $y=-x$, $f=-\tfrac12$ (the min).

Worked Example 2 — Distance / shadow-price meaning

Closest point on line $x+y=4$ to origin. Minimize $f=x^2+y^2$, $g=x+y-4=0$.

$\nabla f=(2x,2y)=\lambda(1,1)$ ⟹ $2x=2y=\lambda$ ⟹ $x=y$. Constraint: $2x=4$, $x=y=2$. $f^*=8$. Check $\lambda$ meaning: with $g=x+y-c$, $f^*=c^2/2$, so $df^*/dc=c=4$, and indeed $\lambda=2x=4$. ✔ The multiplier equals the sensitivity.

Worked Example 3 — Two constraints

Maximize $f=x+y+z$ on the circle that is the intersection of plane $g=x+y+z-1=0$... wait, that's degenerate. Use: Minimize $f=z$ subject to $g=x^2+y^2-z=0$ (paraboloid) and $h=x+y+z-3=0$ (plane).

$\nabla f=(0,0,1)$, $\nabla g=(2x,2y,-1)$, $\nabla h=(1,1,1)$. $\nabla f=\lambda\nabla g+\mu\nabla h$:

• $0=2\lambda x+\mu$
• $0=2\lambda y+\mu$
• $1=-\lambda+\mu$

From first two: $2\lambda x=2\lambda y$ ⟹ $x=y$ (if $\lambda\ne0$). Constraints: $z=2x^2$ and $2x+z=3$ ⟹ $2x+2x^2=3$ ⟹ $2x^2+2x-3=0$ ⟹ $x=\frac{-1\pm\sqrt7}{2}$. Why two roots: the curve dips down then up; one is min, one max of $z$ on the curve.

Flashcards

Connections

• Gradient and directional derivative — why $\nabla f$ gives steepest ascent.
• Tangent planes and normal vectors — $\nabla g$ as surface normal.
• Unconstrained optimization — critical points — the $\lambda=0$ / no-constraint limit.
• KKT conditions — inequality-constraint generalization.
• Level sets and contours — tangency of level curves at the optimum.
• Dual problem and shadow prices — economic meaning of $\lambda$.

Concept Map

Hinglish (regional understanding)