Lagrange multipliers for constrained optimization: Foundations

Right now the symbols $\nabla f$ and "the curve you're stuck on" are just words. Figure s00 below is the whole page in one picture: a red arrow ( $\nabla f$ , where you'd walk to improve) meeting a black fence-curve (the constraint) head-on. Keep this image in mind — every section below earns one piece of it, and §6–§7 return to exactly this picture.

Figure — Lagrange multipliers for constrained optimization

This page builds every symbol the parent note uses, starting from a dot on paper.

0. A point and its coordinates: $x$ , $(x, y)$

Why the topic needs it: everything we optimize — profit, area, distance — is a number that depends on where you stand. No point, nothing to compute.

The plain $x$ (thin) is one coordinate, a single number.
The bold $x$ is the whole point / vector, all coordinates together.

Figure s02 below shows a single point $(3, 2)$ (in red) and the two dashed "walk right then walk up" moves that its coordinates encode — this is the picture behind the bold symbol $x$ .

1. A function of two variables: $f (x, y)$ and $f (x)$

The parent writes $f (x)$ — same thing, using the bold "whole point" name. In the examples:

$f (x, y) = x^{2} + y^{2}$ → a bowl, deepest at the origin (this is distance-to-origin, squared).
$f (x, y) = x y$ → area of a rectangle with sides $x$ and $y$ .

Why the topic needs it: $f$ is the thing we want to make as big or as small as possible — the "objective".

Figure s03 below draws $f = x^{2} + y^{2}$ as seen from directly above: each black ring is one height-level of the bowl, and the red dot marks its deepest point $f = 0$ . Read it as a mountain map of the objective.

2. Contour lines (level curves)

Why the topic needs it: the whole geometric argument ("gradients must align") is easiest to see on contours. Tight-packed contours = steep hill; spread-out = flat. The rings you already saw in Figure s03 are exactly these contours.

3. The dot product $a \cdot b$

Before the gradient can be used, we need the tool that measures agreement between two arrows.

Why this particular recipe? Because that number is also $∣ a ∣ ∣ b ∣ cos θ$ , where $θ$ is the angle between the arrows. So the dot product secretly measures how aligned the arrows are:

same direction ( $θ = 0$ , $cos θ = 1$ ) → as positive as possible,
opposite ( $θ = 18 0^{\circ}$ , $cos θ = - 1$ ) → as negative as possible,
perpendicular ( $θ = 9 0^{\circ}$ ) → exactly zero (because $cos 9 0^{\circ} = 0$ ).

Two facts we lean on hard:

$a \cdot b = 0$ means "the arrows meet at a right angle."
For a fixed-length step $b$ , the dot product $a \cdot b = ∣ a ∣∣ b ∣ cos θ$ is largest when $θ = 0$ — i.e. when $b$ points the same way as $a$ . We use this in §4 to pin down what "steepest" means.

4. The gradient: $\nabla f$

Here a real mathematical tool enters. Why the gradient and not something simpler? Because we need a single object that answers: "From where I stand, which way is uphill, and how steep?" A plain derivative gives steepness along one direction; the gradient packages all directions into one arrow.

Notice we defined $\nabla f$ only by its parts — we have not yet earned the claim that it points uphill. Let's earn it.

Figure s04 below makes this concrete: on one contour ring it draws the red gradient arrow (steepest uphill) and the dashed black tangent direction (along the ring). Notice they meet at $9 0^{\circ}$ — that right angle is the equation $\nabla f \cdot d x = 0$ .

Why the topic needs it: "the gradient of the objective must point the same way as the gradient of the constraint" is the punchline. Without $\nabla$ there is no punchline. This same arrow is what Gradient Descent walks down.

5. The constraint: $g (x, y) = c$

Why the topic needs it: "constrained" means trapped on this curve. Off the curve, points are illegal ("infeasible"). $c$ is just the fixed number that names which curve.

$g (x)$ ::: the constraint function (another hill).
$c$ ::: the constant that fixes which level curve of $g$ is the fence.

6. The tangent step $d x$ and why $\nabla g \cdot d x = 0$

Apply the differential identity from §4 to the constraint function $g$ : a legal step keeps $g$ unchanged (still equal to $c$ ), so $d g = 0$ . Therefore $d g = \nabla g \cdot d x = 0.$ By §3 a zero dot product means perpendicular — so a legal step $d x$ is exactly one perpendicular to $\nabla g$ , i.e. running along the fence. (This needs the fence to have a direction, which is the §5 nondegeneracy condition $\nabla g \neq = 0$ .)

Now the punchline. At a constrained optimum, no legal step can improve $f$ , so $df = \nabla f \cdot d x = 0$ for every legal $d x$ .

Intuition WHY "perpendicular to every legal step" forces

$\nabla f = λ \nabla g$ In the plane, the legal steps all lie along one single line — the fence's tangent direction (they are all the multiples of one tangent arrow, because $\nabla g \neq = 0$ pins that line down). Picture that tangent line drawn on paper. We just learned $\nabla f$ is perpendicular to every arrow on that line. But in a 2D plane there is only one direction perpendicular to a given line — and $\nabla g$ already points exactly that way (§5: $\nabla g$ is perpendicular to the fence too). So $\nabla f$ and $\nabla g$ are forced onto the same perpendicular line. Two arrows on the same line are multiples of each other, which is precisely $\nabla f = λ \nabla g$ for some single number $λ$ . Geometrically: only the arrow-length differs, so one scalar rescales $\nabla g$ onto $\nabla f$ . This is the exact spot Figure s01 drew at the very top.

7. The multiplier $λ$ and the Lagrangian $L$

Let's check the packing by actually differentiating $L$ — the shortcut skips nothing if we do it once by hand.

Why the topic needs it: $L$ is the whole method in one line. $λ$ is also the shadow price — how much the best value shifts if you move the fence, the quantity that reappears in KT Conditions, Support Vector Machines, and Duality Theory.

$λ$ ::: scalar that rescales $\nabla g$ onto $\nabla f$ ; also the sensitivity/shadow price.
$L$ ::: objective minus $λ$ times (constraint − constant).

8. The nabla-with-subscript and $\partial / \partial λ$

$\nabla_{x} L$ means "gradient of $L$ only with respect to the point coordinates" → this gives the alignment equations (worked above).
$\frac{\partial L}{\partial λ}$ means "derivative treating only $λ$ as the variable" → this hands back the constraint $g = c$ (also worked above).

Recall Why differentiating in

$λ$ returns the constraint $L$ contains $- λ (g - c)$ . Nudging $λ$ changes $L$ at rate $- (g - c)$ . Setting that rate to zero means $g - c = 0$ , i.e. $g = c$ . So the "extra" equation is automatically the fence.

Prerequisite map

Equipment checklist

Test yourself — if any line stumps you, re-read its section above.

What does the bold $x$ mean versus the thin $x$ ? ::: Bold $x$ is the whole point/vector $(x, y)$ ; thin $x$ is one coordinate.
What picture is $f (x, y)$ ? ::: The height of a hill above the floor point $(x, y)$ .
What does "continuously differentiable ( $C^{1}$ )" buy us? ::: A hill with no corners or cliffs nearby, so the gradient exists and the identity $df = \nabla f \cdot d x$ holds.
What is a contour line of $f$ ? ::: All points where $f$ has the same value — a horizontal slice / map ring.
How do you compute a dot product $a \cdot b$ , and when is it zero? ::: $a_{1} b_{1} + a_{2} b_{2}$ ; it is zero exactly when the arrows are perpendicular.
Why does $\nabla f$ point in the steepest-uphill direction? ::: Because $df = \nabla f \cdot d x$ is maximized, over fixed-length steps, when $d x$ is parallel to $\nabla f$ ( $cos θ = 1$ ).
What are the two facts about $\nabla f$ ? ::: It points in the steepest-uphill direction and is perpendicular to the contour through that point.
What does $\frac{\partial f}{\partial x}$ measure? ::: The change in $f$ when you nudge only $x$ and freeze $y$ .
What does $g (x) = c$ represent geometrically? ::: A single level curve of $g$ — the fence you are forced to stay on.
Why must $\nabla g \cdot d x = 0$ for a legal step? ::: A legal step keeps $g = c$ so $d g = 0$ ; by the identity $d g = \nabla g \cdot d x$ , and a zero dot product means perpendicular to $\nabla g$ (along the fence).
Why does " $\nabla f ⊥$ every legal step" force $\nabla f = λ \nabla g$ ? ::: In 2D the legal steps fill one line; there is only one direction perpendicular to that line, and $\nabla g$ already points it, so $\nabla f$ must be a multiple of $\nabla g$ .
What nondegeneracy condition must hold, and why? ::: $\nabla g (x^{*}) \neq = 0$ , so the fence has a well-defined tangent direction and $\nabla f = λ \nabla g$ is meaningful.
Which two equations do you get by setting all derivatives of $L$ to zero? ::: $\nabla_{x} L = 0 \Rightarrow \nabla f - λ \nabla g = 0$ , and $\partial L / \partial λ = 0 \Rightarrow g - c = 0$ .
What is the physical meaning of $λ$ ? ::: The shadow price: how much the optimal value changes if you move the fence ( $c$ ) a little.