Lagrange multipliers for constrained optimization: Visual walkthrough

This page unfolds the central result of Lagrange multipliers as a sequence of pictures. Read it in order — each step's figure is the argument.

Step 1 — What "optimize" even means, drawn as a landscape

WHAT. We have a function $f (x, y)$ . Think of it as the height of a hill above every point $(x, y)$ on a flat map. Feeding in a location returns an altitude.

WHY a hill? Because "maximize $f$ " then literally means "find the highest ground," and "minimize $f$ " means "find the lowest valley." Turning numbers into height lets your eyes do the optimizing.

PICTURE. In the figure, the map is the floor; the coloured curves are contour lines — each curve joins points of equal height, exactly like a topographic map. Close-together contours mean a steep slope; far-apart contours mean gentle ground.

Figure — Lagrange multipliers for constrained optimization

Symbol check, term by term in $f (x, y) = k$ :

$f (x, y)$ :: the height machine, "how high am I at this spot."
$k$ :: the one altitude we froze to draw a single curve.

Step 2 — The constraint is a fence you must walk on

WHAT. Now add $g (x, y) = c$ . This is a second landscape $g$ , but we only care about one of its contours — the one at height $c$ . That single curve is drawn on our map as a path. You are only allowed to stand on this path.

WHY. Real problems forbid free roaming: a fixed budget, a fixed perimeter, a required margin. The equation $g (x, y) = c$ is the mathematical fence. Points off the fence are infeasible — they don't exist for us.

PICTURE. The amber curve is the constraint path $g (x, y) = c$ . The greyed-out region is forbidden. Optimizing now means: walk along the amber path only, and find the highest (or lowest) contour of $f$ you touch.

Term by term in $g (x, y) = c$ :

$g (x, y)$ :: a second height machine (budget used, perimeter length, ...).
$c$ :: the fixed level we are locked onto — the fence's exact altitude.

Step 3 — The gradient: the arrow of steepest uphill

WHAT. We introduce a tool: the gradient $\nabla f$ . At any point it is an arrow that points in the direction of fastest increase of $f$ , and whose length says how fast.

WHY this tool and not another? We need to know, at a candidate point, "which way is uphill for $f$ , and which way is uphill for the fence $g$ ?" The gradient is the exact object that answers "which direction increases this function fastest." No other single object packages direction-of-increase this cleanly.

PICTURE. The cyan arrow is $\nabla f$ . Notice a crucial fact drawn in the figure: the gradient is always perpendicular to the contour line through that point. Uphill is the shortest way to the next contour, and the shortest way to a curve is straight across it.

Same story for the fence: $\nabla g$ is an arrow perpendicular to the amber path.

Step 4 — Walking the fence and asking "am I still improving?"

WHAT. Stand on the fence and take a tiny legal step $d x$ that keeps you on the amber path. How much does $f$ change? The answer is the dot product $\nabla f \cdot d x$ .

WHY the dot product? Because we need to know how much of the uphill arrow lies along the direction we actually walk. The dot product $\nabla f \cdot d x$ is precisely "how much of $\nabla f$ points the same way as $d x$ " — it is the change in $f$ for that step. That is the exact question here, so that is the tool.

PICTURE. The figure shows the legal step $d x$ (tangent to the fence, in white) and the uphill arrow $\nabla f$ (cyan). Their dot product is the projection of one onto the other.

Two legal steps in a plane point $+ d x$ and $- d x$ . One of them decreases $f$ unless $df = 0$ . So the optimum demands: $\nabla f \cdot d x = 0 for every legal step d x .$

Step 5 — Legal steps are exactly the ones perpendicular to $\nabla g$

WHAT. Which steps keep us on the fence? A step that stays on $g = c$ must not change $g$ , so $d g = \nabla g \cdot d x = 0$ . Thus legal steps are precisely those perpendicular to $\nabla g$ .

WHY. $\nabla g$ points straight across the fence (Step 3). Anything perpendicular to it runs along the fence — that is the only way to move without leaving $g = c$ .

PICTURE. The white tangent step $d x$ is at a right angle to the amber gradient $\nabla g$ . Every allowed direction lives on this tangent line.

So we now have two facts at the optimum, for every legal $d x$ : $\nabla f \cdot d x = 0 and \nabla g \cdot d x = 0.$

Step 6 — Two arrows perpendicular to the same line must be parallel

WHAT. Both $\nabla f$ and $\nabla g$ are perpendicular to the same tangent line (the set of legal steps). In a 2D plane there is only one direction perpendicular to a given line. So $\nabla f$ and $\nabla g$ must point along the same line — they are parallel.

WHY. This is the whole mechanism. If $\nabla f$ had any sideways component along the fence, that component would be a legal uphill direction, and we could improve $f$ — contradiction. Killing the sideways component forces $\nabla f$ to lie entirely across the fence, exactly where $\nabla g$ already lies.

PICTURE. At the optimum the cyan arrow $\nabla f$ and the amber arrow $\nabla g$ lie on one line — the contour of $f$ kisses (is tangent to) the fence. Away from the optimum they point in noticeably different directions and the contour crosses the fence.

Because they are parallel, one is a scalar multiple of the other: $\nabla f (x^{*}) = λ \nabla g (x^{*})$

Term by term:

$x^{*}$ :: the optimal spot on the fence.
$\nabla f (x^{*})$ :: uphill arrow of the objective there.
$\nabla g (x^{*})$ :: across-fence arrow of the constraint there.
$λ$ :: the stretch factor that makes the two arrows match — the Lagrange multiplier. Its sign says whether they point the same or opposite way; its size says how many $g$ -arrows fit in one $f$ -arrow.

Step 7 — Degenerate cases you must not skip

WHAT / WHY / PICTURE — three failure modes, each drawn.

$λ = 0$ . The arrows still align, but $\nabla f = 0$ by itself. The optimum happens to sit at a flat spot of $f$ that also lies on the fence — the constraint is not "pushing." Interpretation: relaxing the fence would not change the optimal value.
$\nabla g = 0$ at the point. The fence has a cusp or singular point — no well-defined across-fence direction, so "parallel to $\nabla g$ " is meaningless. The method can silently miss such points; they must be checked by hand.
Tangency but a maximum vs minimum. Alignment ( $\nabla f = λ \nabla g$ ) is necessary but does not tell max from min. Two points on a closed loop fence can both satisfy it — one highest, one lowest. You still compare $f$ values.

The one-picture summary

Everything above collapses into a single frame: the fence $g = c$ in amber, the family of $f$ -contours in cyan, and the unique spot where a contour is tangent to the fence — where the two gradient arrows share a line. That tangency is $\nabla f = λ \nabla g$ .

Recall Feynman retelling — say it in plain words

Picture a hill and a footpath painted across it. You're only allowed to walk the path. You wander looking for the lowest point on the path. At the true lowest point, the path can't be heading downhill or uphill anymore — if it did, you'd keep walking. So right there the ground's steepest-downhill arrow points straight off the side of the path, not along it. The path's own "which-way-is-across" arrow also points straight off the side. Two arrows pointing off the same side of the same path must point along the same line. Write "objective's uphill arrow equals some number times the path's across arrow," and that number — how much you stretch one arrow to get the other — is $λ$ . It even tells you a bonus: how much better your best value would get if you shoved the path a little. The only traps: sometimes $λ$ is zero (the path wasn't really squeezing you), sometimes the path has a sharp corner with no clean across-direction, and always remember the highest and lowest points both pass this test — so compare heights at the end.

Recall Quick self-test

Why must $\nabla f$ be perpendicular to the constraint at the optimum? ::: Any component of $\nabla f$ along the fence would be a legal uphill direction, so you could improve $f$ — contradiction with being optimal. What does $λ = 0$ signify geometrically? ::: The optimum is a flat spot of $f$ itself; the constraint isn't binding, and relaxing $c$ doesn't change the optimal value. Does $\nabla f = λ \nabla g$ tell you max from min? ::: No — it flags all tangency candidates; you still compare $f$ values and check singular points.

Connected ideas: the "no downhill along the fence" logic generalises to inequalities in KT Conditions; the multipliers reappear as the dual variables in Duality Theory and as the $α_{i}$ of Support Vector Machines; the same tangency picture underlies the shrinkage geometry of Regularization and the smooth-fence trick of Penalty Methods; when the objective is bowl-shaped the tangency point is the global optimum, studied in Convex Optimization; and following $\nabla f$ directly is the topic of Gradient Descent.