Lagrange multipliers for constrained optimization: Exercises

This page is a self-test. Every problem below has a collapsible solution — try it yourself first, then open the callout to check your full reasoning. Problems climb from recognition (spot the setup) to mastery (build something new). Each is tagged L1–L5.

Everything here builds on the parent note. If a step confuses you, re-read the derivation there. We reuse exactly one machine, over and over:

Here is the whole picture in one image. Everything on this page is a variation of it: the objective's contour lines (blue), the constraint curve (orange), and the special point where they just kiss — where their gradient arrows line up.

Figure — Lagrange multipliers for constrained optimization

Level 1 — Recognition

Goal: read a problem and correctly write down the Lagrangian and the three equations. No heavy algebra.

Exercise 1.1

Write the Lagrangian for: minimize $f (x, y) = 3 x^{2} + y^{2}$ subject to $g (x, y) = x + y = 5$ . Then write the three equations you would solve. (Do not solve them.)

Recall Solution 1.1

What we do: plug straight into the machine. Here $c = 5$ . $L (x, y, λ) = 3 x^{2} + y^{2} - λ (x + y - 5)$ The three equations (two gradient equations + the constraint): $\frac{\partial L}{\partial x} = 6 x - λ = 0$ $\frac{\partial L}{\partial y} = 2 y - λ = 0$ $x + y = 5$ That is it — three equations, three unknowns $x, y, λ$ . The third equation is $\partial L / \partial λ = 0$ .

Exercise 1.2

Which of these is a valid Lagrangian for "maximize $x y$ subject to $x^{2} + y^{2} = 1$ "? Pick and explain. (a) $x y - λ (x^{2} + y^{2})$ (b) $x y - λ (x^{2} + y^{2} - 1)$ (c) $x y \cdot λ (x^{2} + y^{2} - 1)$

Recall Solution 1.2

Answer: (b). The constraint must be written as $g - c$ , i.e. $x^{2} + y^{2} - 1$ , so that setting $\partial L / \partial λ = 0$ gives back exactly $x^{2} + y^{2} = 1$ .

(a) drops the $- 1$ , so $\partial L / \partial λ = - (x^{2} + y^{2}) = 0$ forces $x = y = 0$ — the wrong constraint.
(c) multiplies by $λ$ instead of adding a $λ$ -term. Then $\partial L / \partial λ = x y (x^{2} + y^{2} - 1)$ , which is nonsense as a constraint recovery. The multiplier must enter linearly and additively.

Level 2 — Application

Goal: run the machine end-to-end and get numbers.

Exercise 2.1

Minimize $f (x, y) = x^{2} + y^{2}$ subject to $2 x + y = 10$ . Find $x, y, λ$ and the minimum value.

Recall Solution 2.1

Step 1 — Lagrangian. ( $c = 10$ ) $L = x^{2} + y^{2} - λ (2 x + y - 10)$ Step 2 — gradient equations. $\frac{\partial L}{\partial x} = 2 x - 2 λ = 0 ⟹ x = λ$ $\frac{\partial L}{\partial y} = 2 y - λ = 0 ⟹ y = \frac{λ}{2}$ Step 3 — substitute into the constraint $2 x + y = 10$ : $2 λ + \frac{λ}{2} = 10 ⟹ \frac{5 λ}{2} = 10 ⟹ λ = 4$ So $x = 4$ , $y = 2$ . Minimum value: $f = 4^{2} + 2^{2} = 16 + 4 = 20$ . Geometry: $(4, 2)$ is the point on the line $2 x + y = 10$ closest to the origin — the foot of the perpendicular from $(0, 0)$ .

Exercise 2.2

Maximize $f (x, y) = x y$ subject to $x + 4 y = 8$ . Find $x, y, λ$ and the max value.

Recall Solution 2.2

Step 1. $L = x y - λ (x + 4 y - 8)$ . Step 2. $\frac{\partial L}{\partial x} = y - λ = 0 ⟹ y = λ$ $\frac{\partial L}{\partial y} = x - 4 λ = 0 ⟹ x = 4 λ$ Step 3. Substitute into $x + 4 y = 8$ : $4 λ + 4 λ = 8 ⟹ λ = 1$ So $y = 1$ , $x = 4$ . Max area $= x y = 4$ . Sanity: the two terms $x$ and $4 y$ each equal $4$ — the product is maximized when the constraint's two "weighted halves" are equal. That's a recurring pattern.

Level 3 — Analysis

Goal: interpret $λ$ , handle the sensitivity/shadow-price meaning, deal with quadratic-vs-linear structure.

Exercise 3.1

For minimize $f (x, y) = x^{2} + y^{2}$ subject to $x + y = c$ , find the optimum as a function of $c$ , find $λ (c)$ , and verify the sensitivity relation $\frac{d f ^{*}}{d c} = λ$ .

Recall Solution 3.1

Solve in general. $L = x^{2} + y^{2} - λ (x + y - c)$ . $2 x - λ = 0, 2 y - λ = 0 ⟹ x = y = \frac{λ}{2} .$ Constraint: $\frac{λ}{2} + \frac{λ}{2} = c ⟹ λ = c$ . So $x = y = \frac{c}{2}$ . Optimal value: $f^{*} (c) = (\frac{c}{2})^{2} + (\frac{c}{2})^{2} = \frac{c ^{2}}{2} .$ Sensitivity check: $\frac{d f ^{*}}{d c} = \frac{d}{d c} (\frac{c ^{2}}{2}) = c = λ . ✓$ Meaning: $λ$ is the shadow price — the rate at which the best achievable value changes as you loosen the constraint by one unit. Here relaxing $c$ by $Δ c$ raises $f^{*}$ by about $λ Δ c = c Δ c$ .

Exercise 3.2

In Exercise 2.2 you found $λ = 1$ at $c = 8$ . Predict the new maximum area if the budget rises from $8$ to $8.5$ , using $λ$ . Then compute the exact new area and compare.

Recall Solution 3.2

Prediction using $λ$ : $Δ f^{*} \approx λ Δ c = 1 \times 0.5 = 0.5$ , so predicted new area $\approx 4 + 0.5 = 4.5$ . Exact recompute at $c = 8.5$ : same algebra gives $x = 4 λ, y = λ$ , and $8 λ = 8.5 ⟹ λ = 1.0625$ , so $y = 1.0625$ , $x = 4.25$ . $f^{*} = x y = 4.25 \times 1.0625 = 4.515625.$ Compare: predicted $4.5$ vs exact $4.5156$ . Close — the multiplier gives a first-order (linear) estimate; the small gap ( $0.0156$ ) is the curvature the linear prediction misses.

Level 4 — Synthesis

Goal: combine tools — multiple constraints, symmetry, an SVM-flavored problem.

Exercise 4.1 (two constraints)

Minimize $f (x, y, z) = x^{2} + y^{2} + z^{2}$ subject to both $x + y + z = 3$ and $x - z = 0$ . Find the point and both multipliers.

Recall Solution 4.1

Two constraints ⇒ two multipliers. With $g_{1} = x + y + z - 3$ and $g_{2} = x - z$ : $L = x^{2} + y^{2} + z^{2} - λ_{1} (x + y + z - 3) - λ_{2} (x - z) .$ Gradient equations: $\partial_{x} : 2 x - λ_{1} - λ_{2} = 0$ $\partial_{y} : 2 y - λ_{1} = 0$ $\partial_{z} : 2 z - λ_{1} + λ_{2} = 0$ Constraints: $x + y + z = 3$ , $x = z$ . From $x = z$ and the $\partial_{x}, \partial_{z}$ equations: subtract them, $(2 x - λ_{1} - λ_{2}) - (2 z - λ_{1} + λ_{2}) = 0 ⟹ 2 (x - z) - 2 λ_{2} = 0 ⟹ λ_{2} = 0.$ Then $\partial_{x}$ and $\partial_{z}$ both give $2 x = λ_{1}$ , and $\partial_{y}$ gives $2 y = λ_{1}$ , so $x = y = z$ . Constraint $x + y + z = 3 ⟹ 3 x = 3 ⟹ x = 1$ . Thus $(x, y, z) = (1, 1, 1)$ . Multipliers: $λ_{1} = 2 x = 2$ , $λ_{2} = 0$ . Interpretation of $λ_{2} = 0$ : the symmetry constraint $x = z$ is "free" here — the unconstrained-on-that-axis optimum already satisfies it, so relaxing it changes nothing.

Exercise 4.2 (SVM-style)

Minimize $f (w) = \frac{1}{2} (w_{1}^{2} + w_{2}^{2} + w_{3}^{2})$ subject to $w_{1} + 2 w_{2} + 2 w_{3} = 9$ . Find $w$ , $λ$ , and $∥ w ∥^{2}$ .

Recall Solution 4.2

This is the SVM shape: smallest-norm weight vector meeting a linear margin condition. $L = \frac{1}{2} (w_{1}^{2} + w_{2}^{2} + w_{3}^{2}) - λ (w_{1} + 2 w_{2} + 2 w_{3} - 9) .$ Gradients: $w_{1} - λ = 0, w_{2} - 2 λ = 0, w_{3} - 2 λ = 0.$ So $w = (λ, 2 λ, 2 λ)$ — the weight vector points along the constraint's coefficient vector $(1, 2, 2)$ . That is the whole geometric moral of SVMs: the min-norm solution is parallel to the constraint normal. Constraint: $λ + 2 (2 λ) + 2 (2 λ) = 9 ⟹ λ (1 + 4 + 4) = 9 ⟹ 9 λ = 9 ⟹ λ = 1$ . So $w = (1, 2, 2)$ , and $∥ w ∥^{2} = 1 + 4 + 4 = 9$ , giving $f = \frac{9}{2} = 4.5$ .

Level 5 — Mastery

Goal: reason about when the method breaks, and build a full argument from scratch.

Exercise 5.1 (degenerate gradient)

Try to minimize $f (x, y) = x$ subject to $g (x, y) = x^{2} + y^{2} = 0$ . Set up the Lagrangian equations and explain why the method fails here.

Recall Solution 5.1

Setup. $L = x - λ (x^{2} + y^{2})$ . Gradients: $\partial_{x} : 1 - 2 λ x = 0, \partial_{y} : - 2 λ y = 0.$ The constraint $x^{2} + y^{2} = 0$ forces $x = 0, y = 0$ (only the origin has zero radius). Where it breaks: at $(0, 0)$ , $\nabla g = (2 x, 2 y) = (0, 0)$ — the constraint gradient is the zero vector. The condition $\nabla f = λ \nabla g$ becomes $\nabla f = λ \cdot 0 = 0$ , i.e. $(1, 0) = (0, 0)$ , which is impossible for any $λ$ . Why this matters: the Lagrange condition secretly assumes $\nabla g \neq = 0$ at the optimum (a "regularity" / constraint-qualification condition). When $\nabla g = 0$ , the constraint surface has no well-defined tangent direction and the alignment argument collapses. Lesson: always check $\nabla g \neq = 0$ at candidate points. This same regularity requirement reappears in the KKT conditions for inequality constraints.

Exercise 5.2 (build it yourself)

A rectangular box has a fixed surface area $S = 24$ . Derive the dimensions that maximize its volume, using Lagrange multipliers. (Box with sides $x, y, z$ : volume $x y z$ , surface area $2 (x y + y z + z x)$ .)

Recall Solution 5.2

Objective and constraint. $f = x y z, g = 2 (x y + y z + z x) = 24 ⟺ x y + y z + z x = 12.$ Lagrangian. $L = x y z - λ (x y + y z + z x - 12) .$ Gradients: $\partial_{x} : y z - λ (y + z) = 0$ $\partial_{y} : x z - λ (x + z) = 0$ $\partial_{z} : x y - λ (x + y) = 0$ Solve by symmetry. Multiply the first by $x$ , second by $y$ , third by $z$ : $x y z = λ x (y + z) = λ y (x + z) = λ z (x + y) .$ From $λ x (y + z) = λ y (x + z)$ (with $λ \neq = 0$ ): $x y + x z = x y + y z ⟹ x z = y z ⟹ x = y$ (since $z \neq = 0$ ). Likewise $y = z$ . So $x = y = z$ — a cube. Apply the constraint: $3 x^{2} = 12 ⟹ x^{2} = 4 ⟹ x = 2$ . So $x = y = z = 2$ , volume $= 2^{3} = 8$ , and $λ$ : from $y z = λ (y + z)$ , $4 = λ \cdot 4 ⟹ λ = 1$ . Result: the maximum-volume box for a given surface area is a cube — the 3-D cousin of "square maximizes area for fixed perimeter."

Recall Quick self-check (clozes)

The extra equation you must never forget is ::: the constraint $g (x) = c$ , i.e. $\partial L / \partial λ = 0$ $λ$ equals the derivative of the optimal value with respect to ::: the constraint constant $c$ (the shadow price $d f^{*} / d c$ ) With $m$ constraints you need this many multipliers ::: $m$ The hidden assumption behind $\nabla f = λ \nabla g$ is ::: $\nabla g \neq = 0$ at the optimum (constraint qualification / regularity)

Where to go next: inequality constraints generalize all of this via the KKT conditions; the shadow-price idea becomes duality; the min-norm structure of Exercise 4.2 is the heart of Support Vector Machines; and the " $f$ plus a penalty" viewpoint links to Regularization, Penalty Methods, and Convex Optimization. To reach these optima numerically, see Gradient Descent.