Lagrange multipliers for constrained optimization: Question bank

Recall the setup we are testing: optimize $f (x)$ subject to $g (x) = c$ , using the Lagrangian $L (x, λ) = f (x) - λ (g (x) - c)$ , whose stationarity condition is $\nabla f = λ \nabla g$ .

True or false — justify

Setting

\nabla_{x} L = 0

alone is enough to solve a constrained problem.

False — that gives

n

equations (one per component of

x

) for

n + 1

unknowns (the

n

components plus

λ

); you still need

\partial L / \partial λ = 0

, which is just the constraint

g = c

, to close the system.

At a constrained optimum,

\nabla f

must equal

0

False —

\nabla f

need only be parallel to

\nabla g

, not zero; the constraint surface holds you away from the free optimum where

\nabla f = 0

At a constrained optimum,

\nabla f

and

\nabla g

point in the same direction.

Careful — they are parallel, but

λ

can be negative, so

\nabla f

may point opposite to

\nabla g

; "same line" is guaranteed, "same direction" is not.

A larger

λ

always means a "more important" constraint.

Only after fixing the scaling —

∣ λ ∣

measures sensitivity of the optimal value per unit of $c$ , so rescaling the constraint (e.g.

g = c

2 g = 2 c

) halves

λ

without changing the problem; and where

f

is non-differentiable no single

λ

need exist. Compare importances only for constraints written in the same units.

The Lagrange condition

\nabla f = λ \nabla g

distinguishes maxima from minima.

False — it is a stationarity condition satisfied at maxima, minima, and saddles alike; you need convexity or a second-order test to classify.

f

is convex and

g = c

is linear, any stationary point of

L

is the global optimum.

True — a convex objective over a convex (here affine) feasible set has a unique minimum, and the single stationary point is it.

Changing

L = f - λ (g - c)

L = f + λ (g - c)

changes the optimal

x^{*}

False — it only flips the sign of

λ

; the location

x^{*}

is identical because both encode the same parallel-gradient condition.

The value

λ

is dimensionless.

False — since

λ Δ c

approximates

Δ f

λ

carries units of (objective per unit of constraint), e.g. "profit per dollar."

Spot the error

"To maximize

x y

with

x + y = 10

, I set

\nabla f = \nabla g

and got

y = 1, x = 1

Error — they dropped

λ

; the correct condition is

\nabla f = λ \nabla g

, giving

y = λ, x = λ

and then

x = y = 5

from the constraint.

"I formed

L = f - λ g

(no

- c

), solved

\nabla_{x} L = 0

, and I'm done."

Error — omitting

c

is harmless for

\nabla_{x} L

(the constant vanishes) but you still forgot the

\partial L / \partial λ = 0

equation that reinstates

g = c

\nabla g = 0

at my candidate point, but

\nabla f \neq = 0

, so no

λ

works — therefore no optimum exists here."

Error — a point where

\nabla g = 0

is a degenerate (non-regular) constraint point; the multiplier method's assumption fails there, so you must check it separately, not declare the optimum nonexistent.

"Two constraints, so I'll use one shared

λ

for both."

Error — each equality constraint gets its own multiplier:

L = f - \sum_{i} λ_{i} (g_{i} - c_{i})

; one

λ

cannot balance

\nabla f

against two independent gradient directions.

"The constraint is

g (x, y) \leq c

, so I'll just write

\nabla f = λ \nabla g

as usual."

Error — inequalities require the KT Conditions with

λ \geq 0

and complementary slackness; equality-style Lagrange multipliers apply only when the constraint is active and even then need the sign restriction.

Why questions

Why must

\nabla f

have no component tangent to the constraint surface at an optimum?

Because a nonzero tangential component would let you slide along the surface (staying feasible) in a direction that raises or lowers

f

— contradicting optimality.

Why does the minus sign in

L = f - λ (g - c)

not affect where the optimum sits?

Because

\nabla_{x} L = 0

reads

\nabla f = λ \nabla g

; flipping the sign just relabels

λ \to - λ

, leaving the geometric parallel-gradient equation unchanged.

Why can't we simply substitute the constraint into

f

and minimize freely instead?

We often can for simple explicit constraints, but Lagrange multipliers avoid solving

g = c

for one variable (sometimes impossible or messy) and hand us the sensitivity

λ

for free.

Why does

λ

equal the sensitivity

\partial f^{*} / \partial c

of the optimal value?

By the envelope theorem: write

f^{*} (c) = L (x^{*} (c), λ (c), c)

; differentiating with the chain rule, the

x

- and

λ

-terms vanish because

\nabla_{x} L = 0

and

\partial L / \partial λ = 0

at the optimum, leaving only

\partial L / \partial c = λ

— this is the shadow price.

Why does each new constraint require balancing

\nabla f

against another gradient?

Each constraint carves away one tangent direction of allowed motion;

\nabla f

must be neutralized in every remaining feasible direction, forcing it into the span of all the

\nabla g_{i}

Why is this the same math seen in Support Vector Machines and Regularization?

SVMs minimize

\frac{1}{2} ∥ w ∥^{2}

under margin constraints, and ridge-style penalties are the Lagrangian dual of a norm constraint — both are constrained optimizations whose multipliers become the weights or the penalty strength.

Edge cases

What happens when the constraint gradient

\nabla g = 0

at a feasible point?

Regularity (the constraint qualification defined above) fails: no

λ

may satisfy

\nabla f = λ \nabla g

even at a true optimum, so such points must be examined directly.

What if

\nabla f = 0

already at a feasible point?

Then

λ = 0

works — the unconstrained optimum happens to sit on the constraint, so the constraint is not binding and has zero shadow price.

What if the constraint surface is empty (no

x

satisfies

g = c

There is no feasible set, so the problem is infeasible; the Lagrangian machinery produces meaningless "solutions" and must not be trusted — always check feasibility first.

What if

f

is linear and the feasible region is unbounded?

The optimum may be

\pm \infty

with no finite stationary point;

\nabla f = λ \nabla g

can still have solutions on the boundary but there may be no true extremum — the method finds candidates, not a guarantee of existence.

What if two constraint gradients are parallel to each other at a point?

The constraints are locally dependent (regularity's linear-independence condition fails); their multipliers are not uniquely determined, so the standard sensitivity interpretation of each

λ_{i}

breaks down.

For an equality constraint, can

λ

be negative — and does that break anything?

Yes,

λ

may be any sign for equality constraints (unlike inequality ones where

λ \geq 0

); a negative

λ

merely means tightening

c

raises the optimal objective, which is perfectly valid.

How to classify a stationary point (the third leg)

Once

\nabla f = λ \nabla g

is solved, decide max vs min vs saddle:

Restrict attention to feasible directions — vectors

d

tangent to the constraint, i.e.

\nabla g \cdot d = 0

— and test the bordered Hessian of

L

(the Hessian

\nabla_{x}^{2} L

evaluated on that tangent space).

What does definiteness on the tangent space tell you?

d^{⊤} \nabla_{x}^{2} L d > 0

for all feasible

d \neq = 0

(positive definite on the tangent space) the point is a local minimum; if

< 0

(negative definite) a local maximum; mixed signs mean a saddle.

When can you skip the second-order test entirely?

When

f

is convex and the feasible set is convex (e.g.

f

quadratic-convex with linear constraints): every stationary point is then the global minimum by Convex Optimization, so no Hessian check is needed.

Recall Two-line self-test

Verdict-plus-reason: " $\nabla f = λ \nabla g$ classifies extrema." ::: False — it is stationarity only; classification needs convexity or a second-order (bordered-Hessian) check. Verdict-plus-reason: "For an active inequality constraint I can reuse equality-style multipliers with no sign rule." ::: False — inequalities demand $λ \geq 0$ and complementary slackness (KT Conditions).