KKT conditions for constrained optimization: Visual walkthrough

Before we start, three plain-word promises about the symbols we'll use:

Prerequisites we lean on: Lagrange Multipliers (the equality-only ancestor), and later Convex Optimization and Constraint Qualifications (LICQ, Slater).

Step 1 — What a gradient is, and what "downhill" means

WHAT. Stand at a point on the cost landscape $f (x, y)$ . The gradient $\nabla f$ is the single arrow pointing where the ground rises steepest. Turn it around and you get $- \nabla f$ : the direction water would run — the direction that decreases $f$ the most.

WHY this tool and not another? We could ask "does moving this way lower the cost?" for a thousand directions. That's exhausting. The gradient answers all of them at once. First, plain words: a direction $d$ is just a little arrow saying "which way to step" — say, "one metre east and half a metre north." Take a tiny step along that arrow and the cost changes by (roughly) $dot product \nabla f \cdot d = lengths ∣\nabla f ∣ ∣ d ∣ θ = angle between uphill arrow and step cos θ,$ a single dot product that multiplies the two arrows and the cosine of the angle between them. If that number is negative (you stepped more against the uphill arrow than with it), you went downhill. So the gradient is the compass of optimization; we reach for it because it compresses "which way is down" into one arrow.

PICTURE. The blue curves below are level sets — rings of equal height, like a contour map. The pale-yellow arrow is $\nabla f$ (uphill, always ⟂ to the rings). The pink arrow is $- \nabla f$ (downhill), the way the ball rolls when nothing stops it.

Figure — KKT conditions for constrained optimization

Step 2 — Free minimum: the ball stops where the ground is flat

WHAT. With no fences, where does the ball rest? Exactly where downhill points nowhere — where the ground is flat.

WHY. If $- \nabla f$ were any nonzero arrow, the ball could follow it and drop lower. So resting requires that arrow to vanish. Flatness is the condition $\nabla f = 0$ .

PICTURE. At the bowl's bottom the rings shrink to a point and the gradient arrow has zero length — nothing left to push the ball.

Step 3 — A wire (equality): the ball can only slide along it

WHAT. Now force the ball onto a wire $h (x) = 0$ — a curve it must stay on. It can slide along the wire but not leave it.

WHY the gradient must line up. Split $- \nabla f$ into two parts: the part along the wire and the part across it. If any downhill push runs along the wire, the ball slides that way and lowers $f$ — not resting yet. So at rest, the along-wire push is zero. That leaves $\nabla f$ pointing purely across the wire, i.e. parallel to the wire's own normal $\nabla h$ . "Parallel" means one is a multiple of the other:

$\nabla f (x^{*}) + μ \nabla h (x^{*}) = 0.$

This is exactly ordinary Lagrange Multipliers.

PICTURE. Blue rings (cost), yellow curve (the wire $h = 0$ ). At the resting point the cost-ring is tangent to the wire — they kiss. There, $\nabla f$ (pink) and $\nabla h$ (yellow-outline) lie on the same line, pointing opposite ways; the number $μ$ is how many times longer one is than the other.

Step 4 — A fence (inequality): forbidden only on ONE side

WHAT. Replace the wire with a fence $g (x) \leq 0$ . Crucial difference: one side ( $g < 0$ ) is open ground the ball may enter; the other ( $g > 0$ ) is forbidden. A fence pushes back only from one direction.

WHY this changes everything. With a wire the ball is trapped on the curve. With a fence the ball has a whole region to roam. Two totally different resting scenarios appear, which we split in Steps 5 and 6.

PICTURE. The pink shaded half is forbidden ( $g > 0$ ); the clear half is feasible ( $g \leq 0$ ). The fence line is $g = 0$ . The arrow $\nabla g$ points into the forbidden side (that's the direction $g$ increases).

Step 5 — Case A: fence inactive → it might as well not exist

WHAT. Suppose the free minimum from Step 2 already sits inside the allowed region ( $g < 0$ ). Then the fence never mattered.

WHY the multiplier is zero. We'll write the resting condition as $\nabla f + λ \nabla g = 0$ (like the wire, one multiplier $λ$ ). But here the ball rests on flat ground, so $\nabla f = 0$ , which forces $λ \nabla g = 0$ . Since $\nabla g \neq = 0$ on a real fence, we need $λ = 0$ . Meaning: an untouched fence exerts no force.

PICTURE. The bowl's true bottom lands in clear (feasible) territory; the fence sits far away, slack. The force arrow from the fence has zero length.

Step 6 — Case B: fence active → the wall pushes, and only ONE way

WHAT. Now the free minimum is forbidden, so the ball rolls until it jams against the fence and stops there with $g (x^{*}) = 0$ .

WHY the multiplier must be $\geq 0$ (this is the heart of KKT). At rest against the fence, $\nabla f$ balances the fence's push: $\nabla f (x^{*}) = - λ \nabla g (x^{*}) .$ The only escape directions the ball has are feasible ones — directions where $g$ does not increase, i.e. that point away from $\nabla g$ . For the ball to truly be stuck, none of those escape directions may go downhill. Working the inequality through, the downhill arrow $- \nabla f$ must point into the fence (same side as $+ \nabla g$ ), which forces: $λ \geq 0.$ Contrast Step 3's wire, where $μ$ was free: a fence blocks only one direction, so its multiplier can only have one sign. A negative $λ$ would mean $- \nabla f$ points into open ground — but then the ball could roll off the fence and drop lower. So $λ < 0$ is impossible at a genuine rest.

PICTURE. The ball is jammed on the fence. Downhill $- \nabla f$ (pink) points into the forbidden zone; the fence's inward normal $- \nabla g$ (yellow) exactly cancels it. Their lengths differ by the factor $λ > 0$ .

Step 7 — Fusing both cases: complementary slackness

WHAT. Steps 5 and 6 are two branches. KKT bottles them into one elegant equation: $λ g (x^{*}) = 0.$

WHY it captures both. A product is zero exactly when one factor is zero:

If $g < 0$ (inactive), the product forces $λ = 0$ — that's Step 5.
If $λ > 0$ (real push), the product forces $g = 0$ — that's Step 6.

They can never both be "loose" ( $λ > 0$ and $g < 0$ ) — that would be a wall pushing on a ball it isn't touching. Impossible.

PICTURE. A two-branch fork: the left path (inactive, $λ = 0$ ) and the right path (active, $g = 0$ ) both funnel into the single node $λ g = 0$ .

Step 8 — Stacking many fences and wires: the cone of forces

WHAT. Real problems have several fences and several wires. Before bundling them, let's fix the plain-word bookkeeping so no new symbol sneaks in:

Number the fences $g_{1}, g_{2}, \dots, g_{m}$ — one $g_{i}$ per inequality, with its own non-negative multiplier $λ_{i} \geq 0$ . The little subscript $i$ is just a name-tag running $i = 1, \dots, m$ .
Number the wires $h_{1}, h_{2}, \dots, h_{p}$ — one $h_{j}$ per equality, with its own sign-free multiplier $μ_{j}$ . The subscript $j$ runs $j = 1, \dots, p$ .

So $\sum_{i}$ means "add up over every fence" and $\sum_{j}$ means "add up over every wire." With this in hand we bundle every push and pull into one bookkeeping object, the Lagrangian:

WHY a "cone". Each active fence contributes a push $λ_{i} \nabla g_{i}$ with $λ_{i} \geq 0$ — a non-negative amount along its normal. Adding non-negative amounts of several arrows sweeps out a wedge-shaped region called a cone. The rule says: $- \nabla f$ (downhill) must live inside that cone. If it does, the ball is trapped — every downhill direction is blocked by some wall.

PICTURE. Two active fence-normals $\nabla g_{1}, \nabla g_{2}$ (yellow) open a shaded cone. The downhill arrow $- \nabla f$ (pink) sits inside it — trapped. This is the "geometric soul" the parent note mentioned.

Step 9 — When is "trapped" the SAME as "global best"? (convex + Slater)

WHAT. Everything so far is local — the ball stopped, but is this the lowest point anywhere? For a convex problem, with a mild regularity condition, the answer is yes.

WHY convexity seals it. Convex means the landscape is a single smooth bowl (no secondary dips) and every feasible region is a solid, dent-free blob. In such a world "no downhill escape here" can't be a local trick — there's only one basin. Two precise statements:

KKT ⟹ global minimum (sufficiency): for a convex problem ( $f, g_{i}$ convex, $h_{j}$ affine), any point satisfying the four KKT conditions is a global minimum — no extra assumption needed.
Global minimum ⟹ KKT (necessity): to guarantee the multipliers exist at the optimum you still need a constraint qualification. The convex-friendly one is Slater's condition: some strictly feasible point exists ( $g_{i} (x) < 0$ for all $i$ ). Slater also delivers strong duality, tying these same $λ$ 's to the dual problem.

This "necessary and sufficient (under Slater)" fact is why Support Vector Machines and other Convex Optimization models can trust a KKT point, and why solvers like Gradient Descent and Projected Gradient converge to the true optimum.

PICTURE. Left: a wiggly non-convex landscape where a KKT point sits in a shallow false dip (also flags a mere saddle/max). Right: a clean convex bowl where the one KKT point is the global floor.

The one-picture summary

Everything above, on a single board: the ball at rest against active fences, downhill trapped in the cone of pushes, inactive fences dangling with zero force, and the four KKT stamps labelled where they live.

Recall Feynman retelling of the whole walkthrough

A marble rolls down a bumpy floor toward the lowest spot it can reach, but there are fences. First I learned the marble always wants to roll straight downhill — that arrow is $- \nabla f$ . On open ground it stops only where the floor is flat. Put it on a wire and it slides until the downhill push runs sideways-into the wire, not along it. Fences are trickier: a fence only blocks one side, so it can only push, never pull — that's why its number $λ$ can't go negative. If the marble stops away from a fence, that fence did nothing ( $λ = 0$ ); if it stops against one, the fence is definitely pushing ( $g = 0$ ). "Push only where you touch" is one neat line: $λ g = 0$ . And of course the marble has to actually stay inside the playground the whole time — that's "primal feasibility." To keep the books, we pile cost plus all pushes into one function $L$ and set its $x$ -gradient to zero. With many fences meeting at a corner, their pushes add up into a wedge (a cone), and the marble is stuck exactly when downhill points inside that wedge — provided no fence has a degenerate, direction-less edge (that's what LICQ forbids). Finally, if the whole floor is one smooth bowl (convex) and there's some strictly-inside point (Slater), "stuck here" means "lowest anywhere." That's the entire KKT story.

Recall Why can

$μ$ be negative but not $λ$ ? A wire ( $h = 0$ ) forbids both sides, so it can pull either way → $μ$ free in sign. A fence ( $g \leq 0$ ) forbids only one side, so it can only push one way → $λ \geq 0$ (this is "dual feasibility").

Recall What does

$λ g = 0$ forbid? The combination $λ > 0$ and $g < 0$ — a wall pushing a ball it isn't touching. One of the two must always be zero.

Recall Why does LICQ insist gradients be linearly independent?

Because the balance argument secretly needs each $\nabla g_{i} \neq = 0$ and needs the normals not to collapse onto one line. Linear independence guarantees both, so the cone is full-dimensional and honest multipliers exist and are unique.