Linear Quadratic Regulator (LQR) — Riccati equation, optimal gains: Visual walkthrough

Step 0 — The two words: state and control (before any math)

PICTURE. A cart sits at position $x_{1}$ moving right at speed $x_{2}$ . Your hand pushes it with force $u$ . The target is the origin — the little flag at $0$ . That is the entire universe of this problem.

Figure — Linear Quadratic Regulator (LQR) — Riccati equation, optimal gains

The arrow labelled $u$ is red because it is the only thing under your command. Everything we do from now on answers one question: how hard, and which way, should that red arrow push, given where the cart is right now?

Step 1 — The machine is linear: $\overset{x}{˙} = A x + B u$

WHY care that it's linear? Because a straight rule is the only kind we can fully solve in closed form. The whole payoff of LQR rides on this.

PICTURE. The state lives as a point in a plane (position across, velocity up). At every point an arrow shows which way the machine drifts on its own ( $A x$ ). Your push $B u$ nudges that arrow.

Look at the black arrows: with no push, a cart to the right and moving right ( $x_{1} > 0, x_{2} > 0$ ) drifts further right — it runs away. The red overlay shows how one choice of $u$ bends those arrows back toward the flag.

Step 2 — The bill we pay: the quadratic cost $J$

WHY squares and not absolute values or fourth powers? Two reasons, one practical, one magical.

Practical: a square is smooth and has no sharp corner at zero, so the machine never chatters.
Magical: a linear machine fed a squared bill produces a controller that is itself linear. That coincidence is the entire reason LQR is beautiful. We prove it below.

PICTURE. A squared penalty is a bowl. Being at the bottom (on-target, no push) costs nothing; climbing the wall costs more and more, faster and faster.

The orange bowl is the error bill in $x$ ; the teal bowl is the effort bill in $u$ . Because $Q ⪰ 0$ and $R ≻ 0$ , both bowls open upward — they never dip below the ground. LQR looks for the path that keeps total bowl-height smallest, forever.

Step 3 — The "cost-to-go" is also a bowl: $V (x) = x^{⊤} P x$

WHY must $V$ be a bowl (a quadratic $x^{⊤} P x$ )? Three clues, each forced:

At the target, playing perfectly costs zero, so $V (0) = 0$ — the bowl touches ground at the origin.
Left and right of target are symmetric mirror situations, so no odd/linear term survives.
A straight machine can't manufacture cubes or quartics out of a squared bill — the shape stays a clean bowl.

So $V (x) = x^{⊤} P x$ , where $P$ is an unknown symmetric grid of numbers — the one thing we must find. This choice is exactly the value-function guess.

PICTURE. The bowl $V$ sits over the state plane. Its steepness in each direction is set by $P$ . Find $P$ and you know the price of every situation — and, as we'll see, the perfect push.

Step 4 — Bellman's balance: instant cost + downhill speed = 0

That last line is the HJB equation. The slogan "instant cost cancels the fall of the future bill" is now derived, not merely asserted.

PICTURE. Stand on the inside wall of the bowl. The uphill slope arrow is $\nabla V$ . Your motion $\overset{x}{˙} = A x + B u$ is another arrow. Their alignment (via the chain rule) tells you whether the future bill is dropping (moving downhill) and how fast.

Step 5 — Turn the knob to the bottom: solve for $u$

Inside the bracket, the only thing you control is $u$ . Everything with a $u$ in it: $a bowl in u u^{⊤} R u + a tilt 2 x^{⊤} P B u$ This is a bowl (because $R ≻ 0$ , i.e. $R$ is a genuine upward bowl) tilted by the state. A tilted bowl has exactly one lowest point. Slide to it by setting the slope in $u$ to zero: $2 R u + 2 B^{⊤} P x = 0 ⟹ u^{*} = - R^{- 1} B^{⊤} P x = - K x$

WHY does $R^{- 1}$ appear, and why must $R$ be strictly positive? If effort were free ( $R \to 0$ ) you'd shove infinitely hard — the bowl in $u$ would be flat and have no bottom. A real, strictly-positive $R$ gives the bowl a genuine bottom, so $R^{- 1}$ exists and one finite best push exists. This is the feedback law $u = - K x$ , now derived rather than guessed.

PICTURE. For a fixed state, the cost as a function of your push is a parabola. Its vertex is the perfect push. Steeper parabola (bigger $R$ ) → vertex closer to zero → gentler push.

Step 6 — Put the best push back in: the Riccati equation appears

We now substitute $u^{*} = - R^{- 1} B^{⊤} P x$ into Bellman's balance and carefully collect terms. Start from $0 = x^{⊤} Q x + u^{* ⊤} R u^{*} + 2 x^{⊤} P (A x + B u^{*}) .$

Term (i) — the drift $2 x^{⊤} P A x$ becomes the symmetric pair $A^{⊤} P + P A$ . The number $x^{⊤} P A x$ is a $1 \times 1$ scalar, so it equals its own transpose: $x^{⊤} P A x = (x^{⊤} P A x)^{⊤} = x^{⊤} A^{⊤} P^{⊤} x = x^{⊤} A^{⊤} P x$ (using $P = P^{⊤}$ ). Averaging the two identical copies, $2 x^{⊤} P A x = x^{⊤} P A x + x^{⊤} A^{⊤} P x = x^{⊤} (A^{⊤} P + P A) x .$ So the lone lop-sided $P A$ is rewritten as the symmetric sum $A^{⊤} P + P A$ — the "drift up the bowl" term.

Term (ii) — the two $u^{*}$ pieces merge into $- P B R^{- 1} B^{⊤} P$ . Plug $u^{*} = - R^{- 1} B^{⊤} P x$ into the effort cost and the tilt separately: $u^{* ⊤} R u^{*} = x^{⊤} P B = R^{- 1} R^{- 1} R R^{- 1} B^{⊤} P x = + x^{⊤} P B R^{- 1} B^{⊤} P x,$ $2 x^{⊤} P B u^{*} = - 2 x^{⊤} P B R^{- 1} B^{⊤} P x .$ Add them: $(+ 1 - 2) = - 1$ , so together they give $- x^{⊤} P B R^{- 1} B^{⊤} P x .$ This is exactly what steering saves — the optimizer buys a reduction, hence the minus sign.

Collect (i) + (ii) + the error term $x^{⊤} Q x$ : $0 = x^{⊤} (drift up the bowl A^{⊤} P + P A what steering saves - P B R^{- 1} B^{⊤} P + running error price Q) x .$ This must hold at every state $x$ . A quadratic form that is zero for all $x$ can only come from the flat zero matrix, so the whole grid in the middle must vanish:

WHY nonlinear? The saving term has $P$ multiplied by $P$ — a square. That single squared term is what makes this a Riccati equation rather than a plain linear one.

PICTURE. The equation is a see-saw of grids: drift ( $A^{⊤} P + P A$ ) and error price ( $Q$ ) push $P$ up; the steering saving ( $- P B R^{- 1} B^{⊤} P$ ) pulls it down. The balance point is the $P$ we want.

Step 7 — The degenerate & edge cases (never leave a gap)

Definition Two words we lean on: controllable and observable

Before the edge cases, meet the two health checks that decide whether a good $P$ even exists (full detail in Controllability and Observability):

$(A, B)$ controllable means: your push can eventually reach every direction of the state. If some mode of the machine is deaf to $u$ , no gain can steer it — LQR would be helpless there. Think: can the red arrow, over time, move the cart in every way it might need to?
$(A, Q)$ observable means: every unstable direction of the machine actually shows up in the error bill. $Q$ is any matrix with $Q^{⊤} Q = Q$ ; asking it to be observable is asking that no runaway mode is "invisible" to the cost. If a mode costs nothing yet blows up, the optimizer never notices and the plan diverges.

Together these two guarantee the CARE has a unique stabilizing $P ⪰ 0$ .

PICTURE. Three miniature bowls side by side: cheap fuel (deep narrow bowl, sharp gain), free error (flat wide bowl, lazy gain), and the correct-vs-wrong root (bowl up vs bowl down).

The one-picture summary

Read the five panels below left-to-right — they are exactly Steps 1 → 6 compressed into one strip:

Panel 1 (Step 1) — Linear machine $\overset{x}{˙} = A x + B u$ : the straight cause-and-effect rule.
Panel 2 (Step 2) — Quadratic bill $J = \int_{0}^{\infty} (x^{⊤} Q x + u^{⊤} R u) d t$ : the two upward bowls we pay.
Panel 3 (Step 3) — Cost-to-go bowl $V = x^{⊤} P x$ : forced to be a smooth quadratic; find $P$ .
Panel 4 (Steps 4–5) — Bellman's knob $u = - K x$ : turning the U-curve to its vertex gives a linear law.
Panel 5 (Step 6) — Riccati → $P$ : back-substitution forces $A^{⊤} P + P A - P B R^{- 1} B^{⊤} P + Q = 0$ , whose stabilizing root hands you $K = R^{- 1} B^{⊤} P$ .

Each black arrow between panels is a "therefore": linear + quadratic therefore bowl therefore linear law therefore Riccati therefore one gain.

Recall Feynman: the whole walk in plain words

You're pushing a cart to a flag. First I taught you the two numbers that describe the cart (where, how fast) and the one thing you control (your push). The cart obeys a straight-line rule: your push feeds its speed, its speed feeds its position. Then I handed you a bill — one part for sitting far from the flag, one part for shoving hard — and I squared both so the bill is a smooth bowl that never dips below zero (that's $Q ⪰ 0$ , $R ≻ 0$ ). Standing anywhere, there's a price of your situation, and because everything is straight-and-squared that price is itself a smooth bowl, $x^{⊤} P x$ . Perfect play means the bill you pay each second exactly matches how fast your future bill drops as you slide down that bowl — I got that by splitting time into "the next instant" plus "everything after," which is Bellman's principle, and using the chain rule $d V / d t = \nabla V^{⊤} \overset{x}{˙}$ because the bowl is smooth. Freeze the state and your push-vs-cost is a simple U-curve; its bottom is the perfect push, and it turns out to be just "push in proportion to how far off you are," $u = - K x$ . Feed that best push back into the balance and the algebra coughs up one matrix equation — Riccati — whose good (bowl-up) solution $P$ gives you $K$ once and for all. Cheap fuel → hammer it; ruinously dear fuel → barely tap; free error → laze; runaway cart → still tamed. One recipe, computed once, works everywhere.

Active Recall

Recall Rebuild it without notes

Why must the cost-to-go be a quadratic bowl and not something with a cube in it? ::: Linear dynamics + quadratic cost can't create odd or higher-than-2 powers, and $V (0) = 0$ kills the constant/linear terms — only $x^{⊤} P x$ survives.
In Bellman's balance, what does $\nabla V^{⊤} (A x + B u)$ mean physically, and which rule produces it? ::: Slope of the bowl times your velocity ( $= d V / d t$ ) = how fast the remaining bill is falling; the chain rule produces it, and it needs $V$ differentiable.
Where does $R^{- 1}$ come from in $K$ ? ::: From setting the slope of the U-curve $u^{⊤} R u + 2 x^{⊤} P B u$ to zero: $2 R u + 2 B^{⊤} P x = 0$ .
Why is the Riccati equation nonlinear? ::: The steering-saving term $- P B R^{- 1} B^{⊤} P$ multiplies $P$ by itself.
What goes wrong with the negative Riccati root? ::: The bowl opens downward (not a cost) and the resulting gain destabilizes $A - B K$ .
What does $R \to \infty$ do, versus $R \to 0$ ? ::: $R \to \infty$ drives $K \to 0$ (limp, gentle) — the mirror of $R \to 0$ which drives $K \to \infty$ (hammering).