Linear Quadratic Regulator (LQR) — Riccati equation, optimal gains

WHAT is the problem?

WHY this cost? It is a steel-man of engineering trade-offs. You cannot make error zero instantly — that would need infinite control. You cannot save all fuel — the system would drift. The integral of a quadratic form is the smoothest way to trade the two, and quadratics make the math solvable in closed form.

WHY quadratic (not $|x|$ or $x^4$)? Because with a linear plant + quadratic cost, the optimal control turns out to be linear in the state ($u=-Kx$). That is the magic marriage: Linear + Quadratic → Linear controller.

HOW: Derive the optimal control from first principles

We use dynamic programming / the value function (Hamilton–Jacobi–Bellman).

Step 1 — Guess the value function

Define the optimal "cost-to-go" from state $x$: $V(x) = \min_{u(\cdot)} \int_t^\infty (x^\top Q x + u^\top R u)\,dt.$

Why guess $V(x) = x^\top P x$ (with $P=P^\top \succeq 0$)? Because the cost is quadratic and the dynamics linear, the cost-to-go must also be a quadratic form — there is no linear or constant term (at $x=0$ cost is 0), and no higher powers survive linear dynamics.

Step 2 — Write the HJB equation

The Bellman principle says: at the optimum, the instantaneous cost plus the rate of change of cost-to-go is zero: $0 = \min_u \left[ x^\top Q x + u^\top R u + \nabla V^\top (Ax+Bu) \right].$ With $V = x^\top P x$, we have $\nabla V = 2Px$, so: $0 = \min_u \Big[ x^\top Q x + u^\top R u + 2x^\top P (Ax + Bu)\Big].$

Step 3 — Minimize over $u$ (this gives the gain)

Take $\partial/\partial u$ and set to 0: $\frac{\partial}{\partial u}\Big[u^\top R u + 2x^\top P B u\Big] = 2Ru + 2B^\top P x = 0.$

Why this step? The bracket is convex in $u$ (since $R\succ 0$), so the stationary point is the global minimum. Solving: $\boxed{\,u^* = -R^{-1}B^\top P\, x = -Kx,\qquad K = R^{-1}B^\top P\,}$

Step 4 — Substitute $u^*$ back → the Riccati equation

Put $u^* = -R^{-1}B^\top P x$ into the HJB bracket. Note $u^{*\top}Ru^* = x^\top P B R^{-1} B^\top P x$ and $2x^\top P B u^* = -2x^\top PBR^{-1}B^\top P x$. These combine to $-x^\top PBR^{-1}B^\top P x$: $0 = x^\top\Big(Q + A^\top P + P A - P B R^{-1} B^\top P\Big)x.$ This must hold for all $x$, so the matrix in brackets is zero:

Why is it "algebraic"? In the infinite-horizon steady state $\dot P = 0$, so the differential Riccati equation collapses to this algebraic one. Why nonlinear? The term $-PBR^{-1}B^\top P$ is quadratic in $P$.

Worked Example 1 — Scalar system (do it fully by hand)

Plant: $\dot x = a x + b u$ with $a,b$ scalars; cost weights $q>0,\ r>0$.

Riccati (scalars): $2ap - \dfrac{b^2}{r}p^2 + q = 0.$

Why? $A^\top P + PA = 2ap$; $PBR^{-1}B^\top P = \frac{b^2}{r}p^2$; add $Q=q$.

Solve the quadratic for $p$ (take the positive root, since $P\succeq0$): $p = \frac{ar + \sqrt{a^2 r^2 + q r b^2}}{b^2}.$ Why positive root? $V=px^2$ is a cost — must be non-negative; only the $+$ root gives $p>0$.

Gain: $K = \dfrac{b}{r}\,p$. Closed loop: $\dot x = (a - bK)x$.

Check stability: $a - bK = a - \frac{b^2}{r}p < 0$ always, so the controller always stabilizes even an unstable open-loop plant ($a>0$).

Numbers: $a=1$ (unstable!), $b=1$, $q=1$, $r=1$: $p = 1 + \sqrt{1+1} = 1+\sqrt2 \approx 2.414,\quad K = p \approx 2.414,\quad a-bK \approx -1.414 <0.$ Stable. ✔

Worked Example 2 — Double integrator (a real GNC case)

A satellite axis: position $x_1$, velocity $x_2$, control acceleration $u$: $A=\begin{bmatrix}0&1\\0&0\end{bmatrix},\ B=\begin{bmatrix}0\\1\end{bmatrix},\ Q=\begin{bmatrix}1&0\\0&0\end{bmatrix},\ R=[\rho].$

Let $P=\begin{bmatrix}p_1&p_2\\p_2&p_3\end{bmatrix}$. Expand CARE. Key equations:

• $(1,1):\ 1 - p_2^2/\rho = 0 \Rightarrow p_2 = \sqrt{\rho}.$
• $(1,2):\ p_1 - p_2 p_3/\rho = 0 \Rightarrow p_1 = p_2 p_3/\rho.$
• $(2,2):\ 2p_2 - p_3^2/\rho = 0 \Rightarrow p_3 = \sqrt{2\rho\,p_2} = \sqrt{2}\,\rho^{3/4}.$

Gain: $K = R^{-1}B^\top P = \frac{1}{\rho}[\,p_2,\ p_3\,] = \big[\,\tfrac{1}{\sqrt\rho},\ \tfrac{\sqrt2}{\rho^{1/4}}\big].$

Why this matters: $K=[k_p,\ k_d]$ is exactly a PD controller! The first gain acts on position (spring), the second on velocity (damper). LQR derived PD control optimally. With $\rho=1$: $K=[1,\ \sqrt2]$, giving closed-loop poles at $-\tfrac{1}{\sqrt2}\pm j\tfrac{1}{\sqrt2}$ — a nicely damped response (damping ratio $=1/\sqrt2$).

Active Recall

Connections

• Hamilton-Jacobi-Bellman Equation — LQR is its cleanest closed-form solution.
• Controllability and Observability — needed for a valid stabilizing $P$.
• Pole Placement — alternative to LQR; LQR optimally places poles.
• Kalman Filter — dual Riccati equation; combine → LQG Control.
• PID Controllers — LQR generalizes/derives PD gains optimally.
• Lyapunov Stability — $V=x^\top Px$ is a Lyapunov function for the closed loop.
• State-Space Representation — the $\dot x = Ax+Bu$ foundation.

Concept Map

Hinglish (regional understanding)