Linear Quadratic Regulator (LQR) — Riccati equation, optimal gains: Visual walkthrough

3.5.35 · D2 · Physics › Guidance, Navigation & Control (GNC) › Linear Quadratic Regulator (LQR) — Riccati equation, optimal

Step 0 — Do words: state aur control (koi math se pehle)

PICTURE. Ek cart position $x_{1}$ par baithi hai aur speed $x_{2}$ se right ja rahi hai. Tumhara haath isko force $u$ se push kar raha hai. Target hai origin — woh chhota sa flag $0$ par. Yahi is problem ki puri duniya hai.

Figure — Linear Quadratic Regulator (LQR) — Riccati equation, optimal gains

$u$ wala arrow red hai kyunki yahi ek cheez tumhare command mein hai. Hum ab se jo bhi karte hain woh ek hi sawaal ka jawab deta hai: kitni takat se, aur kis direction mein, woh red arrow push kare, yeh dekhte hue ki cart abhi kahan hai?

Step 1 — Machine linear hai: $\overset{x}{˙} = A x + B u$

LINEAR kyun important hai? Kyunki ek seedha rule hi ek aisa akela kind hai jise hum poori tarah closed form mein solve kar sakte hain. LQR ka poora fayda isi par tika hai.

PICTURE. State ek plane mein ek point ki tarah rehti hai (position across, velocity up). Har point par ek arrow dikhata hai ki machine apne aap kis direction mein drift karti hai ( $A x$ ). Tumhara push $B u$ us arrow ko thoda sa nudge karta hai.

Black arrows dekho: bina push ke, ek cart jo right par hai aur right ja rahi hai ( $x_{1} > 0, x_{2} > 0$ ) aur right drift karti hai — woh bhaag jaati hai. Red overlay dikhata hai ki $u$ ki ek choice un arrows ko flag ki taraf wapas kaise mod deti hai.

Step 2 — Jo bill hum bharte hain: quadratic cost $J$

Squares kyun, absolute values ya fourth powers kyun nahi? Do reasons hain, ek practical, ek magical.

Practical: ek square smooth hota hai aur zero par koi sharp corner nahi hota, isliye machine kabhi chatter nahi karti.
Magical: ek linear machine ko ek squared bill do to controller khud linear nikalta hai. Yahi coincidence reason hai ki LQR beautiful hai. Hum yeh neeche prove karte hain.

PICTURE. Ek squared penalty ek bowl hai. Neeche hona (on-target, no push) kuch nahi costa; wall climb karna zyada se zyada cost karta hai, tezi se tezi.

Orange bowl $x$ mein error bill hai; teal bowl $u$ mein effort bill hai. Kyunki $Q ⪰ 0$ aur $R ≻ 0$ hain, dono bowls upar khulti hain — woh kabhi zameen ke neeche nahi jaatein. LQR woh path dhundhta hai jis par total bowl-height hamesha sabse chhoti rahe.

Step 3 — "Cost-to-go" bhi ek bowl hai: $V (x) = x^{⊤} P x$

$V$ ek bowl (quadratic $x^{⊤} P x$ ) kyun honi chahiye? Teen clues, har ek forced:

Target par, perfectly khelne par zero cost, isliye $V (0) = 0$ — bowl origin par zameen ko touch karta hai.
Target ke left aur right symmetric mirror situations hain, isliye koi odd/linear term survive nahi karta.
Ek seedhi machine ek squared bill se cubes ya quartics nahi bana sakti — shape ek saaf bowl rehti hai.

Isliye $V (x) = x^{⊤} P x$ , jahan $P$ numbers ka ek unknown symmetric grid hai — woh ek cheez jo humein dhundhnee hai. Yeh choice exactly value-function guess hai.

PICTURE. Bowl $V$ state plane ke upar baithi hai. Har direction mein uski steepness $P$ set karta hai. $P$ dhundho aur har situation ki kimat pata chal jaati hai — aur, jaise hum dekhenge, perfect push bhi.

Step 4 — Bellman ka balance: instant cost + downhill speed = 0

Woh last line HJB equation hai. "Instant cost future bill ke fall ko cancel karta hai" ka slogan ab derived hai, sirf assert nahi.

PICTURE. Bowl ki andar ki wall par khade ho. Upar jaata slope arrow $\nabla V$ hai. Tumhari motion $\overset{x}{˙} = A x + B u$ ek aur arrow hai. Unka alignment (chain rule se) bata deta hai ki future bill drop ho raha hai ya nahi (downhill ja rahe ho) aur kitni tezi se.

Step 5 — Knob ko bottom par ghumaao: $u$ ke liye solve karo

Bracket ke andar, sirf ek cheez hai jo tum control karte ho — $u$ . Har woh cheez jisme $u$ hai: $a bowl in u u^{⊤} R u + a tilt 2 x^{⊤} P B u$ Yeh ek bowl hai (kyunki $R ≻ 0$ , matlab $R$ ek genuine upward bowl hai) jise state tilt kar rahi hai. Ek tilted bowl ka exactly ek sabse neecha point hota hai. Slope ko $u$ mein zero set karke wahan slide karo: $2 R u + 2 B^{⊤} P x = 0 ⟹ u^{*} = - R^{- 1} B^{⊤} P x = - K x$

$R^{- 1}$ kyun aata hai, aur $R$ strictly positive kyun hona chahiye? Agar effort free hoti ( $R \to 0$ ) to tum infinitely hard shove karte — $u$ mein bowl flat hoti aur koi bottom nahi hota. Ek real, strictly-positive $R$ bowl ko ek genuine bottom deta hai, isliye $R^{- 1}$ exist karta hai aur ek finite best push exist karta hai. Yeh feedback law $u = - K x$ hai, ab derived, guess nahi.

PICTURE. Ek fixed state ke liye, tumhari push ke function ki tarah cost ek parabola hai. Uska vertex perfect push hai. Steeper parabola (bada $R$ ) → vertex zero ke karib → gentler push.

Step 6 — Best push wapas daalo: Riccati equation appear hoti hai

Ab hum $u^{*} = - R^{- 1} B^{⊤} P x$ ko Bellman ke balance mein substitute karte hain aur carefully terms collect karte hain. Shuru karo yahan se: $0 = x^{⊤} Q x + u^{* ⊤} R u^{*} + 2 x^{⊤} P (A x + B u^{*}) .$

Term (i) — drift $2 x^{⊤} P A x$ symmetric pair $A^{⊤} P + P A$ ban jaata hai. Number $x^{⊤} P A x$ ek $1 \times 1$ scalar hai, isliye yeh apne transpose ke barabar hai: $x^{⊤} P A x = (x^{⊤} P A x)^{⊤} = x^{⊤} A^{⊤} P^{⊤} x = x^{⊤} A^{⊤} P x$ ( $P = P^{⊤}$ use karte hue). Do identical copies ka average lene par, $2 x^{⊤} P A x = x^{⊤} P A x + x^{⊤} A^{⊤} P x = x^{⊤} (A^{⊤} P + P A) x .$ Isliye akela lop-sided $P A$ symmetric sum $A^{⊤} P + P A$ ki tarah rewrite hota hai — "drift up the bowl" term.

Term (ii) — $u^{*}$ ke do pieces merge hokar $- P B R^{- 1} B^{⊤} P$ bante hain. $u^{*} = - R^{- 1} B^{⊤} P x$ ko effort cost aur tilt mein alag alag plug karo: $u^{* ⊤} R u^{*} = x^{⊤} P B = R^{- 1} R^{- 1} R R^{- 1} B^{⊤} P x = + x^{⊤} P B R^{- 1} B^{⊤} P x,$ $2 x^{⊤} P B u^{*} = - 2 x^{⊤} P B R^{- 1} B^{⊤} P x .$ Jodo: $(+ 1 - 2) = - 1$ , isliye milke yeh dete hain: $- x^{⊤} P B R^{- 1} B^{⊤} P x .$ Yeh exactly woh hai jo steering bachata hai — optimizer ek reduction kharidta hai, isliye minus sign.

Collect karo (i) + (ii) + error term $x^{⊤} Q x$ : $0 = x^{⊤} (drift up the bowl A^{⊤} P + P A what steering saves - P B R^{- 1} B^{⊤} P + running error price Q) x .$ Yeh har state $x$ par hold karna chahiye. Ek quadratic form jo sabhi $x$ ke liye zero ho, sirf flat zero matrix se aa sakta hai, isliye beech ka poora grid vanish hona chahiye:

Nonlinear kyun hai? Saving term mein $P$ ko $P$ se multiply kiya gaya hai — ek square. Yeh ek akela squared term hi hai jo ise ek plain linear equation ki jagah ek Riccati equation banata hai.

PICTURE. Equation grids ka ek see-saw hai: drift ( $A^{⊤} P + P A$ ) aur error price ( $Q$ ) $P$ ko upar dhakelte hain; steering saving ( $- P B R^{- 1} B^{⊤} P$ ) ise neeche kheenchti hai. Balance point woh $P$ hai jo hum chahte hain.

Step 7 — Degenerate aur edge cases (koi gap mat chhodna)

Definition Do words jinpar hum rely karte hain: controllable aur observable

Edge cases se pehle, do health checks jaano jo decide karte hain ki ek acha $P$ exist bhi karta hai ya nahi (poori detail Controllability and Observability mein):

$(A, B)$ controllable ka matlab: tumhara push eventually state ke har direction tak pahunch sakta hai. Agar machine ka koi mode $u$ ke liye bahra ho, to koi gain use steer nahi kar sakta — wahan LQR bekaaar hoga. Socho: kya red arrow, waqt ke saath, cart ko har zaruri tarike se move kar sakta hai?
$(A, Q)$ observable ka matlab: machine ki har unstable direction actually error bill mein dikhti hai. $Q$ koi bhi matrix hai jisme $Q^{⊤} Q = Q$ ho; usse observable maangna matlab maangna ki koi runaway mode "invisible" na ho cost ke liye. Agar koi mode kuch nahi costa lekin blow up karta hai, to optimizer notice nahi karta aur plan diverge ho jaata hai.

Saath mein yeh do guarantee karte hain ki CARE ka ek unique stabilizing $P ⪰ 0$ hai.

PICTURE. Teen miniature bowls side by side: sasta fuel (deep narrow bowl, sharp gain), free error (flat wide bowl, lazy gain), aur correct-vs-wrong root (bowl up vs bowl down).

Ek-picture summary

Neeche ke paanch panels left-to-right padho — yeh exactly Steps 1 → 6 hain jo ek strip mein compress ho gaye hain:

Panel 1 (Step 1) — Linear machine $\overset{x}{˙} = A x + B u$ : seedha cause-and-effect rule.
Panel 2 (Step 2) — Quadratic bill $J = \int_{0}^{\infty} (x^{⊤} Q x + u^{⊤} R u) d t$ : wo do upward bowls jo hum bharte hain.
Panel 3 (Step 3) — Cost-to-go bowl $V = x^{⊤} P x$ : smooth quadratic hona forced hai; $P$ dhundho.
Panel 4 (Steps 4–5) — Bellman's knob $u = - K x$ : U-curve ko uske vertex par ghumaane se ek linear law milta hai.
Panel 5 (Step 6) — Riccati → $P$ : back-substitution force karta hai $A^{⊤} P + P A - P B R^{- 1} B^{⊤} P + Q = 0$ , jiska stabilizing root tumhe $K = R^{- 1} B^{⊤} P$ deta hai.

Panels ke beech har black arrow ek "therefore" hai: linear + quadratic therefore bowl therefore linear law therefore Riccati therefore ek gain.

Recall Feynman: poori walk plain words mein

Tum ek cart ko ek flag tak push kar rahe ho. Pehle maine tumhe do numbers sikhaye jo cart describe karte hain (kahan, kitni tezi se) aur woh ek cheez jo tum control karte ho (tumhara push). Cart ek seedha-line rule follow karti hai: tumhara push uski speed ko feed karta hai, uski speed uski position ko. Phir maine tumhe ek bill diya — ek part flag se door baithne ke liye, ek part zyada dhakka dene ke liye — aur dono ko square kiya taaki bill ek smooth bowl ho jo kabhi zero se neeche na jaaye (yahi $Q ⪰ 0$ , $R ≻ 0$ hai). Kahin bhi khade ho, wahan ek tumhari situation ki kimat hai, aur kyunki sab kuch straight-and-squared hai woh kimat khud ek smooth bowl hai, $x^{⊤} P x$ . Perfect play ka matlab hai har second jo bill tum bharte ho woh exactly match kare ki kitni tezi se tumhara future bill gir raha hai jab tum us bowl se slide karte ho — mujhe yeh Bellman's principle use karke mila, time ko "agla instant" aur "uske baad sab kuch" mein tod ke, aur chain rule $d V / d t = \nabla V^{⊤} \overset{x}{˙}$ use karke kyunki bowl smooth hai. State freeze karo aur tumhari push-vs-cost ek simple U-curve hai; uska bottom perfect push hai, aur woh nikal ke aata hai "jitna off ho uske proportion mein push karo," $u = - K x$ . Woh best push wapas balance mein daalo aur algebra ek matrix equation ugalta hai — Riccati — jiska acha (bowl-up) solution $P$ tumhe $K$ ek baar aur hamesha ke liye deta hai. Sasta fuel → hammer karo; bahut mehenga fuel → mushkil se tap karo; free error → aaram karo; runaway cart → phir bhi tame. Ek recipe, ek baar compute karo, har jagah kaam kare.

Active Recall

Recall Bina notes ke rebuild karo

Cost-to-go quadratic bowl kyun honi chahiye, kuch cube wale expression kyun nahi? ::: Linear dynamics + quadratic cost odd ya 2-se-zyada powers create nahi kar sakti, aur $V (0) = 0$ constant/linear terms kill karta hai — sirf $x^{⊤} P x$ survive karta hai.
Bellman ke balance mein, $\nabla V^{⊤} (A x + B u)$ physically kya matlab rakhta hai, aur kaunsa rule ise produce karta hai? ::: Bowl ka slope times tumhari velocity ( $= d V / d t$ ) = kitni tezi se remaining bill gir raha hai; chain rule ise produce karta hai, aur iske liye $V$ differentiable hona chahiye.
$K$ mein $R^{- 1}$ kahan se aata hai? ::: U-curve $u^{⊤} R u + 2 x^{⊤} P B u$ ka slope zero set karne se: $2 R u + 2 B^{⊤} P x = 0$ .
Riccati equation nonlinear kyun hai? ::: Steering-saving term $- P B R^{- 1} B^{⊤} P$ $P$ ko khud se multiply karta hai.
Negative Riccati root se kya galat hota hai? ::: Bowl neeche ki taraf khulti hai (cost nahi) aur resulting gain $A - B K$ ko destabilize karta hai.
$R \to \infty$ kya karta hai, versus $R \to 0$ ? ::: $R \to \infty$ se $K \to 0$ hota hai (limp, gentle) — yeh $R \to 0$ ka mirror hai jo $K \to \infty$ drive karta hai (hammering).