Visual walkthrough — Trust Region Policy Optimization (TRPO)

We will use only a 2-dimensional parameter space the whole time: $\theta =({\theta }_1,{\theta }_2)$. Two numbers. That is enough to see everything; the real thing just has millions of dimensions with the same geometry.

---

Step 1 — The map: what "parameter space" even means

WHAT. Our neural-network policy is controlled by numbers $\theta$. Pick two of them, ${\theta }_1$ and ${\theta }_2$, and treat them as a point on a flat map. Every point on this map = one policy = one way of acting.

WHY. We are about to talk about "taking a step", "a direction", "how far". Those words only make sense once you have a place to stand and a plane to walk on. So first we draw the ground.

PICTURE. The blue dot is our current policy ${\theta }_{\text{old}}$ (the one we already ran and collected data from). The whole plane is every other policy we could jump to.

---

Step 2 — The reward we want to climb (the surrogate hill)

WHAT. For each point on the map we can (approximately) ask: "how much better is this policy than ${\theta }_{\text{old}}$?" That number is the surrogate objective $L(\theta )$ from the parent note. Draw it as height above the map — a hill.

$$L(\theta )=\underset{\text{average over old data}}{\underbrace{{\mathbb{E}}_{s,a\sim {\pi }_{\text{old}}}}}[\underset{\text{how much more likely }\theta \text{ makes this action}}{\underbrace{\frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}}}\cdot \underset{\text{how good the action was}}{\underbrace{A(s,a)}}]$$

• $A(s,a)$ — the advantage (see Advantage Estimation (GAE)): positive = good action, negative = bad action.
• $\frac{{\pi }_{\theta }}{{\pi }_{\text{old}}}$ — the importance ratio (see Importance Sampling): reuses old samples for a new policy by re-weighting them.
• $\mathbb{E}$ — just "average". Nothing scary: add up over your data, divide by count.

WHY draw it as a hill? Because "make the policy better" now becomes the single visual instruction "walk uphill". Everything after this is how far uphill we dare to walk.

PICTURE. Contour lines = lines of equal height, like a topographic map. Closer lines = steeper slope. Our dot sits on the slope; up-and-to-the-right is higher.

---

Step 3 — The gradient: which way is uphill right here

WHAT. Stand on the dot. The direction of steepest uphill is a little arrow called the gradient, written $g$.

$$g={\nabla }_{\theta }L{|}_{{\theta }_{\text{old}}}=(\underset{\text{slope along }{\theta }_1}{\underbrace{\frac{\partial L}{\partial {\theta }_1}}},\underset{\text{slope along }{\theta }_2}{\underbrace{\frac{\partial L}{\partial {\theta }_2}}})$$

• ${\nabla }_{\theta }$ (read "nabla" / "gradient of") — the tool that answers "which way is up?". Why this tool? Because we want to increase $L$ and $\nabla$ literally packages the uphill direction and steepness into one arrow.
• $\frac{\partial L}{\partial {\theta }_1}$ — a partial derivative: "if I nudge only ${\theta }_1$ a hair, how fast does the height change?" The other coordinate held still.

WHY. This is exactly plain Policy Gradient Methods: it tells us the best local direction. If we followed only $g$ we'd be doing vanilla policy gradient — and hit the collapse problem the parent warned about.

PICTURE. The coral arrow $g$ points across the contour lines (steepest way to higher ground). It is the naive answer we are about to correct.

---

Step 4 — The catch: not all steps are equal (distribution vs. parameters)

WHAT. Moving a fixed distance in $\theta$ does not change the policy by a fixed amount. Some directions barely change how the policy behaves; others change it wildly for the same footstep.

WHY this matters. Our real safety rule is "don't change the policy's behaviour too much" — because a wildly changed policy collects wildly different data and can collapse. So a plain circle "$\Vert \Delta \theta \Vert \le r$" around our dot is the wrong notion of "close".

PICTURE. Two footsteps of equal length in $\theta$: the mint one moves along a "cheap" direction (policy barely changes), the coral one along an "expensive" direction (policy changes a lot). Same length, very different danger.

---

Step 5 — The right ruler: KL divergence and its ellipse

WHAT. KL Divergence $D_{\mathrm{KL}}$ measures how different two probability distributions are — here, the old policy vs. the candidate new policy at a state. TRPO's trust region says:

$${\bar{D}}_{\mathrm{KL}}({\theta }_{\text{old}},\theta )\le \delta$$

• ${\bar{D}}_{\mathrm{KL}}$ — average (over states) policy difference, in nats.
• $\delta$ — the size of our leash, e.g. $0.01$. Why a constraint, not a penalty? The parent's exact penalty coefficient $C$ is so huge it forces microscopic steps; a hard cap $\delta$ we can tune is practical.

Now the magic. For small steps, KL is well-approximated by a quadratic bowl:

$${\bar{D}}_{\mathrm{KL}}\approx \frac{1}{2}{s}^{\top }Fs,s=\theta -{\theta }_{\text{old}}$$

• $s$ — the step vector (where we go, relative to the dot).
• $F$ — the Fisher Information Matrix: the curvature of the KL bowl at ${\theta }_{\text{old}}$. Big curvature in a direction = KL grows fast there = that direction is "expensive".

Why quadratic and not linear? Because at ${\theta }_{\text{old}}$ the policy equals itself, so KL is $0$ and its slope is $0$ (you can't be "a little different from yourself"). The first non-zero term is the quadratic one — that is precisely why $F$, the second-order term, is the star.

PICTURE. The constraint $\frac{1}{2}{s}^{\top }Fs\le \delta$ is an ellipse (a squashed circle) around the dot. Cheap directions → the ellipse stretches far. Expensive directions → the ellipse is short. This ellipse is the trust region.

---

Step 6 — The problem, drawn: uphill arrow meets the ellipse

WHAT. Put Steps 3 and 5 together. We want to climb $L$ (go along $g$) but stay inside the KL ellipse. Formally:

$$\underset{s}{\max }\underset{\text{height gained}}{\underbrace{g^{\top }s}}\text{s.t.}\underset{\text{stay in the ellipse}}{\underbrace{\frac{1}{2}{s}^{\top }Fs\le \delta }}$$

• $g^{\top }s$ — a dot product: how much of our step $s$ points along the uphill arrow $g$. Bigger = more height gained. Why a dot product? It's exactly "projection of the step onto the uphill direction" — the linear stand-in for the hill near the dot.

WHY. We reach the highest point we can touch without leaving the ellipse. The answer will sit on the ellipse's boundary (using less of the leash would waste climbing room).

PICTURE. The coral $g$, the ellipse, and the winning point $s^{\star }$ where the highest contour of $g^{\top }s$ just kisses the ellipse.

---

Step 7 — Solving it: the natural gradient falls out

WHAT. At the touching point, the uphill arrow $g$ must point straight out of the ellipse (perpendicular to its boundary) — otherwise you could slide along the boundary and climb more. Writing "gradient of objective = $\lambda$ × gradient of constraint" (the Lagrange condition):

$$g=\lambda Fs⟹s=\frac{1}{\lambda }\underset{\text{natural gradient}}{\underbrace{F^{-1}g}}$$

• $F^{-1}g$ — the natural gradient: the raw uphill arrow $g$ bent by the ellipse's shape. It shortens expensive directions and lengthens cheap ones. This is the corrected direction we hunted for in Step 4.
• $\lambda$ — a scalar we still must pin down.

Fix $\lambda$ by demanding we sit exactly on the boundary, $\frac{1}{2}{s}^{\top }Fs=\delta$. Substituting $s=\frac{1}{\lambda }{F}^{-1}g$ and solving:

$$\boxed{s=\sqrt{\frac{2\delta }{g^{\top }{F}^{-1}g}}{F}^{-1}g}$$

Term by term:

• $F^{-1}g$ — direction (natural gradient).
• $g^{\top }{F}^{-1}g$ — the "length² in the ellipse's ruler" of that direction.
• $\sqrt{2\delta /(g^{\top }{F}^{-1}g)}$ — the exact rescale so the step lands on the KL boundary, using every drop of the $\delta$ leash and not one nat more.

PICTURE. The straightened, correctly-scaled arrow $s$ reaching the ellipse edge — the same arrow the parent note handed you, now earned.

---

Step 8 — Degenerate & edge cases (the map at its limits)

WHAT. What if inputs go extreme? Each has a picture-answer.

1. $g=0$ (flat hill). The uphill arrow vanishes → $s=0$. We stand still. Correct: if nothing improves locally, don't move.
2. $F$ isotropic, $F=cI$ (ellipse is a circle). Then $F^{-1}g=\frac{1}{c}g$: the natural gradient is just the ordinary gradient, rescaled. TRPO gracefully reduces to plain gradient ascent when all directions cost the same.
3. A very cheap direction ($F$ tiny along one axis). The ellipse stretches enormously there; $F^{-1}$ blows that component up — TRPO happily takes a big step where the policy barely reacts. This is the feature, not a bug.
4. Approximation lies (true KL > $\delta$). The quadratic bowl was only local; far out it can under-estimate. So TRPO backtracks: shrink $s\to {\beta }^{j}s$ until the measured KL $\le \delta$ and the surrogate truly rose.

PICTURE. Three mini-panels: circle-ellipse (case 2), stretched ellipse (case 3), and the backtracking dots walking back inside the true region (case 4).

---

The one-picture summary

Everything at once: the hill (surrogate $L$), the naive uphill arrow $g$, the KL ellipse (trust region shaped by $F$), the bent-and-scaled natural-gradient step $s$ landing on the boundary, and a backtracked point pulled inside the true region.

Recall Feynman: tell it back in plain words

We drew every policy as a dot on a map, and "how good it is" as a hill we want to climb. The steepest-up arrow is the ordinary gradient $g$ — but footsteps aren't fair: some directions change the policy's behaviour a lot, some barely at all. So we drew a special leash — an ellipse — that measures distance the honest way (how different the two policies act), using the Fisher matrix $F$ as the ruler. We then asked: climb as high as possible without leaving the ellipse. The answer bends $g$ into $F^{-1}g$ (the natural gradient) and stretches it to just touch the ellipse edge — that stretch factor is the square-root term. If the hill is flat we don't move; if all directions cost the same the ellipse is a circle and we're back to plain gradient ascent; and because our ellipse is only a local guess, we walk the step back until the real leash is satisfied. That's TRPO.

Recall

What geometric object is the TRPO trust region on the parameter map? ::: An ellipse $\frac{1}{2}{s}^{\top }Fs\le \delta$, shaped by the Fisher matrix $F$. Why is the KL constraint quadratic (uses $F$) instead of linear? ::: At ${\theta }_{\text{old}}$ the KL is $0$ and its slope is $0$; the first non-zero term is the second-order (curvature) one, given by $F$. What does multiplying $g$ by $F^{-1}$ accomplish geometrically? ::: It bends the raw uphill arrow to respect the ellipse — shrinking expensive directions, stretching cheap ones — giving the natural gradient. What does the scalar $\sqrt{2\delta /(g^{\top }{F}^{-1}g)}$ do? ::: Rescales the step so it lands exactly on the KL boundary, using all of the leash $\delta$. When does TRPO collapse to ordinary gradient ascent? ::: When $F=cI$ (all directions equally costly), so $F^{-1}g\propto g$.