Trust Region Policy Optimization (TRPO): Worked examples

This page is a drill. We take the parent TRPO note and hit every case the TRPO step can throw at you: positive and negative advantage, easy diagonal Fisher matrices, nasty non-diagonal ones, degenerate inputs (zero gradient, singular Fisher, tiny trust region), the limit of shrinking $δ$ , a real-world word problem, and an exam-style twist. Every number is verified at the bottom.

Nothing here assumes you can already crunch the formula. We re-derive the tools as we need them.

The three earned symbols

Before any example, we pin down the three symbols the TRPO step is made of, so nothing is used before it is defined.

Now the last earned symbol. The expression $g^{⊤} F^{- 1} g$ appears in every example, so let us name what it is.

The scenario matrix

Each row is a "cell" we must cover with at least one worked example.

Cell	What varies	Why it can trip you up	Covered in
A	Diagonal Fisher, positive advantage	The "textbook" easy case	Ex 1
B	Advantage sign flips ( $A < 0$ )	Objective should push probability down	Ex 2
C	Non-diagonal (correlated) Fisher	You must actually invert a $2 \times 2$	Ex 3
D	Degenerate: zero gradient $g = 0$	Step must be zero; formula has $0/0$ risk	Ex 4
D'	Degenerate: singular (non-invertible) $F$	$F^{- 1}$ does not exist at all	Ex 4b
E	Very small $δ$ (limiting behaviour)	Step scales like $δ$ , not $δ$	Ex 5
F	Line search rejects a step	The math step is only a proposal	Ex 6
G	Real-world word problem	Translate a scenario into $g$ , $F$ , $δ$	Ex 7
H	Exam twist: KL of two Gaussians	KL is not symmetric; direction matters	Ex 8

Ex 1 — Cell A: diagonal Fisher, positive advantage

Worked example The clean case

$g = [30]$ , $F = [1009]$ , $δ = 0.005$ . Find the step $s$ .

Forecast: guess before computing — will the step point purely along axis 1, or get tilted? (Since $g$ has no component on axis 2, it should stay on axis 1.)

Invert $F$ . A diagonal matrix inverts by flipping each diagonal entry: $F^{- 1} = [10 0 1/9]$ . Why this step? We need the natural-gradient direction $F^{- 1} g$ , and inverting a diagonal is the cheapest possible inversion.
Natural gradient $F^{- 1} g = [1 \cdot 3 (1/9) \cdot 0] = [30]$ . Why this step? This is the corrected direction — Fisher tells us axis 2 is "steep" in KL, but since $g$ has nothing there, nothing changes.
Compute $q = g^{⊤} F^{- 1} g = 3 \cdot 3 + 0 = 9$ . Why this step? $q$ sets the rescaling; a dot product because we want the length of the step measured by the Fisher geometry.
Scale $= 2 δ / q = 2 \cdot 0.005/9 = 0.001111 \dots \approx 0.03333$ . Why this step? This is the exact factor that puts us on the KL wall $\frac{1}{2} s^{⊤} F s = δ$ .
Step $s = 0.03333 \cdot [30] = [0.1 0]$ . Why this step? Multiplying the natural-gradient direction by the scalar is the final assembly — direction times how-far gives the actual parameter update to apply.

Verify: $\frac{1}{2} s^{⊤} F s = \frac{1}{2} (0. 1^{2} \cdot 1 + 0) = \frac{1}{2} (0.01) = 0.005 = δ .$ ✓ We landed exactly on the boundary, and the step stayed on axis 1 as forecast.

Figure — Trust Region Policy Optimization (TRPO)

What the figure shows: the teal ellipse is the KL wall $\frac{1}{2} s^{⊤} F s = δ$ — every point on it costs exactly $δ$ of KL. Because axis 2 has high Fisher ( $9$ ), the ellipse is squashed vertically (you can move far along axis 1 but only a tiny bit along axis 2 for the same KL). The plum arrow is the raw gradient direction, the orange arrow is our step $s = (0.1, 0)$ , and it touches the ellipse right on axis 1 — visually confirming we spent the whole KL budget in the cheap direction.

Ex 2 — Cell B: negative advantage flips the direction

Worked example Bad action, pushed down

Same $F = [1009]$ , $δ = 0.005$ , but now the raw policy gradient comes out $g = [- 3 0]$ because the sampled action had advantage $A = - 3$ (see Advantage Estimation (GAE)).

Forecast: should the step reverse compared to Ex 1?

Where did $g$ come from? The surrogate contribution of one sample is $\frac{π _{θ}}{π _{old}} A$ (see Importance Sampling). With $A < 0$ , raising that action's probability lowers the objective, so the gradient points to lower its probability. That flipped sign is baked into $g$ . Why this step? To see that the sign of $A$ propagates directly into $g$ ; TRPO's machinery downstream is sign-agnostic.
Natural gradient $F^{- 1} g = [(1) (- 3) (1/9) (0)] = [- 3 0]$ . Why this step? We still bend the gradient through $F^{- 1}$ to get the natural-gradient direction; the negative sign simply rides along unchanged because $F^{- 1}$ is a linear map.
$q = g^{⊤} F^{- 1} g = (- 3) (- 3) = 9$ . Why this step? Note $q$ is still positive — it is a squared length, so it cannot be negative no matter the sign of $g$ . This guarantees the square root is real.
Scale $= 2 \cdot 0.005/9 \approx 0.03333$ (same magnitude as Ex 1).
Step $s = [- 0.1 0]$ — same length, opposite direction.

Verify: $\frac{1}{2} s^{⊤} F s = \frac{1}{2} (0. 1^{2}) = 0.005 = δ .$ ✓ Sign flipped, KL budget identical — exactly what we want: a bad action gets its probability pushed down by the same "distance" a good one gets pushed up.

Ex 3 — Cell C: correlated (non-diagonal) Fisher

Worked example The Fisher couples the two parameters

$g = [11]$ , $F = [2112]$ , $δ = 0.01$ .

Forecast: the raw gradient points diagonally at 45°. Will the natural gradient still point at 45°, or get bent?

Invert the $2 \times 2$ . For $[a c b d]$ , the inverse is $\frac{1}{a d - b c} [d - c - b a]$ . Here $a d - b c = 2 \cdot 2 - 1 \cdot 1 = 3$ , so $F^{- 1} = \frac{1}{3} [2 - 1 - 1 2]$ . Why this step? $F$ is not diagonal, so we cannot cheat — we need the real inverse formula. The determinant $3$ measures how "invertible" $F$ is (nonzero = safe).
Natural gradient $F^{- 1} g = \frac{1}{3} [2 - 1 - 1 + 2] = \frac{1}{3} [11] = [1/3 1/3]$ . Why this step? Because $g$ happens to be an eigenvector of $F$ (equal components), the direction survives at 45° but the magnitude is squashed by the shared curvature. In general the direction bends — this is the whole reason natural gradient differs from vanilla gradient.
$q = g^{⊤} F^{- 1} g = 1 \cdot \frac{1}{3} + 1 \cdot \frac{1}{3} = \frac{2}{3}$ . Why this step? We need the KL-length of the natural gradient before rescaling; the dot product measures it in the Fisher geometry, and it feeds the scale in the next step.
Scale $= 2 \cdot 0.01/ (2/3) = 0.02 \cdot 1.5 = 0.03 \approx 0.17321$ . Why this step? This is the exact rescaling that puts the step on the KL wall $\frac{1}{2} s^{⊤} F s = δ$ ; we divide the budget $2 δ$ by $q$ and take the root because KL is quadratic in step length.
Step $s = 0.17321 \cdot [1/3 1/3] \approx [0.05774 0.05774]$ . Why this step? Final assembly: direction (natural gradient) times how-far (scale) gives the parameter update to actually apply.

Verify: $\frac{1}{2} s^{⊤} F s$ . First $F s = [2112] [0.05774 0.05774] = [0.17321 0.17321]$ , then $s^{⊤} F s = 2 (0.05774) (0.17321) = 0.02$ , halved $= 0.01 = δ .$ ✓

What the figure shows: the teal ellipse is again the KL wall, now tilted because the off-diagonal $1$ in $F$ correlates the two parameters. The plum arrow is the raw gradient (pure 45°); the orange arrow is the natural-gradient step, which here still points at 45° (since $g$ is an eigenvector) but is shortened by the curvature. The picture makes the lesson concrete: Fisher reshapes both the wall and how far you may travel.

Ex 4 — Cell D: degenerate input, zero gradient

Worked example What if

$g = 0$ ? $g = [00]$ , any invertible $F$ , any $δ$ .

Forecast: should we move at all?

Compute $q = g^{⊤} F^{- 1} g = 0$ . Why this step? No gradient means no preferred direction. But now $q = 0$ sits in the denominator $2 δ / q$ — that's a divide-by-zero.
Resolve the $0/0$ carefully. The step is $s = 2 δ / q F^{- 1} g$ . The scalar blows up like $1/ q$ , but $F^{- 1} g = 0$ shrinks to zero faster. Write $s = 2 δ \cdot \frac{F ^{- 1} g}{g ^{⊤} F ^{- 1} g}$ ; as $g \to 0$ along any fixed direction, both numerator and denominator scale linearly in the magnitude of $g$ , so the ratio approaches a finite direction and $s \to 0$ in length. Why this step? We must not blindly plug in — we reason about the limit to avoid a NaN in code.
Conclusion: $s = 0$ . TRPO takes no step. In practice code guards this with if q < 1e-8: skip update.

Verify: $\frac{1}{2} s^{⊤} F s = 0 \leq δ$ for any $δ \geq 0$ . ✓ A zero gradient is a (local) optimum of the surrogate — standing still is correct.

Ex 4b — Cell D': degenerate input, singular Fisher

Worked example What if

$F$ cannot be inverted? $g = [11]$ , $F = [1111]$ , $δ = 0.01$ . Notice both rows of $F$ are identical.

Forecast: the formula needs $F^{- 1}$ — does it even exist here?

Check invertibility via the determinant. $det F = 1 \cdot 1 - 1 \cdot 1 = 0$ . Why this step? A matrix is invertible iff its determinant is nonzero. A zero determinant means $F$ squashes some direction to zero length — two parameters that move the policy identically, so KL cannot tell them apart.
What breaks: $F^{- 1}$ does not exist, so $F^{- 1} g$ is undefined and the raw formula is meaningless. Geometrically, the "KL ellipse" degenerates into an infinite strip — one direction costs zero KL, so the trust region is unbounded there. Why this step? To see that the degeneracy is not a rounding glitch but a genuine modelling issue: the parameterization is redundant.
The fix TRPO actually uses. Solve $F x = g$ with conjugate gradient and add a small damping term: use $F + η I$ (with $η$ tiny, e.g. $1 0^{- 3}$ ) instead of $F$ . Then $det (F + η I) = (1 + η)^{2} - 1 = 2 η + η^{2} > 0$ , so it is invertible and the step is well-defined and finite. Why this step? Damping restores a bounded trust region in the squashed direction; it is standard in every TRPO implementation exactly for this case.

Verify: $det F = 0$ (singular, no inverse); $det (F + η I) = 2 η + η^{2} > 0$ for $η > 0$ (damped matrix is invertible). ✓ Never assume $F$ is invertible — always damp.

Ex 5 — Cell E: the $δ \to 0$ limiting law

Worked example How does step length shrink as we tighten the trust region?

Fix $g = [20]$ , $F = [1001]$ . Compute $∥ s ∥$ for $δ = 0.02, 0.005, 0.00125$ (each a quarter of the previous).

Forecast: if $δ$ drops to a quarter, does $∥ s ∥$ drop to a quarter, or to a half?

General length. With $F = I$ , $F^{- 1} g = g$ , $q = g^{⊤} g = 4$ , so $s = 2 δ /4 \cdot g$ and $∥ s ∥ = 2 δ /4 \cdot ∥ g ∥ = 2 δ /4 \cdot 2 = 2 δ$ . Why this step? We want the scaling law, so we isolate $∥ s ∥$ (Euclidean length) as a function of $δ$ : it is $2 δ$ — a square-root law.
Plug values. $δ = 0.02 \Rightarrow ∥ s ∥ = 0.04 = 0.2$ . $δ = 0.005 \Rightarrow 0.01 = 0.1$ . $δ = 0.00125 \Rightarrow 0.0025 = 0.05$ . Why this step? To confirm the pattern numerically.
Read the law. Each time $δ$ becomes a quarter, $∥ s ∥$ becomes a half ( $0.2 \to 0.1 \to 0.05$ ). Because $1/4 = 1/2$ . Why this step? This is the key intuition behind KL-as-a-speed-limit: halving the step costs a quarter of the KL budget, so KL is a quadratic cost in step length near the old policy.

Verify: $2 \cdot 0.02 = 0.2$ , $2 \cdot 0.005 = 0.1$ , $2 \cdot 0.00125 = 0.05$ . ✓ Square-root scaling confirmed.

What the figure shows: the teal curve plots step length $∥ s ∥ = 2 δ$ against the trust-region size $δ$ . The three marked dots ( $δ = 0.02, 0.005, 0.00125$ ) sit at heights $0.2, 0.1, 0.05$ — each dot half the height of the previous even though $δ$ dropped to a quarter. The curved (not straight) shape is the visual signature of the square-root law: tightening the trust region buys you safety cheaply in step length.

Ex 6 — Cell F: the line search rejects the proposal

Worked example When the quadratic approximation lies

The math step from Ex 1 proposes $s = [0.1 0]$ with target $δ = 0.005$ . Suppose we evaluate the true KL at the new parameters and get $\overset{ˉ}{D}_{KL}^{true} = 0.0092$ — nearly double the target. TRPO uses backtracking factor $β = 0.8$ . How many backtracks until the true KL is within budget, assuming true KL scales quadratically with the step scalar $α$ ?

Forecast: we need to shrink the step. Will one backtrack suffice?

Model the true KL. Backtracking replaces $s \to β^{j} s$ . Since KL is (to leading order) quadratic in step length, $\overset{ˉ}{D}_{KL} (β^{j} s) \approx β^{2 j} \cdot 0.0092$ . Why this step? From Ex 5 we learned KL cost is quadratic in step scale — so shrinking by $β$ multiplies KL by $β^{2}$ .
Try $j = 1$ : $0. 8^{2} \cdot 0.0092 = 0.64 \cdot 0.0092 = 0.005888 > 0.005$ . Still over budget — reject.
Try $j = 2$ : $0. 8^{4} \cdot 0.0092 = 0.4096 \cdot 0.0092 \approx 0.003768 \leq 0.005$ . Accept. Why this step? We also require the surrogate to have actually improved; here we assume it did, so $j = 2$ is the accepted step.
Accepted step $s_{final} = 0. 8^{2} \cdot [0.1 0] = [0.064 0]$ .

Verify: $0. 8^{2} \cdot 0.0092 = 0.005888$ (reject, $> 0.005$ ), $0. 8^{4} \cdot 0.0092 = 0.00376832$ (accept, $\leq 0.005$ ). ✓ The line search is why TRPO is safe even when the local model is optimistic.

Ex 7 — Cell G: real-world word problem

Worked example A warehouse robot's policy update

A pick-and-place robot's policy has two tunable parameters: $θ_{1}$ = "grip force bias", $θ_{2}$ = "arm-speed bias". After one batch we estimate the policy gradient $g = [0.6 0.8]$ (raising both helps) and a diagonal Fisher $F = [4001]$ — grip force is a high-curvature parameter (small changes swing the action distribution a lot), speed is low-curvature. Safety team sets $δ = 0.01$ . Find the update.

Forecast: raw gradient favours speed slightly ( $0.8 > 0.6$ ). After Fisher correction, which parameter moves more?

Invert Fisher: $F^{- 1} = [1/4 0 01]$ . Why this step? High Fisher on grip means "be careful here", so its inverse damps grip moves — exactly the geometry Fisher encodes.
Natural gradient $F^{- 1} g = [0.6/4 0.8] = [0.15 0.8]$ . Why this step? Now speed dominates strongly — grip was throttled because it's dangerous curvature-wise, even though its raw gradient was sizeable. Forecast answer: speed moves more.
$q = g^{⊤} F^{- 1} g = 0.6 \cdot 0.15 + 0.8 \cdot 0.8 = 0.09 + 0.64 = 0.73$ .
Scale $= 2 \cdot 0.01/0.73 = 0.0273973 \approx 0.16552$ .
Step $s = 0.16552 \cdot [0.15 0.8] \approx [0.02483 0.13242]$ .

Verify: $\frac{1}{2} s^{⊤} F s = \frac{1}{2} (4 \cdot 0.0248 3^{2} + 1 \cdot 0.1324 2^{2}) = \frac{1}{2} (0.002466 + 0.017535) = \frac{1}{2} (0.020001) \approx 0.01 = δ .$ ✓ The safe, curvature-aware update nudges the risky grip parameter only a little while advancing speed boldly.

Ex 8 — Cell H: exam twist — KL of two Gaussians (direction matters)

Worked example Compute the constraint value for a Gaussian policy

A continuous-control policy outputs a Gaussian action $N (μ, σ^{2})$ . Old policy: $μ_{old} = 0, σ = 1$ . New policy: $μ_{new} = 0.3, σ = 1$ (same spread). Is the KL step $\overset{ˉ}{D}_{KL} (old ∥ new) \leq δ = 0.05$ ?

Forecast: shifting the mean by $0.3$ standard deviations — small or large in KL?

Recall the formula. For two 1-D Gaussians with equal variance $σ^{2}$ , the KL from old to new simplifies to $D_{KL} = \frac{( μ _{new} - μ _{old} ) ^{2}}{2 σ ^{2}}$ . Why this step? The general Gaussian KL has variance and log terms, but equal $σ$ kills them, leaving a pure mean-shift penalty — quadratic in the shift, echoing Ex 5's "KL is quadratic near the old policy".
Plug in: $D_{KL} = \frac{( 0.3 - 0 ) ^{2}}{2 \cdot 1} = \frac{0.09}{2} = 0.045$ . Why this step? Direct substitution gives the actual KL cost of this proposed move.
Compare to $δ = 0.05$ : $0.045 \leq 0.05$ . ✓ Step is inside the trust region — allowed. Why this step? This is exactly the check TRPO's line search performs, just done analytically because we chose Gaussians.
The twist — is KL symmetric? Here variances are equal so $D_{KL} (old ∥ new) = D_{KL} (new ∥ old) = 0.045$ . But in general KL is not symmetric; if $σ$ also changed, the two directions differ. TRPO always uses $D_{KL} (π_{old} ∥ π_{θ})$ — old first — matching the surrogate's importance-sampling direction. Why this step? An exam favourite: swapping the arguments of KL changes the number, and TRPO's derivation fixes the order.

Verify: $0. 3^{2} /2 = 0.045 \leq 0.05.$ ✓

What the figure shows: two bell curves — the teal old policy $N (0, 1)$ and the orange new policy $N (0.3, 1)$ , same width, shifted right by $0.3$ . The plum arrow marks the mean shift. The visual point: a modest-looking horizontal slide of $0.3$ standard deviations costs only $0.045$ nats of KL, comfortably inside the $δ = 0.05$ wall — matching the quadratic mean-shift law.

Recall Which cells did each example cover?

Ex 1 covers cell ::: A (diagonal Fisher, positive advantage) Ex 2 covers cell ::: B (negative advantage flips the step, $q$ stays positive) Ex 3 covers cell ::: C (non-diagonal Fisher, real $2 \times 2$ inverse) Ex 4 covers cell ::: D (zero gradient, $0/0$ limit gives $s = 0$ ) Ex 4b covers cell ::: D' (singular Fisher, $det F = 0$ , fix with damping $F + η I$ ) Ex 5 covers cell ::: E ( $∥ s ∥ = 2 δ$ , square-root scaling law) Ex 6 covers cell ::: F (line search rejects an optimistic step, quadratic backtracking) Ex 7 covers cell ::: G (word problem: Fisher damps high-curvature parameter) Ex 8 covers cell ::: H (Gaussian KL, quadratic in mean shift, asymmetry twist)

See also: Policy Gradient Methods, Conjugate Gradient Method, Proximal Policy Optimization (PPO).