Before the traps, here is every symbol these questions lean on, in plain words. Nothing below uses a letter not defined here.
Recall Why the quadratic KL ball yields a
unique maximizer (proof sketch)
Setup. Locally, the objective is linear, L≈g⊤s, and the constraint region is the ellipsoid 21s⊤Fs≤δ. Because F is positive-definite, this region is a convex, bounded set (an ellipsoid), and a nonzero linear function attains its maximum on a bounded convex set at exactly one boundary point — the point where the gradient g is normal to the ellipsoid surface. The figure below draws this ellipsoid and the resulting step.
Derivation. Form the Lagrangian g⊤s−λ(21s⊤Fs−δ); the multiplier λ>0 is the price paid per unit of KL budget. Setting the gradient in s to zero: g−λFs=0⇒s=λ1F−1g. Substituting into the active constraint 21s⊤Fs=δ solves for λ and gives the closed form s=g⊤F−1g2δF−1g. Since F is invertible this point is unique — that is the "WHY" behind the trust-region formulation.
The picture below makes the geometry concrete: the black ellipse is the KL trust region 21s⊤Fs≤δ, whose shape is set by the curvature F (wide where the policy barely changes, narrow where it changes fast). The black arrow is the ordinary gradient g; the red arrow is the actual TRPO step s=F−1g scaled to the boundary. Notice the red arrow bends away from g toward the low-curvature (wide) direction — that bend is exactly what the Fisher matrix buys you, and it is why "limit the parameters" (a circle) would be the wrong constraint.
Recall Why
F is never formed: the Fisher-vector-product trick
F is (params × params) — for a deep net that is billions of entries. But Conjugate Gradient Method only ever needs the productFv for chosen vectors v, never F itself. The trick: the mean KL DˉKL (defined above) has gradient ∇θDˉKL (a vector); its directional derivative along v is ∇θ(∇θDˉKL⋅v)=Fv. So Fv = one autodiff of a dot product — a second backward pass. We get every product we need without ever storing the matrix.
The trust region limits how far the parameters θ can move.
False. It limits the change in the policy distribution measured by KL; the same parameter step s can barely move a flat policy yet drastically move a sharp one.
TRPO directly maximizes the true return J(π′).
False. It maximizes the surrogateLπ, which freezes the state-visitation distribution at dπ; this only approximates J near πold, which is why the KL constraint is needed.
A smaller δ always gives faster learning.
False. Small δ makes each step safer but tinier, so learning is slower; larger δ is faster but risks stepping where the surrogate is no longer trustworthy.
TRPO guarantees the true return can never decrease.
False in practice. The theoretical penalty version guarantees monotonic improvement, but TRPO swaps that penalty for a hard δ plus a line search — so it guarantees improvement only up to those approximations.
The Fisher Information Matrix F is the Hessian of the surrogate objective.
False. F is the Hessian of the mean KL divergenceDˉKL (its second-order approximation), not of Lπ; it describes distribution-space curvature, not objective curvature.
The importance ratio πold(a∣s)πθ(a∣s) can be dropped since we sample from πold anyway.
False. That ratio is exactly what re-weights old samples so the expectation matches what the new policy would collect; without it we would optimize the old policy against itself.
TRPO requires computing and inverting F each update.
False. It only needs Fisher-vector products Fv, and solves Fx=g (with g the surrogate gradient) via Conjugate Gradient Method; F is never explicitly formed or inverted.
If the surrogate improves, TRPO always accepts the step.
False. The line search also requires the true KL to stay ≤δ; a step passing surrogate improvement but violating true KL is rejected and shrunk.
The natural gradient F−1g points in the same direction as the ordinary gradient g.
Usually false. They coincide only when F is proportional to identity; otherwise F−1 rotates and rescales g to respect distribution-space geometry.
DˉKL (mean KL) and maxsDKL give identical constraints.
False. The theory uses the harder max KL; TRPO uses the mean KL DˉKL because it is estimable from samples, trading a little rigor for practicality.
"Because DKL(π∥πθ) is a distance, it's symmetric in π and πθ."
Wrong — KL is not symmetric; DKL(p∥q)=DKL(q∥p) in general, so it is a divergence, not a metric. (Its second-order approximation via Fis locally symmetric, which is why the quadratic model works.)
"The surrogate Lπ equals J everywhere, we just approximate it for speed."
Wrong — Lπ matches J only to first order at πold; it diverges from J as π′ moves away because it uses the frozen dπ instead of dπ′.
"We use C=(1−γ)24ϵγ as the actual step-size penalty in TRPO."
Wrong — C is the theoretical guaranteed-improvement coefficient (with ϵ the max advantage magnitude) and is far too large (steps become minuscule); TRPO replaces the penalty with a tunable hard constraint δ.
"The advantage Aπ(s,a) is computed under the new policy π′."
Wrong — it is the old policy's advantage Aπold; that is what we can estimate from collected data via Advantage Estimation (GAE).
"Setting the Lagrangian derivative to zero gives s=λF−1g."
Wrong sign/scale — with λ the Lagrange multiplier it gives s=λ1F−1g; plugging into the KL constraint then fixes the scalar to 2δ/(g⊤F−1g).
"Because g⊤s is linear, the objective has no maximum, so the constraint is irrelevant."
Wrong — the linear objective is precisely why the constraint matters: it would run to infinity unbounded, so the quadratic KL ball 21s⊤Fs≤δ is what makes the maximizer finite and unique.
"We freeze the state distribution because dπ′ doesn't exist yet."
Half-wrong — dπ′ is perfectly well-defined; the real reason is we cannot sample it without already running π′, so we substitute dπ, valid only when π′≈π.
Why does an importance ratio appear at all in the surrogate?
Because samples came from πold but we want the expectation under πθ; Importance Sampling re-weights each old sample by πoldπθ to correct the mismatch.
Why is the KL Hessian F (not the identity) the right geometry for the step s?
Because equal parameter moves cause unequal distribution moves; F measures how sensitive the policy distribution is to each parameter direction, converting a distribution budget δ into the correct parameter step.
Why must the constraint be quadratic in KL rather than linear?
Because at θ=θold the KL and its gradient are both zero (a policy is closest to itself), so the first non-trivial term is the second-order one — the Fisher quadratic form.
Why does TRPO need a backtracking line search on top of the closed-form step?
Because g⊤s (linear) and 21s⊤Fs (quadratic) are only local approximations; the exact KL and surrogate may violate their models, so the line search shrinks the step until the true constraints hold.
Why is a bad update far more dangerous in RL than in supervised learning?
In supervised learning the dataset is fixed, so a bad step just raises loss next batch; in RL the policy generates its own data, so a bad policy collects bad data, compounding the error and possibly preventing recovery.
Why solve Fx=g with conjugate gradient instead of a direct method?
Because F is (params × params) — enormous for deep nets — but Conjugate Gradient Method needs only cheap Fisher-vector products Fv and converges in far fewer iterations than the dimension.
Because the trust-region solution is the natural gradient (steepest ascent in KL geometry) merely rescaled to touch the boundary; TRPO is natural gradient with a principled, adaptive step length.
Why prefer PPO's clipped ratio over TRPO's exact constraint in practice?
Proximal Policy Optimization (PPO) approximates the trust region with a simple clipped surrogate, avoiding Fisher-vector products and line search — cheaper and simpler per step, at the cost of TRPO's cleaner theoretical guarantee.
What happens to the step when the surrogate gradient g=0?
The natural gradient F−1g=0, so s=0: no gradient means no improvement direction, and the scale 2δ/(g⊤F−1g) is a 0/0 that resolves to a null step — the policy is (locally) stationary.
What happens as the discount γ→0?
Only the immediate reward matters, so the visitation distribution dπ concentrates on the start states and the horizon shrinks; the frozen-dπ approximation becomes nearly exact and the penalty C=(1−γ)24ϵγ→0 — the surrogate is trustworthy over larger steps.
What happens as the discount γ→1?
The penalty coefficient C=(1−γ)24ϵγ blows up, forcing arbitrarily tiny theoretical steps; the visitation distribution also has heavier long-horizon weight, making the frozen-dπ approximation harder to trust.
What if πθ assigns near-zero probability where πold did not?
The importance ratio πoldπθ stays finite, but the KL DKL(πold∥πθ) explodes (division-by-near-zero inside the log), so the trust region immediately rejects such a move — exactly the safety we want.
What if the advantage estimate is uniformly zero for all actions?
The surrogate gradient g=0, so no update is taken; with nothing distinguishing actions there is no direction in which return provably increases.
What happens if you set δ extremely large (effectively no constraint)?
TRPO degenerates toward an unconstrained natural-gradient step whose length is bounded only by the line search; the surrogate approximation breaks and performance can collapse — the same failure vanilla policy gradient risks.
What if F is singular or ill-conditioned?
F−1g is undefined or numerically unstable; TRPO adds a small damping term (F+ηI) so conjugate gradient stays well-behaved, trading a bit of geometric accuracy for stability.
Recall One-line summary of the traps
TRPO constrains the distribution (KL), optimizes a surrogateLπ (frozen dπ), moves along the natural gradientF−1g (never inverting F), and verifies with a line search — every misconception comes from forgetting one of these four facts.
