Trust Region Policy Optimization (TRPO) — Hinglish notes

5.2.10 · AI-ML › Deep & Advanced RL

TRPO exist kyun karta hai?

Vanilla policy gradient (REINFORCE / A2C) update karta hai $θ \leftarrow θ + α \nabla_{θ} J$ . Problem yeh hai: gradient sirf locally accurate hota hai. Ek bada step policy ko aisi jagah push kar sakta hai jahan collected data (samples from old policy) environment ko achhi tarah describe hi nahi karta. Result: ek bura update performance ko collapse kar deta hai, aur kyunki RL data policy pe depend karta hai, shayad tum kabhi recover hi na karo.

TRPO ka jawab hai: main kitna bada step le sakta hoon aur phir bhi improvement guarantee kar sakta hoon?

Hum actually optimize kya kar rahe hain?

Hum expected discounted return maximize karna chahte hain $J (π) = E_{τ \sim π} [\sum_{t} γ^{t} r_{t}]$ .

Exact performance-difference identity (first principles)

Key lemma (Kakade & Langford):

J (π^{'}) - J (π) = E_{s \sim d^{π^{'}}, a \sim π^{'}} [A^{π} (s, a)]

jahan $A^{π} (s, a) = Q^{π} (s, a) - V^{π} (s, a)$ purani policy ke under advantage hai, aur $d^{π^{'}}$ nayi policy $π^{'}$ ka discounted state-visitation distribution hai.

Yeh identity itni beautiful kyun hai: yeh kehti hai ki nayi policy ki improvement equals kitna extra advantage woh collect karti hai, purani policy ke advantage function ke against measure kiya gaya. Lekin yeh impractical hai: states $d^{π^{'}}$ se aate hain, jinhe hum sample nahi kar sakte jab tak hamare paas $π^{'}$ na ho.

Surrogate: state distribution ko freeze karo

$d^{π^{'}} \approx d^{π}$ approximate karo (valid jab $π^{'} \approx π$ ). Surrogate objective define karo:

L_{π} (π^{'}) = J (π) + E_{s \sim d^{π}, a \sim π} [\frac{π ^{'} ( a ∣ s )}{π ( a ∣ s )} A^{π} (s, a)]

Approximation error ko bound kaise karte hain? (The trust region)

$L_{π}$ , $J$ se $π$ par first order mein match karta hai, lekin jaise $π^{'}$ door jaata hai, drift karta hai. TRPO ka central theorem drift ko KL divergence se bound karta hai:

J (π^{'}) \geq L_{π} (π^{'}) - C s max D_{KL} (π (\cdot ∣ s) ∥ π^{'} (\cdot ∣ s)), C = \frac{4 ϵ γ}{( 1 - γ ) ^{2}}

jahan $ϵ = max_{s, a} ∣ A^{π} (s, a) ∣$ .

Theoretical penalty coefficient $C$ bahut bada hota hai, jo tiny steps force karta hai. TRPO ki jagah penalty ko ek hard constraint (ek "trust region") mein convert kar deta hai tunable size $δ$ ke saath:

Figure — Trust Region Policy Optimization (TRPO)

Ise solve kaise karte hain? (Natural-gradient step derive karna)

Hum constrained problem ko deep net ke liye exactly solve nahi kar sakte. TRPO locally approximate karta hai:

Objective ko linearize karo $θ_{old}$ ke around: $L (θ) \approx g^{⊤} (θ - θ_{old})$ , jahan $g = \nabla_{θ} L$ policy gradient hai.
KL constraint ko quadratically approximate karo: $\overset{ˉ}{D}_{KL} \approx \frac{1}{2} (θ - θ_{old})^{⊤} F (θ - θ_{old})$ , jahan $F$ Fisher Information Matrix hai (KL ka Hessian, jo first order par $0$ hai — isliye humein quadratic term chahiye).

Problem ban jaati hai:

s max g^{⊤} s s.t. \frac{1}{2} s^{⊤} F s \leq δ, s = θ - θ_{old} .

Lagrangian se solve karo $g^{⊤} s - λ (\frac{1}{2} s^{⊤} F s - δ)$ . Derivative zero set karo: $g - λ F s = 0 \Rightarrow s = \frac{1}{λ} F^{- 1} g$ .

Constraint mein plug karo $\frac{1}{2} s^{⊤} F s = δ$ $λ$ solve karne ke liye:

Yeh smart kyun hai?

Hum kabhi $F^{- 1}$ form nahi karte (bahut bada hai). Hum $F^{- 1} g$ compute karte hain conjugate gradient se, sirf Fisher-vector products $F v$ chahiye (second autodiff se sasta).
Kyunki approximations true constraint violate kar sakti hain, TRPO backtracking line search karta hai: step ko $β^{j}$ se shrink karo jab tak true KL $\leq δ$ aur surrogate actually improve ho.

Worked Examples

Common Mistakes (Steel-manned)

Recall Feynman: 12-saal ke bacche ko explain karo

Socho tum ek recipe adjust kar rahe ho jo tumne already banai hai. Tumne khana taste kiya (data collect kiya) aur jaante ho kaunse changes ise better banate hain. Lekin agar tum ek saath bahut saari ingredients badal do, toh tumhare purane taste-notes apply nahi honge — shayad tum sab kuch barbad kar do. TRPO kehta hai: recipe ko best direction mein change karo, lekin ek time pe sirf thoda sa — itna chhota ki tumhare notes abhi bhi sense banayein. "Kitna zyada zyada hai" measure hota hai is se ki nayi recipe purani se kitni alag hai, na ki tumne kitne numbers change kiye.

Active Recall

Kaun si performance-difference identity par TRPO build karta hai?

J (π^{'}) - J (π) = E_{s \sim d^{π^{'}}, a \sim π^{'}} [A^{π} (s, a)]

— new-minus-old return equals extra advantage collected.

Woh identity directly use karna impractical kyun hai?

States

d^{π^{'}}

se draw hote hain (nayi policy ka visitation), jise hum sample nahi kar sakte jab tak hamare paas

π^{'}

na ho.

Kaun si approximation surrogate objective deti hai?

d^{π^{'}}

d^{π}

se replace karo (valid jab

π^{'} \approx π

), phir actions ko importance-sample karo.

TRPO surrogate objective likho.

E_{s, a \sim π_{o l d}} [\frac{π _{θ} ( a ∣ s )}{π _{o l d} ( a ∣ s )} A^{π_{o l d}} (s, a)]

TRPO constraint kya hai?

Mean KL divergence

\overset{ˉ}{D}_{K L} (θ_{o l d}, θ) \leq δ

— policy-distribution space mein ek hard trust region.

∥Δ θ ∥

ki jagah KL constrain kyun karo?

Equal parameter steps unequal distribution changes cause karte hain; KL distribution space mein distance measure karta hai, jo surrogate ki validity ke liye actually matter karta hai.

KL constraint ko quadratically kaunsa matrix approximate karta hai?

Fisher Information Matrix

F

(

θ_{o l d}

par KL ka Hessian).

Closed-form TRPO step do.

s = \frac{2 δ}{g ^{⊤} F ^{- 1} g} F^{- 1} g

, yaani natural gradient trust boundary tak scale kiya gaya.

F

ko invert kiye bina

F^{- 1} g

compute kaise karte hain?

Conjugate gradient use karke Fisher-vector products

F v

ke saath.

CG step ke baad backtracking line search kyun?

Linear/quadratic approximations true KL constraint violate kar sakti hain ya surrogate improve karne mein fail ho sakti hain; line search step ko tab tak shrink karta hai jab tak dono hold na ho jayein.

F^{- 1} g

(natural gradient) geometrically kya represent karta hai?

Euclidean parameter geometry ki jagah KL geometry mein measure kiya gaya steepest-ascent direction.

Theory kya guarantee deta hai?

True

J

ka monotonic (non-decreasing) improvement jab lower bound

L - C \cdot D_{K L}

optimize karo.

Connections

Policy Gradient Methods — TRPO unki instability fix karta hai large steps ke under.
Natural Gradient Descent — $F^{- 1} g$ direction natural gradient hai.
Fisher Information Matrix — KL ka curvature, trust-region geometry define karta hai.
KL Divergence — trust region define karne wala distance measure.
Proximal Policy Optimization (PPO) — sasta successor: hard KL constraint ki jagah ratio clip karta hai.
Conjugate Gradient Method — $F$ form kiye bina $F x = g$ solve karta hai.
Advantage Estimation (GAE) — practice mein $A^{π}$ kaise estimate karte hain.
Importance Sampling — nayi policy score karne ke liye purane samples reweight karta hai.