Proximal Policy Optimization (PPO) — Hinglish notes

5.2.9 · AI-ML › Deep & Advanced RL

PPO exist KYU karta hai?

Intuition Woh core problem jo isse solve karni hai

Policy gradients mein hum chahte hain ki bade improvement steps lein taaki jaldi seekhein. Lekin policy gradients sirf locally valid hote hain — gradient tumhe improvement ki direction batata hai current policy par. Zyada bada step lo toh estimate bekar ho jaata hai, aur policy collapse kar sakti hai.

Vanilla policy gradient (REINFORCE): data ke ek batch par ek gradient step → wasteful, high variance.
TRPO: ek hard KL-divergence trust region enforce karta hai taaki steps "trustworthy" rahein — lekin second-order optimization chahiye (conjugate gradient, Fisher matrix). Complicated.
PPO: ~TRPO ki reliability sirf first-order SGD se achieve karta hai, trust region ko ek clipped objective mein bake karke. Yeh 80/20 win hai: TRPO ka 90% benefit, 10% complexity.

HUM ACTUALLY KYA optimize kar rahe hain? (Scratch se derivation)

Step 1 — Policy gradient objective

Hum expected return maximize karna chahte hain $J (θ) = E_{τ \sim π_{θ}} [R (τ)]$ . Policy gradient theorem deta hai:

$\nabla_{θ} J (θ) = E [\nabla_{θ} lo g π_{θ} (a ∣ s) \hat{A} (s, a)]$

Kyun? $\hat{A}$ (advantage) kehta hai "yeh action average se kitna better tha." Hum $lo g π$ ko achhe actions ke liye push up karte hain ( $\hat{A} > 0$ ), bure ke liye push down.

Step 2 — Importance sampling se purana data reuse karo

Hum data ek purani policy $π_{θ_{o l d}}$ se collect karte hain, phir ek nayi $π_{θ}$ optimize karte hain. Purane samples reuse karne ke liye hum reweight karte hain:

$E_{a \sim π_{θ_{o l d}}} [\frac{π _{θ} ( a ∣ s )}{π _{θ_{o l d}} ( a ∣ s )} \hat{A}]$

Probability ratio define karo:

$r_{t} (θ) = \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{θ_{o l d}} ( a _{t} ∣ s _{t} )}$

Ratio kyun? $r_{t} > 1$ matlab nayi policy is action ko purani se zyada likely banati hai. Yeh surrogate objective $L^{C P I} = E [r_{t} \hat{A}_{t}]$ "conservative policy iteration" objective hai.

Step 3 — Clipped surrogate objective

$L^{C L I P} (θ) = E_{t} [min (r_{t} (θ) \hat{A}_{t}, clip (r_{t}, 1 - ϵ, 1 + ϵ) \hat{A}_{t})]$

Intuition

min KYU aur yeh KYU kaam karta hai $\hat{A}_{t}$ ke sign ke hisaab se split karo:

$\hat{A}_{t} > 0$ (achha action): hum $r_{t}$ upar chahte hain. Lekin objective $(1 + ϵ) \hat{A}_{t}$ par capped hai. Jaise hi $r_{t} > 1 + ϵ$ , gradient = 0 → aage push karne ka koi incentive nahi. Achha, lekin bahut zyada achha nahi.
$\hat{A}_{t} < 0$ (bura action): hum $r_{t}$ neeche chahte hain. $(1 - ϵ) \hat{A}_{t}$ par capped. Jaise hi $r_{t} < 1 - ϵ$ , gradient = 0 → hum over-suppress karna band kar dete hain.
Asymmetry (kyun sirf clip nahi, min kyun): min objective ko freely worse hone deta hai (koi clip nahi) lekin artificially better nahi. Agar update ne ek achhe action ko less likely bana diya ( $r_{t} < 1$ ), toh unclipped term smaller hai, toh min use rakhta hai — tum galti correct karne ke liye allowed ho chahe trust region ke bahar ho.

Figure — Proximal Policy Optimization (PPO)

ADVANTAGE ESTIMATE KAISE hota hai? (GAE)

PPO Generalized Advantage Estimation use karta hai. TD error $δ_{t} = r_{t} + γ V (s_{t + 1}) - V (s_{t})$ ke saath:

$\hat{A}_{t}^{G A E} = \sum_{l = 0}^{\infty} (γ λ)^{l} δ_{t + l}$

Kyun? $λ$ bias vs variance trade karta hai: $λ = 0$ → $\hat{A} = δ_{t}$ (low variance, biased, 1-step); $λ = 1$ → full Monte-Carlo return (unbiased, high variance). Typical $λ \approx 0.95$ .

Poora PPO loss

$L (θ) = policy L^{C L I P} - c_{1} value loss (V_{θ} (s) - V_{t}^{t a r g e t})^{2} + c_{2} entropy bonus H [π_{θ}]$

Entropy $H$ kyun? Policy ko jaldi deterministic hone se rokta hai, exploration encourage karta hai. Ek shared step kyun? Actor aur critic aksar ek network share karte hain; hum usi collected batch par K epochs of minibatch SGD run karte hain — yeh REINFORCE par sample-efficiency ka win hai.

Worked examples

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum basketball hoops shoot karna seekh rahe ho. Kuch shots ke baad tumhe pata hai kaun se moves kaam aaye aur kaun se nahi. Tum apni technique badalna chahte ho — lekin agar tum kuch lucky shots ki wajah se sab kuch ek saath badal do, toh tumhari form kharab ho jaayegi. Toh tum ek rule banate ho: "Main apni technique adjust karunga, lekin ek baar mein thodi si hi." Agar koi move kaam aaya, main thoda aur karta hoon — lekin rukta hoon jab pehle se kaafi change kar chuka hoon. Agar koi move bura tha, thoda kam karta hoon — lekin woh bhi, sirf thoda. Yeh "sirf thoda badlo" wala rule hi PPO mein clip hai. Yeh tumhe jaldi seekhne deta hai bina jo pehle se kaam karta hai use barbaad kiye.

Active recall

PPO mein probability ratio

r_{t}

kya hai?

r_{t} (θ) = π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{o l d}} (a_{t} ∣ s_{t})

— nayi policy sampled action ko purani policy ke comparison mein kitna zyada/kam likely banati hai.

PPO clipped objective likho.

L^{C L I P} = E_{t} [min (r_{t} \hat{A}_{t}, clip (r_{t}, 1 - ϵ, 1 + ϵ) \hat{A}_{t})]

PPO clipped aur unclipped terms ka min kyun leta hai?

Ek pessimistic lower bound banane ke liye: yeh bahut door jaane ka incentive remove karta hai jab fayda ho, lekin phir bhi galat direction mein jaane wale updates ko poora correct karne deta hai.

\hat{A} > 0

aur

r_{t} > 1 + ϵ

ke liye, objective gradient kya hai?

Zero — clipped branch active hai, toh

r_{t}

ko aur badhane ka koi incentive nahi.

Clipping TRPO ki kaun si problem replace karta hai?

Hard KL trust-region constraint; clipping isse saste first-order updates se approximate karta hai.

ϵ

kya control karta hai aur iska typical value kya hai?

Trust-region width

[1 - ϵ, 1 + ϵ]

; typically

ϵ = 0.2

GAE kya hai aur

λ

kya trade-off karta hai?

Generalized Advantage Estimation,

\hat{A}_{t} = \sum (γ λ)^{l} δ_{t + l}

;

λ

bias (low) vs variance (high) trade karta hai, typically 0.95.

PPO ek batch par multiple SGD epochs kyun run kar sakta hai?

Importance sampling (

r_{t}

) ise purani policy se collect kiya data reuse karne deta hai, REINFORCE par sample efficiency deta hai.

Full PPO loss ke teen terms kya hain?

Clipped policy surrogate, value-function MSE loss, aur exploration ke liye entropy bonus.

Entropy bonus kyun include kiya jaata hai?

Policy ko stochastic rakhne aur exploration encourage karne ke liye, taaki woh premature deterministic policy mein collapse na ho jaye.

Connections

Policy Gradient Theorem — woh objective jise PPO reweight karta hai.
Trust Region Policy Optimization (TRPO) — PPO ka second-order predecessor.
Generalized Advantage Estimation (GAE) — $\hat{A}_{t}$ kaise compute hota hai.
Actor-Critic Methods — PPO ek actor-critic algorithm hai.
Importance Sampling — purani policy ka data reuse karne ko justify karta hai.
KL Divergence — trust region clipping jise approximate karta hai.
Entropy Regularization — loss mein exploration term.