Proximal Policy Optimization (PPO)

WHAT are we actually optimizing? (Derivation from scratch)

Step 1 — The policy gradient objective

We want to maximize expected return $J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]$ . The policy gradient theorem gives:

$\nabla_\theta J(\theta)=\mathbb{E}\big[\nabla_\theta \log\pi_\theta(a|s)\,\hat A(s,a)\big]$

Why? $\hat A$ (the advantage) says "how much better than average was this action." We push up $\log\pi$ for good actions ( $\hat A>0$ ), push down for bad ones.

Step 2 — Reuse old data via importance sampling

We collect data with an old policy $\pi_{\theta_{old}}$ , then optimize a new $\pi_\theta$ . To reuse the old samples, we reweight:

$\mathbb{E}_{a\sim\pi_{\theta_{old}}}\!\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)}\hat A\right]$

Define the probability ratio:

$r_t(\theta)=\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$

Why a ratio? $r_t>1$ means the new policy makes this action more likely than the old one did. This surrogate objective $L^{CPI}=\mathbb{E}[r_t\hat A_t]$ is the "conservative policy iteration" objective.

Step 3 — The clipped surrogate objective

$\boxed{L^{CLIP}(\theta)=\mathbb{E}_t\Big[\min\big(r_t(\theta)\hat A_t,\ \text{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t\big)\Big]}$

Intuition WHY the

min and WHY it works Split by sign of $\hat A_t$ :

$\hat A_t>0$ (good action): we want $r_t$ up. But the objective is capped at $(1+\epsilon)\hat A_t$ . Once $r_t>1+\epsilon$ , gradient = 0 → no incentive to push further. Good, but not too good.
$\hat A_t<0$ (bad action): we want $r_t$ down. Capped at $(1-\epsilon)\hat A_t$ . Once $r_t<1-\epsilon$ , gradient = 0 → we stop over-suppressing.
The asymmetry (why min not just clip): the min lets the objective get worse freely (no clip) but not artificially better. If the update made a good action less likely ( $r_t<1$ ), the unclipped term is smaller, so min keeps it — you're allowed to correct a mistake even outside the trust region.

Figure — Proximal Policy Optimization (PPO)

HOW is the advantage estimated? (GAE)

PPO uses Generalized Advantage Estimation. With TD error $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$ :

$\hat A_t^{GAE}=\sum_{l=0}^{\infty}(\gamma\lambda)^l\,\delta_{t+l}$

Why? $\lambda$ trades bias vs variance: $\lambda=0$ → $\hat A=\delta_t$ (low variance, biased, 1-step); $\lambda=1$ → full Monte-Carlo return (unbiased, high variance). Typical $\lambda\approx0.95$ .

The full PPO loss

$L(\theta)=\underbrace{L^{CLIP}}_{\text{policy}}-\;c_1\underbrace{(V_\theta(s)-V_t^{target})^2}_{\text{value loss}}+\;c_2\underbrace{H[\pi_\theta]}_{\text{entropy bonus}}$

Why entropy $H$ ? Encourages exploration by keeping the policy from becoming deterministic too fast. Why one shared step? Actor and critic often share a network; we run K epochs of minibatch SGD on the same collected batch — that's the sample-efficiency win over REINFORCE.

Worked examples

Recall Feynman: explain to a 12-year-old

Imagine you're learning to shoot basketball hoops. After some shots you know which moves helped and which didn't. You want to change your technique — but if you change everything at once based on a few lucky shots, you'll ruin your form. So you make a rule: "I'll adjust my technique, but never by more than a little bit at a time." If a move worked, I do a bit more of it — but I stop once I've already changed enough. If a move was bad, I do a bit less — but again, only a little. That "only change a little" rule is the clip in PPO. It lets you learn fast without wrecking what already works.

Active recall

What is the probability ratio

r_t

in PPO?

r_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{old}}(a_t|s_t)

— how much more/less likely the new policy makes the sampled action vs the old policy.

Write the PPO clipped objective.

L^{CLIP}=\mathbb{E}_t[\min(r_t\hat A_t,\ \text{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t)]

Why does PPO take the min of clipped and unclipped terms?

To form a pessimistic lower bound: it removes incentive to move too far when it helps, but still lets you fully correct updates that move in the wrong direction.

For

\hat A>0

and

r_t>1+\epsilon

, what is the objective gradient?

Zero — the clipped branch is active, so no incentive to increase

r_t

further.

What problem does clipping replace from TRPO?

The hard KL trust-region constraint; clipping approximates it with cheap first-order updates.

What does

\epsilon

control and its typical value?

The trust-region width

[1-\epsilon,1+\epsilon]

; typically

\epsilon=0.2

What is GAE and what does

\lambda

trade off?

Generalized Advantage Estimation,

\hat A_t=\sum(\gamma\lambda)^l\delta_{t+l}

;

\lambda

trades bias (low) vs variance (high), typically 0.95.

Why can PPO run multiple SGD epochs on one batch?

Importance sampling (

r_t

) lets it reuse data collected by the old policy, giving sample efficiency over REINFORCE.

What are the three terms of the full PPO loss?

Clipped policy surrogate, value-function MSE loss, and entropy bonus for exploration.

Why is the entropy bonus included?

To keep the policy stochastic and encourage exploration, preventing premature collapse to a deterministic policy.

Connections

Policy Gradient Theorem — the objective PPO reweights.
Trust Region Policy Optimization (TRPO) — PPO's second-order predecessor.
Generalized Advantage Estimation (GAE) — how $\hat A_t$ is computed.
Actor-Critic Methods — PPO is an actor-critic algorithm.
Importance Sampling — justifies reusing old-policy data.
KL Divergence — the trust region clipping approximates.
Entropy Regularization — exploration term in the loss.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, PPO ka core idea bahut simple hai. Policy gradient methods me hum policy ko improve karte hain, par problem yeh hai ki agar ek hi update me policy ko bahut zyada change kar diya, toh policy collapse ho jaati hai — kyunki gradient sirf current policy ke aas-paas hi reliable hota hai. Toh PPO bolta hai: "improve toh karo, par thoda-thoda, apne purane policy ke close raho." Isko hi trust region kehte hain.

Iske liye PPO ek ratio use karta hai: $r_t = \pi_{new}/\pi_{old}$ — matlab naya policy us action ko kitna zyada ya kam likely bana raha hai. Fir yeh ratio ko $[0.8, 1.2]$ me clip kar deta hai (jab $\epsilon=0.2$ ). Agar action accha tha ( $\hat A > 0$ ) aur ratio bahut upar chala gaya, PPO ka gradient zero ho jaata hai — matlab "bas, ab aur mat badha." Agar action bura tha, toh usko zyada suppress karne se rokta hai. Par ek important baat: agar galti se humne bure action ko zyada likely kar diya, toh min wali trick usko fully penalize karti hai — yaani apni galti sudharne ki hamesha permission hai.

Yeh matter isliye karta hai kyunki PPO ek hi data batch pe multiple SGD epochs chala sakta hai (importance sampling ki wajah se), toh REINFORCE se zyada sample-efficient hai. Aur TRPO jaisi complicated second-order maths (Fisher matrix, conjugate gradient) ki zaroorat nahi — sirf normal Adam optimizer se kaam ho jaata hai. Isiliye PPO aaj real-world RL (robotics, ChatGPT ka RLHF) me sabse zyada use hota hai — simple, stable, aur efficient.

WHY does PPO exist?

WHAT are we actually optimizing? (Derivation from scratch)

Step 1 — The policy gradient objective

Step 2 — Reuse old data via importance sampling

Step 3 — The clipped surrogate objective

HOW is the advantage estimated? (GAE)

The full PPO loss

Worked examples

Active recall

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Deep & Advanced RL

Connections