Exercises — Proximal Policy Optimization (PPO)

Numerator = the new policy ${\pi }_{\theta }$ (the one we are updating). Denominator = the old policy ${\pi }_{{\theta }_{old}}$ that collected the data. $r_t=0.8$ means the new policy makes this sampled action ==$20\%$ less likely== than the old policy did. It sits exactly on the lower clip edge when $ϵ=0.2$.

$\mathrm{clip}$ just squeezes the number back into $[0.8,1.2]$:

• $0.65<0.8\Rightarrow 0.8$
• $1.0$ is inside $\Rightarrow 1.0$
• $1.35>1.2\Rightarrow 1.2$

${\hat{A}}_t>0$ = better-than-average action, so we want it more likely ⇒ push $r_t$ up (above $1$). Recall from the Policy Gradient Theorem that we scale $\nabla \log \pi$ by $\hat{A}$: positive advantage pushes probability up.

• Unclipped: $1.1\times 4=4.4$
• Clip: $\mathrm{clip}(1.1,0.8,1.2)=1.1\Rightarrow 1.1\times 4=4.4$
• $\min (4.4,4.4)=\boxed{4.4}$. Both branches equal ⇒ gradient flows, PPO acts like plain policy gradient here.

• Unclipped: $1.6\times 4=6.4$
• Clip: $\mathrm{clip}(1.6,0.8,1.2)=1.2\Rightarrow 1.2\times 4=4.8$
• $\min (6.4,4.8)=\boxed{4.8}$. Clipped branch active ⇒ $\partial L/\partial r_t=0$: no reward for pushing further.

• Unclipped: $0.5\times (-3)=-1.5$
• Clip: $\mathrm{clip}(0.5,0.8,1.2)=0.8\Rightarrow 0.8\times (-3)=-2.4$
• $\min (-1.5,-2.4)=\boxed{-2.4}$. The more negative value wins the min. Clipped branch active ⇒ gradient $=0$: PPO refuses to suppress a bad action even more in one step.

• Unclipped: $1.4\times (-3)=-4.2$
• Clip: $\mathrm{clip}(1.4,0.8,1.2)=1.2\Rightarrow -3.6$
• $\min (-4.2,-3.6)=\boxed{-4.2}$. Unclipped active ⇒ gradient flows to push $r_t$ down: we are always allowed to undo a mistake.

$\hat{A}>0$: the objective rises with $r_t$ until it hits the cap $(1+ϵ)\hat{A}$. So gradient is zero for $r_t>1+ϵ=1.2$. Below that it flows. $\hat{A}<0$: objective is $r_t\hat{A}$ (a decreasing line since $\hat{A}<0$). We want it larger, i.e. $r_t$ smaller. The clip caps the "reward" once $r_t<1-ϵ=0.8$. So gradient is zero for $r_t<0.8$. Summary of the flat (zero-gradient) region: $\hat{A}>0:r_t>1.2,\hat{A}<0:r_t<\mathrm{0.8.}$ These are exactly the "you've moved far enough in the helpful direction" zones.

For $\hat{A}>0$, $\min (r_t\hat{A},\mathrm{clip}(r_t)\hat{A})$ compares two increasing multiples of $\hat{A}$; the min tracks the smaller multiplier:

For $\hat{A}<0$, multiplying by a negative number flips which multiplier gives the smaller product; the min now picks the larger multiplier:

Ratios are computed from log-probs to stay numerically stable: $r_t=\exp (\log {\pi }_{\theta }-\log {\pi }_{{\theta }_{old}})=\exp (-0.90-(-1.20))=\exp (0.30)\approx \mathrm{1.3499.}$ Since $r_t\approx 1.35>1.2$ and $\hat{A}>0$:

• Unclipped $=1.3499\times 2=2.6997$
• Clipped $=1.2\times 2=2.4$
• $\min \Rightarrow \boxed{2.4}$, clipped branch active, gradient $=0$.

Per-sample (from L2 logic):

1. $\min (2.2,2.2)=2.2$
2. $\min (3.0,2.4)=2.4$
3. $\min (-1.0,-1.6)=-1.6$
4. $\min (-2.6,-2.4)=-2.6$ Mean $=\frac{2.2+2.4-1.6-2.6}{4}=\frac{0.4}{4}=\boxed{0.10}$.

• Value loss: $(5.0-6.5{)}^2=(-1.5{)}^2=2.25$; times $c_1=0.5\Rightarrow 1.125$.
• Entropy bonus: $0.01\times 0.7=0.007$. (This term, from Entropy Regularization, keeps exploration alive.) $L=2.4-1.125+0.007=\boxed{1.282}.$ Why signs: value loss is subtracted (we minimise squared error while maximising $L$); entropy is added (we reward randomness).

PPO stops rewarding a positive-advantage action once $r_t>1+ϵ$. To cap the rewarded range at $1.25$, set $1+ϵ=1.25\Rightarrow \boxed{ϵ=0.25}$. What it does NOT constrain: for $\hat{A}<0$, ratios above $1.2(=1.25$ here$)$ are left open by design — so a bad action wrongly made more likely can still have $r_t$ arbitrarily large in the objective; the gradient (correctly) drives it back down but PPO does not hard-cap it. Clipping is a soft trust region, unlike TRPO's hard KL constraint.

(a) $\hat{A}>0,r_t=0.7<1.2$ ⇒ objective $=r_t\hat{A}$, slope $\hat{A}>0$ ⇒ push $r_t$ up. (b) $\hat{A}>0,r_t=1.3>1.2$ ⇒ flat ⇒ zero. (c) $\hat{A}<0,r_t=0.6<0.8$ ⇒ flat ⇒ zero (already suppressed enough). (d) $\hat{A}<0,r_t=1.1\ge 0.8$ ⇒ objective $=r_t\hat{A}$, slope $\hat{A}<0$ ⇒ increasing $r_t$ lowers $L$, so gradient pushes $r_t$ down.

Mechanism: the Importance Sampling ratio $r_t$ is only trustworthy near ${\pi }_{{\theta }_{old}}$. With no clip, the optimizer keeps pushing $r_t\to \infty$ for the largest (often noisy) positive $\hat{A}$. After a couple of epochs the new policy is far from the old one, so the surrogate no longer approximates the true return $J(\theta )$ — the "improvement" is fictitious and the real policy degrades (collapse). Fix: reinstate $L^{CLIP}$ (or add a KL penalty). The clip zeroes the gradient once $r_t>1+ϵ$ for good actions, keeping every epoch's update inside the region where the surrogate is valid — this is precisely PPO's cheap stand-in for the Actor-Critic Methods + TRPO trust region.