Visual walkthrough — Proximal Policy Optimization (PPO)

---

Step 1 — What is a policy, and what does "make an action more likely" mean?

WHAT. A policy is the brain of our agent. Given a situation (a state, written $s$), it outputs a probability for each possible move (an action, written $a$). We write this probability as ${\pi }_{\theta }(a\mid s)$ — read "pi of $a$ given $s$". The little $\theta$ (the Greek letter theta) is just the collection of knobs (neural-network weights) that shape those probabilities.

WHY. We cannot improve what we cannot measure. The whole of PPO is about nudging these probabilities: raising them for moves that turned out well, lowering them for moves that turned out badly. So the object we push around is a number between 0 and 1 attached to each action.

PICTURE. Two bars for the same state: the old policy (violet) and a slightly-changed new policy (magenta). Notice one action got taller — the new policy makes it more likely.

---

Step 2 — The advantage $\hat{A}$: was this action better or worse than average?

WHAT. After taking an action we get a number ${\hat{A}}_t$ (read "A-hat at time $t$"), the advantage. It answers one question: compared to what my agent usually does in this state, was this particular action better or worse?

• ${\hat{A}}_t>0$ → better than average (a good action, we want more of it).
• ${\hat{A}}_t<0$ → worse than average (a bad action, we want less of it).
• ${\hat{A}}_t=0$ → exactly average (no opinion).

WHY. If we simply rewarded high-reward actions, a state where every action is great would push all of them up equally — no learning signal. Subtracting the average (that is what "advantage" does) leaves only the relative judgement, which is exactly the direction to nudge. The hat $\hat{}$ means "estimated" — we don't know the true value, we approximate it (via GAE).

PICTURE. A horizontal zero-line. Green arrow up for $\hat{A}>0$, red arrow down for $\hat{A}<0$. The sign is the only thing that will steer the whole objective.

---

Step 3 — The ratio $r_t$: how far has the new policy moved?

WHAT. We collected our experience with the old policy ${\pi }_{{\theta }_{old}}$, but we are now tuning a new policy ${\pi }_{\theta }$. To compare them on the same sampled action we form the probability ratio:

$r_t(\theta )=\underset{\text{new prob}}{\underbrace{{\pi }_{\theta }(a_t\mid s_t)}}/\underset{\text{old prob}}{\underbrace{{\pi }_{{\theta }_{old}}(a_t\mid s_t)}}$

WHY a ratio and not a difference? We are reusing old data — a trick called Importance Sampling. Importance sampling says: to estimate an average under the new policy using samples from the old one, reweight each sample by exactly this ratio. So $r_t$ is forced on us by the maths of reuse, not chosen for taste. And a ratio has a clean reading:

• $r_t=1$ → policies agree, no change on this action.
• $r_t>1$ → new policy makes this action more likely.
• $r_t<1$ → new policy makes it less likely.
• $r_t=0$ → new policy would never take it (degenerate — see Step 8).

PICTURE. A number line for $r_t$ anchored at $1$. Left of 1 = "less likely", right of 1 = "more likely". We will spend the whole derivation drawing a fence on this line.

---

Step 4 — Why we cannot just maximize $r_t{\hat{A}}_t$

WHAT. Suppose we hand $L^{CPI}=r_t{\hat{A}}_t$ to an optimizer with no restrictions. For a good action ($\hat{A}>0$) the optimizer discovers it can make the objective bigger forever by pushing $r_t\to \infty$.

WHY that is a disaster. The ratio $r_t$ is only a trustworthy estimate of the true objective near the old policy (importance sampling degrades as the two distributions separate). Far away, $r_t{\hat{A}}_t$ is a fantasy. Worse, $\hat{A}$ itself is noisy — the optimizer will happily march to infinity on a wrong advantage estimate, and the policy collapses.

PICTURE. The straight line $r_t{\hat{A}}_t$ for a good action shoots off the top of the frame — an unbounded ramp with no brakes. This is the runaway we must stop.

---

Step 5 — The clip: build a fence around $r_t$

WHAT. The function $\text{clip}(r_t,1-ϵ,1+ϵ)$ takes $r_t$ and squashes it into a band $[1-ϵ,1+ϵ]$:

$\text{clip}(r_t,1-ϵ,1+ϵ)=\{\begin{array}{ll}1-ϵ& r_t<1-ϵ\\ r_t& 1-ϵ\le r_t\le 1+ϵ\\ 1+ϵ& r_t>1+ϵ\end{array}$

Here $ϵ$ (the Greek epsilon) is a small width, typically $ϵ=0.2$, giving the band $[0.8,1.2]$.

WHY. Inside the band, clip does nothing — we behave like a normal policy gradient. Outside the band, clip flattens the value to a constant, and a constant has zero slope. Zero slope means zero gradient, means the optimizer gets no reward for pushing further. That flat shelf is our trust region, enforced with nothing but a first-order operation (the whole point versus Trust Region Policy Optimization (TRPO), which needed a costly KL Divergence constraint).

PICTURE. A staircase: rising diagonally through the band, then flat shelves on both sides. The flat parts are where "the fence" is.

---

Step 6 — Combine with min: the good-action side ($\hat{A}>0$)

WHAT. The clipped objective takes the minimum of the unclipped and clipped terms:

$L^{CLIP}={\mathbb{E}}_t\left[\min \right(\underset{\text{unclipped}}{\underbrace{r_t{\hat{A}}_t}},\underset{\text{clipped}}{\underbrace{\text{clip}(r_t,1-ϵ,1+ϵ){\hat{A}}_t}}\left)\right]$

For a good action, ${\hat{A}}_t>0$. Multiplying both terms by a positive number keeps their ordering, so we plot the objective against $r_t$:

• For $r_t\le 1+ϵ$: both terms agree, the objective rises with $r_t$ — improvement flows.
• For $r_t>1+ϵ$: the clipped term becomes the flat $(1+ϵ){\hat{A}}_t$, which is now the smaller one, so min picks it → flat shelf, gradient = 0.

WHY min and not just clip? With plain clip we'd cap both directions symmetrically. The min is deliberately pessimistic: it lets the objective get worse freely but refuses to let it look better than the honest capped value. For a good action that means: reward improvement up to the fence, then stop.

PICTURE. The green $\hat{A}>0$ curve: a rising ramp that hits a ceiling at $r_t=1+ϵ$ and goes flat. Past the fence, no incentive to push.

---

Step 7 — The bad-action side ($\hat{A}<0$) and the mistake-correction gift

WHAT. Now ${\hat{A}}_t<0$. Multiplying by a negative flips everything, so the shape is a mirror image. Trace it left-to-right along $r_t$:

• We want $r_t$ down (make a bad action less likely).
• If $r_t<1-ϵ$ (already suppressed a lot): the clipped term $(1-ϵ){\hat{A}}_t$ is the smaller (more negative) one, min picks it → flat shelf, gradient = 0. We stop over-suppressing.
• If $r_t>1+ϵ$ (we accidentally made a bad action more likely!): the unclipped $r_t{\hat{A}}_t$ is more negative than the clipped one, so min keeps the unclipped term → gradient flows, pushing $r_t$ back down.

WHY this is the clever bit. Even though $r_t$ is outside the trust region, the min still lets us fully punish a move in the wrong direction. You are always allowed to undo damage — the fence only stops you from over-committing to something that already worked, never from correcting a mistake.

PICTURE. The magenta $\hat{A}<0$ curve: flat shelf on the left (over-suppression frozen), then a steep downward slope on the right (mistake correction always active).

---

Step 8 — Every case on one map (including degenerate inputs)

WHAT. Combine both signs and cover the corners no one shows:

${\hat{A}}_t$	region of $r_t$	active branch	gradient?	meaning
$>0$	$r_t<1-ϵ$	unclipped	flows	correct: action got too rare, push up
$>0$	in band	either (equal)	flows	normal policy gradient
$>0$	$r_t>1+ϵ$	clipped	0	good enough — stop
$<0$	$r_t<1-ϵ$	clipped	0	already suppressed enough — stop
$<0$	in band	either (equal)	flows	normal policy gradient
$<0$	$r_t>1+ϵ$	unclipped	flows	mistake! push back down
$=0$	anywhere	—	0	no advantage, no signal
$r_t=0$	new policy killed the action	unclipped ($0\cdot \hat{A}=0$)	flat there	degenerate — see below

WHY cover $\hat{A}=0$ and $r_t=0$?

• ${\hat{A}}_t=0$: both terms are $0$, the whole objective is flat — the action is exactly average, nothing to learn. Correct behaviour, not a bug.
• $r_t=0$: the new policy assigns probability zero to a sampled action. Then $\log {\pi }_{\theta }=-\infty$ in the underlying gradient; in practice PPO clips ratios and uses stable log-probs so this never sends the update to infinity. The clip's flat shelves are exactly what keep such extremes harmless.
• $r_t=1$ at the very first inner epoch: new = old, so $r_t=1$ everywhere and $L^{CLIP}={\hat{A}}_t$ — plain policy gradient. Clipping only "wakes up" once repeated SGD epochs move ${\pi }_{\theta }$ away from ${\pi }_{{\theta }_{old}}$.

PICTURE. The full two-curve diagram: green (good action) and magenta (bad action) plotted together over $r_t$, with the flat shelves and live slopes labelled, and the vertical fences at $1\pm ϵ$.

---

The one-picture summary

Below: the entire PPO objective as one shape. Read it as a decision map — pick your $\hat{A}$ sign (colour), find your $r_t$ (horizontal), and the slope tells you instantly whether the update flows or freezes. This one figure is the derivation.

Recall Feynman retelling of the whole walkthrough

A policy is a set of dials that say how likely each move is. After playing, each move gets a report card: the advantage — a plus if it beat the average, a minus if it lagged. We reuse our old playthroughs by measuring the ratio $r_t$: how much more (or less) the new dials favour a move versus the old dials. We'd love to just crank good moves way up — but that ratio is only honest near where we started, so cranking blindly wrecks the policy. So we build a fence at $0.8$ and $1.2$: inside it we learn normally; outside it the objective goes flat, so the optimizer gets no reward for straying. The magic is the min: it flattens things only when straying would look falsely good, but keeps full punishment when we've made a bad move more likely — meaning we can always undo a mistake, but never over-commit to a lucky one. That single fenced, one-way shape is PPO.

Active recall