Proximal Policy Optimization (PPO): Worked examples

Intuition What this page is

The parent note showed you the clipped objective $L^{C L I P}$ and four examples. Here we build a complete map of every situation the clip can face — every sign of the advantage, every position of the ratio (inside the band, above it, below it, exactly on the edge), the zero-advantage case, and a real-world word problem. Then we walk through worked examples that touch every single cell of that map, so you never meet a scenario we didn't show.

Before we start, three plain-English reminders from Proximal Policy Optimization (PPO):

The advantage $\hat{A}_{t}$ is a number telling us "how much better than average this action turned out." Positive = good surprise. Negative = bad surprise. Zero = exactly average.
The ratio $r_{t} = \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{θ_{o l d}} ( a _{t} ∣ s _{t} )}$ is a positive number: bigger than 1 means the new policy makes this action more likely than the old one did; smaller than 1 means less likely; exactly 1 means "no change yet." (This ratio comes from Importance Sampling.)
clip is a machine that squeezes a number into a box. $clip (r, 0.8, 1.2)$ means: if $r < 0.8$ output $0.8$ ; if $r > 1.2$ output $1.2$ ; otherwise output $r$ unchanged. Throughout we use $ϵ = 0.2$ , so the box is $[0.8, 1.2]$ .

The objective we evaluate every time

Two words we will use constantly:

"Gradient flows" = this sample is still teaching the policy to change.
"Gradient stopped" = this sample has been silenced; PPO has decided we've moved far enough for this action.

The scenario matrix

The behaviour depends on two things only: the sign of $\hat{A}_{t}$ , and where $r_{t}$ sits relative to the band edges $1 - ϵ$ and $1 + ϵ$ . That gives a small grid. The last rows are the special/real-world cases.

Cell	Sign of $\hat{A}_{t}$	Ratio $r_{t}$	Which branch wins	Gradient?	Meaning
A	$\hat{A} > 0$ (good)	inside band $[0.8, 1.2]$	either (equal)	flows	normal policy gradient
B	$\hat{A} > 0$ (good)	above band ( $r > 1.2$ )	clipped	stopped	"good enough, stop boosting"
C	$\hat{A} > 0$ (good)	below band ( $r < 0.8$ )	unclipped	flows	undoing an over-cut of a good action
D	$\hat{A} < 0$ (bad)	inside band	either (equal)	flows	normal suppression
E	$\hat{A} < 0$ (bad)	below band ( $r < 0.8$ )	clipped	stopped	"suppressed enough, stop"
F	$\hat{A} < 0$ (bad)	above band ( $r > 1.2$ )	unclipped	flows	undoing accidental boost of a bad action
G	$\hat{A} = 0$	anywhere	tie (both zero)	none	average action → no learning signal
H (edge)	either	$r_{t}$ exactly on $1 \pm ϵ$	boundary	undefined at kink; 0 on clipped side, $\hat{A}_{t}$ on unclipped side	limiting behaviour (both edges)
I (word)	real task	—	—	—	trading-bot decision
J (exam twist)	mixed batch	—	—	—	which samples are "dead"?

Figure — Proximal Policy Optimization (PPO)

Intuition How to read figure s01 (the master picture)

The horizontal axis is the probability ratio $r_{t}$ ; the vertical axis is the per-sample objective $L_{t}$ . The two dashed grey lines mark the band edges $0.8$ and $1.2$ ; the dotted line at $r_{t} = 1$ is "no change yet."

The blue curve is $L_{t}$ for a good action ( $\hat{A} > 0$ ). Read it left to right: it rises normally until $r_{t} = 1.2$ , then goes flat — that flat shelf is the clipped region where the gradient is zero (cell B). To the far left (below $0.8$ ) it is still sloped: that is the "undo an over-cut" region (cell C).
The orange curve is $L_{t}$ for a bad action ( $\hat{A} < 0$ ). It slopes downward and goes flat for $r_{t} < 0.8$ (clipped, cell E), but stays sloped for $r_{t} > 1.2$ (the "undo a boosted bad action" region, cell F).
Wherever a curve is flat, the gradient has died; wherever it slopes, the sample is still teaching. That single visual — flat vs sloped — is the whole page in one picture.

Worked examples

Worked example Example 1 — Cell A: good action, ratio inside the band

Statement: $\hat{A}_{t} = + 3$ , $r_{t} = 1.05$ , $ϵ = 0.2$ . Compute $L_{t}$ and say whether the gradient flows.

Forecast: $1.05$ is comfortably inside $[0.8, 1.2]$ , so clipping should do nothing and PPO should behave like plain policy gradient. Guess: $L_{t} = 3.15$ , gradient flows.

Unclipped term $= r_{t} \hat{A}_{t} = 1.05 \times 3 = 3.15$ . Why this step? This is the raw reweighted Policy Gradient Theorem objective — always our first candidate.
Clip the ratio: $clip (1.05, 0.8, 1.2) = 1.05$ (already inside the box, unchanged). Why this step? We must check whether the ratio left the trust region; here it didn't.
Clipped term $= 1.05 \times 3 = 3.15$ .
Take the min: $min (3.15, 3.15) = 3.15$ . Both branches equal → the $min$ selects the unclipped-equivalent, so $\partial L_{t} / \partial r_{t} = \hat{A}_{t} = 3 \neq = 0$ : gradient flows. Why this step? Inside the band there is no penalty; PPO = normal policy gradient.

Verify: Inside $[0.8, 1.2]$ the two terms are identical, so $L_{t} = r_{t} \hat{A}_{t} = 3.15$ . Sanity: a good action ( $\hat{A} > 0$ ) that we've barely nudged should still be encouraged. ✓

Worked example Example 2 — Cell B: good action, ratio too high

Statement: $\hat{A}_{t} = + 3$ , $r_{t} = 1.6$ .

Forecast: We've made a good action much more likely already. The clip should kick in and freeze the gradient. Guess: clipped branch wins.

Unclipped: $1.6 \times 3 = 4.8$ .
Clip: $clip (1.6, 0.8, 1.2) = 1.2$ (pulled down to the ceiling). Why this step? $r_{t} > 1 + ϵ$ , so the ratio left the trust region on the high side.
Clipped term: $1.2 \times 3 = 3.6$ .
Min: $min (4.8, 3.6) = 3.6$ → the clipped branch won. Why this step? Because $\hat{A} > 0$ , the smaller number is the capped one; the $min$ keeps it.
Since the clipped term $1.2 \times \hat{A}_{t}$ does not vary with $r_{t}$ here, $\partial L_{t} / \partial r_{t} = 0$ : gradient stopped.

Verify: For $\hat{A} > 0$ , whenever $r_{t} > 1.2$ the objective is stuck at $1.2 \hat{A}_{t} = 3.6$ ; increasing $r_{t}$ to $2.0$ would still give clipped $= 3.6$ . Confirmed no reward for going further. ✓

Worked example Example 3 — Cell C: good action, but we over-cut it (subtle, the "undo" case)

Statement: $\hat{A}_{t} = + 3$ , $r_{t} = 0.5$ .

Forecast: A good action's probability got slashed (new policy makes it less likely). We should be allowed to fix that — gradient should flow to push $r_{t}$ back up.

Unclipped: $0.5 \times 3 = 1.5$ .
Clip: $clip (0.5, 0.8, 1.2) = 0.8$ (raised up to the floor).
Clipped term: $0.8 \times 3 = 2.4$ .
Min: $min (1.5, 2.4) = 1.5$ → the unclipped branch won. Why this step? Since $\hat{A} > 0$ , the unclipped value $1.5$ is the smaller (more pessimistic) one, so the $min$ keeps it — and it does depend on $r_{t}$ .
$\partial L_{t} / \partial r_{t} = \hat{A}_{t} = 3 > 0$ : gradient flows, pushing $r_{t}$ up. Why this step? PPO never blocks you from increasing a good action's probability back toward normal, even when you're outside the band.

Verify: The whole point of the $min$ is a pessimistic lower bound. Here $1.5 < 2.4$ , so the min chose the smaller, and that smaller branch still teaches. Correcting an over-suppressed good action is allowed. ✓

Worked example Example 5 — Cell E: bad action, already over-suppressed

Statement: $\hat{A}_{t} = - 3$ , $r_{t} = 0.6$ .

Forecast: We already cut this bad action's probability to 60% of before. Clip should stop us from over-suppressing further. Guess: clipped wins, gradient stopped.

Unclipped: $0.6 \times (- 3) = - 1.8$ .
Clip: $clip (0.6, 0.8, 1.2) = 0.8$ (raised to the floor). Why this step? $r_{t} < 1 - ϵ$ : the ratio left the trust region on the low side.
Clipped term: $0.8 \times (- 3) = - 2.4$ .
Min: $min (- 1.8, - 2.4) = - 2.4$ → the clipped branch won. Why this step? With $\hat{A} < 0$ , the clipped value is more negative and hence the smaller; the $min$ keeps it.
Clipped branch is constant in $r_{t}$ here → $\partial L_{t} / \partial r_{t} = 0$ : gradient stopped.

Verify: For $\hat{A} < 0$ with $r_{t} < 0.8$ , objective is frozen at $0.8 \times (- 3) = - 2.4$ . Pushing $r_{t}$ to $0.3$ would still give clipped $- 2.4$ . No reward for further suppression. ✓

Worked example Example 6 — Cell F: bad action we accidentally

boosted (the other "undo" case) Statement: $\hat{A}_{t} = - 3$ , $r_{t} = 1.4$ .

Forecast: We made a bad action more likely — a genuine mistake. PPO should fully penalize this even though $r_{t}$ is outside the band. Gradient should flow to pull $r_{t}$ down.

Unclipped: $1.4 \times (- 3) = - 4.2$ .
Clip: $clip (1.4, 0.8, 1.2) = 1.2$ .
Clipped term: $1.2 \times (- 3) = - 3.6$ .
Min: $min (- 4.2, - 3.6) = - 4.2$ → the unclipped branch won. Why this step? $- 4.2$ is more negative (smaller); the $min$ keeps the harsher unclipped penalty, which depends on $r_{t}$ .
$\partial L_{t} / \partial r_{t} = \hat{A}_{t} = - 3 < 0$ : gradient flows, pushing $r_{t}$ down.

Verify: This is the $min$ "always let it get worse" behaviour: the mistake of boosting a bad action is punished in full, unclipped. You are always allowed to undo damage. ✓

Worked example Example 8 — Cell H: exactly on both boundaries (limiting behaviour)

Statement: With $\hat{A}_{t} = + 3$ we probe the upper edge, and with $\hat{A}_{t} = - 3$ we probe the lower edge, in each case comparing the value at the edge to the value a hair past it.

Forecast: At the exact edge nothing is clipped yet, so it looks like normal PG. A hair past the edge the clip snaps on and (in the "reward" direction) the gradient dies. So the gradient should be discontinuous at each edge, while the value stays continuous.

Sub-case (a) — upper edge $r_{t} = 1 + ϵ = 1.2$ , good action $\hat{A}_{t} = + 3$ :

At $r_{t} = 1.2$ : $clip (1.2, 0.8, 1.2) = 1.2$ ; unclipped $= 1.2 \times 3 = 3.6$ ; clipped $= 3.6$ ; $min = 3.6$ . Both equal → left-hand gradient $= \hat{A}_{t} = 3$ .
At $r_{t} = 1.2 + η$ , $η \to 0^{+}$ : clip snaps to $1.2$ ; clipped $= 3.6$ ; unclipped $= (1.2 + η) \cdot 3 > 3.6$ ; $min$ picks clipped $= 3.6$ ; right-hand gradient $= 0$ . Why this step? Value $\to 3.6$ from both sides (continuous), but slope jumps $3 \to 0$ : the trust-region "ceiling" for a good action.

Sub-case (b) — lower edge $r_{t} = 1 - ϵ = 0.8$ , bad action $\hat{A}_{t} = - 3$ : 3. At $r_{t} = 0.8$ : $clip (0.8, 0.8, 1.2) = 0.8$ ; unclipped $= 0.8 \times (- 3) = - 2.4$ ; clipped $= - 2.4$ ; $min = - 2.4$ . Both equal → right-hand gradient (coming from inside the band) $= \hat{A}_{t} = - 3$ . 4. At $r_{t} = 0.8 - η$ , $η \to 0^{+}$ : clip snaps up to $0.8$ ; clipped $= - 2.4$ ; unclipped $= (0.8 - η) \cdot (- 3) > - 2.4$ (less negative, i.e. larger); $min$ picks clipped $= - 2.4$ ; left-hand gradient $= 0$ . Why this step? Value $\to - 2.4$ from both sides (continuous), slope jumps $- 3 \to 0$ : the trust-region "floor" for a bad action.

Verify: Upper edge: $lim_{r \to 1. 2^{-}} L = 3.6 = lim_{r \to 1. 2^{+}} L$ , slopes $3 \neq = 0$ . Lower edge: $lim_{r \to 0. 8^{+}} L = - 2.4 = lim_{r \to 0. 8^{-}} L$ , slopes $- 3 \neq = 0$ . Both edges continuous but kinked — the two walls of the trust region in figure s01. ✓

Worked example Example 9 — Cell I: real-world word problem (trading bot)

Statement: A trading agent's old policy took the action "buy" in state $s$ with probability $π_{o l d} = 0.40$ . After an SGD update the new policy assigns $π_{θ} = 0.60$ . The measured advantage of that buy was $\hat{A}_{t} = + 1.5$ (it beat baseline). With $ϵ = 0.2$ , is this sample still driving the update?

Forecast: Probability rose from 0.40 to 0.60 — that's a 50% relative jump, likely outside the 20% band. For a good action that means "boosted enough, stop." Guess: clipped, gradient stopped.

Ratio: $r_{t} = \frac{π _{θ}}{π _{o l d}} = \frac{0.60}{0.40} = 1.5$ . Why this step? The ratio is literally new-probability over old-probability; this is the Importance Sampling weight.
Band check: $1.5 > 1.2$ → outside the trust region on the high side.
Unclipped term: $r_{t} \hat{A}_{t} = 1.5 \times 1.5 = 2.25$ . Why this step? Our first candidate is always the raw reweighted objective.
Clip: $clip (1.5, 0.8, 1.2) = 1.2$ ; clipped term $= 1.2 \times 1.5 = 1.8$ . Why this step? The ratio left the band on the high side, so it is capped at the ceiling $1.2$ .
Min: $min (2.25, 1.8) = 1.8$ → the clipped branch won → $\partial L_{t} / \partial r_{t} = 0$ : this sample is now silent. Why this step? PPO decided that boosting "buy" from 40% to 60% in one update is already the most it will reward; pushing to 80% earns nothing extra, protecting against over-committing on one lucky trade.

Verify: $0.60/0.40 = 1.5 > 1.2$ ✓; clipped value $1.8 < 2.25$ unclipped, so $min$ chose clipped and $L_{t} = 1.8$ . The bot won't chase this action further this update — exactly the stability Trust Region Policy Optimization (TRPO) buys with a hard KL constraint, here for free. ✓

Worked example Example 10 — Cell J: exam twist (which samples are dead in a batch?)

Statement: A minibatch has four samples, all with $ϵ = 0.2$ :

#	$\hat{A}_{t}$	$r_{t}$
1	$+ 2$	$1.3$
2	$- 2$	$0.7$
3	$+ 2$	$0.9$
4	$- 2$	$1.5$

Which samples contribute zero gradient (are "dead"), and what is the summed $L^{C L I P}$ ? This is Cell J: the whole point is that "outside the band" alone does not decide life or death — the sign of $\hat{A}$ does.

Forecast: Recall the rule: a sample is dead only when the clipped branch wins. That happens for good actions above the band and bad actions below the band. So samples 1 and 2 look dead; 3 and 4 alive.

Sample 1 ( $\hat{A} > 0$ , $r = 1.3 > 1.2$ ): unclipped $2.6$ , clipped $1.2 \times 2 = 2.4$ , $min = 2.4$ → clipped, dead. Why this step? Good action pushed above the ceiling → clip caps the reward, gradient dies.
Sample 2 ( $\hat{A} < 0$ , $r = 0.7 < 0.8$ ): unclipped $- 1.4$ , clipped $0.8 \times (- 2) = - 1.6$ , $min = - 1.6$ → clipped, dead. Why this step? Bad action suppressed below the floor → clip stops further suppression.
Sample 3 ( $\hat{A} > 0$ , $r = 0.9$ inside): unclipped $=$ clipped $= 1.8$ , $min = 1.8$ → alive. Why this step? Inside the band, PPO = plain policy gradient.
Sample 4 ( $\hat{A} < 0$ , $r = 1.5 > 1.2$ ): unclipped $- 3.0$ , clipped $1.2 \times (- 2) = - 2.4$ , $min = - 3.0$ → unclipped, alive (undoing a boosted bad action). Why this step? Sample 4 is the trap: $r$ is outside the band, which fools people into calling it "clipped/dead", but for a bad action above the band the $min$ keeps the harsher unclipped term, so it's alive.
Summed objective: $2.4 + (- 1.6) + 1.8 + (- 3.0) = - 0.4$ . Why this step? $L^{C L I P}$ is the sum (or mean) over the batch; here we add the four per-sample $L_{t}$ .

Verify: Dead samples $= {1, 2}$ ; alive $= {3, 4}$ . Sum $= 2.4 - 1.6 + 1.8 - 3.0 = - 0.4$ . The exam lesson: "outside the band" ≠ "dead"; whether the $min$ picks the clipped branch depends on the sign of $\hat{A}$ . ✓

Recall Fast decision procedure

Given $\hat{A}_{t}$ and $r_{t}$ , is the sample dead? Dead only if: ( $\hat{A} > 0$ and $r > 1 + ϵ$ ) or ( $\hat{A} < 0$ and $r < 1 - ϵ$ ). ::: In all other cases the gradient flows.

Worked examples — Proximal Policy Optimization (PPO)

The objective we evaluate every time

The scenario matrix

Worked examples

Active recall