Proximal Policy Optimization for LLMs: Worked examples

This is a Deep Dive child of Proximal Policy Optimization for LLMs. The parent built the clipped surrogate and the KL leash. Here we do only one thing: we grind through every case the PPO objective can throw at you — every sign of the advantage, both directions the probability ratio can move, the exact spot where the clip bites, the degenerate ratio-equals-one case, and a couple of real training-style word problems.

Before anything, let us re-anchor the two numbers that drive every single example on this page. Everything below is arithmetic on just these two things, so let us make sure they are pictures, not symbols.

Definition The two dials: ratio and advantage

The probability ratio $ρ_{t}$ (say "rho") answers: "How much MORE (or less) likely does the NEW policy make this token, compared to the OLD one?" $ρ_{t} = \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{θ_{old}} ( a _{t} ∣ s _{t} )} = \frac{new probability of this token}{old probability of this token}$ $ρ_{t} = 1$ means "no change". $ρ_{t} = 1.6$ means "the new policy makes this token $1.6 \times$ as likely — it got MORE eager". $ρ_{t} = 0.5$ means "the new policy halved this token's chance — it backed off".
The advantage $A_{t}$ answers: "Was this token BETTER or WORSE than average from this state?" $A_{t} > 0$ = good surprise, $A_{t} < 0$ = bad surprise, $A_{t} = 0$ = exactly as expected. It comes from GAE and the critic — see Policy Gradient Methods & REINFORCE for why we subtract that baseline.

Everything the clip does is decided by the sign of $A_{t}$ crossed with which way $ρ_{t}$ moved. That is a $2 \times$ -something grid, so let us literally draw the grid first, then fill in each cell.

The scenario matrix

The clipped surrogate for one token is $L_{t}^{CLIP} = min (ρ_{t} A_{t}, clip (ρ_{t}, 1 - ϵ, 1 + ϵ) A_{t}), ϵ = 0.2 \Rightarrow band = [0.8, 1.2] .$

Here $min$ = "pick the smaller of the two numbers", and $clip (x, lo, hi)$ = "if $x$ is below lo return lo, if above hi return hi, else return $x$ itself" — it drags a stray value back to the nearest edge of the safe band.

Every token you will ever feed PPO lands in exactly one row below:

#	Cell (case class)	Sign of $A_{t}$	Where is $ρ_{t}$ ?	Does the clip bite?	What the update does
1	Good token, eager but inside band	$A_{t} > 0$	$ρ_{t} \in [1, 1.2]$	no	keeps pushing prob up
2	Good token, TOO eager	$A_{t} > 0$	$ρ_{t} > 1.2$	yes (upper)	gradient zeroed, stop pushing
3	Good token, policy backed OFF	$A_{t} > 0$	$ρ_{t} < 1$	no (min keeps unclipped)	correction still allowed
4	Bad token, policy already dropped it	$A_{t} < 0$	$ρ_{t} < 0.8$	yes (lower, via min)	keeps letting prob fall
5	Bad token, policy got MORE eager	$A_{t} < 0$	$ρ_{t} > 1$	no (min keeps unclipped)	strong corrective push down
6	Neutral / degenerate	$A_{t} = 0$ or $ρ_{t} = 1$	anywhere / exactly $1$	irrelevant	zero contribution
7	Real-world: KL leash shaving reward	(any)	—	—	reward reshaped before advantage
8	Exam twist: identical numbers, opposite conclusions	both signs	same $ρ_{t}$	depends on sign	shows sign decides everything

The single most important picture on this page is the two curves of $L_{t}^{CLIP}$ as a function of $ρ_{t}$ — one for good tokens, one for bad tokens. Study it; every example is just a dot on one of these curves.

Figure — Proximal Policy Optimization for LLMs

Read the figure: the flat plateaus are exactly where the clip has zeroed the gradient (no reward for moving further). The sloped part still active is where PPO still lets the policy move. Notice the flats are on opposite sides for good vs bad tokens — that asymmetry is the whole trick.

The six sign-and-ratio cases

Worked example Example 1 — Cell #1: good token, eager but still inside the band

$A_{t} = + 2$ , $ρ_{t} = 1.15$ , $ϵ = 0.2$ (band $[0.8, 1.2]$ ).

Forecast: the ratio $1.15$ is below the ceiling $1.2$ — do you think the clip changes anything? Guess yes/no before reading.

Unclipped surrogate $= ρ_{t} A_{t} = 1.15 \times 2 = 2.30$ . Why this step? This is the raw "reward-weighted" push before any safety.
Clip the ratio: $clip (1.15, 0.8, 1.2) = 1.15$ (it is already inside the band, untouched). Why this step? The clip only acts on values outside the band; $1.15$ is fine.
Clipped surrogate $= 1.15 \times 2 = 2.30$ .
Take the min: $min (2.30, 2.30) = 2.30$ . Why this step? The min is how PPO picks the pessimistic estimate; here both agree, so nothing is lost.

Answer: $L_{t}^{CLIP} = 2.30$ , gradient still active — the model is still encouraged to make this good token even more likely.

Verify: on the good-token curve in the figure, $ρ = 1.15$ is still on the rising line, not yet on the plateau. Consistent. ✓

Worked example Example 2 — Cell #2: good token, TOO eager (clip bites)

$A_{t} = + 2$ , $ρ_{t} = 1.6$ . (This is the parent note's Example 1 — we re-derive it as our canonical "upper clip" case.)

Forecast: the ratio blew past the $1.2$ ceiling. Will PPO reward that extra eagerness, or refuse?

Unclipped $= 1.6 \times 2 = 3.2$ .
Clip the ratio: $clip (1.6, 0.8, 1.2) = 1.2$ (dragged down to the ceiling). Why this step? $1.6 > 1.2$ , so the clip snaps it to the top edge — this is the safety cap acting.
Clipped surrogate $= 1.2 \times 2 = 2.4$ .
Take the min: $min (3.2, 2.4) = 2.4$ . Why this step? Since $A_{t} > 0$ , the clipped value is the smaller one; the min forces us to use it, so pushing $ρ$ past $1.2$ earns nothing extra.

Answer: $L_{t}^{CLIP} = 2.4$ ; gradient w.r.t. $ρ$ beyond $1.2$ is zero — PPO refuses to reward over-eagerness.

Verify: on the figure this dot sits on the flat plateau of the good-token curve, at height $1.2 \times 2 = 2.4$ . ✓

Worked example Example 3 — Cell #3: good token, but the policy accidentally BACKED OFF

$A_{t} = + 2$ , $ρ_{t} = 0.7$ (new policy made this good token less likely — a mistake we want to correct).

Forecast: $0.7$ is below the lower edge $0.8$ . Does the clip block the correction, or let PPO fix it?

Unclipped $= 0.7 \times 2 = 1.4$ .
Clip the ratio: $clip (0.7, 0.8, 1.2) = 0.8$ .
Clipped surrogate $= 0.8 \times 2 = 1.6$ .
Take the min: $min (1.4, 1.6) = 1.4$ → the unclipped value wins. Why this step? For a good token we want the ratio to go back up toward and past $1$ . The min keeps the unclipped branch, whose gradient is still nonzero, so PPO allows the correction. The clip did not block us from fixing an under-eager good token.

Answer: $L_{t}^{CLIP} = 1.4$ , gradient active — correction toward more probability is permitted.

Verify: on the good-token curve, $ρ = 0.7$ is on the sloped (active) part to the left of $1$ , not on any plateau. ✓

Worked example Example 4 — Cell #4: bad token, policy already dropped it (parent's Example 2)

$A_{t} = - 3$ , $ρ_{t} = 0.5$ .

Forecast: the token is bad and the policy already halved its probability. Will PPO keep pushing it down, or say "far enough"?

Unclipped $= 0.5 \times (- 3) = - 1.5$ .
Clip the ratio: $clip (0.5, 0.8, 1.2) = 0.8$ .
Clipped surrogate $= 0.8 \times (- 3) = - 2.4$ .
Take the min: $min (- 1.5, - 2.4) = - 2.4$ → the clipped value wins (it is more negative). Why this step? With $A_{t} < 0$ the min picks the more negative number. Because $- 2.4$ comes from the clipped-at- $0.8$ branch, and $0.5$ is already below $0.8$ , the gradient here still pushes the probability further down — clipping only limits over-optimistic moves, never corrective decreases of a bad token.

Answer: $L_{t}^{CLIP} = - 2.4$ , correction still allowed to drive the bad token's probability lower.

Verify: on the bad-token curve the active (sloped) region is on the left; $ρ = 0.5$ sits there, correction ongoing. ✓

Worked example Example 5 — Cell #5: bad token, policy got MORE eager (the clip must brake hard)

$A_{t} = - 3$ , $ρ_{t} = 1.4$ (new policy is increasing the chance of a bad token — dangerous).

Forecast: should the safety cap kick in here, or let the surrogate run?

Unclipped $= 1.4 \times (- 3) = - 4.2$ .
Clip the ratio: $clip (1.4, 0.8, 1.2) = 1.2$ .
Clipped surrogate $= 1.2 \times (- 3) = - 3.6$ .
Take the min: $min (- 4.2, - 3.6) = - 4.2$ → unclipped wins. Why this step? For a bad token that the policy made more likely, the unclipped value is the more negative one, so the min keeps it. Its gradient is still active and large, so PPO gives a strong push to reduce the probability. The clip does not shield a bad-and-getting-worse token.

Answer: $L_{t}^{CLIP} = - 4.2$ , strong corrective gradient.

Verify: on the bad-token curve, $ρ = 1.4$ lies on the still-active (rising-loss-in-magnitude) part above $1$ . ✓

Worked example Example 6 — Cell #6: the degenerate cases (

$A_{t} = 0$ and $ρ_{t} = 1$ ) Two sub-cases that must never surprise you.

(a) $A_{t} = 0$ , any $ρ_{t}$ (say $ρ_{t} = 1.9$ ).

Unclipped $= 1.9 \times 0 = 0$ . Clipped $= 1.2 \times 0 = 0$ . $min (0, 0) = 0$ . Why? If a token was exactly as good as average, there is nothing to learn from it — zero contribution regardless of how the ratio moved.

(b) $ρ_{t} = 1$ exactly (new policy identical to old for this token), $A_{t} = + 2$ .

Unclipped $= 1 \times 2 = 2$ . Clipped $= clip (1, 0.8, 1.2) \times 2 = 1 \times 2 = 2$ . $min (2, 2) = 2$ . Why? Right at the start of each PPO epoch, $π_{θ} = π_{θ_{old}}$ , so every ratio is $1$ and we sit exactly at the middle of the plateau-free zone — the surrogate reduces to plain advantage-weighted policy gradient. This is the sanity check that PPO starts as ordinary policy gradient and only diverges as $θ$ moves.

Answers: (a) $0$ , (b) $2$ .

Verify: in the figure both curves pass through the point $(ρ = 1)$ with value $A_{t}$ , and both are flat-zero along the $A_{t} = 0$ line. ✓

Case #7 — Real-world word problem: the KL leash reshapes reward

Before any advantage is computed, the per-token reward already has the KL penalty baked in (see KL Divergence and Reinforcement Learning from Human Feedback (RLHF)). We must show how a raw reward-model score becomes the number PPO actually optimizes.

Worked example Example 7 — Cell #7: KL-shaped reward across a 3-token response

A response of 3 tokens gets a terminal reward-model score $r_{ϕ} = 1.0$ (paid only at the last token). The KL coefficient is $β = 0.2$ . Per-token new/reference probabilities:

token $t$	$π_{θ}$	$π_{ref}$	$lo g (π_{θ} / π_{ref})$
1	$0.9$	$0.3$	$lo g 3 = 1.0986$
2	$0.5$	$0.5$	$lo g 1 = 0$
3	$0.4$	$0.8$	$lo g 0.5 = - 0.6931$

The per-token reward is $r_{t} = r_{ϕ} [t = T] - β lo g \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{ref} ( a _{t} ∣ s _{t} )}$ .

Forecast: which token gets punished by the leash, and which one actually gets a tiny bonus? Guess before computing.

Token 1 (not terminal): $r_{1} = 0 - 0.2 (1.0986) = - 0.2197$ . Why? The policy got greedy — $0.9$ vs the reference's $0.3$ . Positive log-ratio ⇒ the leash shaves reward to discourage drift.
Token 2: $r_{2} = 0 - 0.2 (0) = 0$ . Why? Policy agrees with the reference (ratio $1$ ) — no drift, no penalty.
Token 3 (terminal, $t = T$ ): $r_{3} = 1.0 - 0.2 (- 0.6931) = 1.0 + 0.1386 = 1.1386$ . Why? Here the policy is less eager than the reference (log-ratio negative), so $- β \times (negative)$ is a small bonus — moving back toward the reference is gently encouraged, and the terminal RM score lands here.

Answers: $r_{1} = - 0.2197$ , $r_{2} = 0$ , $r_{3} = 1.1386$ . Total shaped reward $= 0.9189$ .

Verify: total $= 1.0 - 0.2 (1.0986 + 0 - 0.6931) = 1.0 - 0.2 (0.4055) = 1.0 - 0.0811 = 0.9189$ . Matches the token sum. Units: all in "reward-model points". ✓

Case #8 — Exam twist: same numbers, opposite verdicts

Worked example Example 8 — Cell #8: identical

$ρ_{t} = 1.5$ , flip the sign of $A_{t}$ An exam gives you $ρ_{t} = 1.5$ (above the $1.2$ ceiling). Compute $L_{t}^{CLIP}$ for (a) $A_{t} = + 2$ and (b) $A_{t} = - 2$ , and explain why the clip helps in one case but not the other.

Forecast: the ratio is identical. Does the clip bite in both cases, or only one?

(a) $A_{t} = + 2$ :

Unclipped $= 1.5 \times 2 = 3.0$ ; Clipped $= 1.2 \times 2 = 2.4$ .
$min (3.0, 2.4) = 2.4$ → clip bites, gradient zeroed. Why? Good token pushed too far up ⇒ PPO caps the reward. (Cell #2 behaviour.)

(b) $A_{t} = - 2$ :

Unclipped $= 1.5 \times (- 2) = - 3.0$ ; Clipped $= 1.2 \times (- 2) = - 2.4$ .
$min (- 3.0, - 2.4) = - 3.0$ → unclipped wins, clip does NOT bite. Why? Bad token that got more likely ⇒ PPO wants the full corrective push, so the more-negative unclipped value is kept. (Cell #5 behaviour.)

Answers: (a) $2.4$ , (b) $- 3.0$ .

The lesson: the same ratio produces a clipped result when $A_{t} > 0$ and an unclipped result when $A_{t} < 0$ . The sign of the advantage — not the ratio alone — decides whether the clip acts. This is the single most common exam trap, and it is exactly the asymmetry drawn in Figure 1.

Verify: on Figure 1, drop a vertical line at $ρ = 1.5$ : it hits the plateau of the good curve (2.4) but the still-sloped part of the bad curve ( $- 3.0$ ). ✓

Recall Scenario matrix — cover and recall

Clip bites for a good token when ::: $ρ_{t} > 1 + ϵ$ (ratio above the ceiling). Clip bites for a bad token when ::: $ρ_{t} < 1 - ϵ$ (ratio below the floor), through the min picking the clipped-and-more-negative branch. If $A_{t} = 0$ the surrogate contribution is ::: exactly $0$ , for any ratio. If $ρ_{t} = 1$ the surrogate equals ::: $A_{t}$ itself — PPO reduces to plain advantage-weighted policy gradient. Same $ρ_{t} = 1.5$ , $A_{t} = + 2$ vs $A_{t} = - 2$ give ::: $2.4$ (clipped) and $- 3.0$ (unclipped) — the sign of $A$ decides. The KL leash punishes a token when ::: $π_{θ} > π_{ref}$ (positive log-ratio, policy drifting away from reference).

Related deep tools you may want next: Direct Preference Optimization (DPO) (skips PPO's rollout loop entirely), Supervised Fine-Tuning (SFT) (the reference model), and Reward Hacking & Specification Gaming (why the KL leash exists at all).