Proximal Policy Optimization for LLMs: Exercises

Before we start, one shared reminder so nothing is used unexplained.

Definition The five quantities you will keep meeting

Probability ratio $ρ_{t} = \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{θ_{old}} ( a _{t} ∣ s _{t} )}$ — "how much more (or less) likely the new policy makes this token compared to the old policy that generated the data." $ρ = 1$ means no change; $ρ = 2$ means twice as likely; $ρ = 0.5$ means half as likely.
Advantage $A_{t}$ — a single number: positive = this token was better than average here, negative = worse than average. Built as $A = Q - V$ (see Generalized Advantage Estimation (GAE)).
Clip band $[1 - ϵ, 1 + ϵ]$ — the safe zone for $ρ$ . With the usual $ϵ = 0.2$ that is $[0.8, 1.2]$ .
Clipped surrogate $L^{CLIP} = min (ρ A, clip (ρ, 1 - ϵ, 1 + ϵ) A)$ .
Per-token KL penalty $- β lo g \frac{π _{θ} ( a _{t} ∣ s _{t} )}{π _{ref} ( a _{t} ∣ s _{t} )}$ — the leash to the frozen reference (see KL Divergence).

The figure below is your reference picture for every clip question — keep it open.

Figure — Proximal Policy Optimization for LLMs

Level 1 — Recognition

Exercise 1.1

Match each symbol to its plain-English meaning: (a) $ρ_{t}$ , (b) $A_{t}$ , (c) $β$ , (d) $ϵ$ .

Recall Solution

(a) $ρ_{t}$ = ratio of new-policy to old-policy probability for one token.
(b) $A_{t}$ = advantage — how much better than average that token was.
(c) $β$ = strength of the KL leash pulling the policy toward the reference.
(d) $ϵ$ = half-width of the clip band; $ρ$ is allowed to roam in $[1 - ϵ, 1 + ϵ]$ .

Exercise 1.2

True or false: "The clip caps how much the loss can grow when we push a good token ( $A_{t} > 0$ ) further." Explain in one sentence.

Recall Solution

True. For $A_{t} > 0$ the surrogate is capped at $(1 + ϵ) A_{t}$ ; once $ρ$ exceeds $1 + ϵ$ the term stops growing, so there is no reward for pushing that token's probability higher — the gradient there becomes zero.

Exercise 1.3

Which single term in the full LLM-RLHF objective is responsible for keeping the model "in-distribution" so the reward model stays trustworthy?

Recall Solution

The per-token KL penalty to the reference, $- β lo g \frac{π _{θ}}{π _{ref}}$ . The reward model is only accurate near the Supervised Fine-Tuning (SFT) distribution; the KL leash keeps you there. This is the mitigation for Reward Hacking & Specification Gaming.

Level 2 — Application

Exercise 2.1

Given $A_{t} = + 2$ , $ϵ = 0.2$ , $ρ_{t} = 1.6$ . Compute the unclipped surrogate, the clipped surrogate, and the value PPO actually uses.

Recall Solution

Unclipped: $ρ_{t} A_{t} = 1.6 \times 2 = 3.2$ .
Clip $ρ$ : $clip (1.6, 0.8, 1.2) = 1.2$ , so clipped surrogate $= 1.2 \times 2 = 2.4$ .
PPO takes the min: $min (3.2, 2.4) = 2.4$ . The min discards the over-optimistic $3.2$ : we refuse to reward pushing $ρ$ past $1.2$ .

Exercise 2.2

Given $A_{t} = - 3$ , $ϵ = 0.2$ , $ρ_{t} = 0.5$ . Compute all three quantities.

Recall Solution

Unclipped: $0.5 \times (- 3) = - 1.5$ .
Clip: $clip (0.5, 0.8, 1.2) = 0.8$ , clipped $= 0.8 \times (- 3) = - 2.4$ .
$min (- 1.5, - 2.4) = - 2.4$ . For a bad token we take the more negative (pessimistic) value, so the policy is still permitted to keep lowering this token's probability. Clipping restrains over-optimism, never correction.

Exercise 2.3

A full response gets reward-model score $r_{ϕ} = 1.0$ . On one token the new policy assigns probability $0.9$ while the reference assigned $0.3$ , and $β = 0.2$ . Compute the KL penalty and the effective end-of-sequence reward.

Recall Solution

Log-ratio: $lo g (0.9/0.3) = lo g 3 \approx 1.0986$ .
KL penalty: $β \times 1.0986 = 0.2 \times 1.0986 = 0.2197$ .
Effective reward: $1.0 - 0.2197 = 0.7803$ . The policy got greedy on a token the reference was unsure about, so PPO shaves reward to discourage drift. See Reward Modeling from Human Preferences.

Exercise 2.4

With $ϵ = 0.2$ and $A_{t} = + 2$ , at what value of $ρ_{t}$ does the clipped surrogate first stop increasing?

Recall Solution

At $ρ_{t} = 1 + ϵ = 1.2$ . For $ρ_{t} \geq 1.2$ the clipped term is frozen at $(1.2) (2) = 2.4$ , and since $ρ A \geq 2.4$ there, the min stays at $2.4$ : gradient $= 0$ .

Level 3 — Analysis

Exercise 3.1

For a good token ( $A_{t} > 0$ ), sketch/describe the surrogate $L^{CLIP}$ as a function of $ρ$ over $[0, 2]$ . Where is the gradient nonzero, and where is it zero? Reference the figure.

Recall Solution

Look at the left panel of the figure (magenta $A > 0$ curve).

For $ρ \leq 1 + ϵ$ (i.e. $ρ \leq 1.2$ ): the min equals the unclipped line $ρ A$ , which rises — gradient nonzero, so we keep encouraging the token.
For $ρ > 1.2$ : the min equals the flat clipped value $(1.2) A$ — gradient zero, no further push. So for good tokens PPO rewards improvement only up to the ceiling $1 + ϵ$ , then flatlines.

Exercise 3.2

Now the bad token ( $A_{t} < 0$ ). Describe the same surrogate shape (right panel) and say where the gradient dies.

Recall Solution

Look at the right panel (violet $A < 0$ curve). Because $A < 0$ :

For $ρ \geq 1 - ϵ$ (i.e. $ρ \geq 0.8$ ): min equals the unclipped line $ρ A$ , which falls as $ρ$ rises — gradient nonzero, so the model keeps lowering the probability of the bad token.
For $ρ < 0.8$ : min equals the flat clipped value $(0.8) A$ — gradient zero. Once we've already cut the probability far enough ( $ρ$ below the floor), PPO stops over-penalizing. Symmetric story: the flat region is on the opposite side of $ρ = 1$ compared to the good-token case.

Exercise 3.3

A student claims: "Because $ρ$ is clipped to $[0.8, 1.2]$ , one PPO update can change any token's probability by at most 20%." Refute this precisely.

Recall Solution

False. Clipping does not clamp $ρ$ itself — the actual new probability can move $ρ$ anywhere; clipping only clamps the value used inside the surrogate, which zeroes the gradient contribution of that token once $ρ$ leaves the band on the improving side. But:

Other tokens in the sequence still contribute gradient.
We take several gradient steps per batch (importance sampling, Step 4 of parent), so cumulative movement compounds. Hence the extra KL penalty is required as the true global leash. This is exactly Mistake A in the parent note.

Level 4 — Synthesis

Exercise 4.1

Combine both mechanisms. A token has $A_{t} = + 1.5$ , $ρ_{t} = 1.4$ , $ϵ = 0.2$ . The same token has $π_{θ} = 0.8$ , $π_{ref} = 0.5$ , $β = 0.1$ . Compute (i) the clipped surrogate contribution and (ii) the KL penalty, and state the net "shaped" signal $= L^{CLIP} - KL$ .

Recall Solution

(i) Clipped surrogate: unclipped $= 1.4 \times 1.5 = 2.1$ ; clipped $= clip (1.4, 0.8, 1.2) \times 1.5 = 1.2 \times 1.5 = 1.8$ ; min $= 1.8$ . (ii) KL penalty: $β lo g (0.8/0.5) = 0.1 \times lo g (1.6) = 0.1 \times 0.4700 = 0.04700$ . (iii) Net shaped signal $= 1.8 - 0.04700 = 1.7530$ . The clip capped the greedy over-reward at $1.8$ ; the KL further shaves a little because the policy over-committed relative to the reference.

Exercise 4.2

Contrast with Direct Preference Optimization (DPO): PPO needs a reward model + rollouts + clip + KL, DPO folds preference and KL into a single closed-form loss. In one sentence, which PPO ingredient does DPO's implicit-reward derivation replace, and what does DPO lose?

Recall Solution

DPO replaces the explicit reward model + PPO sampling loop by deriving the optimal KL-regularized policy in closed form, so it needs no rollouts or clip. What it loses: the ability to learn from fresh on-policy samples and from a separately reusable reward model, and it is more sensitive to the fixed preference dataset. (See Reinforcement Learning from Human Feedback (RLHF) for the pipeline both sit inside.)

Exercise 4.3

Explain why the baseline/critic $V (s)$ can be subtracted "for free." Show the expectation argument in one line.

Recall Solution

The extra term is $E_{a \sim π_{θ}} [V (s) \nabla_{θ} lo g π_{θ} (a ∣ s)]$ . Since $V (s)$ does not depend on $a$ , pull it out: $V (s) E [\nabla_{θ} lo g π_{θ}] = V (s) \nabla_{θ} \sum_{a} π_{θ} (a ∣ s) = V (s) \nabla_{θ} 1 = 0$ . So subtracting the baseline changes the expected gradient by zero, while slashing its variance. See Policy Gradient Methods & REINFORCE.

Level 5 — Mastery

Exercise 5.1

Design. You observe PPO training where reward keeps rising but human raters say outputs are getting worse (classic Reward Hacking & Specification Gaming). You have two knobs: $ϵ$ (clip width) and $β$ (KL strength). Which do you change and why? What is the failure mode if you push it too far?

Recall Solution

Increase $β$ (tighten the KL leash). Rising RM reward with falling human quality means the policy has drifted out of the region where the reward model is calibrated and is now exploiting RM blind spots. The clip $ϵ$ only affects per-token step size, not global drift, so it cannot fix distributional wandering. Too far: if $β$ is huge the policy is pinned to the reference and never improves — you get SFT behavior back with wasted compute. Tune $β$ so the running KL sits in a target band (adaptive-KL controllers do exactly this).

Exercise 5.2

Prove the clip is one-sided. Show that for $A_{t} > 0$ the surrogate is bounded above by $(1 + ϵ) A_{t}$ but is not bounded below by the clip (small $ρ$ still contributes), and state the symmetric fact for $A_{t} < 0$ .

Recall Solution

Let $A = A_{t} > 0$ . Define $g (ρ) = min (ρ A, clip (ρ, 1 - ϵ, 1 + ϵ) A)$ .

For $ρ \geq 1 + ϵ$ : $clip = 1 + ϵ$ , and $ρ A \geq (1 + ϵ) A$ , so $min = (1 + ϵ) A$ . Hence $g \leq (1 + ϵ) A$ — an upper bound; pushing $ρ$ higher adds nothing.
For $ρ \leq 1 - ϵ$ : $clip = 1 - ϵ$ , but $ρ A \leq (1 - ϵ) A$ , so $min = ρ A$ (the smaller one), which keeps falling with $ρ$ — no clip protection on the low side. So for good tokens the clip caps over-optimism only. For $A < 0$ everything mirrors: the clip caps at $(1 - ϵ) A$ on the low- $ρ$ side, while large $ρ$ keeps producing the unclipped (more negative) value — the correction stays alive. This one-sidedness is the whole point of the $min$ .

Exercise 5.3

Synthesis under a degenerate case. What is $L^{CLIP}$ when $A_{t} = 0$ ? What does this say about tokens the model already rates as exactly average?

Recall Solution

$L^{CLIP} = min (ρ \cdot 0, clip (ρ, \dots) \cdot 0) = min (0, 0) = 0$ for every $ρ$ . Its derivative in $ρ$ is $0$ everywhere, so such tokens produce no gradient. Interpretation: PPO only moves probability mass toward better-than-average or away from worse-than-average tokens; a token exactly at the baseline is left untouched — the advantage, not the raw reward, is the true learning signal.