Proximal Policy Optimization for LLMs: Question bank

This is a child of Proximal Policy Optimization for LLMs. Terms are used as defined there: the ratio $ρ_{t} = π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{old}} (a_{t} ∣ s_{t})$ (new prob over old prob), the advantage $A_{t}$ (how much better than average this token was), the clip band $[1 - ϵ, 1 + ϵ]$ , and the KL leash to a frozen reference. If any symbol feels unfamiliar, re-read the parent's "Building the objective" section first.

True or false — justify

Every item: state true/false AND the reason.

The clip in PPO directly guarantees the policy moves at most

ϵ

per update

False — the clip only zeroes the gradient once the ratio leaves the band on the reward-improving side; across many tokens and many epochs the policy can still drift far, which is why a separate KL leash exists.

Subtracting the value baseline

V (s)

changes the true expected policy gradient

False — the baseline is independent of the action, so its contribution has expectation zero; it only shrinks variance, leaving the expected gradient unchanged.

PPO needs a critic (value network); REINFORCE does not

True — REINFORCE can run baseline-free, but PPO's advantage

A_{t} = Q - V

requires a learned value estimate

V

to compute, so a critic is standard.

The KL penalty and the clip do the same job, so one is redundant

False — the clip is a local per-update step-size guard (it stops over-optimistic ratio moves), while the KL-to-reference is a global leash keeping the whole policy near the SFT distribution; removing either breaks a different failure mode.

Reward hacking means PPO has a bug

False — it is an inherent hazard of optimizing a learned, imperfect reward model; PPO's KL-to-reference term is the built-in mitigation, see Reward Hacking & Specification Gaming.

Because we take

min

of two surrogate terms, PPO always applies a smaller (more punishing) update

False — the

min

picks the pessimistic surrogate, which combined with the sign of

A_{t}

often yields exactly zero gradient; it is a brake, not a penalty.

We can backpropagate gradients from the reward model directly into

π_{θ}

False — token generation is discrete sampling, which is non-differentiable; that is precisely why we use the log-derivative (policy-gradient) trick instead, see Policy Gradient Methods & REINFORCE.

Importance sampling in PPO exists to make updates more accurate

False — its purpose is data reuse: LLM rollouts are expensive, so the ratio reweighting lets us take several gradient steps on one batch while keeping the estimate valid.

The reward model produces a score for every single token

False — the RM scores the full response; that terminal score is placed at the last token (

t = T

), and the only per-token signal along the way is the KL penalty.

Entropy bonus and KL penalty push in the same direction

False (subtly) — the entropy bonus encourages exploration/spread in the current policy, while the KL leash pulls the policy toward a fixed reference; they can conflict, and the coefficients

c_{2}, β

balance them.

A higher

ϵ

(wider clip band) makes training more stable

False — a wider band permits larger ratio moves before clipping bites, so updates get more aggressive and less stable; small

ϵ \approx 0.2

is the safe default.

Spot the error

Each line: find what is wrong in the claim.

"With

A_{t} > 0

and

ρ_{t} = 1.6

ϵ = 0.2

, PPO uses the value

3.2

to push harder."

Wrong — clipping caps the ratio at

1.2

, giving surrogate

2.4

, and

min (3.2, 2.4) = 2.4

; the gradient past

ρ_{t} = 1.2

is zero, so it does not push harder.

"For a bad token (

A_{t} < 0

), the clip stops the model from lowering that token's probability."

Wrong — clipping only limits over-optimistic moves; for negative advantage the

min

selects the more-negative surrogate, so the model is still free to keep decreasing that probability (corrective moves are allowed).

"The advantage is just the reward-model score for the response."

Wrong — the advantage is

A = Q - V

, a relative quantity measuring better/worse than the critic's baseline; equating it with raw reward reintroduces the high variance the baseline was meant to remove.

"The ratio

ρ_{t}

compares the new policy to the frozen reference model."

Wrong —

ρ_{t}

compares the new policy to

π_{θ_{old}}

(the rollout-collecting policy). The reference

π_{ref}

appears only in the KL penalty, and those are usually different snapshots.

"Because clip caps the ratio to

[0.8, 1.2]

, one token can never change probability by more than 20%."

Wrong — the cap is on the surrogate objective's gradient contribution, not on the underlying probability; over multiple epochs on the same batch the actual token probability can move well beyond 20%.

"PPO maximizes reward directly, so more reward is always better for alignment."

Wrong — maximizing an imperfect learned reward invites reward hacking; the KL leash deliberately sacrifices some reward to stay in-distribution where the RM is trustworthy.

"Increasing

β

(KL coefficient) makes the model chase reward more aggressively."

Wrong — a larger

β

tightens the leash to the reference, penalizing drift more, so the model chases reward less aggressively.

Why questions

Why do we subtract a baseline instead of just using raw reward in the gradient?

Raw reward gives enormous variance and only tells the model "big/small number"; subtracting

V (s)

(mean-zero in expectation) makes the signal "better/worse than average", which is far more stable, see Generalized Advantage Estimation (GAE).

Why is PPO preferred over vanilla REINFORCE for a 7B LLM?

REINFORCE takes large noisy steps that can push the policy into gibberish (catastrophic forgetting); PPO gives trust-region-like stability via clipping while staying first-order (plain SGD), see Policy Gradient Methods & REINFORCE.

Why add a KL-to-reference penalty when we already clip?

The RM is only accurate near the SFT distribution; clipping bounds each local update but lets the policy drift globally, so the KL leash keeps the whole policy in-distribution where the reward is meaningful, see KL Divergence.

Why use the

min

of clipped and unclipped surrogates instead of just the clipped one?

The

min

makes the estimate pessimistic: it removes any incentive to overshoot on the reward-improving side while still allowing corrective moves, giving a one-sided guard that a plain clip alone would not.

Why can't we just fine-tune on reward with ordinary supervised gradients?

There is no differentiable path from "a human/RM liked this sampled text" back to the weights; the reward is a black box over discrete samples, forcing the policy-gradient formulation, see Reinforcement Learning from Human Feedback (RLHF).

Why does DPO skip PPO's clip-and-KL machinery entirely?

DPO reparameterizes the RLHF objective so preferences train the policy directly with a supervised-style loss, eliminating the online rollouts, critic, ratio and clip — see Direct Preference Optimization (DPO).

Why place the reward only at the final token rather than spreading it?

The reward model judges the whole response, so its score is only defined at completion (

t = T

); the intermediate tokens get their per-step signal from the KL term, not from partial RM scores.

Why does the reference model stay frozen during PPO?

It provides a fixed anchor for the KL leash; if it moved with the policy, "distance from reference" would shrink automatically and the leash would stop restraining drift, defeating its purpose.

Edge cases

A_{t} = 0

for a token, what does PPO do to it?

Nothing — every surrogate term is

ρ_{t} \cdot 0 = 0

, so the gradient contribution is zero regardless of the ratio; the update ignores neutral tokens.

ρ_{t} = 1

exactly (new policy matches old on this token), is the clip active?

No —

1

sits inside

[1 - ϵ, 1 + ϵ]

, so clip and unclip agree and the surrogate is simply

A_{t}

; the clip only matters once the ratio leaves the band.

What happens as

ϵ \to 0

The clip band collapses to

{1}

, so almost any ratio movement is clipped and gradients vanish — the policy freezes, learning nothing;

ϵ

must be positive to make progress.

What happens as

β \to 0

(KL weight off)?

The leash disappears, so the policy is free to drift far from the reference and exploit the reward model's blind spots — the classic setup for reward hacking, see Reward Hacking & Specification Gaming.

What if

π_{ref}

assigns near-zero probability to a token the policy now favors?

The KL term

lo g (π_{θ} / π_{ref})

blows up, producing a large per-token penalty that strongly discourages that drift — this is the leash biting hard on out-of-distribution moves.

If we only ever do ONE gradient step per batch, is importance sampling still needed?

No practical benefit — with a single step

ρ_{t} = 1

for all tokens (new policy equals old), so the ratio machinery is only worthwhile when reusing a batch for multiple epochs.

What if the value network

V

is badly miscalibrated?

The advantage estimates become inaccurate and noisy, so even though the expected gradient is unbiased, effective variance rises and training destabilizes — a good critic is a practical necessity, see Generalized Advantage Estimation (GAE).

For a large negative

A_{t}

, does a wide clip band protect against a big update?

No — on the negative-advantage side the

min

selects the more-negative (unclipped-or-clipped) surrogate to allow correction, so the clip provides little protection there; the protection is one-sided toward optimistic moves.

Recall One-line self-test after this bank

Clip vs KL — which is local and which is global? ::: Clip is the local per-update step guard; KL is the global leash to the reference.

NONE