Foundations — Proximal Policy Optimization for LLMs

This page assumes nothing. If the parent note Proximal Policy Optimization for LLMs used a symbol, we build it here from a picture first. Read top to bottom — each item is a brick for the next.

The cast of characters, in build order

1. A "token" — the atom of text

A language model does not write whole sentences at once. It writes one piece at a time: a word, or part of a word, called a token. Think of a train being built one carriage at a time.

Figure 1 below shows this: five boxes, one per token, with an arrow marking the next token as the thing we're about to choose.

Why the topic needs it: everything PPO does is decided per token. The reward, the leash, the clip — all act on individual carriages of the train.

2. State $s$ and action $a$ — the fork in the road

At any moment the model has written some tokens already. That "everything written so far, including the prompt" is the state $s$. The next token it is about to choose is the action $a$.

In Figure 2 the blue road is the state $s$; the three coloured arrows leaving the white "fork" dot are the possible actions $a$. Picking one action lands you in a new state (old road + one more token) — so states and actions chain into a whole path, called a trajectory $\tau$.

Why "state" and "action"? These are borrowed from reinforcement learning, where an agent in a state chooses an action. Framing text generation this way is exactly what lets us use RL tools. See Policy Gradient Methods & REINFORCE.

3. The policy ${\pi }_{\theta }(a\mid s)$ — a probability, not a decision

At the fork, the model does not pick one branch with certainty. It assigns a probability to each possible next token. That whole rulebook — "given state $s$, how likely is each action $a$?" — is the policy, written ${\pi }_{\theta }(a\mid s)$.

Let us earn every piece of that notation:

Why the topic needs it: PPO's entire job is to adjust $\theta$ so good tokens get higher probability. The policy is the thing being trained.

4. The reward — per-token $r_t$ vs the response score $r_{\varphi }$

We want to know: was this good? A reward is a single number saying "how much we liked it". But when is it handed out? This trips people up, so we pin it down.

How partial trajectories are handled: because the RM only judges finished text, intermediate tokens get no task reward of their own — only the KL term. Credit for the final score is spread back over earlier tokens through the advantage (§7), not by the reward itself.

Why a learned judge? We cannot differentiate "a human liked this." A learned $r_{\varphi }$ turns fuzzy human taste into a computable number. The catch, flagged later, is Reward Hacking & Specification Gaming: a learned judge can be fooled.

5. Expectation $\mathbb{E}[\cdot ]$ and the objective $J(\theta )$

The model is random (it samples tokens), so a single response tells us little. We care about the average score across many samples. That average is written $\mathbb{E}$ (a fancy "E" for Expectation).

Now we can state precisely what PPO maximizes — the average cumulative reward over a full generation:

Why the topic needs it: $J(\theta )$ is the mountain we climb. Everything after this — gradients, advantage, clipping — is machinery for climbing it safely.

6. The gradient ${\nabla }_{\theta }$ and the policy-gradient estimator

We want to increase $J$. In which direction should we nudge the knobs $\theta$? The gradient ${\nabla }_{\theta }J$ is an arrow that points in the direction of steepest increase of $J$. Follow it a little, and $J$ goes up (Figure 3).

Putting the trick together with the advantage (defined next) gives the estimator every policy-gradient method — and PPO — is built on:

Why calculus here and not guess-and-check? With billions of knobs, random guessing is hopeless. The gradient gives the single best direction to step. Foundations live in Policy Gradient Methods & REINFORCE.

7. Value $V_{\psi }(s)$, action-value $Q(s,a)$, and advantage $A(s,a)$

Raw reward is noisy. We want a sharper question: was this token better or worse than what we usually expect from here?

• The value $V_{\psi }(s)$ = the average discounted reward we expect from state $s$ onward, before choosing an action. It is estimated by a learned network, the critic, whose own knobs we call $\psi$ (Greek "psi") — hence the subscript, to keep it separate from the policy's $\theta$ and the reward model's $\varphi$.
• The action-value $Q(s,a)$ = expected discounted reward if we take action $a$ in state $s$, then continue.
• The advantage $A(s,a)=Q(s,a)-V_{\psi }(s)$ = "how much better than average was this action."

Figure 4 draws this: the dashed blue line is the baseline $V_{\psi }(s)$; each coloured stem is a token's $Q$, and the gap between stem and line is its advantage — green above (good), red below (bad), yellow flush (average).

Why subtract $V_{\psi }$ (the "baseline")? Subtracting a quantity that doesn't depend on the action leaves the average gradient unchanged but shrinks its jitter enormously. The model learns "better than usual" instead of "big raw number." Practical estimation uses Generalized Advantage Estimation (GAE).

8. The ratio ${\rho }_t(\theta )$ — new-you vs old-you

PPO gathers responses from an old copy of the policy ${\pi }_{{\theta }_{\text{old}}}$, then updates several times. To keep the maths honest when reusing that old data, we compare the new and old probabilities of the same token as a ratio.

Why a ratio and not a difference? Reusing old samples correctly (called importance sampling) requires reweighting each sample by exactly this ratio of "how likely now" over "how likely then." A difference would not correct the reuse; the ratio does.

9. The clip, $ϵ$, and the surrogate objective $L^{\text{CLIP}}$

If ${\rho }_t$ balloons, one token could yank the whole model. The clip flattens the ratio so it can't leave a safe band $[1-ϵ,1+ϵ]$.

First, let us nail down exactly which average the ${\mathbb{E}}_t$ below means:

Now assemble the ratio, the advantage, and the clip into the quantity PPO adjusts $\theta$ to make large:

Why clip? Once you've improved a token "enough," clipping makes the gradient go flat there — no reward for pushing further. It's a brake on over-eager moves, applied per token. Concretely: when $A_t>0$ the term is capped at $(1+ϵ)A_t$, so once ${\rho }_t$ passes $1+ϵ$ the gradient for that token becomes zero; when $A_t<0$ it is capped at $(1-ϵ)A_t$, limiting how hard a correction is pushed. The $\min$ makes the choice always the more conservative one.

10. KL divergence, $\beta$, and the reference ${\pi }_{\text{ref}}$ — the global leash

The clip is a local guard. Across thousands of tokens the model could still wander far from sensible language. So we add a global leash: a penalty for drifting away from a frozen reference policy ${\pi }_{\text{ref}}$.

The distance between two probability rulebooks is measured by the KL divergence. Full theory lives in KL Divergence; here is the piece we need.

Why the leash? The reward model $r_{\varphi }$ is only accurate near where the reference lives. Stray too far and the judge is fooled (Reward Hacking & Specification Gaming). The KL keeps the model in trustworthy territory. This whole loop — reward model + PPO + KL leash — is Reinforcement Learning from Human Feedback (RLHF). A leash-free alternative is Direct Preference Optimization (DPO).

How the foundations feed the topic

Every arrow is a "you need this before that." Follow any path top to bottom and you reconstruct the parent note's derivation.

Equipment checklist

Cover the right side and self-test. If any answer surprises you, re-read that section above.