Proximal Policy Optimization (PPO): Foundations

Before you can read the Proximal Policy Optimization (PPO) page, you meet a wall of symbols: $π_{θ}$ , $\hat{A}_{t}$ , $r_{t}$ , $γ$ , $λ$ , $ϵ$ , $E$ , $\nabla$ , $lo g$ . This page builds each one from nothing, in an order where every symbol is earned before it is used.

The stage: an agent living in a loop

Figure s01 (below) draws exactly this loop: the four rounded boxes are the four things that happen every tick — read State (top-left) → Policy picks Action (top-right, the agent decides arrow) → Reward (bottom-right) → World gives a new state (bottom-left, the world reacts arrow) → back up to State. Trace the arrows once and you have seen the entire game an RL agent plays.

Figure — Proximal Policy Optimization (PPO)

We give names to the pieces of this loop. Each name is a symbol you will see everywhere in PPO.

A whole run: the trajectory $τ$ and its return $R (τ)$

The decision-maker: the policy $π$

Figure s02 (below) shows one such policy as a bar chart: four bars (up/down/left/right), heights are the probabilities, and they sum to $1$ . Notice "up" is tallest (most likely) while "left" is a short-but-nonzero bar — that surviving small bar is the exploration the entropy bonus later protects.

The subscript $θ$ : the knobs of the brain

Averaging over uncertainty: $E$

Which actions were good? The advantage $\hat{A}$

Figure s03 (below) makes this concrete: three actions all give returns near $+ 100$ (mint bars), and the red dashed line is the baseline $V (s) = 100$ . The label above each bar is $\hat{A} = return - V (s)$ — only $+ 3, - 2, + 1$ survive. That subtraction is what separates good from bad when raw numbers all look alike.

The discount $γ$ and the trade-off knob $λ$

The learning move: $lo g$ , $\nabla$ , and the ratio $r_{t}$

Formula The policy gradient — and WHY it holds

$\nabla_{θ} J (θ) = E [\nabla_{θ} lo g π_{θ} (a ∣ s) \hat{A} (s, a)]$ Where it comes from (the log-derivative trick). We want $\nabla_{θ} J = \nabla_{θ} E_{τ \sim π_{θ}} [R (τ)]$ . The trouble: the randomness itself depends on $θ$ (the paths $τ$ are drawn from $π_{θ}$ ), so we can't just push the gradient inside the average. The fix is one identity: $\nabla_{θ} π_{θ} = π_{θ} \nabla_{θ} lo g π_{θ} (since \nabla lo g π = \frac{\nabla π}{π}).$ Multiplying and dividing the gradient of the average by $π_{θ}$ using this identity re-expresses $\nabla_{θ} J$ as another average — one we can estimate from sampled runs: $\nabla_{θ} J = E [R (τ) \nabla_{θ} lo g π_{θ} (a ∣ s)] .$ Why swap $R (τ)$ for $\hat{A}$ ? Replacing the whole-run score $R (τ)$ by the per-action advantage $\hat{A} (s, a)$ (subtracting the baseline $V (s)$ ) leaves the average unchanged but slashes the noise — exactly the variance trick from the advantage section. What it says in words: nudge the knobs so good actions ( $\hat{A} > 0$ ) get more log-probability and bad ones ( $\hat{A} < 0$ ) get less. Full derivation lives in the Policy Gradient Theorem; every symbol here you now own.

Figure s04 (below) graphs this: the dotted diagonal is the untouched ratio $r$ , the lavender curve is $clip (r, 0.8, 1.2)$ . Inside the mint band the two coincide (clip does nothing); outside, the lavender line goes flat — a ratio of $1.5$ is pinned to $1.2$ , a ratio of $0.5$ is lifted to $0.8$ .

Where all the symbols plug in: PPO's actual loss

Now that every symbol is defined, here is the objective PPO maximizes — the clipped surrogate:

How the foundations feed PPO

Equipment checklist

Test yourself — cover the right side.

What does

s_{t}

stand for?

The state (snapshot of where the agent is / what it sees) at time-step

t

What is a trajectory

τ

One complete run through the loop — the ordered list

(s_{0}, a_{0}, r_{0}, s_{1}, a_{1}, r_{1}, \dots)

; a random object since the policy is random.

Write

R (τ)

and say what it measures.

R (τ) = \sum_{t \geq 0} γ^{t} r_{t}

— the total discounted reward of one whole run.

What does

J (θ)

equal?

J (θ) = E_{τ \sim π_{θ}} [R (τ)]

— the average return over all paths the policy might take.

What does

π (a ∣ s)

output, and what must the outputs sum to?

The probability of each action given state

s

; they sum to

1

What is the difference between

θ

and

π_{θ}

θ

is the list of network knobs;

π_{θ}

is the policy those knobs produce.

Why keep a frozen

π_{θ_{o l d}}

It's the reference the new policy must not drift too far from; it's also the policy that collected the data.

What does

E [X]

mean and equal for a fair die reward?

The long-run average of

X

; for a die it is

3.5

Why does the policy gradient use

\nabla_{θ} lo g π_{θ}

The log-derivative identity

\nabla π = π \nabla lo g π

turns

\nabla_{θ} J

into an average we can estimate from sampled runs.

Why do we subtract

V (s)

to form the advantage?

To remove the state's baseline so only the difference between actions remains — cutting variance without bias.

Give the TD error formula.

δ_{t} = r_{t} + γ V (s_{t + 1}) - V (s_{t})

Write the full GAE advantage.

\hat{A}_{t}^{G A E} = \sum_{l \geq 0} (γ λ)^{l} δ_{t + l}

What does

λ

trade off in GAE?

Bias vs variance (

λ = 0

low-variance biased one-step;

λ = 1

unbiased noisy full return).

What does the gradient

\nabla_{θ}

point toward?

The direction in knob-space of steepest increase of the quantity.

Write the probability ratio and say what

r_{t} = 0.8

means.

r_{t} (θ) = π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{o l d}} (a_{t} ∣ s_{t})

;

r_{t} = 0.8

means the new policy makes the action 20% less likely.

Write the PPO clipped-surrogate objective.

L^{C L I P} = E_{t} [min (r_{t} \hat{A}_{t}, clip (r_{t}, 1 - ϵ, 1 + ϵ) \hat{A}_{t})]

What does

clip (r, 0.8, 1.2)

do to

r = 1.5

and to

r = 1.05

Squashes

1.5 \to 1.2

; leaves

1.05

unchanged.