Foundations — Policy gradient methods

This page assumes you have seen nothing. Every letter in the parent note Policy gradient methods is unpacked below, in an order where each idea leans only on the ones before it. If a symbol confused you upstairs, its full story is here.

0. The reinforcement-learning stage (the picture everything lives on)

Before any symbol, look at the loop in the figure. There is an agent (the learner) and an environment (the world). At each tick of the clock:

1. The environment shows the agent a situation — we call it a state.
2. The agent picks a move — an action.
3. The environment answers with a score — a reward — and a new situation.

Everything below is just careful names for the arrows in this loop. Keep this picture in mind; every symbol is a label on one of these arrows.

1. State $s$ — "where am I?"

• Picture: the "situation" box in the loop above.
• Why the topic needs it: the policy must decide based on something. That something is $s$. Without a state there is no "given" to condition on.

2. Action $a$ — "what do I do?"

• Picture: the arrow the agent chooses to fire.
• Two flavours (this distinction is the whole reason policy gradients exist):
  • Discrete — finitely many moves (left / right / jump).
  • Continuous — a knob you can turn to any real number (steering angle $=12.7°$). There are infinitely many such actions.
• Why the topic needs it: Value-based methods (Q-learning) must scan every action to find the best one. With infinitely many actions that scan is impossible — this is exactly why we learn the policy directly instead.

3. Reward $r$ and return $R(\tau )$ — "was that good?"

The little subscript $t$ just means "the reward at time-step number $t$" ($t=0,1,2,\dots$).

Let us earn every symbol in that formula.

• ${\sum }_{t=0}^T$ — the big Greek "S" ($\Sigma$, sigma) means "add up". The label underneath ($t=0$) and on top ($T$) say "let $t$ run from $0$ to $T$ and add every term". So this is $r$-at-step-0 plus $r$-at-step-1 plus ... up to step $T$.
• $T$ — the last time-step (the end of the episode).
• $\gamma$ (Greek gamma) — the discount factor, a number between $0$ and $1$.
• ${\gamma }^t$ — gamma multiplied by itself $t$ times. Because $\gamma <1$, this shrinks as $t$ grows.

4. Trajectory $\tau$ — "the whole story of one run"

• Picture: unroll the loop of §0 into a single left-to-right chain.
• Why the topic needs it: the return $R(\tau )$ is a property of the whole run, so we must name the run itself. $\tau$ is that name.

5. The policy ${\pi }_{\theta }(a\mid s)$ — the star of the show

This is the single most important symbol. We build it piece by piece.

The bar $\mid$. Read $a\mid s$ as "$a$ given $s$" — "the action, given we are in state $s$". The bar means "conditioned on".

The letter $\pi$ (Greek pi) is just the traditional name for a policy; here it does not mean $3.14$.

• Picture: a set of dice, one per state, whose faces are the actions. Roll the dice to pick your move.

A probability distribution is just a bar chart of non-negative bars that add up to $1$ (there is a $100\%$ chance you pick some action).

6. The parameters $\theta$ — the shape of the dice

• Picture: knobs on the side of the dice-machine. Turn a knob → the bar chart in the figure above tilts.
• Why the topic needs it: learning = searching for the best $\theta$. The subscript in ${\pi }_{\theta }$ shouts "this policy is controlled by $\theta$".

7. Expectation $\mathbb{E}[\cdot ]$ — "the average outcome"

• The squiggle $\sim$ means "is sampled from" — "$\tau \sim {\pi }_{\theta }$" = "$\tau$ is a random run generated by the policy".
• Why the topic needs it: we cannot see every possible run — there are too many. So we settle for the average run. The whole objective $J(\theta )={\mathbb{E}}_{\tau }[R(\tau )]$ is "the average return".
• Estimating it: replace the true average by the average of $N$ sampled runs. This Monte-Carlo idea is expanded in Variance reduction in Monte Carlo.

8. The objective $J(\theta )$ — "how good is this policy?"

• Picture: a single height. Slide the knobs $\theta$ and this height goes up or down — we want the tallest spot.
• Why the topic needs it: you can only optimize what you can score. $J$ is the score.

9. The gradient ${\nabla }_{\theta }$ — "which way is uphill?"

The upside-down triangle $\nabla$ (called nabla or del) is the workhorse of learning.

• Picture: stand on a hill $J$; ${\nabla }_{\theta }J$ is the compass arrow pointing straight uphill.
• Gradient ascent: the update $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J$ takes a small step uphill. The little $\alpha$ (alpha) is the step size (how far each step goes).
• Why the topic needs it: the entire game of the parent note is computing this arrow when $J$ hides its $\theta$ inside a random process.

10. Logarithm $\log$ and the log-derivative trick

The parent's key move uses $\log$. Here is the honest picture.

You do not need to master this here — the parent derivation walks it fully. You only need the one slope fact above.

11. Baseline $b(s)$, value $V(s)$, advantage $A(s,a)$ — the variance fighters

These are the "part 2" symbols. Short pictures now; full stories in Advantage function and Actor-Critic methods.

• Reward-to-go $G_t$ — the return counted from time $t$ onward only: $G_t={\sum }_{t^{\prime }\ge t}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}$. Picture: the shrinking-bars figure, but starting at bar $t$ instead of bar $0$.
• Value $V(s)$ — the average reward-to-go you'd expect starting from state $s$. Picture: the "typical height" of a state.
• Baseline $b(s)$ — any number you subtract that depends only on the state, not the action. Subtracting it does not bias the gradient (it averages to zero).
• Advantage $A(s,a)=Q(s,a)-V(s)$ — "how much better than average was this specific action?" Positive = above average, negative = below. Picture: the height of one action's bar measured from the state's average line.

Prerequisite map

Read it top-down: the raw pieces (state, action, parameters) build the policy; the policy plus reward build trajectories; averaging over trajectories gives the objective; the gradient of that objective, powered by the log-derivative trick and tamed by baselines, is the policy gradient theorem.

Equipment checklist

Cover the right side and answer aloud; reveal to check.

Connections

• Parent: Policy gradient methods — this page unpacks every symbol it assumes.
• Log-derivative trick and Score function estimator — the calculus at the heart.
• Variance reduction in Monte Carlo — why baselines and reward-to-go matter.
• Advantage function, Actor-Critic methods — where $V(s)$ and $A(s,a)$ grow up.
• Value-based methods (Q-learning) — the contrast that motivates learning $\pi$ directly.