Foundations — Actor-critic methods

This page assumes you have seen none of the notation. We build each symbol from a picture, in an order where every new symbol only uses symbols already defined.

0. The playground: an agent in a world

Before any symbol, picture the world the parent note lives in.

Figure s01 shows the whole game as a loop. The white box on the left is the agent (the robot's "brain"); the white box on the right is the environment (the world). The red arrow is the one thing the agent controls — the action it sends into the world. The black arrow returning is what the world sends back: a reward number and a next state. Follow the red arrow, then the black arrow, and you have lived one full step of the game.

1. $s$ and $a$ — state and action (and where they live)

• $s$ (read "ess") is a symbol for the current state — the snapshot in the picture above.
• $a$ (read "ay") is a symbol for the action the agent takes in that state.
• We write $s_t$ and $a_t$ where the subscript $t$ (read "at time $t$") means "at step number $t$." Step $0$, step $1$, step $2$, … So $s_{t+1}$ is simply the next state.

2. $r$ — reward, and $\gamma$ — the discount

• $r_{t+1}$ is the reward the world pays after taking action $a_t$ in state $s_t$. (It carries subscript $t+1$ because it arrives with the next state.)
• $\gamma$ (Greek letter "gamma," read "gam-uh") is the discount factor, a number $0\le \gamma \le 1$.

Figure s02 plots the shrink-factor ${\gamma }^k$ against how many steps $k$ into the future a reward sits. The red bar at $k=0$ is the reward right now, weighted a full $1$. Each black bar to the right is shorter — the same reward, worth less the later it arrives. This falling staircase is exactly why the sum below converges (see the definition).

3. $\pi$ — the policy (this is the ACTOR)

• $\pi$ (Greek "pi," read "pie") is the agent's policy: its rule for choosing actions.
• $\pi (a\mid s)$ reads "the probability of action $a$ given state $s$." The vertical bar $\mid$ means "given" — it separates what we're guessing ($a$) from what we already know ($s$).
• $a\sim \pi (\cdot \mid s)$ reads "$a$ is sampled from the policy in state $s$." The squiggle $\sim$ means "drawn at random according to the probabilities listed by whatever is on its right." So $a\sim \pi (\cdot \mid s)$ = "roll the policy's weighted dice for state $s$ and let $a$ be the action that comes up." The dot $\cdot$ is a placeholder standing for "the action slot" — we're naming the whole distribution over actions, not one specific action yet.

• $\theta$ (Greek "theta," read "thay-tuh") is the bag of tunable numbers (weights) inside the policy, $\theta \in {\mathbb{R}}^n$ (a list of $n$ real numbers). We write ${\pi }_{\theta }$ to say "the policy shaped by settings $\theta$." Change $\theta$ → the dials move.

4. $\mathbb{E}$ — the expectation symbol (before we can score anything)

We must define averaging before we write value functions, because a "how-good" score is literally an average.

• $\mathbb{E}[\cdot ]$ (a stylised "E") means expected value = the long-run average of the thing in brackets over all the randomness.
• ${\mathbb{E}}_{\pi }[\cdot ]$ reads "average over everything random when actions come from policy $\pi$." Concretely the randomness has three sources: which state you're in, which action $\pi$ samples ($a\sim \pi (\cdot \mid s)$, the squiggle just defined in §3), and which next state the environment transitions to.

5. $V$ and $Q$ — value functions (this is the CRITIC)

Two "how-good" scores. Both are averages of the return $G_t$ (using the $\mathbb{E}$ just defined), but they condition on different things.

Figure s03 makes the difference visual. The single left circle is the state $s$; the label above it, $V(s)$, scores the whole spot. The three branches are the three possible actions; each landing circle carries its own $Q(s,a)$. The red branch ($a_1$) is highlighted to show that $Q$ is one number per action, while $V$ is one number for the state — literally the average of the $Q$'s weighted by how often the policy picks each branch.

• $V_w(s)$ and $Q_w(s,a)$: the subscript $w$ (with $w\in {\mathbb{R}}^m$) is another bag of tunable numbers — the critic's weights. $V_w$ is our learned estimate of the true $V^{\pi }$.

6. The advantage $A(s,a)$ — the star of the show

Figure s04 shows why we subtract. The three black bars are $Q(s,a)$ for three actions — all towering near $100$. The red horizontal line is the pedestal $V(s)=100$. What the actor cares about is only the little bit sticking above or below the red line — the advantage $A=Q-V$, printed above each bar ($+3,0,-3$). Its sign cleanly says "push this action's probability up" ($+$) or "down" ($-$). This is the advantage function, and the pedestal-removal is the whole reason the critic reduces variance.

7. ${\delta }_t$ — the TD error (turning $Q$ into $V$, the shared signal)

First: why $Q$ can be written from $V$ (the one-step recursion)

We rarely know $Q$ exactly, and we do not want to learn a separate $Q_w$. The trick is a small derivation. Start from the definitions in §5 and split the return $G_t$ into "the first reward" plus "everything after":

$G_t=r_{t+1}+\gamma (r_{t+2}+\gamma r_{t+3}+\cdots )=r_{t+1}+\gamma G_{t+1}.$

Why this split? The return is defined as a sum where every term past the first already carries one factor of $\gamma$; factoring that $\gamma$ out leaves exactly $G_{t+1}$ — the return starting one step later. Now take the average ${\mathbb{E}}_{\pi }[\cdot \mid s_t=s,a_t=a]$ of both sides. On the right, ${\mathbb{E}}_{\pi }[G_{t+1}\mid s_{t+1}]$ is by the very definition of $V^{\pi }$ just $V^{\pi }(s_{t+1})$. This gives the one-step recursion (a Bellman equation):

$\boxed{Q^{\pi }(s,a)=\mathbb{E}[r_{t+1}+\gamma V^{\pi }(s_{t+1})]}$

In words: the value of an action = the immediate reward you collect, plus the discounted value of wherever you land. This is the bridge that lets a $V$-only critic estimate $Q$.

Then: the sample estimate

Dropping the average and using the critic $V_w$ in place of the unknown true $V^{\pi }$, a one-sample estimate of the advantage $A(s_t,a_t)=Q(s_t,a_t)-V(s_t)$ is:

8. Gradients, the objective, the credit signal ${\Psi }_t$, and the update rules

• ${\nabla }_{\theta }$ (the symbol $\nabla$ is "nabla" or "grad") is the gradient with respect to $\theta$: an arrow pointing in the direction that most increases whatever follows it, as you wiggle the knobs $\theta$.
• $\log$ here means the natural logarithm (base $e\approx 2.718$), the convention used everywhere in machine learning. ${\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)$ is "the direction in knob-space that makes action $a$ more probable."

• $\alpha$ (Greek "alpha") = the learning rate / step size: how big a nudge each update makes. We use ${\alpha }_{\theta }$ for the actor and ${\alpha }_w$ for the critic (typically ${\alpha }_w>{\alpha }_{\theta }$ so the critic keeps ahead).

Prerequisite map

Read it bottom-up: states and rewards define returns; the expectation averages them into values and the objective $J$; values minus values give the advantage; the advantage collapses to the TD error; the TD error fills the credit slot ${\Psi }_t$ and, times the gradient-of-log direction, is the actor-critic update.

Equipment checklist

Cover the right side and see if you can state each from memory.

Connections

• Parent: Actor-critic methods — everything here feeds it.
• REINFORCE — uses $G_t$ directly; motivates the critic.
• Policy Gradient Theorem — where ${\nabla }_{\theta }\log {\pi }_{\theta }$, ${\Psi }_t$ and $J(\theta )$ come from.
• Temporal Difference Learning — the origin of ${\delta }_t$.
• Advantage function — the advantage $A(s,a)$ in depth.
• Generalized Advantage Estimation (GAE), Proximal Policy Optimization (PPO), A3C and A2C — where these foundations lead next.