Foundations — Advantage Actor-Critic (A2C - A3C)

This page assumes nothing. If the parent note Advantage Actor-Critic (A2C - A3C) used a symbol, we build it here from a picture first. Read top to bottom; each block earns the next.

0 · The stage: what a reinforcement-learning problem even is

Before any symbol, picture the world as a loop of four things repeating forever:

• State $s$ — a snapshot of the world right now. Picture the chessboard as it currently sits, or a robot's current position. It is "everything you'd need to know to decide what to do next."
• Action $a$ — the move you pick from this state. Picture an arrow you choose to follow out of the current square.
• Reward $r$ — a single number the world hands you after an action. Picture a little scoreboard blipping "+2" or "−1". Good = big positive, bad = negative.
• Next state $s^{\prime }$ (also written $s_{t+1}$) — where you land after acting. The board after your move.

Why we need this: every symbol later is defined on this loop. $t$ is just the tick counter — a subscript $x_t$ means "the value of $x$ at tick $t$." No mystery.

1 · The policy $\pi$ — the player

Picture a spinner whose slices are the possible actions; a state that makes "shoot" look great gives "shoot" a fat slice.

• The subscript $\theta$ (Greek "theta") is the bag of tunable numbers (the neural-network weights) that shape the spinner. Learning = slowly turning the $\theta$ dials so good actions get fatter slices.
• Because it outputs probabilities, every slice is $\ge 0$ and they sum to 1: ${\sum }_a{\pi }_{\theta }(a\mid s)=1.$

Why probabilities and not one fixed action? Because early on we don't know what's best, so the player must explore — try things — and probabilities let it. This "sum to 1" fact is the exact hinge of the baseline proof in the parent note, so hold onto it.

This is the same ${\pi }_{\theta }$ you first met in REINFORCE and the Policy Gradient Theorem.

2 · Return $G_t$ — adding up the future, discounted

One reward isn't enough — a move that scores now might ruin you later. So we care about the whole future from this moment on.

Why discount at all? Two reasons. (1) Money/points now beat money later. (2) Without shrinking, an infinite loop would sum to infinity — $\gamma <1$ keeps the total a finite, comparable number. The bars in the figure show each reward already faded by its ${\gamma }^k$; the return is their total height.

$G_t$ is exactly the noisy signal REINFORCE uses — and the noise is the problem A2C fixes.

3 · Expectation $\mathbb{E}$ — "average over the randomness"

The world is random: the spinner picks different actions, the environment may hand different rewards. So $G_t$ is a random number — run the same start twice, get two answers.

The subscript says what is random: ${\mathbb{E}}_{a\sim \pi }[\cdot ]$ means "average over actions $a$ drawn from the policy $\pi$." The symbol $\sim$ reads "is drawn from."

Why we need it: we can't optimise a random number directly, so every objective is written as an expectation — the average outcome — which is a fixed target to push up.

4 · Value $V$ and action-value $Q$ — the coach's two guesses

Now the two guesses at the heart of A2C.

The difference is the whole point:

• $V$ looks at the state (before choosing) — the flat baseline "how good is here."
• $Q$ looks at a state–action pair (after committing to $a$) — the height of one specific branch.

The subscript $\varphi$ in $V_{\varphi }$ (Greek "phi") is the critic network's own bag of weights — the coach also learns, separately from the actor's $\theta$.

Why subtract instead of using $Q$ alone? Because "this state is worth 100" tells you nothing about which action to pick if every action is worth ~100. Subtracting $V$ centres the numbers around zero, so only the relative choice remains. That centring is exactly what slashes the noise — the Bias-Variance Tradeoff lever.

5 · The gradient ${\nabla }_{\theta }$ and the log — how the player learns

Why this tool and not another? We want to make good outcomes more likely. "Which way should I nudge every dial to raise the score fastest?" is answered by exactly one object — the gradient. Learning = repeatedly stepping a little way along ${\nabla }_{\theta }$ (this is gradient ascent).

Put it together and the parent's central update reads in plain words: ${\nabla }_{\theta }J=\mathbb{E}\left[\underset{\text{"which way makes }a\text{ more likely"}}{\underbrace{{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)}}\underset{\text{"was }a\text{ good?"}}{\underbrace{A(s,a)}}\right].$ Direction times goodness. Good action → step that way; bad action → step the other way.

6 · Bootstrapping & the TD error ${\delta }_t$ — guessing from a guess

Waiting for the full return $G_t$ is slow and noisy. Instead, replace "the rest of the future" with the coach's own guess $V(s_{t+1})$. This is bootstrapping — improving a guess using another guess.

This is the engine of Temporal-Difference Learning, and stacking many ${\delta }_t$ smoothly gives Generalized Advantage Estimation (GAE). The hat in ${\hat{A}}_t$ just means "our estimate of $A_t$," not the true value.

Why bootstrap? It lets you learn every step instead of waiting for the episode to end, and it trades a little bias for a big drop in variance — the dial the parent's n-step section turns.

Prerequisite map

Each arrow is a "you need the left before the right." A2C sits at the bottom because it needs all of it: the loop to act in, the policy to nudge, values to compare against, the gradient to step, and the TD error to estimate advantage cheaply.

Equipment checklist

Test yourself — cover the right side and answer aloud.