Visual walkthrough — Actor-critic methods

---

Step 0 — The words we will use (drawn, not assumed)

Before any math, three plain-language ideas, each pinned to the first figure.

• A state $s$ is "the situation right now" — where the little robot is standing on a board.
• An action $a$ is "the move it picks" — up/down/left/right.
• A reward $r$ is "the points the world hands back" right after the move.
• A policy ${\pi }_{\theta }(a\mid s)$ is a dice the robot rolls: given the situation $s$, it gives a probability to each action $a$. The little $\theta$ (theta) is just the set of knobs we can turn to bias those dice.

---

Step 1 — The goal, and the gradient that chases it

WHAT. We want to turn the knobs $\theta$ so the robot's average return is as big as possible. Call that average $J(\theta )$. The Policy Gradient Theorem tells us which way to turn the knobs: ${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\pi }\left[\underset{\text{"how to make }{a}_t\text{ more likely"}}{\underbrace{{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)}}\cdot \underset{\text{"how good }{a}_t\text{ was"}}{\underbrace{{\Psi }_t}}\right]$

WHY this shape. Read it as a product of two arrows.

• ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$ is a vector in knob-space pointing in the direction that makes the action we actually took more probable. (${\nabla }_{\theta }$, "nabla theta", just means "the slope with respect to every knob at once.")
• ${\Psi }_t$ is a scalar weight — a single number saying how much we want that action repeated. Positive → push the action up; negative → push it down.

PICTURE. Figure s02 shows knob-space with the "make-$a_t$-likelier" arrow, and the weight ${\Psi }_t$ stretching or flipping it.

REINFORCE's choice is ${\Psi }_t=G_t$. Correct on average, but $G_t$ is that whole-future dice roll. We will now transform ${\Psi }_t$ step by step: first $G_t$, then $G_t-b(s)$, then the advantage $A(s,a)$, and finally the TD error ${\delta }_t$. Watch the same slot ${\Psi }_t$ get better and better.

---

Step 2 — Picture the variance villain

WHAT. Plot the same good action taken many times; each time $G_t$ lands somewhere different because the rest of the episode is random luck.

WHY. If the weight ${\Psi }_t$ jumps between $+40$ and $-10$ for an identically good action, the knob-pushes fight each other and learning crawls. The average is right; the scatter is the problem.

PICTURE. Figure s03: a cloud of $G_t$ dots with a wide spread, and one honest "true expected value" line through the middle. The gap between any dot and that line is pure noise we want to delete.

---

Step 3 — We are allowed to subtract a baseline (the zero-trick)

WHAT. Replace ${\Psi }_t=G_t$ with ${\Psi }_t=G_t-b(s_t)$, where $b(s)$ is any number that depends only on the state, not the action. Claim: this changes nothing about the average gradient.

WHY it's legal. The extra piece averages to exactly zero: ${\mathbb{E}}_{a\sim \pi }[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]=b(s){\sum }_a{\pi }_{\theta }(a\mid s)\frac{{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)}{{\pi }_{\theta }(a\mid s)}=b(s){\nabla }_{\theta }\underset{=1}{\underbrace{\sum _a{\pi }_{\theta }(a\mid s)}}=b(s){\nabla }_{\theta }1=0.$

• ${\sum }_a{\pi }_{\theta }(a\mid s)=1$ because probabilities over all actions must add to one.
• The slope of the constant $1$ is $0$. Done — the baseline vanishes in expectation, so the update stays unbiased.

PICTURE. Figure s04: the arrows for each action, summed, cancel to zero when weighted by $b(s)$ — a balanced mobile hanging level.

---

Step 4 — Name the middle: the value $V(s)$, and the advantage

WHAT. Define the state-value $V(s)$ = the average return you expect starting from $s$ and following the policy. Choose the baseline $b(s)=V(s)$. And define the action-value $Q(s,a)$ = the average return if you take action $a$ first, then follow the policy. Our weight slot is now ${\Psi }_t=A(s,a)$, the advantage: $A(s,a)=\underset{\text{value of }a\text{ specifically}}{\underbrace{Q(s,a)}}-\underset{\text{value of an average action}}{\underbrace{V(s)}}$

WHY. Subtracting the state's own average removes the "is this state rich or poor at all?" nuisance, leaving only how much action $a$ beats the average action here. That sign is all the actor needs.

PICTURE. Figure s05: a number line at state $s$. $V(s)$ is the middle mark; $Q(s,a)$ for several actions sit above/below it; the coloured gap $A(s,a)$ is what we keep.

---

Step 5 — We can't compute $Q$ and $V$ exactly, so bootstrap (the TD trick)

WHAT. $Q(s_t,a_t)$ needs the full future — the very thing we're avoiding. The Bellman relation splits it into "one step, then the rest" — but it holds only as an average over the randomness of the reward and the next state: $Q(s_t,a_t)=\mathbb{E}[\underset{\text{reward we actually saw}}{\underbrace{r_{t+1}}}+\underset{\text{value of where we landed}}{\underbrace{\gamma V(s_{t+1})}}|s_t,a_t]$

• The $\mathbb{E}[\cdots \mid s_t,a_t]$ says: average over which reward and which next state the world hands back after taking $a_t$ in $s_t$. Only on average does one step plus the rest equal $Q$.

On a single sampled transition $(s_t,a_t,r_{t+1},s_{t+1})$ we drop the average and use the one draw we actually got as a noisy estimate. Replace the true, unknown $V$ with a learned estimate $V_w$ (the critic — $w$ is its own set of knobs). Our weight slot becomes: $A(s_t,a_t)\approx \underset{\text{TD error }{\delta }_t}{\underbrace{r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t)}}(\text{one-sample estimate})$

WHY this tool and not full returns? Full returns wait for the episode to end (high variance, offline). Bootstrapping — trusting your own next-step estimate — turns the infinite future into one real reward plus one guess. This is exactly Temporal Difference Learning: learn a guess from a guess.

Term by term:

• $r_{t+1}$ — the one honest fact this step gives us.
• $\gamma V_w(s_{t+1})$ — discounted critic-guess of the rest of the future from the new state.
• $-V_w(s_t)$ — subtract what we expected the whole thing to be worth. The difference is the surprise.

PICTURE. Figure s06: two stacked bars — "what we predicted" $V_w(s_t)$ versus "what we now believe" $r_{t+1}+\gamma V_w(s_{t+1})$ — with the gap labelled ${\delta }_t$.

Recall The whole chain of substitutions into the same slot

${\Psi }_t$ ${\Psi }_t=G_t$ (REINFORCE) ::: then ${\Psi }_t=G_t-b(s)$ (baseline, Step 3) ::: then ${\Psi }_t=A(s,a)=Q-V$ (Step 4) ::: then ${\Psi }_t={\delta }_t=r+\gamma V_w(s^{\prime })-V_w(s)$ (one-sample bootstrap, Step 5). Same slot, four upgrades.

---

Step 6 — One number, two jobs (and where the update rules come from)

WHAT. The single scalar ${\delta }_t$ drives both updates. First, a word on the step sizes ${\alpha }_w$ and ${\alpha }_{\theta }$:

WHY the critic update has that exact form. The critic wants its guess $V_w(s_t)$ to match the one-step target. Measure the miss with a squared-error loss (squared so over- and under-shooting both count, and big misses hurt more): $L(w)=\frac{1}{2}{\delta }_t^2=\frac{1}{2}(\underset{\text{target, treated as FIXED}}{\underbrace{r_{t+1}+\gamma V_w(s_{t+1})}}-V_w(s_t){)}^2.$ Take the slope with respect to $w$, treating the target as a frozen constant (only the $-V_w(s_t)$ end is differentiated): ${\nabla }_{w}L={\delta }_t\cdot (-{\nabla }_w{V}_w(s_t))=-{\delta }_t{\nabla }_w{V}_w(s_t).$ Gradient descent moves against the slope, $w\leftarrow w-{\alpha }_w{\nabla }_{w}L$, which flips the sign back to $+$:

WHY the same ${\delta }_t$ twice.

• To the critic, ${\delta }_t$ is "you mis-predicted by this much" → move $V_w(s_t)$ toward the truth.
• To the actor, ${\delta }_t$ is the one-sample advantage that now fills the ${\Psi }_t$ slot from Step 1 → scale the "make-$a_t$-likelier" arrow by it.

Stop-gradient warning (a case you must not fumble): the target $r_{t+1}+\gamma V_w(s_{t+1})$ is a frozen constant, as we used above. This is semi-gradient TD; differentiating the target too creates chasing-your-own-tail instability.

PICTURE. Figure s07: ${\delta }_t$ in the centre, two arrows fanning out — left to the critic net, right to the actor net.

---

Step 7 — Walk every sign (no scenario left unshown)

The whole method hinges on the sign of ${\delta }_t$. Three cases, all pictured together in s08 with the numbers from the parent's worked examples.

Case	Numbers ($\gamma =0.9$)	${\delta }_t$	Actor does	Critic does
Positive	$r=1,V_w(s)=5,V_w(s^{\prime })=6$	$1+0.9(6)-5=+1.4$	make $a$ more likely	raise $V_w(s)$
Negative	$r=1,V_w(s)=5,V_w(s^{\prime })=2$	$1+0.9(2)-5=-2.2$	make $a$ less likely	lower $V_w(s)$
Zero	$r=1,V_w(s)=5,V_w(s^{\prime })=\frac{40}{9}$	$1+0.9(\frac{40}{9})-5=0$	no push	leave $V_w(s)$

Notice case two: a positive reward still gives a negative ${\delta }_t$, because the future collapsed (we expected $5$, realized only $2.8$). Advantage compares against expectation, not against zero — the entire reason this beats raw reward.

---

The one-picture summary

Figure s09 compresses all seven steps into one blueprint: state → actor rolls the policy dice → world returns $r$ and $s^{\prime }$ → critic computes ${\delta }_t$ → the same ${\delta }_t$ pushes both nets → loop. Every symbol on it was earned above.

Recall Feynman: retell the whole walkthrough in plain words

We wanted the robot's dice (${\pi }_{\theta }$) to favour good moves. The honest way (REINFORCE) rewards each move by the whole random future — correct but wildly noisy. So we subtract a "how good is this spot normally?" number $V(s)$; that subtraction is free because it averages to zero, and it deletes most of the noise. The leftover is the advantage: how much this move beats the typical move. We can't know the future exactly, so we peek one real step ($r$), guess the rest with the critic $V_w(s^{\prime })$, and compare to what we predicted $V_w(s)$. That gap is the TD error ${\delta }_t$ — one number that says "you were surprised this much." It tells the critic to fix its guess (it's the slope of a squared-error loss) and tells the actor to repeat (or avoid) the move, scaled by the surprise. The same slot ${\Psi }_t$ from the very first formula just kept getting sharper: return → return-minus-baseline → advantage → TD error. Positive $\delta$ pushes up, negative pushes down, zero does nothing — even when the reward looked great.

---

---

Connections

• Actor-critic methods — the parent; this page is its derivation in pictures.
• Policy Gradient Theorem — Step 1's starting arrow.
• REINFORCE — the high-variance baseline we improved on (Step 2).
• Advantage function — formalised in Steps 3–4.
• Temporal Difference Learning — the bootstrap of Step 5.
• A3C and A2C — same ${\delta }_t$ math, parallel plumbing.
• Generalized Advantage Estimation (GAE) — smoother multi-step version of Step 5's advantage.
• Proximal Policy Optimization (PPO) — clips the actor step of Step 6 for stability.