Exercises — Actor-critic methods

The actor (policy ${\pi }_{\theta }$) is being trained — its parameters $\theta$ change, stepped by the actor learning rate ${\alpha }_{\theta }$. The quantity acting as the advantage estimate is the ==TD error ${\delta }_t$==. It is a one-sample stand-in for $A(s,a)$: positive ${\delta }_t$ pushes the action's probability up, negative pushes it down.

• (a) TD error $\to$ (iii) $r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t)$.
• (b) advantage $\to$ (i) $Q(s,a)-V(s)$.
• (c) critic target $\to$ (ii) $r_{t+1}+\gamma V_w(s_{t+1})$ — the fixed number the critic tries to match with $V_w(s_t)$.

True. REINFORCE needs the full Monte-Carlo return $G_t$ (defined in the Symbol reminder: the discounted sum of all future rewards), which requires the whole trajectory; actor-critic replaces $G_t$ with the bootstrapped $r_{t+1}+\gamma V_w(s_{t+1})$, which is available immediately after one step (the Temporal Difference Learning trick).

$\delta =r+\gamma V_w(s^{\prime })-V_w(s)=2+0.9(5)-4=2+4.5-4=\mathrm{2.5.}$ $\delta =2.5>0$ ⇒ the transition beat expectation ⇒ actor increases $\log {\pi }_{\theta }(a\mid s)$, making $a$ more likely (green bar in Figure s01). The critic also nudges $V_w(s)$ upward (it under-predicted).

$\delta =1+0.9(3)-10=1+2.7-10=-\mathrm{6.3.}$ $\delta =-6.3<0$ ⇒ discouraged (red bar in Figure s01). Positive reward is not enough: we expected value $10$ but the realized estimate is only $r+\gamma V_w(s^{\prime })=3.7$. Advantage compares against expectation, not against zero.

$\delta =r+\gamma \cdot 0-V_w(s)=1+0-2=-1.$ At a terminal state there is no future, so the bootstrap term vanishes. Here the episode ended worse than the critic's optimistic $V_w(s)=2$, so $\delta =-1<0$.

${\delta }_0=r_1+\gamma V_w(s_1)-V_w(s_0)=1+0.5(4)-3=1+2-3=0.$ ${\delta }_1=r_2+\gamma V_w(s_2)-V_w(s_1)=2+0.5(6)-4=2+3-4=1.$ A ${\delta }_0=0$ means the critic predicted $s_0$ perfectly (no surprise there); the surprise all lives at step 1.

Recap of Step 1 (self-contained). For any function $b(s)$ that depends only on the state (not the action), its expected contribution to the policy gradient is exactly zero: ${\mathbb{E}}_{a\sim \pi }[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]=b(s){\sum }_a{\pi }_{\theta }(a\mid s)\frac{{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)}{{\pi }_{\theta }(a\mid s)}=b(s){\nabla }_{\theta }{\sum }_a{\pi }_{\theta }(a\mid s)=b(s){\nabla }_{\theta }1=0.$ The key move is that probabilities sum to $1$, so their gradient sums to $0$. Because this contribution is zero in expectation, subtracting $b(s)$ leaves the expected gradient unchanged ⇒ unbiased. Setting $b(s)=V(s)$ (the true value) removes the "how valuable is this state at all" offset shared by every action, leaving only the action-specific part $A(s,a)=Q-V$. Since the removed offset was a big, action-independent number multiplying a noisy score, cutting it shrinks the variance of the estimator. See Advantage function.

• Critic A: ${\delta }_A=6-5.0=1.0>0$ ⇒ correctly says "better than average," encourages the action.
• Critic B: ${\delta }_B=6-8.0=-2.0<0$ ⇒ wrongly discourages a genuinely good action, because B massively over-predicted $V_w(s)$ relative to the true $V(s)=5$. A biased critic flips the sign of the advantage, so the actor learns in the wrong direction. This is exactly why the critic should be trained reliably (and often faster) before the actor trusts it.

Recall ${\alpha }_w$ is the critic's step size and ${\alpha }_{\theta }$ the actor's. The actor's gradient is scaled by ${\delta }_t$, which is only meaningful if $V_w$ is a decent estimate. If the actor changes the policy faster than the critic can track its value, the critic is always chasing an outdated policy, so the advantages it feeds the actor are stale/garbage — the actor optimizes noise. Making the critic faster (${\alpha }_w>{\alpha }_{\theta }$) keeps the value estimate roughly correct for the current policy, giving the actor trustworthy gradients.

With $\gamma =1$, the weight on ${\delta }_{t+l}$ is ${\lambda }^l=0.9^l$: ${\hat{A}}_0={\delta }_0+0.9{\delta }_1+0.9^2{\delta }_2=1+0.9(0.5)+0.81(-1).$ $=1+0.45-0.81=\mathrm{0.64.}$ $\lambda$ trades bias against variance (Figure s02): $\lambda =0$ recovers the pure one-step ${\delta }_0$ (low variance, more bias); $\lambda =1$ recovers the full Monte-Carlo advantage (unbiased, high variance).

• $\lambda =0$: every term with $l\ge 1$ carries factor $0^l=0$, leaving only $l=0$: ${\hat{A}}_t={\delta }_t$.
• $\lambda =1,\gamma =1$: weights all $1$, so ${\hat{A}}_t={\sum }_l{\delta }_{t+l}={\sum }_l(r_{t+l+1}+V(s_{t+l+1})-V(s_{t+l}))$. The $V$ terms telescope: $-V(s_t)+(V(s_{t+1})-V(s_{t+1}))+\cdots ={\sum }_l{r}_{t+l+1}-V(s_t)=G_t-V(s_t)$. That is exactly the Monte-Carlo advantage — the return $G_t$ (total discounted future reward) minus the baseline, i.e. the REINFORCE signal with a baseline.

Derivation (the WHY). Write $\log \pi (a_1\mid s)=h(s,a_1)-\log {\sum }_j{e}^{h(s,a_j)}$. Differentiate with respect to a logit $h(s,a_k)$: $\frac{\partial \log \pi (a_1)}{\partial h(s,a_k)}=\underset{\text{from }h(s,a_1)}{\underbrace{[k=1]}}-\frac{e^{h(s,a_k)}}{{\sum }_j{e}^{h(s,a_j)}}=[k=1]-\pi (a_k),$ where $[k=1]$ is $1$ if $k=1$ and $0$ otherwise. The second term comes from the chain rule on the log-sum-exp normaliser, and it is exactly $\pi (a_k)$. Intuition: the chosen action's logit gets a $+1$ "raise," and every logit (including the chosen one) gets pulled down by its own current probability — a competition where more-probable actions are pulled harder. So for the chosen action $a_1$: $1-\pi (a_1)$; for the other $a_2$: $-\pi (a_2)$.

Numbers. Softmax: $\pi (a_1)=\frac{e^2}{e^2+e^1}=\frac{7.389}{7.389+2.718}=0.731$, $\pi (a_2)=0.269$. Update on logit $h(s,a_1)$: $\delta (1-\pi (a_1))=1\cdot (1-0.731)=+0.269$ (raise the chosen action's logit). Update on logit $h(s,a_2)$: $\delta (-\pi (a_2))=1\cdot (-0.269)=-0.269$ (lower the other). The magnitude of the whole move is set by $\delta$: a bigger surprise moves the logits more.

With $\gamma =0$: ${\delta }_t=r_{t+1}+0-V_w(s_t)=r_{t+1}-V_w(s_t)$. The bootstrap disappears; the critic just learns the mean immediate reward $\mathbb{E}[r_{t+1}\mid s]$, and the advantage is "did this action beat the average immediate reward in this state?" The actor becomes a myopic, one-step (contextual-bandit) optimizer — it ignores all long-term consequences.

• (a) Semi-gradient: treat $r+\gamma V_w(s^{\prime })$ as a constant target, so ${\nabla }_{w}L=-{\delta }_t{\nabla }_w{V}_w(s)$, giving $w\leftarrow w+{\alpha }_w{\delta }_t{\nabla }_w{V}_w(s)$ — exactly the parent's rule.
• (b) Full gradient would also differentiate the target: it adds $+{\delta }_t\gamma {\nabla }_w{V}_w(s^{\prime })$. This term makes the critic chase a moving target it is itself creating, which empirically destabilizes learning and is not the TD fixed-point being solved. The advantage estimate we derived (Step 3) treats $r+\gamma V_w(s^{\prime })$ as the sampled value of $Q$, i.e. a fixed number — so semi-gradient is the mathematically consistent choice.

Mean $=\frac{2-3+4-1-2}{5}=\frac{0}{5}=0$. Deviations from mean: $[2,-3,4,-1,-2]$; squares $[4,9,16,1,4]$, sum $=34$; variance $=34/5=6.8$. A healthy batch of advantages should have mean $\approx 0$, because $A(s,a)=Q-V$ is defined relative to the average — over the actions actually taken, the mean advantage is near zero (better-than-average and worse-than-average cancel). This is exactly why practitioners often normalize advantages: subtract the batch mean and divide by the batch standard deviation before the actor update. Here that means dividing the centred values by $\sqrt{6.8}\approx 2.608$, which keeps the step sizes well-scaled and stable without changing the direction of improvement.

A3C (Asynchronous Advantage Actor-Critic) runs many workers asynchronously: each worker holds its own copy of the parameters, plays its own rollout, computes gradients locally, and pushes them to a shared parameter server whenever it finishes, then pulls the latest parameters back. Because workers update at different, uncoordinated times, no two workers are guaranteed to be on the same parameter version — updates are "lock-free" and slightly stale.

A2C (Advantage Actor-Critic) is the synchronous version: a central controller waits for all workers to finish their rollouts, concatenates their transitions into one big batch, computes the averaged advantages/gradients, and applies a single update. Every worker then starts the next rollout from the same fresh parameters.

So the plumbing difference is asynchronous, per-worker, staggered updates (A3C) versus synchronous, batched, single-update steps (A2C). The underlying math is identical in both: each worker uses the actor step ${\delta }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$, the critic step minimizing ${\delta }_t^2$, and the advantage $A=Q-V$. Only when and how gradients are combined and applied differs — not the estimator itself.

${\delta }_t$ train both networks? ${\delta }_t$ is simultaneously the critic's prediction error (minimize ${\delta }_t^2$) and a one-sample advantage estimate scaling the actor's ${\nabla }_{\theta }\log {\pi }_{\theta }$.