Exercises — Advantage Actor-Critic (A2C - A3C)

(a) → (ii): the actor chooses actions. (b) → (i): the critic estimates a state's value. (c) → (iii): the TD error is the surprise — did the action do better or worse than the critic predicted? A positive ${\delta }_t$ means "better than expected."

The blank is $V(s)$, the state value (the value averaged over actions). So $A=Q-V$ measures "how much better is this specific action than the state's average action?" Subtracting a baseline that depends only on the state reduces variance without changing the expected gradient — this was proven in the parent note (Policy Gradient Theorem): ${\sum }_a{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)={\nabla }_{\theta }1=0$.

False. A2C/A3C are on-policy — every gradient must come from the current policy. They decorrelate data with parallel environments/workers, not a stale buffer. (See the parent-note mistake callout.)

${\delta }_t=r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)=3+0.95(8)-10=3+7.6-10=\mathrm{0.6.}$ ${\delta }_t=0.6>0$ → the action did better than expected, so the actor loss $-\log \pi \cdot 0.6$ pushes to make $a_t$ more likely.

${\delta }_t=-2+0.95(5)-10=-2+4.75-10=-\mathrm{7.25.}$ Strongly negative → the outcome was far worse than average from $s_t$, so the actor loss $-\log \pi \cdot (-7.25)=+7.25\log \pi$ decreases this action's probability. We steer away from $a_t$.

${\hat{A}}_t^{(3)}=\left({\sum }_{k=0}^2{\gamma }^k{r}_{t+k}\right)+{\gamma }^3{V}_{\varphi }(s_{t+3})-V_{\varphi }(s_t).$ Why the ${\gamma }^3$ on the bootstrap? Each reward is discounted by how far in the future it lands: $r_t$ by ${\gamma }^0$, $r_{t+1}$ by ${\gamma }^1$, $r_{t+2}$ by ${\gamma }^2$. After using 3 real rewards, the next thing we count is the critic's estimate of everything from step $t+3$ onward — and step $t+3$ is three steps ahead, so its whole value must be discounted by ${\gamma }^3$ to sit on the same footing as the rewards. Using $\gamma$ instead of ${\gamma }^3$ would treat a far-future estimate as if it happened right now. Real rewards: $0.9^0(1)+0.9^1(0)+0.9^2(2)=1+0+1.62=\mathrm{2.62.}$ Bootstrap tail: $0.9^3(10)=0.729\times 10=\mathrm{7.29.}$ ${\hat{A}}_t^{(3)}=2.62+7.29-4=\mathrm{5.91.}$

Uniform: $H=-(0.5\ln 0.5+0.5\ln 0.5)=-\ln 0.5=\ln 2\approx \mathrm{0.6931.}$ Peaked: $H=-(0.99\ln 0.99+0.01\ln 0.01)\approx -(0.99(-0.01005)+0.01(-4.6052))\approx 0.0100+0.0461=\mathrm{0.0560.}$ The uniform policy has much higher entropy (more exploration); the peaked one is nearly collapsed. The $-\beta H$ bonus in the actor loss rewards the high-entropy state, keeping the door open.

First verify the scores form a valid gradient of $\log \pi$: they must satisfy ${\sum }_a{\pi }_a{g}_a=0$ because ${\sum }_a{\pi }_a\nabla \log {\pi }_a=\nabla {\sum }_a{\pi }_a=\nabla 1=0$. Check: $0.6(1)+0.4(-1.5)=0.6-0.6=0.$ ✓ Now the baseline term: ${\sum }_a{\pi }_{a}b{g}_a=b{\sum }_a{\pi }_a{g}_a=7\times 0=0.$ Interpretation: the baseline factors out and multiplies a sum that is structurally zero. That is why we can subtract $V(s)$ for free — the parent-note proof, made concrete.

• ${\hat{A}}^{(1)}$ (1-step) uses one real reward and leans heavily on the critic $V_{\varphi }$. If the critic is wrong, this estimate is biased, but it is low variance (only one random reward inside).
• ${\hat{A}}^{(3)}$ (3-step) uses three real rewards. It trusts the (possibly noisy) environment more and the critic less → less bias, more variance. They differ numerically because they are different points on the same bias–variance dial; Generalized Advantage Estimation (GAE) blends all $n$ smoothly to get the best of both.

With a stop-gradient, only $V_{\varphi }(s_t)$ moves toward a fixed target — genuine learning. Without it, the optimizer can shrink the loss by moving both ends together: it can push $V_{\varphi }(s_{t+1})$ (inside the target) toward $V_{\varphi }(s_t)$, minimizing the squared difference without either being correct. You'd be "moving the goalposts to meet the ball." The value function collapses toward a self-consistent but meaningless constant. Hence: treat the target as a constant.

A2C average: $\bar{g}=\frac{1}{4}(0.8-0.2+0.5+0.1)=\frac{1}{4}(1.2)=\mathrm{0.3.}$ One synchronous step of size $\propto 0.3$. A3C is asynchronous: each worker pushes its own gradient to the shared network the moment it's ready — no waiting, no averaging. This gives speed on many CPUs but noisier, order-dependent updates. A2C = "A3C Awaiting All workers."

Set $n=1$. The sum has a single term $k=0$: ${\gamma }^0{r}_t=r_t$. The bootstrap becomes ${\gamma }^1{V}_{\varphi }(s_{t+1})$. So ${\hat{A}}_t^{(1)}=r_t+\gamma V_{\varphi }(s_{t+1})-V_{\varphi }(s_t)={\delta }_t.$ The n-step family contains the TD error as its simplest member — exactly the Temporal-Difference Learning connection.

Increase $\beta$, the entropy coefficient. Mechanism:

1. A collapsed policy has near-zero entropy $H$.
2. The term $-\beta H$ in the loss becomes a large penalty when $H$ is small (minimizing $-\beta H$ means maximizing $H$).
3. Gradient descent therefore pushes $\pi$ back toward a spread-out distribution → the agent explores again and can escape the bad local optimum. This is the parent-note "keeps the door open" idea, quantified.

Take the expectation over the reward and next state, holding $s_t,a_t$ fixed: $\mathbb{E}[{\delta }_t\mid s_t,a_t]=\mathbb{E}[r_t+\gamma V(s_{t+1})\mid s_t,a_t]-V(s_t).$ By the definition of the action-value function, $\mathbb{E}[r_t+\gamma V(s_{t+1})\mid s_t,a_t]=Q(s_t,a_t)$ (this is the Bellman expectation for $Q$: reward now plus discounted value of wherever we land). Hence $\mathbb{E}[{\delta }_t\mid s_t,a_t]=Q(s_t,a_t)-V(s_t)=A(s_t,a_t).■$ So ${\delta }_t$ is an unbiased advantage estimate when the critic is perfect — the theoretical justification for "the TD error IS the advantage."

(a) ${\hat{A}}_t=4+0.9(5)-6=4+4.5-6=\mathrm{2.5.}$ (b) $-\log (0.3)\times 2.5=-(-1.20397)\times 2.5=1.20397\times 2.5=\mathrm{3.0099.}$ Positive advantage → minimizing this loss increases $\log \pi$ → action becomes more likely. ✓ (c) target $=r_t+\gamma V_{\varphi }(s_{t+1})=4+4.5=8.5$. Loss $=\frac{1}{2}(8.5-6{)}^2=\frac{1}{2}(2.5{)}^2=\frac{1}{2}(6.25)=\mathrm{3.125.}$ Critic pushes $V_{\varphi }(s_t)$ from $6$ up toward $8.5$.

Algebra (telescoping). Write out the sum of discounted TD errors: ${\sum }_{k=0}^{n-1}{\gamma }^k{\delta }_{t+k}={\sum }_{k=0}^{n-1}{\gamma }^k(r_{t+k}+\gamma V_{\varphi }(s_{t+k+1})-V_{\varphi }(s_{t+k})).$ Split into three running pieces: $=\underset{\text{real rewards}}{\underbrace{\sum _{k=0}^{n-1}{\gamma }^k{r}_{t+k}}}+\underset{\text{"forward" values}}{\underbrace{\sum _{k=0}^{n-1}{\gamma }^{k+1}{V}_{\varphi }(s_{t+k+1})}}-\underset{\text{"backward" values}}{\underbrace{\sum _{k=0}^{n-1}{\gamma }^k{V}_{\varphi }(s_{t+k})}}.$ Now look at the two value sums. The forward sum, reindexed with $j=k+1$, runs over $j=1\dots n$ giving terms ${\gamma }^j{V}_{\varphi }(s_{t+j})$. The backward sum runs over $k=0\dots n-1$ giving terms ${\gamma }^k{V}_{\varphi }(s_{t+k})$. Every interior value ${\gamma }^m{V}_{\varphi }(s_{t+m})$ for $m=1\dots n-1$ appears once with $+$ (from forward) and once with $-$ (from backward), so they cancel. What survives is only the two endpoints:

• the forward sum's last term ${\gamma }^n{V}_{\varphi }(s_{t+n})$,
• the backward sum's first term $-{\gamma }^0{V}_{\varphi }(s_t)=-V_{\varphi }(s_t)$.

Therefore ${\sum }_{k=0}^{n-1}{\gamma }^k{\delta }_{t+k}={\sum }_{k=0}^{n-1}{\gamma }^k{r}_{t+k}+{\gamma }^n{V}_{\varphi }(s_{t+n})-V_{\varphi }(s_t)={\hat{A}}_t^{(n)}.■$ Numeric check ($n=2$). Directly: ${\hat{A}}^{(2)}=1+0.9(2)+0.9^2(8)-5=1+1.8+6.48-5=\mathrm{4.28.}$ Via TD errors: ${\delta }_t=1+0.9(3)-5=1+2.7-5=-1.3,{\delta }_{t+1}=2+0.9(8)-3=2+7.2-3=\mathrm{6.2.}$ ${\delta }_t+\gamma {\delta }_{t+1}=-1.3+0.9(6.2)=-1.3+5.58=\mathrm{4.28.}✓$ Both routes give $4.28$ — the identity holds.

2.1 → $0.6$ · 2.2 → $-7.25$ · 2.3 → $5.91$ · 2.4 → $0.6931,0.0560$ · 4.1 → $0.3$ · 5.2 → $2.5,3.0099,3.125$ · 5.3 → $4.28$ (both ways).