Advantage Actor-Critic (A2C - A3C)

---

WHY does A2C exist? (The problem it fixes)

Pure policy gradient (REINFORCE) updates the policy with: $\nabla_\theta J = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a_t\mid s_t)\, G_t\big]$ where $G_t = \sum_{k\geq 0}\gamma^k r_{t+k}$ is the full Monte-Carlo return.

• WHAT is wrong? $G_t$ is an unbiased but extremely noisy estimate. One lucky reward far in the future can swamp the credit for the action you actually took now.
• WHY noisy? It sums many random future rewards. Variance piles up.

The fix: subtract a baseline $b(s_t)$ that depends only on the state (not the action). This does not change the expected gradient (proof below) but slashes variance. The best baseline is the state value $V(s_t)$, giving the advantage: $A(s_t,a_t) = Q(s_t,a_t) - V(s_t)$

So we need two networks:

• Actor $\pi_\theta(a\mid s)$ — chooses actions.
• Critic $V_\phi(s)$ — estimates state value, used to compute the advantage.

That coupling actor learns, critic judges is the whole "actor-critic" idea.

---

Derivation from first principles

Step 1 — Policy gradient theorem

Start from the objective (expected return from start distribution): $J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\Big[\sum_t \gamma^t r_t\Big].$ The log-derivative trick: for any distribution, $\nabla_\theta p_\theta(x) = p_\theta(x)\nabla_\theta \log p_\theta(x)$. Applying it to the trajectory distribution and noting the dynamics don't depend on $\theta$: $\nabla_\theta J = \mathbb{E}\Big[\sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\; G_t\Big].$ Why this step? We can't differentiate through sampling, so we turn "gradient of an expectation" into "expectation of a gradient × a score."

Step 2 — Baselines don't add bias

Claim: for any $b(s_t)$, $\mathbb{E}_{a_t\sim\pi}\big[\nabla_\theta \log\pi_\theta(a_t\mid s_t)\,b(s_t)\big]=0.$ Proof: pull $b(s_t)$ out (constant in $a_t$):

= b(s_t)\sum_a \nabla_\theta \pi_\theta(a\mid s_t) = b(s_t)\,\nabla_\theta \underbrace{\sum_a \pi_\theta(a\mid s_t)}_{=1}=0.$$ *Why this matters:* we may subtract $V(s_t)$ for free — **same expected gradient, lower variance**. ### Step 3 — Replace $G_t$ with the advantage Since $\mathbb{E}[G_t\mid s_t,a_t]=Q(s_t,a_t)$, we get: $$\boxed{\nabla_\theta J = \mathbb{E}\big[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,A(s_t,a_t)\big]},\quad A=Q-V.$$ ### Step 4 — Estimate $A$ without a $Q$ network We only trained a critic $V_\phi$. Use the **TD (temporal-difference) estimate** of the advantage: $$\hat A_t = \underbrace{r_t + \gamma V_\phi(s_{t+1})}_{\text{bootstrap target of }Q} - V_\phi(s_t) = \delta_t.$$ This $\delta_t$ is exactly the **TD error**. So *the critic's TD error IS the advantage estimate.* Beautiful. > [!formula] The three update ingredients > **TD error (advantage):** $\;\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$ > **Actor loss:** $\;\mathcal L_\pi = -\log\pi_\theta(a_t\mid s_t)\,\hat A_t \;-\; \beta\,H(\pi_\theta(\cdot\mid s_t))$ > **Critic loss:** $\;\mathcal L_V = \tfrac12\big(\underbrace{r_t+\gamma V_\phi(s_{t+1})}_{\text{stop-grad target}} - V_\phi(s_t)\big)^2$ > $H$ is the policy **entropy** ($-\sum\pi\log\pi$); the $-\beta H$ bonus *encourages exploration* (keeps $\pi$ from collapsing too early). > [!definition] n-step advantage (bias–variance dial) > Instead of a 1-step bootstrap, use $n$ real rewards then bootstrap: > $$\hat A_t^{(n)} = \Big(\sum_{k=0}^{n-1}\gamma^k r_{t+k}\Big) + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t).$$ > Small $n$ → low variance, more bias (relies on critic). Large $n$ → high variance, less bias (relies on real returns). GAE interpolates all $n$ smoothly. --- ## A3C vs A2C — the "parallel" part - **A3C (Asynchronous Advantage Actor-Critic):** many worker threads each hold a copy of the network, roll out their own environment, compute gradients, and **asynchronously** push them to a shared global network. The diversity of workers ==decorrelates the data==, playing the role a replay buffer plays in DQN — but on-policy. - **A2C (Advantage Actor-Critic):** the *synchronous* version. Wait for **all** workers to finish a rollout, average their gradients, do one update. Simpler, more GPU-efficient, often equal or better than A3C. "A2C = A3C minus the asynchrony." > [!mistake] "A3C needs a replay buffer like DQN." > **Why it feels right:** DQN drilled into us that decorrelated samples are essential. > **The fix:** A2C/A3C are **on-policy** — every gradient must come from the *current* policy. A stale replay buffer holds *old-policy* actions, which would make $\nabla\log\pi$ wrong. Instead they decorrelate via **parallel environments**, not a buffer. > [!mistake] Letting gradients flow through the critic target. > **Why it feels right:** the target $r+\gamma V_\phi(s')$ contains $V_\phi$, so it "seems" natural to differentiate it. > **The fix:** treat the target as a **constant (stop-gradient)**. Otherwise you optimize $V_\phi(s')$ downward to trivially minimize the loss — you'd be moving the goalposts, not learning value. > [!mistake] Sharing one optimizer step but forgetting entropy → premature collapse. > **Why it feels right:** entropy looks like a minor "regularizer." > **The fix:** without the $\beta H$ term the policy can spike to a near-deterministic bad action early, stop exploring, and get stuck. Entropy keeps the door open. --- ## Worked examples > [!example] Example 1 — computing one advantage > $\gamma=0.9$, $V_\phi(s_t)=5$, reward $r_t=2$, $V_\phi(s_{t+1})=6$. > $\delta_t = 2 + 0.9(6) - 5 = 2 + 5.4 - 5 = 2.4$. > **Why this step?** Positive $\delta$ means the action did *better than the critic expected* → the actor loss $-\log\pi\cdot 2.4$ pushes to make this action **more likely**. The critic loss pushes $V_\phi(s_t)$ up toward $7.4$. > [!example] Example 2 — a negative advantage > Same critic, but $r_t=-1$, $V_\phi(s_{t+1})=4$: $\delta_t=-1+0.9(4)-5=-1+3.6-5=-2.4$. > Actor loss gradient becomes $-\log\pi\cdot(-2.4)=+2.4\log\pi$ → **decreases** this action's probability. **Why?** The outcome was worse than average from $s_t$, so we steer away from $a_t$. > [!example] Example 3 — 2-step advantage > $\gamma=0.9$, rewards $r_t=1, r_{t+1}=2$, $V_\phi(s_t)=5$, $V_\phi(s_{t+2})=8$. > $\hat A^{(2)}_t = 1 + 0.9(2) + 0.9^2(8) - 5 = 1 + 1.8 + 6.48 - 5 = 4.28.$ > **Why this step?** Using two real rewards trusts the environment more and the (possibly rough) critic less — more variance, less bias than the 1-step version. --- > [!recall]- Feynman: explain to a 12-year-old > Imagine you're learning basketball. A **coach (critic)** watches and guesses how good your position is ("from here you usually score about 5 points"). You take a shot (the **actor's action**) and actually get 7. The *surprise* — 7 minus the expected 5 = **+2** — tells you "that move was better than my usual, do it more!" If you'd gotten 3, the surprise is −2, "do that less." A3C is just having **many kids practicing at once** on different courts and sharing what they learn, so the team improves faster and doesn't all copy the same mistake. > [!mnemonic] Remember the pieces > **"A CAT VE"** → > **A**dvantage = **CAT** *Critic's TD error* = **V**alue-next minus **V**alue-now (plus reward) → drives the **E**ntropy-bonused actor. > And: **A2C = A3C Awaiting All (synchronous)**. --- ## Active recall #flashcards/ai-ml What is the advantage function $A(s,a)$? ::: $A(s,a)=Q(s,a)-V(s)$ — how much better action $a$ is than the state's average. Why subtract a baseline in policy gradient? ::: It reduces variance without changing the expected gradient (baseline term has zero expectation). Prove the baseline adds no bias (key line). ::: $\sum_a \nabla_\theta \pi_\theta(a|s)=\nabla_\theta\sum_a\pi=\nabla_\theta 1=0$. What single quantity estimates the advantage from a value critic? ::: The TD error $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$. Actor loss in A2C? ::: $-\log\pi_\theta(a_t|s_t)\hat A_t - \beta H(\pi)$. Critic loss in A2C? ::: $\tfrac12(r_t+\gamma V(s_{t+1})-V(s_t))^2$ with a stop-gradient on the target. Role of the entropy bonus? ::: Encourages exploration; stops the policy collapsing to a deterministic action too early. A2C vs A3C in one line? ::: A2C is the synchronous version (wait for all workers, then average gradients); A3C is asynchronous. How do A2C/A3C decorrelate data instead of a replay buffer? ::: By running many parallel environments/workers (on-policy). n-step advantage trade-off? ::: Larger n → lower bias, higher variance (more real rewards); smaller n → lower variance, more critic bias. Why must the critic target be a stop-gradient? ::: Otherwise the optimizer moves $V(s')$ to trivially shrink the loss instead of learning true value. --- ## Connections - [[REINFORCE]] — the Monte-Carlo ancestor; A2C = REINFORCE + baseline + bootstrapping. - [[Policy Gradient Theorem]] — the identity we differentiate. - [[Temporal-Difference Learning]] — where $\delta_t$ comes from. - [[Generalized Advantage Estimation (GAE)]] — smooths all n-step advantages. - [[DQN]] — contrast: off-policy + replay buffer vs on-policy + parallel workers. - [[PPO]] — descendant that adds a clipped surrogate for stable large steps. - [[Bias-Variance Tradeoff]] — the n dial embodies it. ## 🖼️ Concept Map ```mermaid flowchart TD REINFORCE[REINFORCE policy gradient] -->|uses| Gt[MC return G_t] Gt -->|is| HighVar[High variance signal] HighVar -->|fixed by| Baseline[Subtract baseline b s] Baseline -->|best choice| V[State value V s] Baseline -->|proof| NoBias[Adds no bias] V -->|defines| Adv[Advantage A = Q - V] Adv -->|low variance signal| PG[Policy gradient update] Actor[Actor pi theta] -->|chooses actions| PG Critic[Critic V phi] -->|estimates value| V Adv -->|estimated via| TD[TD error delta_t] TD -->|bootstraps| Critic Actor -->|learns from| Critic ``` ## 🔊 Hinglish (regional understanding) > [!intuition]- Hinglish mein samjho > Dekho, REINFORCE me hum poore episode ka return $G_t$ use karte hain, lekin wo bahut **noisy** hota hai — ek future me mila lucky reward pura signal kharab kar deta hai. Isko theek karne ke liye hum ek **baseline** minus karte hain, aur best baseline hota hai state ki value $V(s)$. Jo bacha usko bolte hain **advantage** $A = Q - V$ — matlab "ye action average se kitna behtar tha". Yahi advantage actor ko batata hai kaunsa action zyada karo, kaunsa kam. > > Ab do dimaag chahiye: **Actor** ($\pi_\theta$) jo action choose karta hai, aur **Critic** ($V_\phi$) jo judge karta hai. Kamaal ki baat: advantage ko separately compute karne ki zaroorat nahi — Critic ka **TD error** $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ hi advantage ka estimate hai! Agar $\delta$ positive hai to action ko zyada probable banao, negative hai to kam. Critic khud ko TD target ke taraf move karta hai (target pe **stop-gradient** lagana mat bhoolna, warna cheating ho jaati hai). > > **A3C vs A2C** ka farak sirf itna: A3C me bahut saare workers alag-alag environment me parallel chalte hain aur asynchronously gradients global network me daalte hain — isse data decorrelated ho jaata hai (DQN ke replay buffer ka kaam yahan parallelism karta hai, kyunki ye **on-policy** hai). A2C wahi cheez synchronous tareeke se karta hai: sab workers ka gradient average karke ek update. Aur ek chhoti si magar important baat — **entropy bonus** ($\beta H$) zaroor add karo taaki policy jaldi collapse na ho aur exploration chalta rahe. ![[audio/5.2.08-Advantage-Actor-Critic-(A2C---A3C).mp3]]