Advantage Actor-Critic (A2C - A3C): Visual walkthrough

Step 1 — A number line of luck: what a "return" is

WHAT. An agent sitting in a situation (a state, written $s_{t}$ — the subscript $t$ just means "at time-step $t$ ", like a clock tick) picks an action $a_{t}$ . The world then hands it a stream of small rewards $r_{t}, r_{t + 1}, r_{t + 2}, \dots$ (little numbers: +1 for a good thing, −1 for a bad thing). The return $G_{t}$ is the whole future pile, but with far-off rewards shrunk by a discount $γ$ (a knob between 0 and 1):

$G_{t} = reward now r_{t} + shrink γ r_{t + 1} + shrink twice γ^{2} r_{t + 2} + \dots$

Each symbol: $r_{t + k}$ is the reward $k$ ticks later; $γ^{k}$ (gamma to the power $k$ ) is how much we still care about it — since $γ < 1$ , raising it to bigger powers makes it tinier, so the distant future fades.

WHY. We want one number that says "how much total good followed this moment?" A single reward isn't enough — a move can pay off many steps later.

PICTURE. Look at s01: each future reward is a chalk bar on a number line, and the pale-yellow curve $γ^{k}$ presses the far bars down toward zero.

Step 2 — The same move, wildly different scores (why $G_{t}$ is noisy)

WHAT. Run the exact same state and action many times. The environment is random (dice rolls, opponent moves, luck), so the future rewards differ each run — and therefore $G_{t}$ differs each run. We get a whole spread of returns for one action.

WHY. REINFORCE updates the policy using $G_{t}$ directly. If $G_{t}$ jumps around from +50 to −30 for the same action, the learning signal is drowning in luck. This spread is the variance — see Bias-Variance Tradeoff.

PICTURE. s02: a histogram of many $G_{t}$ samples for one identical action. It's fat and messy. The tall blue line marks its average, which is the truly informative quantity.

Step 3 — Two averages, and the gap between them

WHAT. Two "centre" numbers matter:

$Q (s_{t}, a_{t})$ — average return if you take this specific action $a_{t}$ , then behave normally.
$V (s_{t})$ — average return from this state over all the actions the policy usually picks. The state's "typical" score.

Their difference is the star of the show, the advantage:

$A (s_{t}, a_{t}) = this action’s average Q (s_{t}, a_{t}) - state’s average V (s_{t})$

Read it as a subtraction of two heights: "how far above (or below) the room's average did this action land?"

WHY. $Q$ alone is biased for choosing: every action in a great state looks great. What we need is relative goodness — better or worse than the norm. That is exactly $Q - V$ .

PICTURE. s03: two horizontal chalk lines, $V (s_{t})$ (blue, the baseline height) and $Q (s_{t}, a_{t})$ (pink). The pale-yellow bracket between them is $A$ — positive if the action beat the baseline, negative if it fell short.

Step 4 — Why subtracting the baseline is free

WHAT. We swap the noisy $G_{t}$ for $A = Q - V$ inside the policy-gradient update. Subtracting $V (s_{t})$ (a number that depends on the state only, not the action) shifts the signal but does not bias the gradient. This is the Policy Gradient Theorem baseline result.

WHY. Here's the one line that makes it free. The action-probabilities always sum to 1, and the gradient of a constant (1) is 0:

= \nabla_\theta \underbrace{\sum_a \pi_\theta(a\mid s)}_{=\,1} = \nabla_\theta(1) = 0.$$ Term by term: $\pi_\theta(a\mid s)$ is the probability the policy gives action $a$; $\theta$ (theta) is the tunable weights inside the policy; $\nabla_\theta$ ("nabla theta") means "the direction in weight-space that increases this." Because every state's probabilities must total 1, nudging $\theta$ can only *shuffle* probability between actions — the total never changes — so the baseline term vanishes. **PICTURE.** s04: an arrow pushing up on one action's probability bar automatically pushes the others down; the total stays pinned at 1. A constant baseline $b(s)$ multiplies this zero, so it adds nothing. > [!intuition] What we bought > Same expected gradient, **smaller spread**. We centred the signal around zero: good actions get positive nudges, bad ones negative, and "average" actions get ≈0 instead of a big absolute number that just tells us "the state was nice." --- ## Step 5 — We only trained ONE network. How do we get $A = Q - V$? **WHAT.** We have a **critic** $V_\phi(s)$ — a network with weights $\phi$ (phi) that estimates $V$. We never trained a $Q$ network. The trick: peel *one real reward* off the return and let the critic estimate the rest. $$Q(s_t,a_t) \;\approx\; \underbrace{r_t}_{\text{one real step}} \;+\; \gamma\,\underbrace{V_\phi(s_{t+1})}_{\text{critic guesses the rest}}$$ This is **bootstrapping** from [[Temporal-Difference Learning]]: use your own next-state estimate as a stand-in for "everything after the first step." **WHY.** $r_t + \gamma V_\phi(s_{t+1})$ is a *sample-cheap* estimate of $Q$: it needs just one observed reward plus one critic call, instead of rolling the whole noisy future. **PICTURE.** s05: a chain $s_t \to s_{t+1}$. The first hop is a solid pink reward $r_t$ (real, observed); everything past $s_{t+1}$ is folded into the blue box $V_\phi(s_{t+1})$ (the critic's guess), scaled by $\gamma$. --- ## Step 6 — The punchline: the advantage IS the TD error **WHAT.** Plug the bootstrap estimate of $Q$ (Step 5) into $A = Q - V$ (Step 3): $$\hat A_t = \big[\,r_t + \gamma V_\phi(s_{t+1})\,\big] - V_\phi(s_t) = \underbrace{\delta_t}_{\text{TD error}}$$ Term by term: - $r_t$ ::: the reward we actually got. - $\gamma V_\phi(s_{t+1})$ ::: the critic's discounted guess for the future *after* the move. - $V_\phi(s_t)$ ::: the critic's guess for *before* the move — what it expected. - $\delta_t$ (delta) ::: the **surprise**: outcome minus expectation. So the critic's own prediction error, $\delta_t$, is exactly our advantage estimate. No extra network needed. **WHY.** $\delta_t > 0$ means "things went *better* than the critic predicted" → push the actor to make $a_t$ more likely. $\delta_t < 0$ means "worse than predicted" → make it less likely. $\delta_t \approx 0$ means "as expected" → barely nudge. That is precisely the centred, low-variance signal we wanted. **PICTURE.** s06: the critic draws a dashed "expected" bar at height $V_\phi(s_t)$; the realized $r_t + \gamma V_\phi(s_{t+1})$ lands above (pink) or below (blue). The gap $\delta_t$ is the advantage arrow. > [!formula] The one identity to remember > $$\hat A_t \;=\; \delta_t \;=\; r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$ > Actor push: $-\log\pi_\theta(a_t\mid s_t)\,\hat A_t$. Positive $\hat A_t$ ⇒ raise this action's probability. --- ## Step 7 — Worked numbers: positive, negative, and multi-step **WHAT.** We turn the crank on three concrete cases so no scenario surprises you. **Case A — a pleasant surprise** ($\gamma=0.9$, $V_\phi(s_t)=5$, $r_t=2$, $V_\phi(s_{t+1})=6$): $$\delta_t = 2 + 0.9(6) - 5 = 2 + 5.4 - 5 = 2.4 \;>\;0 \;\Rightarrow\; \text{make } a_t \text{ more likely.}$$ **Case B — a disappointment** ($r_t=-1$, $V_\phi(s_{t+1})=4$): $$\delta_t = -1 + 0.9(4) - 5 = -1 + 3.6 - 5 = -2.4 \;<\;0 \;\Rightarrow\; \text{make } a_t \text{ less likely.}$$ **Case C — the degenerate "no surprise"** ($r_t=0.5$, $V_\phi(s_{t+1})=5$): $$\delta_t = 0.5 + 0.9(5) - 5 = 0.5 + 4.5 - 5 = 0 \;\Rightarrow\; \text{no nudge; the critic was exactly right.}$$ **WHY.** Case C is the important edge case: when the critic is perfect, the advantage is zero and the actor stops changing — the system is at a fixed point for that transition. Good. **PICTURE.** s07: three side-by-side gaps — pink up (A), blue down (B), flat (C) — sharing the same dashed baseline $V_\phi(s_t)=5$. > [!example] Extending to $n$ steps (the bias–variance dial) > Trust more real rewards before bootstrapping. With $\gamma=0.9$, $r_t=1$, $r_{t+1}=2$, $V_\phi(s_t)=5$, $V_\phi(s_{t+2})=8$: > $$\hat A_t^{(2)} = 1 + 0.9(2) + 0.9^2(8) - 5 = 1 + 1.8 + 6.48 - 5 = 4.28.$$ > More real rewards ⇒ less reliance on the critic ⇒ **less bias, more variance**. [[Generalized Advantage Estimation (GAE)]] blends every $n$ at once; [[PPO]] uses it downstream. --- ## The one-picture summary s08 compresses the whole journey: the noisy return $G_t$ (fat cloud) → its average $Q$ → subtract the state baseline $V$ → the centred advantage $A$ → estimated cheaply as the critic's TD surprise $\delta_t$ → which points the actor's arrow up or down. > [!recall]- Feynman: the whole walkthrough in plain words > You do a move and see how it turned out. But "how it turned out" bounces around because of luck (Step 1–2), so the raw number is a bad teacher. The fair thing to ask is: *did this move beat what I normally get from here?* That "beat the normal" number is the advantage (Step 3). Subtracting your usual score is free — it can't lie to you, because probabilities always add up to one, so shifting them never changes the total (Step 4). You don't even need a separate "how good is this exact move" gadget: just take the one reward you really saw, add your coach's guess for the rest, and compare it to your coach's guess for before you moved (Step 5). That comparison — outcome minus expectation — is the *surprise*, and the surprise IS the advantage (Step 6). Positive surprise: do it more. Negative: do it less. Zero: the coach nailed it, don't change (Step 7). That's A2C, drawn end to end. --- ## Active recall Why is the raw return $G_t$ a noisy teacher? ::: It sums many random future rewards, so the same action yields a wide spread of $G_t$ values (high variance). What is the advantage $A(s,a)$ in words? ::: How much better (or worse) this action is than the state's average score: $Q(s,a)-V(s)$. Why does subtracting a state baseline add no bias? ::: Action probabilities sum to 1, so $\nabla_\theta\sum_a\pi=\nabla_\theta 1=0$; the baseline term has zero expectation. How do we get $A$ with only a value critic? ::: Bootstrap $Q\approx r_t+\gamma V_\phi(s_{t+1})$, so $\hat A_t = r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t)=\delta_t$. What does $\delta_t=0$ mean? ::: The critic predicted the outcome exactly; the advantage is zero and the actor is not nudged. Effect of using more real rewards ($n$-step)? ::: Less bias (trusts the critic less), more variance (trusts noisy real returns more).