Advantage Actor-Critic (A2C - A3C): Worked examples

This is the "grind through every case" companion to the parent A2C/A3C note. There we derived that the TD error is the advantage estimate: $δ_{t} = what actually + expected r_{t} + γ V_{ϕ} (s_{t + 1}) - what we predicted V_{ϕ} (s_{t}) .$ Here we make sure you never meet a case you haven't seen: positive/negative/zero advantage, terminal states, $γ = 0$ and $γ = 1$ , the $n$ -step dial, GAE smoothing, entropy, and an exam twist.

The scenario matrix

Every A2C advantage computation falls into one of these cells. The examples below are each tagged with the cell they cover.

Cell	What makes it special	Covered by
C1 Positive advantage	reality beat the critic, $δ_{t} > 0$	Ex 1
C2 Negative advantage	reality was worse, $δ_{t} < 0$	Ex 2
C3 Zero advantage	critic was exactly right, $δ_{t} = 0$	Ex 3
C4 Terminal state	no next state, so $V_{ϕ} (s_{t + 1}) = 0$	Ex 4
C5 $γ$ extremes	$γ = 0$ (myopic) and $γ = 1$ (undiscounted)	Ex 5
C6 $n$ -step dial	use $n$ real rewards then bootstrap	Ex 6
C7 GAE blend	smoothly mix all $n$ via $λ$	Ex 7
C8 Entropy + loss sign	full actor loss with the $β H$ bonus	Ex 8
C9 Word problem	real story maps onto numbers	Ex 9
C10 Exam twist	stop-gradient / on-policy trap	Ex 10

Figure — Advantage Actor-Critic (A2C - A3C)

The picture above is the mental model for every cell: a number line where the critic's prediction sits at zero, and $δ_{t}$ is the signed distance of reality from it.

Worked examples

Worked example Example 1 — Positive advantage (cell C1)

$γ = 0.9$ , $V_{ϕ} (s_{t}) = 5$ , $r_{t} = 2$ , $V_{ϕ} (s_{t + 1}) = 6$ . Find $δ_{t}$ and say which way the actor moves.

Forecast: guess the sign before computing — the next state is valued higher (6 vs 5) and we also got a reward, so… ?

Compute the bootstrap target $r_{t} + γ V_{ϕ} (s_{t + 1}) = 2 + 0.9 (6) = 2 + 5.4 = 7.4$ . Why this step? This is our best one-step estimate of $Q (s_{t}, a_{t})$ — "reward now plus discounted value of where we landed."
Subtract the old prediction: $δ_{t} = 7.4 - 5 = 2.4$ . Why this step? $δ_{t} = Q - V$ is the advantage: how much better the action turned out than the state's average.
Actor direction: loss $= - lo g π \cdot δ_{t} = - 2.4 lo g π$ . Minimizing this raises $lo g π$ , i.e. makes $a_{t}$ more likely. Why this step? Positive surprise ⇒ reinforce.

Verify: target $7.4 > 5$ , so $δ_{t} > 0$ ✓. Critic will nudge $V_{ϕ} (s_{t})$ up toward $7.4$ . Sign matches forecast.

Worked example Example 7 — GAE blend (cell C7)

Generalized Advantage Estimation (GAE) blends all $n$ -step advantages with weight $λ$ : $\hat{A}_{t}^{GAE} = \sum_{l \geq 0} (γ λ)^{l} δ_{t + l} .$ Take a 3-step window with per-step TD errors $δ_{t} = 1.0, δ_{t + 1} = 0.5, δ_{t + 2} = - 0.2$ , $γ = 0.9$ , $λ = 0.95$ .

Forecast: $λ$ close to 1 — does GAE weight the later errors heavily or lightly?

Compute the decay factor $γ λ = 0.9 \times 0.95 = 0.855$ . Why this step? This single number controls how fast future TD errors are down-weighted — it's the GAE "memory."
Weighted sum: $\hat{A}_{t}^{GAE} = 1.0 + 0.855 (0.5) + 0.85 5^{2} (- 0.2) .$
Evaluate: $1.0 + 0.4275 + 0.731025 (- 0.2) = 1.0 + 0.4275 - 0.146205 = 1.281295$ . Why this step? GAE ≈ an exponentially-weighted average of every $n$ -step estimate at once — one number, smoothly tunable.

Verify: with $λ = 0$ GAE collapses to the 1-step $δ_{t} = 1.0$ ; our $λ = 0.95$ answer ( $\approx 1.28$ ) is larger because later positive error $δ_{t + 1}$ still counts ✓.

Worked example Example 8 — Entropy in the full actor loss (cell C8)

A 2-action policy at $s_{t}$ : $π = (0.8, 0.2)$ , we took action 1 (prob 0.8), $\hat{A}_{t} = 2.4$ , entropy coefficient $β = 0.1$ .

Forecast: the entropy bonus $- β H$ — does it raise or lower the loss for a spread-out policy?

Policy-gradient term: $- lo g π (a_{t} ∣ s_{t}) \hat{A}_{t} = - lo g (0.8) \cdot 2.4$ . With $lo g (0.8) \approx - 0.22314$ , this is $- (- 0.22314) (2.4) = 0.53554$ . Why this step? Standard actor loss; a positive advantage on a confident action gives a positive loss whose gradient still increases that action's probability.
Entropy $H = - \sum π lo g π = - (0.8 lo g 0.8 + 0.2 lo g 0.2)$ . Compute: $- (0.8 (- 0.22314) + 0.2 (- 1.60944)) = - (- 0.17851 - 0.32189) = 0.50040$ . Why this step? $H$ measures how "spread out" the policy is; higher $H$ = more exploration.
Full loss $L_{π} = 0.53554 - β H = 0.53554 - 0.1 (0.50040) = 0.53554 - 0.05004 = 0.48550$ . Why this step? The $- β H$ term subtracts value for staying spread out, so minimizing the loss resists collapsing to a deterministic policy.

Verify: max entropy for 2 actions is $lo g 2 \approx 0.693$ ; our $H = 0.500$ is below that (policy isn't uniform) ✓, and the entropy term shaved $0.05$ off the loss as expected.

Worked example Example 9 — Word problem (cell C9)

A delivery robot in state $s_{t}$ = "at the depot." The critic thinks the depot is worth $V_{ϕ} = 40$ (expected future packages). The robot takes a shortcut (its action), immediately loses $r_{t} = - 5$ (traffic fine) but reaches a hub the critic values at $V_{ϕ} (s_{t + 1}) = 55$ . Discount $γ = 0.9$ . Should the actor learn to take that shortcut again?

Forecast: short-term penalty but a much better next position — reinforce or avoid?

Target $= - 5 + 0.9 (55) = - 5 + 49.5 = 44.5$ . Why this step? We trade the fine against the discounted value of a great next state.
$δ_{t} = 44.5 - 40 = 4.5 > 0$ . Why this step? Despite the fine, the move beat the depot's average expectation.
Actor loss $- lo g π \cdot 4.5$ ⇒ shortcut becomes more likely. Why this step? Positive advantage ⇒ the long-term gain outweighs the short-term cost.

Verify: flip intuition — if the hub were only worth $V_{ϕ} (s_{t + 1}) = 45$ , then target $= - 5 + 40.5 = 35.5$ , giving $δ_{t} = - 4.5 < 0$ and the robot would learn to avoid the shortcut ✓. The sign hinges entirely on whether the future gain covers the fine.

Worked example Example 10 — Exam twist: the stop-gradient trap (cell C10)

The critic target is $y = r_{t} + γ V_{ϕ} (s_{t + 1})$ and the critic loss is $\frac{1}{2} (y - V_{ϕ} (s_{t}))^{2}$ . A student differentiates through both $V_{ϕ}$ terms. With $γ = 0.9$ , $r_{t} = 2$ , $V_{ϕ} (s_{t + 1}) = 6$ , $V_{ϕ} (s_{t}) = 5$ , contrast the intended vs buggy behaviour.

Forecast: should the gradient try to change $V_{ϕ} (s_{t + 1})$ or leave it fixed?

Correct: stop-gradient the target. $y = 7.4$ is treated as a constant. Loss $= \frac{1}{2} (7.4 - 5)^{2} = \frac{1}{2} (2.4)^{2} = 2.88$ ; gradient only pushes $V_{ϕ} (s_{t})$ up toward $7.4$ . Why this step? We want $V_{ϕ} (s_{t})$ to predict the target, not to move the target — otherwise we're chasing our own tail.
Buggy: differentiating $y$ too lets the optimizer lower $V_{ϕ} (s_{t + 1})$ to shrink the loss trivially, corrupting values (the "moving goalposts" bug from the parent note). Why this step? Minimizing $(y - V_{ϕ} (s_{t}))^{2}$ by dragging $y$ down is cheating, not learning value.

Verify: with stop-grad, loss $= \frac{1}{2} (2.4)^{2} = 2.88$ ✓. The advantage used by the actor, $δ_{t} = 2.4$ , is exactly Ex 1 — the actor and critic share the same TD error, but the critic target must stay frozen for the actor's signal to remain meaningful ✓.

How the cases connect

Active recall

Recall

When $δ_{t} = 0$ , what does the actor do? ::: Nothing — the gradient $- lo g π \cdot 0 = 0$ ; the critic already predicted perfectly. What do you plug in for $V_{ϕ} (s_{t + 1})$ at a terminal state? ::: Zero — there is no future after the episode ends. With $γ = 0$ , the advantage reduces to what? ::: $r_{t} - V_{ϕ} (s_{t})$ (only the immediate reward, no bootstrap). Why treat the critic target as a constant (stop-gradient)? ::: So the network predicts the target instead of moving it down to trivially shrink the loss. GAE decay factor per step? ::: $γ λ$ — it exponentially down-weights later TD errors.