5.2.8 · D5Deep & Advanced RL
Question bank — Advantage Actor-Critic (A2C - A3C)
True or false — justify
True or false: subtracting a baseline (a state-only number) changes the direction the policy gradient points.
False — a state-only baseline has zero expectation against , so the expected gradient is untouched; it only shrinks variance (the noise around that direction).
True or false: the best possible baseline is exactly .
Roughly true in practice and the one A2C uses, but the variance-minimizing baseline is actually a return-weighted value, not plain ; is chosen because it makes the leftover term the interpretable advantage .
True or false: a positive advantage means the action gave a positive reward.
False — advantage is relative. A reward of can still give a negative advantage if the critic expected from that state; what matters is better-than-average, not better-than-zero.
True or false: A2C is off-policy because it reuses worker rollouts.
False — every worker rolls out with the current policy and the gradient is consumed immediately; nothing old-policy is stored, so it stays strictly on-policy.
True or false: the entropy bonus (the term rewarding randomness) is just a cosmetic regularizer you can drop for speed.
False — dropping it lets spike to a near-deterministic action early, killing exploration and locking in a bad local optimum; it materially changes what is learned, not just how neatly.
True or false: A3C usually outperforms A2C because asynchrony adds useful randomness.
False — A2C (the synchronous averaging version) is typically equal or better and more GPU-efficient; asynchrony was an engineering choice for CPU clusters, not a learning advantage.
True or false: the TD error is used both as the advantage estimate for the actor and as the loss signal for the critic.
True — the same multiplies in the actor update and is squared (with a stop-gradient target) as the critic loss; one quantity, two jobs.
True or false: because the critic appears in both the target and the estimate of the TD error, it cancels out and the critic is irrelevant.
False — the two evaluations are at different states ( vs ), so they don't cancel; a good critic is what makes a low-variance, informative signal.
Spot the error
"Since (the observed discounted return) is unbiased, REINFORCE is strictly better than A2C." — what's wrong?
Unbiased ≠ good: has huge variance from summing many random future rewards, so gradients are noisy and slow. A2C accepts a little bias (from bootstrapping) to cut that variance — see Bias-Variance Tradeoff.
"To minimize the critic loss faster, let gradients flow through the target ." — what's wrong?
You'd be moving the goalposts: the optimizer could shrink to trivially make the loss small instead of learning value. The target must be a stop-gradient constant.
"A3C decorrelates data with a replay buffer, just like DQN." — what's wrong?
A replay buffer holds old-policy actions, which makes point the wrong way for an on-policy method. A3C/A2C decorrelate via parallel environments, not a buffer.
"The critic outputs (the action-value), and we subtract to get the advantage." — what's wrong?
A2C's critic outputs only ; there is no network. The advantage is estimated by the TD error , which stands in for .
"Increasing in the -step advantage always makes learning more accurate." — what's wrong?
Larger lowers bias but raises variance (more random real rewards summed); it's a dial, not a free improvement. The sweet spot depends on the critic's quality, and GAE (Generalized Advantage Estimation) tunes it smoothly.
"The actor loss increases the action's probability whenever we take it." — what's wrong?
Only when the advantage estimate . If the sign flips and the update decreases that action's probability — the advantage's sign decides push-toward vs push-away.
"Entropy should be minimized to make the policy confident." — what's wrong?
We add a bonus (with coefficient ), encouraging higher entropy early. Minimizing it would collapse exploration prematurely.
Why questions
Why is the advantage a better learning signal than the raw value ?
tells you how good the state is but says nothing about which action to prefer; the advantage isolates each action's edge over the state's average, which is exactly what the policy needs.
Why does using as the baseline turn the return into the advantage?
Because , so has expectation ; the baseline subtraction recenters the noisy return around the state's mean value.
Why do we call it "actor-critic" rather than just "policy gradient with a value net"?
Because the two roles genuinely interact: the actor acts, the critic judges each action via , and that judgment feeds straight back into improving the actor — a coupled learning loop, not a passive add-on.
Why can A2C get away without a replay buffer when DQN needs one?
DQN is off-policy and must decorrelate stale bootstrapped samples; A2C is on-policy and gets fresh, decorrelated data from many parallel environments at once, which serves the same de-correlation purpose without storing old data.
Why does the TD error equal the advantage estimate at all?
is a one-step bootstrap estimate of ; subtracting gives . So the TD error literally is a sample estimate of .
Why does averaging gradients across synchronous workers (A2C) reduce variance?
Each worker's gradient is a noisy sample from a different trajectory; averaging independent noisy estimates shrinks their variance roughly like , giving a cleaner update direction.
Why must every gradient in A2C come from the current policy?
The score function only points toward improvement for actions sampled from the present ; feeding it actions from an older policy corrupts the direction — this is the on-policy constraint of the Policy Gradient Theorem.
Why can subtracting a baseline ever hurt if it depends on the action?
An action-dependent "baseline" no longer sums to zero against , so it injects bias and changes the expected gradient — the zero-expectation proof requires the baseline be a function of the state only.
Edge cases
Edge case: the environment gives every step. Is the advantage always zero?
No — can be nonzero because value can still change between states (e.g. moving closer to a distant reward); zero reward ≠ zero advantage.
Edge case: is terminal. What happens to the bootstrap?
You set (no future beyond the episode's end), so ; forgetting this leaks a phantom future value and biases the target.
Edge case: the critic is perfect, exactly. Does learning stop?
The critic stops changing, but the actor keeps improving: still measures each action's advantage, so is still nudged toward better-than-average actions.
Edge case: the policy has already collapsed to one action ( nearly deterministic). Can A2C recover?
Only weakly — with almost no entropy the policy rarely samples alternatives, so it can't discover a better action; this is exactly why the bonus must be present before collapse, not added after.
Edge case: . What does the advantage become?
— the agent becomes purely myopic, judging actions only by immediate reward versus expected value, ignoring all future consequences.
Edge case: with no episode termination. Why is this dangerous?
The bootstrap can grow unboundedly (infinite-horizon sums don't converge), making value targets unstable; discounting or termination is needed to keep finite.
Edge case: one worker's environment is buggy and returns garbage rewards. Effect on A2C vs A3C?
In A2C the bad gradient is averaged with good ones, diluting its harm; in A3C it can be pushed asynchronously and immediately corrupt the shared network before others correct it.
Edge case: in the -step advantage (full episode). What does A2C reduce to?
The bootstrap term vanishes and — i.e. plain REINFORCE with a value baseline: maximal variance, minimal reliance on the critic.
Recall One-line self-test
If you can state, for advantage: (1) why it beats raw return, (2) why the baseline is bias-free, (3) why the target is stop-gradient, and (4) why no replay buffer — you have the whole trap set covered. Advantage recentered around what? ::: The state's average value , so only better-than-average actions get reinforced.