Actor-critic methods: Question bank — AI-ML deep dive

Before you start, a one-line refresher on every symbol used below, so nothing is unearned:

Definition Symbols on this page

$π_{θ} (a ∣ s)$ — the actor: probability the policy (with parameters $θ$ ) picks action $a$ in state $s$ .
$r_{t + 1}$ (written $r$ for short) — the immediate reward: the scalar payoff the environment hands back one step after you take action $a_{t}$ in state $s_{t}$ . It is a single-step quantity, not a whole trajectory's worth.
$G_{t}$ — the return: the actual total (discounted) reward collected from time $t$ onward along one sampled trajectory, i.e. $G_{t} = r_{t + 1} + γ r_{t + 2} + γ^{2} r_{t + 3} + \dots$ . This is REINFORCE's raw, high-variance learning signal.
$V (s)$ — the true state value: the expected return starting from $s$ , i.e. $V (s) = E [G_{t} ∣ s]$ . The critic $V_{w} (s)$ (parameters $w$ ) is a learned estimate of it.
$Q (s, a)$ — the action value: the expected return if you take action $a$ in $s$ and then follow the policy, i.e. $Q (s, a) = E [G_{t} ∣ s, a]$ . So $V (s)$ is the action-average of $Q (s, a)$ .
$δ_{t} = r_{t + 1} + γ V_{w} (s_{t + 1}) - V_{w} (s_{t})$ — the TD error: realized-minus-expected, the "surprise". It is a one-step bootstrapped estimate of the advantage.
$A (s, a) = Q (s, a) - V (s)$ — the advantage: how much better action $a$ is than the average action in $s$ .
$J (θ)$ — the policy objective: the expected return the policy earns, $J (θ) = E_{π_{θ}} [G_{0}]$ . Training the actor means climbing $J$ via gradient ascent, so $\nabla_{θ} J$ is the direction that most increases expected return.
$α_{w}, α_{θ}$ — the learning rates (step sizes) for the critic and the actor respectively: how big a step each network takes along its gradient per update.
$h_{θ} (s, a)$ — the logit (raw score) the network assigns to action $a$ in state $s$ before the softmax turns scores into probabilities: $π_{θ} (a ∣ s) = softmax (h_{θ} (s, \cdot))$ .
$Ψ_{t}$ — the generic credit signal slotted into the Policy Gradient Theorem $\nabla_{θ} J = E [\nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) Ψ_{t}]$ . Choosing $Ψ_{t} = G_{t}$ gives REINFORCE; $Ψ_{t} = A (s, a)$ gives actor-critic.
$γ \in [0, 1]$ — the discount: how much we care about future reward vs. immediate.

The figure below traces the loop: the actor sends an action, the environment returns $r$ and $s^{'}$ , and the critic forms one $δ_{t}$ that then flows up to the actor (as an advantage) and down into itself (as an error to reduce). Several answers below (WQ1, EC1, SE5) refer back to specific arrows in it.

True or false — justify

TF1. In REINFORCE, the return

G_{t}

is a biased estimate of the policy gradient.

False.

G_{t}

(the actual trajectory return defined above) is unbiased — correct on average; its problem is high variance, because it depends on an entire random trajectory — see REINFORCE.

TF2. Subtracting a baseline

b (s)

that depends only on the state changes the expected gradient.

False. Its expected contribution is exactly zero because

E_{a} [\nabla_{θ} lo g π_{θ} (a ∣ s)] = \nabla_{θ} 1 = 0

, so the gradient stays unbiased while variance can drop.

TF3. A baseline that depends on the action,

b (s, a)

, is still safe to subtract.

False. Only state-only baselines cancel cleanly; an action-dependent term does not integrate to zero and would bias the gradient (unless carefully corrected, as in some control-variate schemes).

TF4. The true advantage

A (s, a) = Q (s, a) - V (s)

can be negative even when the immediate reward

r

is positive.

True.

A

compares the action's full value

Q (s, a)

against the state average

V (s)

; if the action leads to poor futures,

Q (s, a) < V (s)

and

A < 0

despite

r > 0

. (In practice we estimate this via the one-step

δ

, which can likewise be negative when

γ V (s^{'})

is low.)

TF5. Actor-critic must wait until the episode ends to update, like Monte-Carlo REINFORCE.

False. It is online/incremental: the bootstrapped

δ_{t}

needs only

(s_{t}, r_{t + 1}, s_{t + 1})

, so it updates every step — that's the whole point of using Temporal Difference Learning.

TF6. In the update

w \leftarrow w + α_{w} δ_{t} \nabla_{w} V_{w} (s_{t})

we should also differentiate the target

V_{w} (s_{t + 1})

False. The target

r + γ V_{w} (s_{t + 1})

is treated as fixed (stop-gradient) — this is semi-gradient TD; differentiating the target invites instability.

TF7. A2C and A3C use fundamentally different mathematics from vanilla actor-critic.

False. Same advantage math; A2C batches synchronous advantages, A3C runs asynchronous parallel workers — different plumbing, identical objective (see A3C and A2C).

TF8. The variance-minimizing state-only baseline is exactly

V (s)

True (to the standard approximation used in derivations). For the pure policy-gradient objective the optimal constant-per-state baseline is

E [G ∣ s] = V (s)

— subtracting it centers the return's mean to zero in each state, which is why

V (s)

is the canonical baseline, not just a convenient one.

TF9. If the critic is perfect, the actor's gradient becomes zero.

False. A perfect critic gives accurate advantages, not zero ones — the gradient vanishes only when the policy is optimal so that

A (s, a) \leq 0

for all improvements.

Spot the error

SE1. "I'll set

α_{w} = α_{θ}

to keep things simple."

The critic should usually learn faster (

α_{w} > α_{θ}

) so the advantage the actor trusts is reasonably accurate — a lagging critic feeds the actor garbage. This is the two-timescale requirement.

SE2. "Advantage and reward are basically the same, so I'll just multiply the log-prob by

r

Raw reward has no baseline, so it carries the high-variance "how good is this state at all" term. Subtracting

V (s)

cancels that, cutting variance while staying unbiased.

SE3. "I updated the actor with

∣ δ_{t} ∣

so bad actions still get pushed strongly."

The sign of

δ_{t}

carries the direction; using

∣ δ_{t} ∣

would push worse-than-expected actions up, breaking the credit signal entirely.

SE4. "The critic outputs

Q_{w} (s, a)

, so I don't need

V

— I'll use

Q

directly as the actor's weight."

Then you lose the baseline and its variance reduction. You want the advantage

Q - V

, not

Q

; using

Q

alone reintroduces the state-value nuisance term.

SE5. "My TD error is

δ = r + γ V (s^{'}) - V (s^{'})

The subtracted term must be

V (s_{t})

(the state you left), not

V (s_{t + 1})

— the figure's "fit V" arrow reads

V_{w} (s)

, the start state. As written

δ

collapses to

r

and destroys the bootstrap.

SE6. "I use

γ = 1

in a never-ending (continuing) task."

With

γ = 1

and no episode end, the bootstrapped value can diverge (infinite undiscounted sum). Continuing tasks need

γ < 1

or an average-reward formulation.

SE7. "The actor's softmax gradient just raises the chosen action's logit

h_{θ} (s, a)

It also lowers every other logit in proportion to its probability:

\nabla_{θ} lo g π = \nabla_{θ} h_{θ} (s, a) - \sum_{a^{'}} π (a^{'} ∣ s) \nabla_{θ} h_{θ} (s, a^{'})

. It's a competition, not a solo raise (

h_{θ} (s, a)

= the raw pre-softmax score of action

a

Why questions

WQ1. Why does the same

δ_{t}

appear in both the actor and critic updates?

It is one quantity with two readings: to the critic it's "you mispredicted by

δ_{t}

" (error to reduce), and to the actor it's "this action beat expectation by

δ_{t}

" (advantage estimate). In the figure this is the single

δ_{t}

box whose two arrows split up to the actor and down to the critic — see Advantage function.

WQ2. Why does actor-critic have lower variance than REINFORCE but potentially some bias?

Bootstrapping replaces the noisy full return

G_{t}

with

r + γ V_{w} (s^{'})

, slashing variance; but because

V_{w}

is only an estimate, the signal is no longer perfectly unbiased — a bias/variance trade dial explored by Generalized Advantage Estimation (GAE).

WQ3. Why is a state-only baseline "free" (no bias) yet still useful?

Its expected pull on the gradient is exactly zero (TF2), so on average it changes nothing — but it re-centers the noisy per-sample signal around zero, shrinking its spread.

WQ4. Why can't pure value methods (Q-learning) easily handle continuous action spaces, motivating the actor?

They need

max_{a} Q (s, a)

, an intractable search over a continuum; the actor sidesteps this by directly parameterizing a policy you can sample from.

WQ5. Why does the advantage's sign matter more than its magnitude for direction?

Positive

\Rightarrow

action was better than the state's average

\Rightarrow

raise its probability; negative

\Rightarrow

worse

\Rightarrow

lower it. Magnitude only scales the step size, direction is set by sign.

WQ6. Why does PPO still keep a critic despite clipping the policy update?

It needs advantage estimates to weight the (clipped) policy-gradient; the critic supplies

V_{w}

for computing those advantages — see Proximal Policy Optimization (PPO).

WQ7. Why does the Policy Gradient Theorem permit any baseline

b (s)

inside the credit signal

Ψ_{t}

Because the theorem's expectation factorizes so that any state-only additive term multiplied by

\nabla lo g π

averages to zero — so replacing

Ψ_{t} = G_{t}

with

Ψ_{t} = G_{t} - b (s)

(or

Ψ_{t} = A (s, a)

) leaves the gradient

\nabla_{θ} J

unbiased. That identity licenses the whole baseline trick.

Edge cases

EC1. Terminal state

s_{t + 1}

: what happens to the bootstrap term

γ V_{w} (s_{t + 1})

It is set to zero (there is no future beyond terminal), so

δ_{t} = r_{t + 1} - V_{w} (s_{t})

— in the figure, the "r, s'" arrow returns a terminal

s^{'}

whose value contributes nothing. Forgetting this leaks phantom future value into the last step.

EC2. Reward is identically zero on a transition but

s^{'}

has high value — is

δ

zero?

Not necessarily.

δ = 0 + γ V (s^{'}) - V (s)

; if

γ V (s^{'}) > V (s)

the transition is still "surprisingly good" and

δ > 0

EC3. The critic is randomly initialized at the start of training — is early actor learning meaningful?

Mostly noise at first, since advantages are near-random; this is why the critic is given a faster learning rate to become useful before the actor commits hard.

EC4. Deterministic actor (zero policy entropy): what breaks?

Exploration collapses —

\nabla lo g π

pushes probability toward one action, so no alternatives are sampled and the advantage signal can't discover better actions. Entropy regularization counters this.

EC5.

γ = 0

(myopic agent): what does the advantage reduce to?

δ = r_{t + 1} - V_{w} (s_{t})

; the critic learns immediate expected reward and the actor is trained purely on "did this beat my expected immediate reward" — no lookahead at all.

EC6. Advantages all equal (constant

δ

across actions in a state): does the actor move?

The softmax gradient's competition term subtracts the mean, so a constant advantage produces exactly zero net update in that state — only differences between actions drive change.

EC7. Critic overfits and predicts

V_{w} (s)

far too high everywhere: effect on actor?

Advantages become systematically negative, so nearly every action gets pushed down — the policy can collapse or stall until the critic's bias corrects.

Connections

Actor-critic methods — the parent this bank interrogates.
REINFORCE, Policy Gradient Theorem — the baseline and variance claims.
Temporal Difference Learning, Advantage function — the $δ_{t}$ machinery.
Generalized Advantage Estimation (GAE), Proximal Policy Optimization (PPO), A3C and A2C — where these ideas scale up.