Actor-critic methods

WHAT is an actor-critic method?

WHY do we need it? (Steel-manning REINFORCE first)

The policy gradient theorem says: $\nabla_\theta J(\theta) = \mathbb{E}_{\pi}\!\left[\, \nabla_\theta \log \pi_\theta(a_t\mid s_t)\; \Psi_t \,\right]$

where $\Psi_t$ is any "credit" signal. REINFORCE uses $\Psi_t = G_t$ (the full Monte-Carlo return). This is unbiased but high variance — $G_t$ depends on a whole random trajectory.

Deriving the advantage estimator from first principles

Step 1 — We may subtract any baseline $b(s)$

Claim: for any function $b(s)$ that does not depend on the action, $\mathbb{E}_{a\sim\pi}\big[\nabla_\theta \log\pi_\theta(a\mid s)\, b(s)\big] = 0.$

Why this step? Because the gradient of a probability distribution integrates to zero: $\mathbb{E}_a[\nabla_\theta \log\pi_\theta(a\mid s)] = \sum_a \pi_\theta(a\mid s)\frac{\nabla_\theta \pi_\theta(a\mid s)}{\pi_\theta(a\mid s)} = \nabla_\theta \sum_a \pi_\theta(a\mid s) = \nabla_\theta 1 = 0.$

So subtracting $b(s)$ leaves the gradient unbiased but can shrink variance. The variance-minimizing baseline is close to $V(s)$ .

Step 2 — Choose the baseline $b(s)=V(s)$

With $\Psi_t = Q(s_t,a_t) - V(s_t)$ we get the advantage function: $\boxed{A(s,a) = Q(s,a) - V(s)}$

Step 3 — Bootstrap $Q$ with the critic (the TD trick)

We don't want to wait for full returns. Using $Q(s_t,a_t)=r_{t+1}+\gamma V(s_{t+1})$ : $A(s_t,a_t) \approx \underbrace{r_{t+1} + \gamma V_w(s_{t+1}) - V_w(s_t)}_{\text{TD error } \delta_t}$

Formula The actor-critic update rules

Critic (minimize squared TD error, target treated as fixed): $\delta_t = r_{t+1} + \gamma V_w(s_{t+1}) - V_w(s_t)$ $w \leftarrow w + \alpha_w\,\delta_t\,\nabla_w V_w(s_t)$ Actor (policy-gradient ascent, using $\delta_t$ as an advantage estimate): $\theta \leftarrow \theta + \alpha_\theta\,\delta_t\,\nabla_\theta \log\pi_\theta(a_t\mid s_t)$

Why $\delta_t$ appears in BOTH? The same TD error tells the critic "you mis-predicted by $\delta_t$ " and tells the actor "this action beat expectation by $\delta_t$ ." One quantity, two jobs.

HOW it runs (one online step)

In state $s_t$ , actor samples $a_t\sim\pi_\theta$ .
Environment returns $r_{t+1}, s_{t+1}$ .
Critic computes $\delta_t = r_{t+1}+\gamma V_w(s_{t+1}) - V_w(s_t)$ .
Update critic toward reducing $\delta_t^2$ .
Update actor in direction $\delta_t \nabla_\theta\log\pi_\theta(a_t|s_t)$ .
$s_t \leftarrow s_{t+1}$ , repeat.

This is fully online and incremental — no waiting for episode end, unlike REINFORCE.

Worked Examples

Common Mistakes (Steel-manned)

Active Recall

Recall What role does the critic play, and how does it reduce variance?

The critic estimates $V(s)$ and serves as a baseline, so the actor is trained on the advantage (surprise relative to expectation) rather than the raw high-variance return $G_t$ . Subtracting a state-only baseline keeps the gradient unbiased (its expected contribution is 0) while shrinking variance.

Recall Why does the SAME TD error

$\delta_t$ update both networks? $\delta_t = r+\gamma V_w(s') - V_w(s)$ is simultaneously (a) the critic's prediction error to be minimized and (b) a one-sample estimate of the advantage $A(s,a)$ that scales the actor's policy-gradient step.

Recall Feynman: explain actor-critic to a 12-year-old

Picture a kid learning basketball. The actor is the kid's arm choosing how to shoot. The critic is a coach who, before the ball lands, says "that's better than your usual shot" or "worse than usual." The kid doesn't wait for the final score of the whole game (that's too random) — they trust the coach's quick judgment on each shot and adjust immediately. Over time the coach's judgments also get sharper, and the kid gets better together with the coach.

Mnemonic

Connections

Policy Gradient Theorem — actor-critic is a variance-reduced instance of it.
REINFORCE — the Monte-Carlo ancestor (no critic → high variance).
Temporal Difference Learning — where the bootstrapped $\delta_t$ comes from.
Advantage function — $A=Q-V$ , the heart of the actor's signal.
A3C and A2C — parallel/batched implementations.
Generalized Advantage Estimation (GAE) — interpolates between TD and Monte-Carlo advantage.
Proximal Policy Optimization (PPO) — a modern clipped actor-critic.

What are the two components of an actor-critic method?

The actor (policy

\pi_\theta

) that chooses actions, and the critic (value function

V_w

Q_w

) that evaluates them.

Why does subtracting a baseline

b(s)

keep the policy gradient unbiased?

Because

\mathbb{E}_a[\nabla_\theta\log\pi_\theta(a|s)\,b(s)] = b(s)\nabla_\theta\sum_a\pi_\theta(a|s)=b(s)\nabla_\theta 1 = 0

Define the advantage function.

A(s,a)=Q(s,a)-V(s)

: how much better action

a

is than the state's average action.

Write the TD-error advantage estimate.

\delta_t = r_{t+1} + \gamma V_w(s_{t+1}) - V_w(s_t)

What is the critic's update rule?

w \leftarrow w + \alpha_w\,\delta_t\,\nabla_w V_w(s_t)

(semi-gradient, target fixed).

What is the actor's update rule?

\theta \leftarrow \theta + \alpha_\theta\,\delta_t\,\nabla_\theta\log\pi_\theta(a_t|s_t)

Why is REINFORCE high variance and actor-critic lower?

REINFORCE uses full random return

G_t

; actor-critic replaces it with a bootstrapped, baseline-subtracted advantage (only the surprise), cutting variance.

Should you backprop through the TD target

r+\gamma V_w(s')

No — it's treated as a fixed (stop-gradient) target; this is semi-gradient TD.

Why is a two-timescale learning rate (critic faster) used?

So the advantage estimate the actor relies on is accurate; a lagging critic gives the actor misleading gradients.

What does the sign of the advantage tell the actor to do?

A>0

: increase the action's probability;

A<0

: decrease it.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RL me do camp hote hain: policy-based (REINFORCE) jo directly action seekhte hain, aur value-based (Q-learning) jo state ki value estimate karte hain. REINFORCE ka problem yeh hai ki wo pura random return $G_t$ use karta hai as learning signal — yeh bahut noisy/high variance hota hai, jaise poore semester ki luck pe grade dena. Actor-critic dono ko jodta hai: actor decide karta hai kya karna hai ( $\pi_\theta$ ), aur critic batata hai wo kitna accha tha ( $V_w$ ).

Magic yahaan hai ki hum raw reward ki jagah advantage use karte hain, $A(s,a)=Q(s,a)-V(s)$ — matlab "expectation se kitna better/worse rahe". Baseline $V(s)$ subtract karne se gradient unbiased rehta hai (proof: policy ke gradient ka sum zero hota hai) par variance kaafi kam ho jaata hai. Practically hum bootstrap karte hain: $\delta_t = r_{t+1} + \gamma V_w(s_{t+1}) - V_w(s_t)$ . Yeh $\delta_t$ hi TD error hai.

Sabse pyaari baat: ek hi $\delta_t$ dono ko train karta hai. Critic ke liye yeh uska prediction error hai (isko minimize karo), aur actor ke liye yeh advantage estimate hai (agar $\delta>0$ to us action ki probability badhao, agar $\delta<0$ to ghatao). Example 2 dekho — reward positive tha par future bura tha, to $\delta$ negative aa gaya, action discourage hua. Yeh hi difference hai raw reward aur advantage me.

Do cheezein yaad rakhna: (1) TD target $r+\gamma V_w(s')$ ko fixed maano, uspe backprop mat karo (semi-gradient). (2) Critic ko thoda fast learn karao (bada learning rate) taaki actor ko accha signal mile. Yehi foundation hai A2C, A3C, PPO jaise modern algorithms ka.

WHAT is an actor-critic method?

WHY do we need it? (Steel-manning REINFORCE first)

Deriving the advantage estimator from first principles

Step 1 — We may subtract any baseline $b(s)$

Step 2 — Choose the baseline $b(s)=V(s)$

Step 3 — Bootstrap $Q$ with the critic (the TD trick)

HOW it runs (one online step)

Worked Examples

Common Mistakes (Steel-manned)

Active Recall

Mnemonic

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Deep & Advanced RL

Connections