Actor-critic methods

5.2.7 · AI-ML › Deep & Advanced RL

WHAT hai ek actor-critic method?

WHY chahiye yeh? (Pehle REINFORCE ko steel-man karte hain)

Policy gradient theorem kehta hai: ${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\pi }\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\Psi }_t\right]$

jahan ${\Psi }_t$ koi bhi "credit" signal ho sakta hai. REINFORCE ${\Psi }_t=G_t$ (full Monte-Carlo return) use karta hai. Yeh unbiased hai lekin high variance — $G_t$ poore random trajectory par depend karta hai.

Advantage estimator ko first principles se derive karna

Step 1 — Hum koi bhi baseline $b(s)$ subtract kar sakte hain

Claim: kisi bhi function $b(s)$ ke liye jo action par depend nahi karta, ${\mathbb{E}}_{a\sim \pi }[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]=0.$

Yeh step kyun? Kyunki ek probability distribution ka gradient integrate hokar zero hota hai: ${\mathbb{E}}_a[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)]={\sum }_a{\pi }_{\theta }(a\mid s)\frac{{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)}{{\pi }_{\theta }(a\mid s)}={\nabla }_{\theta }{\sum }_a{\pi }_{\theta }(a\mid s)={\nabla }_{\theta }1=0.$

Toh $b(s)$ subtract karna gradient ko unbiased rakhta hai lekin variance shrink kar sakta hai. Variance-minimizing baseline $V(s)$ ke close hota hai.

Step 2 — Baseline $b(s)=V(s)$ choose karo

${\Psi }_t=Q(s_t,a_t)-V(s_t)$ ke saath hume advantage function milta hai: $\boxed{A(s,a)=Q(s,a)-V(s)}$

Step 3 — Critic ke saath $Q$ ko Bootstrap karo (TD trick)

Hum full returns ka wait nahi karna chahte. $Q(s_t,a_t)=r_{t+1}+\gamma V(s_{t+1})$ use karke: $A(s_t,a_t)\approx \underset{\text{TD error }{\delta }_t}{\underbrace{r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t)}}$

HOW chalti hai yeh (ek online step)

1. State $s_t$ mein, actor $a_t\sim {\pi }_{\theta }$ sample karta hai.
2. Environment $r_{t+1},s_{t+1}$ return karta hai.
3. Critic ${\delta }_t=r_{t+1}+\gamma V_w(s_{t+1})-V_w(s_t)$ compute karta hai.
4. Critic ko ${\delta }_t^2$ reduce karne ki direction mein update karo.
5. Actor ko ${\delta }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)$ direction mein update karo.
6. $s_t\leftarrow s_{t+1}$, repeat karo.

Yeh fully online aur incremental hai — REINFORCE ki tarah episode end ka wait nahi karna.

Worked Examples

Common Mistakes (Steel-manned)

Active Recall

Mnemonic

Connections

• Policy Gradient Theorem — actor-critic iska ek variance-reduced instance hai.
• REINFORCE — Monte-Carlo ancestor (koi critic nahi → high variance).
• Temporal Difference Learning — jahan se bootstrapped ${\delta }_t$ aata hai.
• Advantage function — $A=Q-V$, actor ke signal ka dil.
• A3C and A2C — parallel/batched implementations.
• Generalized Advantage Estimation (GAE) — TD aur Monte-Carlo advantage ke beech interpolate karta hai.
• Proximal Policy Optimization (PPO) — ek modern clipped actor-critic.

Concept Map