Advantage Actor-Critic (A2C - A3C) — Hinglish notes

5.2.8 · AI-ML › Deep & Advanced RL

A2C exist kyun karta hai? (Woh problem jo yeh fix karta hai)

Pure policy gradient (REINFORCE) policy ko is tarah update karta hai: $\nabla_{θ} J = E [\nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) G_{t}]$ jahan $G_{t} = \sum_{k \geq 0} γ^{k} r_{t + k}$ full Monte-Carlo return hai.

Kya problem hai? $G_{t}$ ek unbiased lekin bahut zyada noisy estimate hai. Future mein koi ek lucky reward abhi aapne jo action li uska credit dubaa sakta hai.
Noisy kyun hai? Yeh bahut saare random future rewards ko sum karta hai. Variance badhta jaata hai.

Fix: ek baseline $b (s_{t})$ subtract karo jo sirf state par depend kare (action par nahi). Yeh expected gradient ko nahi badalta (proof neeche hai) lekin variance ko kaat deta hai. Sabse accha baseline state value $V (s_{t})$ hai, jisse advantage milta hai: $A (s_{t}, a_{t}) = Q (s_{t}, a_{t}) - V (s_{t})$

Toh humein do networks chahiye:

Actor $π_{θ} (a ∣ s)$ — actions choose karta hai.
Critic $V_{ϕ} (s)$ — state value estimate karta hai, advantage compute karne ke liye use hota hai.

Yeh coupling actor seekhta hai, critic judge karta hai — yahi poora "actor-critic" idea hai.

Figure — Advantage Actor-Critic (A2C - A3C)

First principles se derivation

Step 1 — Policy gradient theorem

Objective se shuru karo (start distribution se expected return): $J (θ) = E_{τ \sim π_{θ}} [\sum_{t} γ^{t} r_{t}] .$ Log-derivative trick: kisi bhi distribution ke liye, $\nabla_{θ} p_{θ} (x) = p_{θ} (x) \nabla_{θ} lo g p_{θ} (x)$ . Isse trajectory distribution par apply karo aur note karo ki dynamics $θ$ par depend nahi karte: $\nabla_{θ} J = E [\sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) G_{t}] .$ Yeh step kyun? Hum sampling ke through differentiate nahi kar sakte, toh hum "expectation ka gradient" ko "gradient ka expectation × score" mein convert karte hain.

Step 2 — Baselines bias nahi add karte

Claim: kisi bhi $b (s_{t})$ ke liye, $E_{a_{t} \sim π} [\nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) b (s_{t})] = 0.$ Proof: $b (s_{t})$ ko bahar nikalo ( $a_{t}$ mein constant hai):

= b(s_t)\sum_a \nabla_\theta \pi_\theta(a\mid s_t) = b(s_t)\,\nabla_\theta \underbrace{\sum_a \pi_\theta(a\mid s_t)}_{=1}=0.$$ *Yeh kyun matter karta hai:* hum $V(s_t)$ free mein subtract kar sakte hain — **same expected gradient, lower variance**. ### Step 3 — $G_t$ ki jagah advantage use karo Kyunki $\mathbb{E}[G_t\mid s_t,a_t]=Q(s_t,a_t)$ hai, hum paate hain: $$\boxed{\nabla_\theta J = \mathbb{E}\big[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,A(s_t,a_t)\big]},\quad A=Q-V.$$ ### Step 4 — $Q$ network ke bina $A$ estimate karo Humne sirf critic $V_\phi$ train kiya hai. Advantage ka **TD (temporal-difference) estimate** use karo: $$\hat A_t = \underbrace{r_t + \gamma V_\phi(s_{t+1})}_{\text{bootstrap target of }Q} - V_\phi(s_t) = \delta_t.$$ Yeh $\delta_t$ exactly **TD error** hai. Toh *critic ka TD error HI advantage estimate hai.* Kamal hai. > [!formula] Teen update ingredients > **TD error (advantage):** $\;\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$ > **Actor loss:** $\;\mathcal L_\pi = -\log\pi_\theta(a_t\mid s_t)\,\hat A_t \;-\; \beta\,H(\pi_\theta(\cdot\mid s_t))$ > **Critic loss:** $\;\mathcal L_V = \tfrac12\big(\underbrace{r_t+\gamma V_\phi(s_{t+1})}_{\text{stop-grad target}} - V_\phi(s_t)\big)^2$ > $H$ policy ki **entropy** hai ($-\sum\pi\log\pi$); $-\beta H$ bonus *exploration encourage karta hai* (policy ko bahut jaldi collapse hone se rokta hai). > [!definition] n-step advantage (bias–variance dial) > 1-step bootstrap ki jagah, $n$ real rewards use karo phir bootstrap karo: > $$\hat A_t^{(n)} = \Big(\sum_{k=0}^{n-1}\gamma^k r_{t+k}\Big) + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t).$$ > Chhota $n$ → low variance, zyada bias (critic par depend karta hai). Bada $n$ → high variance, kam bias (real returns par depend karta hai). GAE saare $n$ ko smoothly interpolate karta hai. --- ## A3C vs A2C — "parallel" waala part - **A3C (Asynchronous Advantage Actor-Critic):** bahut saare worker threads hote hain, har ek ke paas network ki copy hoti hai, woh apna environment roll out karte hain, gradients compute karte hain, aur unhe **asynchronously** ek shared global network par push karte hain. Workers ki diversity data ko ==decorrelates the data==, wahi role play karti hai jo DQN mein replay buffer ka hota hai — lekin on-policy. - **A2C (Advantage Actor-Critic):** *synchronous* version hai. Saare workers ka rollout khatam hone ka wait karo, unke gradients average karo, ek update karo. Simpler hai, zyada GPU-efficient hai, aksar A3C ke barabar ya behtar hota hai. "A2C = A3C minus asynchrony." > [!mistake] "A3C ko DQN ki tarah ek replay buffer chahiye." > **Kyun sahi lagta hai:** DQN ne humein sikhaya ki decorrelated samples zaroori hain. > **Fix:** A2C/A3C **on-policy** hain — har gradient current policy se aana chahiye. Ek stale replay buffer *old-policy* actions rakhta hai, jo $\nabla\log\pi$ ko galat bana deta. Iske bajay woh **parallel environments** ke through decorrelate karte hain, buffer ke through nahi. > [!mistake] Gradients ko critic target ke through flow karne dena. > **Kyun sahi lagta hai:** target $r+\gamma V_\phi(s')$ mein $V_\phi$ hai, toh isse differentiate karna "natural lagta" hai. > **Fix:** target ko ek **constant (stop-gradient)** treat karo. Warna optimizer $V_\phi(s')$ ko trivially loss minimize karne ke liye neeche move karega — tum goalpost hila rahe ho, value nahi seekh rahe. > [!mistake] Ek optimizer step share karna lekin entropy bhool jaana → premature collapse. > **Kyun sahi lagta hai:** entropy ek chhoti si "regularizer" lagti hai. > **Fix:** $\beta H$ term ke bina policy jaldi kisi bure action par near-deterministic ho sakti hai, exploring band kar deti hai, aur stuck ho jaati hai. Entropy darwaza khula rakhti hai. --- ## Worked examples > [!example] Example 1 — ek advantage compute karna > $\gamma=0.9$, $V_\phi(s_t)=5$, reward $r_t=2$, $V_\phi(s_{t+1})=6$. > $\delta_t = 2 + 0.9(6) - 5 = 2 + 5.4 - 5 = 2.4$. > **Yeh step kyun?** Positive $\delta$ matlab action ne *critic ki expectation se behtar* kiya → actor loss $-\log\pi\cdot 2.4$ is action ko **zyada likely** banane ke liye push karta hai. Critic loss $V_\phi(s_t)$ ko $7.4$ ki taraf upar push karta hai. > [!example] Example 2 — ek negative advantage > Same critic, lekin $r_t=-1$, $V_\phi(s_{t+1})=4$: $\delta_t=-1+0.9(4)-5=-1+3.6-5=-2.4$. > Actor loss gradient ban jaata hai $-\log\pi\cdot(-2.4)=+2.4\log\pi$ → is action ki probability **decrease** hoti hai. **Kyun?** Outcome $s_t$ se average se worse tha, toh hum $a_t$ se door steer karte hain. > [!example] Example 3 — 2-step advantage > $\gamma=0.9$, rewards $r_t=1, r_{t+1}=2$, $V_\phi(s_t)=5$, $V_\phi(s_{t+2})=8$. > $\hat A^{(2)}_t = 1 + 0.9(2) + 0.9^2(8) - 5 = 1 + 1.8 + 6.48 - 5 = 4.28.$ > **Yeh step kyun?** Do real rewards use karna environment par zyada trust karta hai aur (possibly rough) critic par kam — 1-step version se zyada variance, kam bias. --- > [!recall]- Feynman: ek 12-saal ke bachche ko samjhao > Socho tum basketball seekh rahe ho. Ek **coach (critic)** dekhta hai aur andaza lagata hai ki tumhari position kitni achi hai ("yahan se tum usually 5 points score karte ho"). Tum ek shot lete ho (**actor ka action**) aur actually 7 aata hai. Woh *surprise* — 7 minus expected 5 = **+2** — batata hai "woh move meri aadat se behtar tha, zyada karo!" Agar 3 aata, toh surprise hai −2, "woh kam karo." A3C bas alag courts par **bahut saare bacche ek saath practice** kar rahe hain aur jo seekhte hain share karte hain, taaki team jaldi improve kare aur sab ek hi galti copy na karein. > [!mnemonic] Pieces yaad karo > **"A CAT VE"** → > **A**dvantage = **CAT** *Critic's TD error* = **V**alue-next minus **V**alue-now (plus reward) → **E**ntropy-bonused actor ko drive karta hai. > Aur: **A2C = A3C Awaiting All (synchronous)**. --- ## Active recall #flashcards/ai-ml Advantage function $A(s,a)$ kya hai? ::: $A(s,a)=Q(s,a)-V(s)$ — action $a$ state ke average se kitna behtar hai. Policy gradient mein baseline kyun subtract karte hain? ::: Yeh variance reduce karta hai bina expected gradient ko change kiye (baseline term ka zero expectation hota hai). Prove karo ki baseline koi bias nahi add karta (key line). ::: $\sum_a \nabla_\theta \pi_\theta(a|s)=\nabla_\theta\sum_a\pi=\nabla_\theta 1=0$. Value critic se advantage estimate karne wala ek quantity kya hai? ::: TD error $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)$. A2C mein actor loss? ::: $-\log\pi_\theta(a_t|s_t)\hat A_t - \beta H(\pi)$. A2C mein critic loss? ::: $\tfrac12(r_t+\gamma V(s_{t+1})-V(s_t))^2$ target par stop-gradient ke saath. Entropy bonus ka role? ::: Exploration encourage karta hai; policy ko ek deterministic action par bahut jaldi collapse hone se rokta hai. A2C vs A3C ek line mein? ::: A2C synchronous version hai (saare workers ka wait karo, phir gradients average karo); A3C asynchronous hai. A2C/A3C replay buffer ki jagah data kaise decorrelate karte hain? ::: Bahut saare parallel environments/workers chalaa ke (on-policy). n-step advantage trade-off? ::: Bada n → kam bias, zyada variance (zyada real rewards); chhota n → kam variance, zyada critic bias. Critic target stop-gradient kyun hona chahiye? ::: Warna optimizer $V(s')$ ko trivially loss shrink karne ke liye move karta hai sach mein value seekhne ki bajay. --- ## Connections - [[REINFORCE]] — Monte-Carlo ancestor; A2C = REINFORCE + baseline + bootstrapping. - [[Policy Gradient Theorem]] — woh identity jise hum differentiate karte hain. - [[Temporal-Difference Learning]] — jahan se $\delta_t$ aata hai. - [[Generalized Advantage Estimation (GAE)]] — saare n-step advantages ko smooth karta hai. - [[DQN]] — contrast: off-policy + replay buffer vs on-policy + parallel workers. - [[PPO]] — descendant jo stable large steps ke liye ek clipped surrogate add karta hai. - [[Bias-Variance Tradeoff]] — n dial isse embody karta hai. ## 🖼️ Concept Map ```mermaid flowchart TD REINFORCE[REINFORCE policy gradient] -->|uses| Gt[MC return G_t] Gt -->|is| HighVar[High variance signal] HighVar -->|fixed by| Baseline[Subtract baseline b s] Baseline -->|best choice| V[State value V s] Baseline -->|proof| NoBias[Adds no bias] V -->|defines| Adv[Advantage A = Q - V] Adv -->|low variance signal| PG[Policy gradient update] Actor[Actor pi theta] -->|chooses actions| PG Critic[Critic V phi] -->|estimates value| V Adv -->|estimated via| TD[TD error delta_t] TD -->|bootstraps| Critic Actor -->|learns from| Critic ```