Exercises — Policy gradient methods

WHAT survives: only ${\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$. WHY: $\log$ turns the product into a sum: $\log P_{\theta }(\tau )=\log \rho (s_0)+{\sum }_t\log P(s_{t+1}\mid s_t,a_t)+{\sum }_t\log {\pi }_{\theta }(a_t\mid s_t)$. The gradient ${\nabla }_{\theta }$ asks "how does this change when I wiggle the knobs $\theta$?" The initial distribution and the dynamics belong to the environment — they contain no $\theta$ — so their slope is $0$. Only the policy carries $\theta$. Consequence: we never need a model of the world → policy gradients are model-free.

Ascent: $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J$. $J$ is return we want to maximize, and the gradient points uphill. (Equivalently: minimize the loss $-J$.)

Step 1 — the log-prob gradient. $\log {\pi }_{\theta }(a_1)=\log \sigma (\theta )$. A standard identity: $\frac{d}{d\theta }\log \sigma (\theta )=1-\sigma (\theta )$. Why? ${\sigma }^{\prime }=\sigma (1-\sigma )$, and $\frac{d}{d\theta }\log \sigma =\frac{{\sigma }^{\prime }}{\sigma }=1-\sigma$. Step 2 — plug in $\theta =0$. $\sigma (0)=\frac{1}{1+e^0}=0.5$, so ${\nabla }_{\theta }\log {\pi }_{\theta }(a_1)=1-0.5=0.5$. Step 3 — assemble. $\Delta \theta =\alpha \cdot 0.5\cdot R=0.5\cdot 0.5\cdot 2=\boxed{0.5}$. What it looks like: $\theta$ rises → $\sigma (\theta )$ rises above $0.5$ → action $a_1$ becomes more probable. A rewarded action was reinforced. ✓

WHAT we do: sum rewards from $t=1$ onward, each discounted by how far in the future it is relative to $t=1$. $G_1={\gamma }^0{r}_1+{\gamma }^1{r}_2+{\gamma }^2{r}_3=1(0)+0.9(2)+0.81(-1)=0+1.8-0.81=\boxed{0.99}.$ WHY start the exponent at $0$: the reward at $t=1$ is "now" from $t=1$'s viewpoint, so it gets ${\gamma }^0=1$. $r_0$ is excluded because an action at $t=1$ cannot affect the past (causality).

Step 1 — write the expectation as a sum over actions (discrete case): ${\mathbb{E}}_a[{\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s)]={\sum }_a{\pi }_{\theta }(a\mid s){\nabla }_{\theta }\log {\pi }_{\theta }(a\mid s)b(s).$ Step 2 — undo the log-derivative trick. Since $\pi \nabla \log \pi =\pi \frac{\nabla \pi }{\pi }=\nabla \pi$: $=b(s){\sum }_a{\nabla }_{\theta }{\pi }_{\theta }(a\mid s)=b(s){\nabla }_{\theta }\underset{=1}{\underbrace{\sum _a{\pi }_{\theta }(a\mid s)}}.$ Step 3 — probabilities sum to 1, and the gradient of a constant is $0$: $=b(s){\nabla }_{\theta }1=b(s)\cdot 0=0.■$ WHY $b(s)$ came out front: it does not depend on $a$, so it is a constant with respect to the sum over $a$. That is the whole reason a state-only baseline is allowed but an action-dependent one is not.

The update weight on each action is $(G-b)$.

• $b=0$: weights $\{100,102,101\}$, mean $101$, spread around a huge common offset.
• $b=101$: weights $\{-1,+1,0\}$ — the offset is removed; only the differences that matter remain. WHY variance drops: the Monte-Carlo estimator's variance scales with the magnitude of these weights (roughly $\mathbb{E}[(G-b{)}^2\Vert \nabla \log \pi {\Vert }^2]$). With $b=0$ every weight is $\sim 100$, so sampling noise on $\nabla \log \pi$ gets amplified $100\times$. Subtracting $\bar{G}$ centers the weights near zero without changing the expected gradient (Q3.1), so we keep the signal and kill the amplified noise.

  

  The figure shows the same three weights before (tall bars far from zero) and after (short bars straddling zero) subtracting the baseline — same shape, dramatically smaller magnitude.

Step 1 — baseline $V(s)$: the natural estimate is the average return, $V(s)=\frac{8+2+5}{3}=5$. Step 2 — advantages: $A(s,a_1)=8-5=+3,A(s,a_2)=2-5=-3,A(s,a_3)=5-5=0.$ Step 3 — interpretation via the update $\theta +=\alpha \nabla \log \pi \cdot A$:

• $a_1$: $A>0$ → pushed up (better than average).
• $a_2$: $A<0$ → pushed down (worse than average).
• $a_3$: $A=0$ → no change (exactly average). WHY this is the seed of Actor-Critic methods: replace the hand-averaged $V(s)$ with a learned critic $V_{\varphi }(s)$, and you turn REINFORCE-with-baseline into actor-critic. The advantage $A$ is the shared currency; see Advantage function and Generalized Advantage Estimation.

Step 1 — accuracy = variance of the average. For $N$ i.i.d. samples the mean's variance is (per-sample variance)$/N$.

• Baselined, $N=20$: variance $=\frac{s^2/9}{20}=\frac{s^2}{180}$.
• Un-baselined, $N^{\prime }$: variance $=\frac{s^2}{N^{\prime }}$. Step 2 — match them: $\frac{s^2}{N^{\prime }}=\frac{s^2}{180}\Rightarrow N^{\prime }=180$. Answer: $\boxed{N^{\prime }=180}$ un-baselined trajectories — $9\times$ more data for the same accuracy. This is the practical payoff of Variance reduction in Monte Carlo.

(a) Gradient of the losing branch. $\log {\pi }_{\theta }(a_2)=\log (1-\sigma (\theta ))$. Since $\frac{d}{d\theta }(1-\sigma )=-\sigma (1-\sigma )$, ${\nabla }_{\theta }\log {\pi }_{\theta }(a_2)=\frac{-\sigma (1-\sigma )}{1-\sigma }=-\sigma (\theta ).$ (Compare Q2.1's ${\nabla }_{\theta }\log {\pi }_{\theta }(a_1)=+(1-\sigma )$ — the two branches have opposite-signed gradients, as they must, since raising one lowers the other.)

(b) Sequential updates ($\alpha =1$, $b=0$).

• Episode 1 at ${\theta }_0=0$, $\sigma =0.5$: sampled $a_1$, weight $G=2$. $\Delta \theta =1\cdot (1-\sigma )\cdot 2=1\cdot 0.5\cdot 2=+1.0$. New ${\theta }_1=1.0$.
• Episode 2 at ${\theta }_1=1.0$, $\sigma (1)=\frac{1}{1+e^{-1}}\approx 0.7311$: sampled $a_2$, weight $G=2$. $\Delta \theta =1\cdot (-\sigma )\cdot 2=1\cdot (-0.7311)\cdot 2\approx -1.4621$. New ${\theta }_2\approx 1.0-1.4621=\boxed{-0.4621}$.

(c) What went wrong. Both actions gave the same return $+2$, so neither is genuinely better — yet we still shoved $\theta$ around by more than a full unit each step. With $b=0$, every positive return is treated as "reinforce me," so the updates are dominated by the common offset, not by real differences. The net wandered to $-0.46$ purely from ordering/noise, not from any signal.

The baseline fix. Set $b=2$. Now each weight is $G-b=0$, so both updates are exactly $0$: $\theta$ stays put at $0$. That is the correct behavior — when two actions are equally good, the policy should not move. The baseline stripped the meaningless common offset, leaving only the (here, zero) advantage. This is exactly Q3.1's unbiasedness paying off as Q3.2's variance reduction.