Question bank — REINFORCE algorithm

False. It learns the policy directly by nudging $\theta$; any value function here is only an optional baseline, not the thing used to pick actions.

False. $P$ does not depend on $\theta$, so ${\nabla }_{\theta }\log P=0$ and it drops out — REINFORCE is model-free.

False. As long as $b$ doesn't depend on the action, $\mathbb{E}[{\nabla }_{\theta }\log \pi \cdot b(s)]=b(s){\nabla }_{\theta }{\sum }_a\pi =b(s)\cdot 0$, so it changes only the variance.

False. By causality — action $a_t$ cannot influence rewards that were already earned before step $t$, so those past rewards are statistically independent of ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$ and contribute zero in expectation; both estimators point the same way, but $G_t$ has lower variance.

False. Optimizers minimize, but we want to maximize the objective $J(\theta )$ (average total return); minimizing $-J$ is exactly gradient ascent on $J$. The minus is essential.

True. The expectation ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}$ is over trajectories $\tau$ drawn from the current ${\pi }_{\theta }$, so samples from an old policy are not valid without importance corrections.

False. It only means "push its probability down this update"; it stays available and can recover if later experience favors it.

True. The value function $V(s)$ — the expected reward-to-go from state $s$ — is the natural, unbiased state-dependent baseline and (to a very close approximation) the one that minimizes the estimator's variance, which is exactly why it gives the advantage $A_t=G_t-V(s_t)$.

True. It estimates the gradient expectation by sampling whole trajectories $\tau$ and averaging — no bootstrapping from other estimates.

False. A baseline can't remove variance from the return itself $G_t$; it only removes the state-average component. Residual variance from stochastic trajectories remains.

Error: wrong sign. Minimizing $+\log \pi \cdot A$ decreases good-action probabilities. Correct loss is $-{\sum }_t\log \pi \cdot A_t$.

Error: no reward exists before $t=0$, and more generally each $a_t$ should be credited only with its reward-to-go $G_t$, not reward already banked before it (recall $R(\tau )=G_0$, only correct for the very first action).

Error: the expectation's distribution $p_{\theta }(\tau )$ depends on $\theta$. You cannot push ${\nabla }_{\theta }$ through ${\mathbb{E}}_{\tau \sim {\pi }_{\theta }}$; that dependence on $\theta$ is exactly why the log-derivative trick is needed.

Error: an action-dependent baseline breaks unbiasedness. The proof that the baseline term vanishes needs $b$ constant across actions in a state.

Error: its gradient is zero. Although $P$ sits inside $p_{\theta }(\tau )$, it has no $\theta$ in it, so ${\nabla }_{\theta }\log P=0$; only ${\nabla }_{\theta }\log {\pi }_{\theta }$ survives.

Partial error. Subtracting the mean is just a constant baseline and is provably unbiased (it leaves the true gradient direction unchanged). Dividing by the standard deviation, however, rescales the whole update like an adaptive learning rate — it is a heuristic that is not part of the unbiased gradient and can change the effective step size per batch.

Error: Actor-Critic bootstraps. It replaces the Monte Carlo $G_t$ with a learned critic estimate (e.g. $r_t+\gamma V(s_{t+1})$), trading variance for bias; REINFORCE uses full-episode returns.

Start from $J=\int p_{\theta }(\tau )R(\tau )d\tau$; differentiating gives $\int {\nabla }_{\theta }{p}_{\theta }(\tau )R(\tau )d\tau$, but ${\nabla }_{\theta }{p}_{\theta }(\tau )$ is not a probability, so we can't sample it. The identity ${\nabla }_{\theta }p=p{\nabla }_{\theta }\log p$ (just $\frac{d}{dx}\log f=\frac{f^{\prime }}{f}$ rearranged) rewrites it as $\int p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )R(\tau )d\tau ={\mathbb{E}}_{\tau }[{\nabla }_{\theta }\log p_{\theta }\cdot R]$ — now it's an average we can estimate by sampling trajectories.

A Gaussian Policy outputs a mean ${\mu }_{\theta }(s)$ and spread ${\sigma }_{\theta }(s)$ and samples $a\sim \mathcal{N}(\mu ,{\sigma }^2)$; the gradient ${\nabla }_{\theta }\log \pi$ has a clean closed form, so we never need $\arg {\max }_{a}Q(s,a)$ over an infinite set — an optimization that value methods cannot do exactly in continuous spaces.

In expectation $\mathbb{E}[\nabla \log \pi \cdot b(s)]=b(s)\cdot 0=0$, so the mean is untouched; but per-sample, if raw returns are all near $+500$, every $\nabla \log \pi$ term points up by a huge amount and the useful differences between actions are drowned out. Centering by $b(s)$ turns $+500$-ish numbers into small $\pm$ advantages, so good actions get a positive push and bad ones a negative push — a cleaner, lower-variance signal.

$G_t$ is a single-sample Monte Carlo estimate summing many random actions and transitions; its spread grows with episode length, so one trajectory's gradient can point far from the true ${\nabla }_{\theta }J$, forcing tiny learning rates and slow, unstable training.

Softmax sets $\pi (a)=\frac{e^{z_a}}{{\sum }_{a^{\prime }}{e}^{z_{a^{\prime }}}}$, and these probabilities must sum to 1; raising the numerator for $R$ raises the shared denominator too, so every other action's fraction — including $L$ — must shrink. You get "punish the untaken action" for free.

The object it differentiates is $J(\theta )$, the expected return, taken with respect to the policy parameters $\theta$ — see Policy Gradient Methods. Value functions enter only as optional baselines for variance reduction, never as the thing being maximized.

Every action-independent baseline contributes $b(s)\cdot 0$ in expectation, so all share the same mean gradient (unbiased); but each reshapes the per-sample advantages differently, giving different variances — and lower variance means smoother, faster convergence. This is the core idea of Variance Reduction in RL.

Every $\log \pi$ term is multiplied by its $G_t$; if all $G_t=0$ the gradient contribution is zero — that episode teaches nothing. This is why sparse-reward tasks train slowly.

The advantage $A_t=0$, so that timestep contributes nothing to this update — expected across samples it still gives the correct centered gradient.

Then $\log \pi \to 0$ and its gradient shrinks, so exploration collapses; the agent can get stuck because it never samples alternative actions to learn about them.

$G_t={\sum }_{k=t}^T{\gamma }^{k-t}{r}_k$ collapses to just $r_t$ (only the $k=t$ term survives, since ${\gamma }^0=1$ and every later ${\gamma }^{>0}=0$), so each action is credited solely with its immediate reward — a myopic agent ignoring all future consequences.

$G_t={\sum }_{k\ge t}{r}_k$ (no shrinking, since ${\gamma }^{k-t}=1$) can diverge to infinity, so returns are ill-defined; you need $\gamma <1$ or an episodic (terminating) task for finite $G_t$.

No; with one sample the value estimate is essentially the single noisy return itself, so the advantage barely reduces variance there. Baselines help most for frequently-visited states.

Expected advantages are all $\approx 0$, so the policy is (correctly) not pushed to prefer any action — REINFORCE leaves indifferent states alone.

No — high variance already means noisy gradients; a large $\alpha$ amplifies that noise and destabilizes training. High variance calls for smaller steps or better variance reduction.