Policy gradient methods: Question bank — AI-ML deep dive

This bank assumes the parent note's build-up: the objective $J (θ) = E_{τ \sim π_{θ}} [R (τ)]$ , the Log-derivative trick, reward-to-go $G_{t}$ , and baselines. If a symbol here feels unfamiliar, re-read the parent first — nothing new is introduced, only stress-tested.

True or false — justify

The policy gradient is zero because

R (τ)

contains no

θ

False —

θ

shapes

P_{θ} (τ)

, the distribution we sample trajectories from. The gradient flows through

P_{θ}

via

\nabla_{θ} lo g P_{θ}

, so nudging

θ

changes which rewards are likely even though reward values are fixed by the environment.

Policy gradient methods need a model of the transition dynamics

P (s^{'} ∣ s, a)

False — dynamics appear in

P_{θ} (τ)

but carry no

θ

, so

\nabla_{θ} lo g P (s^{'} ∣ s, a) = 0

and they drop out. Only

\nabla_{θ} lo g π_{θ} (a ∣ s)

survives, which is exactly why PG is model-free.

Subtracting a baseline

b (s)

makes the gradient estimate biased.

False —

E_{a \sim π} [\nabla_{θ} lo g π_{θ} (a ∣ s) b (s)] = b (s) \nabla_{θ} \sum_{a} π_{θ} = b (s) \nabla_{θ} 1 = 0

, so subtracting it changes nothing in expectation. It only lowers variance.

For

J (θ)

we should run gradient descent, like every other ML loss.

False —

J

is expected return we want to maximize, so we do gradient ascent (

θ \leftarrow θ + α \nabla_{θ} J

), or equivalently descend on

- J

REINFORCE and Q-learning both ultimately learn a value function.

False — Q-learning learns

Q (s, a)

and acts greedily; REINFORCE learns the policy parameters directly and never requires a value function (though a learned baseline can help). See Value-based methods (Q-learning) for the contrast.

Reward-to-go changes the expected gradient compared to using full

R (τ)

False — dropping past rewards is unbiased because an action at time

t

has zero expected correlation with rewards before

t

(causality). The expectation is unchanged; only the variance drops.

A deterministic policy can be optimal, so stochastic policies are just a training convenience.

False in general — under partial observability or in games like rock-paper-scissors, the optimal policy is genuinely stochastic. A greedy value policy cannot represent that; a stochastic

π_{θ}

can.

The advantage baseline

b (s) = V (s)

is chosen because it makes the estimator unbiased.

False — any action-independent

b (s)

is already unbiased.

V (s)

is chosen because it best centres the returns around zero, minimizing variance; unbiasedness is not the reason.

Spot the error

\nabla_{θ} lo g P_{θ} (τ) = \nabla_{θ} lo g ρ (s_{0}) + \sum_{t} \nabla_{θ} lo g P (s_{t + 1} ∣ s_{t}, a_{t}) + \sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t})

, keep all three."

The first two groups are zero (

ρ

and

P

have no

θ

), so only the policy term remains:

\nabla_{θ} lo g P_{θ} (τ) = \sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t})

"The baseline should depend on the action taken,

b (s, a)

, so it can rate each action."

b

depends on

a

it no longer factors out of the expectation and the estimator becomes biased. The baseline must be action-independent,

b (s)

; the advantage

A (s, a) = G_{t} - b (s)

is where the action-rating happens.

"To reduce variance, just scale every weight by a small constant

c < 1

Scaling by a constant biases the gradient direction's magnitude and is not variance reduction — you'd need

α / c

to compensate. Real reduction comes from causality and a well-chosen baseline, which shrink weight magnitude without changing the expectation.

\nabla_{θ} lo g π_{θ} (a_{1})

for softmax

π = σ (θ)

σ (θ)

It is

1 - σ (θ)

\frac{d}{d θ} lo g σ (θ) = 1 - σ (θ)

. Using

σ

would push probability the wrong way for the sampled action.

"Since the log-derivative trick gives an expectation, one sampled trajectory gives the exact gradient."

One sample is an unbiased but noisy estimate. You average over

N

trajectories; the single-sample estimate has high variance, which is the whole motivation for Variance reduction in Monte Carlo tricks.

"Advantage

A (s, a) = V (s) - Q (s, a)

Sign flipped —

A (s, a) = Q (s, a) - V (s)

, "how much better than average this action is." With the wrong sign you'd reinforce bad actions.

Why questions

Why does the Log-derivative trick use

lo g

specifically and not some other transform?

Because

\nabla lo g f = \nabla f / f

produces exactly the factor

P_{θ}

needed to rewrite

\nabla_{θ} P_{θ}

P_{θ} \nabla_{θ} lo g P_{θ}

, turning a gradient-of-a-density into a plain expectation you can sample. No other elementary function gives that self-cancelling structure.

Why can policy gradients handle continuous action spaces where Q-learning struggles?

Q-learning needs

ar g max_{a} Q (s, a)

, which is intractable over an infinite action set. PG parameterizes and samples

π_{θ} (a ∣ s)

directly (e.g. a Gaussian), so no maximization over actions is ever required.

Why is smooth (gradient) policy improvement often more stable than greedy value updates?

Small

θ

changes give small changes in

π_{θ}

, so behaviour drifts gently. Greedy value updates can flip the whole policy from a tiny

Q

change, causing oscillation.

Why does subtracting the mean return already help even before a state-dependent

V (s)

When all returns are large and positive (e.g.

100, 102, 101

), every action is pushed up hard and the informative differences drown in noise. Centring by the mean turns them into

- 1, + 1, 0

, isolating the signal — same expectation, far less variance.

Why is

V (s)

a better baseline than a global constant?

A constant only removes a per-batch offset.

V (s)

removes the state-specific offset, so high-value states don't inflate every weight there; this is the seed of Actor-Critic methods and links to the Advantage function.

Why do we still call it "the score function" estimator?

\nabla_{θ} lo g π_{θ} (a ∣ s)

is the score function of the policy distribution; the whole estimator weights scores by returns, hence Score function estimator is a synonym for the REINFORCE gradient.

Edge cases

What happens to the update if the sampled action already has probability

\approx 1

The score

\nabla_{θ} lo g π_{θ}

shrinks toward zero (e.g.

1 - σ (θ) \to 0

), so the update naturally saturates — a near-certain action can't be reinforced much further, preventing runaway probabilities.

What if every trajectory in the batch has the same return

R

The gradient becomes

R \cdot E [\sum_{t} \nabla_{θ} lo g π_{θ}]

, whose expectation is

R \cdot \nabla_{θ} 1 = 0

— no learning signal, because nothing distinguishes good from bad. A baseline that equals

R

makes this explicit: all advantages are zero.

What if the return

R (τ)

is negative for the sampled action?

The weight is negative, so the update decreases

lo g π_{θ}

of those actions — bad choices are made less likely. Sign of the return, not just its size, drives direction.

What if

γ = 0

in the reward-to-go

G_{t} = \sum_{t^{'} \geq t} γ^{t^{'} - t} r_{t^{'}}

Only the immediate reward

r_{t}

survives; each action is credited solely by its instantaneous reward. This is myopic — fine when rewards are dense, poor when they're delayed.

At an optimum

θ^{⋆}

, is the policy gradient necessarily zero?

At a local optimum,

\nabla_{θ} J = 0

in expectation, but sampled estimates are still noisy and nonzero. Convergence means the expected update vanishes, not that any single batch update does — another reason large step sizes destabilize, motivating Trust Region and PPO.

Can two very different

θ

values give the same policy and hence the same

J

Yes — networks are over-parameterized, so

J (θ)

has flat directions where

θ

moves but

π_{θ}

(and the gradient) don't change. The gradient is zero along those directions, which is harmless.

What does the gradient do at a state the policy visits with near-zero probability?

That state rarely appears in sampled trajectories, so it receives almost no gradient signal — a coverage/exploration problem, not a formula problem. Poorly-visited states learn slowly, which is why exploration and Generalized Advantage Estimation-style credit assignment matter.

Connections

Parent: Policy gradient methods and its Hinglish version.
Mechanism: Log-derivative trick · Score function estimator.
Variance: Variance reduction in Monte Carlo · Advantage function · Generalized Advantage Estimation.
Algorithms: REINFORCE algorithm · Actor-Critic methods · Trust Region and PPO.
Contrast: Value-based methods (Q-learning).