Policy gradient methods: Worked examples — AI-ML deep dive

This page is the practice arena for Policy gradient methods. The parent note built the theory; here we grind through every kind of situation the policy gradient can throw at you: positive and negative returns, the do-nothing (zero-gradient) case, degenerate deterministic policies, continuous actions, a full multi-step trajectory, a baseline rescue, and an exam twist.

Before a single number: recall the one object we keep computing, the score $\nabla_{θ} lo g π_{θ} (a ∣ s)$ — literally "how would the log-probability of the action I took change if I wiggled $θ$ ?" Every example is that score times a weight (return, reward-to-go, or advantage). If any symbol here feels unearned, re-read the parent's derivation and the Log-derivative trick.

The scenario matrix

Every policy-gradient exercise lands in one of these cells. Each worked example below is tagged with the cell(s) it covers.

#	Cell class	What's special	Covered by
C1	Positive return, discrete	good action, push up	Ex 1
C2	Negative return, discrete	bad action, push down	Ex 2
C3	Zero / degenerate weight	return $= 0$ or baseline cancels → no update	Ex 3
C4	Deterministic-limit policy	$π \to 0$ or $\to 1$ : score blows up / vanishes	Ex 4
C5	Multi-action softmax	full gradient over $> 2$ actions, reward-to-go	Ex 5
C6	Continuous action (Gaussian)	$ar g max$ impossible; score of a normal density	Ex 6
C7	Baseline rescue	large-offset returns, variance killed by $b = V$	Ex 7
C8	Full trajectory, causality	discounting + reward-to-go across time	Ex 8
C9	Exam twist	"gradient is zero because reward has no $θ$ " trap, sign of $γ$ -weighting	Ex 9

Building blocks we reuse

Two policies appear everywhere below. Let's earn their scores once.

The figure shows why the score is $(1 - σ)$ for $a_{1}$ : when $σ$ is already near 1, there's little room to grow, so the nudge is small; when $σ$ is near 0, the score is near 1 — a huge push. Look at the orange curve's slope.

The worked examples

Worked example Ex 4 — Deterministic-limit policy

(C4) Sigmoid policy pushed to the edge: $θ = 6$ , so $σ (6) \approx 0.9975$ (nearly deterministic toward $a_{1}$ ). Compare the score for sampling $a_{1}$ vs. sampling the rare $a_{2}$ .

Forecast: which sampled action produces the bigger score magnitude?

Score if $a_{1}$ sampled: $1 - σ (6) = 1 - 0.9975 = 0.0025$ — almost zero. Why? $a_{1}$ is already near-certain; there's almost no room to make it more likely.
Score if $a_{2}$ sampled: $- σ (6) = - 0.9975$ — nearly $- 1$ , huge. Why? A surprising action carries a strong gradient; if it earned reward it dramatically reshapes the policy.
Limit reading. As $θ \to \infty$ : the confident action's score $\to 0$ (learning stalls), the rare action's score $\to - 1$ (saturated but bounded). Why this matters: over-confident policies stop exploring — the vanishing score is the mathematical face of "premature convergence."

Verify: $∣ - 0.9975 ∣ > ∣0.0025∣$ ; ratio $\approx 399$ . ✓ Rare actions dominate the gradient near a deterministic limit — exactly why entropy bonuses exist.

Worked example Ex 5 — Multi-action softmax

(C5) Three actions with logits $θ = (z_{1}, z_{2}, z_{3}) = (1, 0, - 1)$ . We sample $a_{1}$ and get $G = + 1$ . Compute the gradient w.r.t. all three logits.

Forecast: the taken action $a_{1}$ 's logit rises — but what happens to $a_{2}, a_{3}$ ?

Softmax probabilities. $e^{1}, e^{0}, e^{- 1} = 2.718, 1, 0.368$ ; sum $= 4.086$ . $π = (0.665, 0.245, 0.090)$ . Why? We need current probabilities to evaluate the score $1 [i = k] - π (a_{i})$ .
Scores (chosen $k = 1$ ). $\partial_{z_{1}} lo g π (a_{1}) = 1 - 0.665 = 0.335$ $\partial_{z_{2}} lo g π (a_{1}) = 0 - 0.245 = - 0.245$ $\partial_{z_{3}} lo g π (a_{1}) = 0 - 0.090 = - 0.090$ Why negatives? Boosting $a_{1}$ 's probability necessarily steals mass from the others.
Weight by $G = + 1$ → gradient $= (0.335, - 0.245, - 0.090)$ . Why? Positive reward, so we move along the score.

Verify: the three scores sum to $0.335 - 0.245 - 0.090 = 0$ . ✓ (Softmax scores always sum to zero — probabilities can only be redistributed, not created.)

Worked example Ex 6 — Continuous action, Gaussian policy

(C6) The action is a real number (e.g. steering angle). Policy: $a \sim N (μ_{θ}, σ^{2})$ with fixed $σ = 1$ and mean $μ_{θ} = θ$ . We sample $a = 3$ while $θ = 1$ , and the return is $R = + 2$ . Learning rate $α = 0.1$ .

Forecast: Q-learning would need an $ar g max$ over infinitely many angles — impossible. The policy gradient just needs the score of a bell curve. Which way does $μ$ move?

Log-density of a normal. $lo g π_{θ} (a) = - \frac{( a - μ _{θ} ) ^{2}}{2 σ ^{2}} + const$ . Why this tool? The density is smooth in $θ$ , so its log has a clean derivative — no maximization needed.
Score w.r.t. the mean. $\nabla_{θ} lo g π_{θ} (a) = \frac{a - μ _{θ}}{σ ^{2}} = \frac{3 - 1}{1} = 2$ . Why? The score points toward the sampled action: if the action was good, shift the mean toward it.
Weighted update. $θ \leftarrow 1 + 0.1 \times 2 \times 2 = 1.4$ . Why $\times R$ ? We only move toward $a = 3$ because it paid off ( $R > 0$ ).

Verify: new mean $1.4$ is closer to the rewarded action $3$ than the old $1$ . ✓ Continuous actions handled with zero $ar g max$ — see Value-based methods (Q-learning) for the contrast this dodges.

The figure shows the Gaussian centred at $μ = 1$ ; the sampled $a = 3$ sits on the right tail; the arrow shows the mean sliding right because $R > 0$ . If $R$ had been negative, the arrow would flip left — the score's sign is fixed, the reward decides the direction.

Worked example Ex 7 — Baseline rescue

(C7) Three actions all return large positive numbers: $G = (100, 102, 101)$ . Softmax at $θ = (0, 0, 0)$ so $π = (\frac{1}{3}, \frac{1}{3}, \frac{1}{3})$ . Compare the effective weights on each action's gradient with and without baseline $b = \overset{ˉ}{G}$ .

Forecast: without a baseline, which action gets pushed down?

No baseline. Weights are the raw returns $(100, 102, 101)$ — all strongly positive. Why bad? Every action's probability is pushed up hard; the informative differences ( $\pm 1$ ) are swamped by the shared offset of $\sim 101$ . High variance.
Baseline $b = \overset{ˉ}{G} = \frac{100 + 102 + 101}{3} = 101$ . Why $\overset{ˉ}{G}$ ? It's the sample estimate of $V (s)$ , the best variance-minimizing baseline.
Advantages. $A = (100 - 101, 102 - 101, 101 - 101) = (- 1, + 1, 0)$ . Why unbiased? Parent proof: $E [score \cdot b] = 0$ , so subtracting $b$ leaves the gradient's expectation unchanged.

Verify: the advantages sum to $- 1 + 1 + 0 = 0$ and their spread ( $\pm 1$ ) is tiny versus the raw $\sim 101$ magnitude. ✓ Now action 2 is clearly promoted, action 1 demoted, action 3 untouched — see Variance reduction in Monte Carlo and Advantage function.

Worked example Ex 8 — Full trajectory with causality & discounting

(C8) A 3-step episode, rewards $r_{0} = 1, r_{1} = 0, r_{2} = 2$ , discount $γ = 0.9$ . Compute the reward-to-go weight $G_{t}$ that multiplies each step's score.

Forecast: should the last action be weighted by the full episode reward or only by what came after it?

Reward-to-go definition. $G_{t} = \sum_{t^{'} \geq t} γ^{t^{'} - t} r_{t^{'}}$ . Why not full $R$ ? Causality: an action at $t$ can't change rewards before $t$ ; including them is pure noise.
Compute each. $G_{2} = r_{2} = 2$ . $G_{1} = r_{1} + γ r_{2} = 0 + 0.9 \times 2 = 1.8$ . $G_{0} = r_{0} + γ r_{1} + γ^{2} r_{2} = 1 + 0 + 0.81 \times 2 = 2.62$ . Why discount? Future reward is worth less; $γ^{t^{'} - t}$ shrinks distant terms.
Assemble the estimator. $\nabla_{θ} J \approx \sum_{t} \nabla_{θ} lo g π_{θ} (a_{t} ∣ s_{t}) G_{t}$ with weights $(2.62, 1.8, 2)$ . Why per-term weights? Each action is credited only with the (discounted) future it could influence.

Verify: $G_{0} = 1 + 0.9 \cdot G_{1} = 1 + 0.9 \times 1.8 = 2.62$ (recursion consistent) and $G_{1} = 0 + 0.9 \times 2 = 1.8$ . ✓ Compare with Generalized Advantage Estimation for the bias–variance dial on these targets.

Worked example Ex 9 — Exam twist

(C9) "A student writes: since $R (τ)$ contains no $θ$ , $\nabla_{θ} J = \nabla_{θ} E [R] = E [\nabla_{θ} R] = E [0] = 0$ . Find the error and give the correct one-line reason. Then: with sigmoid policy $θ = 0$ , action $a_{2}$ sampled, return $R = + 4$ , compute the update ( $α = 0.25$ )."

Forecast: where exactly does the " $= 0$ " argument break?

Locate the fallacy. The step $E [\nabla_{θ} R]$ is illegal: the expectation itself is taken over $τ \sim π_{θ}$ , which does depend on $θ$ . You cannot pull $\nabla_{θ}$ inside without also differentiating the sampling distribution. Why? $J = \int P_{θ} (τ) R (τ) d τ$ — the gradient hits $P_{θ}$ , not $R$ (parent Step 1).
Correct one-liner. $\nabla_{θ} J = E_{τ} [\nabla_{θ} lo g P_{θ} (τ) R (τ)]$ via the Log-derivative trick; $θ$ acts through which trajectories appear, not their reward values.
Now the number. Score for $a_{2}$ : $\nabla_{θ} lo g (1 - σ (0)) = - σ (0) = - 0.5$ . Weight $= + 4$ . Update: $θ \leftarrow 0 + 0.25 \times (- 0.5) \times 4 = - 0.5$ . Why negative even though $R > 0$ ? We rewarded $a_{2}$ , whose probability is $1 - σ$ ; increasing $a_{2}$ means decreasing $θ$ .

Verify: at $θ = - 0.5$ , $1 - σ (- 0.5) = σ (0.5) \approx 0.622 > 0.5$ . The rewarded action $a_{2}$ became more likely. ✓ Sign handled correctly.

Recall Rapid self-check

Which sub-quantity carries the sign of "good vs bad"? ::: The weight (return / advantage), not the score. Near a deterministic policy, whose score dominates the gradient? ::: The rare, surprising action's score ( $\approx \pm 1$ ); the confident action's score $\to 0$ . Do softmax scores over all actions sum to? ::: Zero — probability mass is only redistributed. Why does baseline subtraction leave the gradient unbiased? ::: $E_{a \sim π} [score \cdot b (s)] = b (s) \nabla_{θ} 1 = 0$ . For continuous actions, what replaces the impossible $ar g max$ ? ::: The score of the action's density, e.g. $(a - μ) / σ^{2}$ for a Gaussian.

Connections

Policy gradient methods (index 5.2.5) — the parent derivation these examples exercise.
REINFORCE algorithm — the Monte-Carlo estimator used in Ex 1–2, 5, 8–9.
Score function estimator and Log-derivative trick — the machinery behind every score.
Advantage function · Variance reduction in Monte Carlo — Ex 7.
Actor-Critic methods · Generalized Advantage Estimation — where the baseline becomes learned.
Value-based methods (Q-learning) — the $ar g max$ Ex 6 sidesteps.