REINFORCE algorithm: Worked examples — AI-ML deep dive

Before we start, one reminder of the single rule everything below obeys:

Formula The one update we keep applying

$θ \leftarrow θ + α \nabla_{θ} lo g π_{θ} (a ∣ s) (G_{t} - b (s))$

$θ$ = the numbers inside the policy (weights). Picture: dials we turn.
$π_{θ} (a ∣ s)$ = probability the policy assigns to action $a$ in state $s$ .
$lo g π_{θ}$ = the log of that probability. We differentiate the log, not the probability, thanks to the Log-Derivative Trick.
$\nabla_{θ}$ = "how does this quantity change if I nudge each dial?" — a vector of slopes, one per dial.
$r_{t}$ = the per-step reward: the single number the environment hands back right after the action at time $t$ (e.g. $+ 1$ for grabbing the cup, $0$ otherwise).
$τ$ = a trajectory / episode: the whole rollout $(s_{0}, a_{0}, r_{0}, s_{1}, a_{1}, r_{1}, \dots, s_{T})$ from start to finish.
$R (τ)$ = the full return of that episode: the (discounted) sum of all its per-step rewards, $R (τ) = \sum_{t} γ^{t} r_{t}$ .
$γ$ = the discount factor, a number in $[0, 1]$ . It multiplies each reward by $γ$ for every step of delay, so a reward $k$ steps in the future is worth $γ^{k}$ of its face value. Picture: a "patience knob" — $γ = 1$ cares about the far future equally, $γ \to 0$ cares only about the immediate reward.
$G_{t}$ = reward-to-go: the discounted reward collected from time $t$ onward, $G_{t} = \sum_{k = t}^{T} γ^{k - t} r_{k}$ .
$b (s)$ = baseline: an "average day" score we subtract so we measure relative goodness. Often $b (s) = \hat{V} (s)$ , the Value Function V(s).
$α$ = learning rate (step size). Picture: how far we turn the dials each time.

The whole game: the sign of $(G_{t} - b)$ decides the direction, and $\nabla_{θ} lo g π_{θ}$ decides which dials move.

We first need the two policy families that appear, so we build them once — before the matrix names them.

Definition The two policies we differentiate

Softmax Policy (discrete actions). Give each action $a$ a score (logit) $θ_{a}$ . Then $π_{θ} (a) = \frac{e ^{θ_{a}}}{\sum _{a^{'}} e ^{θ_{a^{'}}}}$ Its log-gradient has a clean form — for the score of action $a$ : $\frac{\partial}{\partial θ _{a^{'}}} lo g π_{θ} (a) = 1 if it’s the taken action 1 [a^{'} = a] - π_{θ} (a^{'})$ Read it as: "raise the taken action's score by (1 − its own prob), and lower every other score by its prob." Everything sums to zero, so probabilities stay a valid distribution.

Gaussian Policy (continuous actions). Output a mean $μ_{θ} (s)$ and pick $a \sim N (μ, σ^{2})$ . Then $\frac{\partial}{\partial μ} lo g π_{θ} (a) = \frac{a - μ}{σ ^{2}}, \frac{\partial}{\partial σ} lo g π_{θ} (a) = \frac{( a - μ ) ^{2} - σ ^{2}}{σ ^{3}}$ Read the $μ$ term: if the action taken was above the mean and it worked, move the mean toward it. Read the $σ$ term: if the rewarded action landed far from the mean ( $(a - μ)^{2} > σ^{2}$ ), widen the spread; if it landed close in ( $(a - μ)^{2} < σ^{2}$ ), narrow it.

The scenario matrix

Every REINFORCE question is one (or a blend) of these cells. Each example below is tagged with the cell it hits.

#	Cell	What's special	Example
A	Positive return, discrete action	push chosen action up	Ex 1
B	Negative return, discrete action	push chosen action down	Ex 2
C	Zero / degenerate signal	gradient vanishes — no learning	Ex 3
D	Continuous action (Gaussian)	mean and spread both move	Ex 4
E	Baseline centering	same returns, opposite pushes	Ex 5
F	Reward-to-go vs full return	causality kills past rewards	Ex 6
G	Discounting $γ < 1$	far-future reward shrinks	Ex 7
H	Real-world word problem	full multi-step episode end to end	Ex 8
I	Exam twist: baseline that hurts	wrong constant raises variance	Ex 9

Example 1 — Cell A: positive return, discrete action

Compute the log-gradient for each logit. Using the softmax formula with taken action $R$ : $\partial_{θ_{R}} lo g π (R) = 1 - π (R) = 1 - 0.5 = 0.5, \partial_{θ_{L}} lo g π (R) = 0 - π (L) = - 0.5$ Why this step? The update only touches $lo g π$ of the action we actually took. The softmax gradient tells us it raises $R$ 's score and lowers $L$ 's in equal-and-opposite amounts (the magenta and violet arrows in the figure above).
Multiply by $G$ and $α$ , then add to each logit. $θ_{R} \leftarrow 0 + 0.5 \cdot (0.5) \cdot (+ 1) = 0.25, θ_{L} \leftarrow 0 + 0.5 \cdot (- 0.5) \cdot (+ 1) = - 0.25$ Why this step? The gradient ascent rule; the positive $G$ keeps the sign so $R$ goes up.
Recompute $π (R)$ . $π (R) = \frac{e ^{0.25}}{e ^{0.25} + e ^{- 0.25}} \approx 0.622$ Why this step? We want to confirm the probability actually rose, not just the logit.

Example 2 — Cell B: negative return, discrete action

Log-gradients are identical to Ex 1 (they depend on the policy, not the reward): $\partial_{θ_{R}} = + 0.5, \partial_{θ_{L}} = - 0.5$ . Why this step? The direction of "push $R$ up" is baked into $\nabla lo g π$ ; the reward only supplies a sign multiplier.
The negative $G$ flips the sign. $θ_{R} \leftarrow 0 + 0.5 (0.5) (- 1) = - 0.25, θ_{L} \leftarrow 0 + 0.5 (- 0.5) (- 1) = + 0.25$ Why this step? $G < 0$ means "do that less," so the chosen action's score drops.
Recompute: $π (R) = \frac{e ^{- 0.25}}{e ^{- 0.25} + e ^{0.25}} \approx 0.378$ .

Example 3 — Cell C: zero / degenerate signal

Case (i), $G = 0$ . Update $= α \nabla lo g π \cdot 0 = 0$ . No dial moves. Why this step? The reward is the magnitude of the push; a zero reward means "no evidence," so REINFORCE stays put.
Case (ii), $π (R) = 1$ . Softmax gradient for the taken action: $1 - π (R) = 1 - 1 = 0$ . $update = α \cdot 0 \cdot G = 0$ Why this step? Once an action has probability 1, the log-gradient vanishes — the policy is saturated and cannot learn, even from real reward. This is the "premature determinism" trap: exploration died.
Contrast the two zeros. In (i) the signal is zero; in (ii) the gradient is zero. Same result (no update) but different cause. Why this step? Diagnosing which zero you hit tells you the fix: (i) needs a nonzero reward or a baseline that centers; (ii) needs entropy/exploration to keep $π$ off the boundary.
Why a baseline can rescue case (i) — the formal reason. A single reward of $0$ looks like "no evidence" only relative to a baseline of $0$ . But the true signal is $G - b (s)$ , not $G$ alone. If the average return in this state is $\hat{V} (s) = + 2$ (rewards are usually positive here) and this time we got the unusually bad $G = 0$ , then $G - b (s) = 0 - 2 = - 2 \neq = 0$ — a nonzero, negative signal that correctly says "this action did worse than usual, push it down." So the baseline does not create signal from nothing; it reveals that " $G = 0$ " is only uninformative against a zero reference, and informative against the true average. This is why centering is the fix for case (i) but not case (ii) (where the gradient itself is zero and no baseline can revive it). Why this step? To show precisely how subtracting $b (s)$ can turn a $G = 0$ into a nonzero update — it is the contrast $G - b$ , not $G$ , that drives learning.

Example 4 — Cell D: continuous action (Gaussian policy), mean AND spread

Compute the $μ$ -gradient of the log. $\partial_{μ} lo g π (a) = \frac{a - μ}{σ ^{2}} = \frac{2 - 0}{1} = 2$ Why this step? Gaussian log-prob is $- \frac{( a - μ ) ^{2}}{2 σ ^{2}} + const$ ; its $μ$ -slope is $(a - μ) / σ^{2}$ . It points from the mean toward the sampled action (the magenta arrow on the bell in the figure above).
Apply the $μ$ -update. $μ \leftarrow 0 + 0.1 \cdot 2 \cdot 3 = 0.6$ Why this step? Positive $G$ says "that action was good," so the mean slides toward the good action $a = 2$ .
Compute the $σ$ -gradient of the log. $\partial_{σ} lo g π (a) = \frac{( a - μ ) ^{2} - σ ^{2}}{σ ^{3}} = \frac{( 2 - 0 ) ^{2} - 1 ^{2}}{1 ^{3}} = \frac{4 - 1}{1} = 3$ Why this step? Cell D promises both parameters move. This term compares how far the action landed from the mean $((a - μ)^{2} = 4)$ against the current spread $(σ^{2} = 1)$ .
Apply the $σ$ -update. $σ \leftarrow 1 + 0.1 \cdot 3 \cdot 3 = 1.9$ Why this step? The rewarded action was far from the mean, so REINFORCE widens the spread — it says "good things happen out here too, explore more broadly." Had the rewarded action been close in ( $(a - μ)^{2} < σ^{2}$ ), the $σ$ -gradient would be negative and the spread would shrink, sharpening the policy.

Example 5 — Cell E: baseline centering

No baseline. Weights are $G_{A} = + 105$ and $G_{B} = + 95$ — both positive, so both actions get pushed up. Why this step? Raw returns don't say which action is relatively better; both look great.
With baseline $b = 100$ . $A_{A} = 105 - 100 = + 5, A_{B} = 95 - 100 = - 5$ Now $A$ is pushed up, $B$ pushed down. Why this step? Subtracting the average exposes the contrast. This is Variance Reduction in RL and, crucially, it does not bias the gradient (the baseline term has zero expectation).
Check the advantages are balanced. Add the two advantages directly: $A_{A} + A_{B} = (+ 5) + (- 5) = 0$ — the baseline redistributed the pushes symmetrically around zero without inventing net bias. Why this step? Summing the advantages (not the raw returns) is the clean sanity check that centering is balanced.

Example 6 — Cell F: reward-to-go vs full return

Full return $R (τ) = \sum_{t} γ^{t} r_{t} = 5 + 0 + 10 = 15$ . Naively every action, including $a_{1}$ , is weighted by 15. Why this step? This is the "clean derivation" form — and it's noisy.
Reward-to-go for $a_{1}$ . $G_{1} = \sum_{k \geq 1} γ^{k - 1} r_{k} = 0 + 10 = 10$ . Why this step? Action $a_{1}$ can only influence rewards from time 1 onward. The $r_{0} = 5$ was already collected — by causality it carries zero expected credit for $a_{1}$ , so dropping it lowers variance with no bias.
Reward-to-go for $a_{0}$ . $G_{0} = 5 + 0 + 10 = 15$ (here it equals full return since $a_{0}$ is first). Why this step? Confirms reward-to-go and full return agree at $t = 0$ and diverge only for later actions.

Example 7 — Cell G: discounting $γ < 1$

Apply $G_{t} = \sum_{k \geq t} γ^{k - t} r_{k}$ . $G_{2} = 10, G_{1} = γ \cdot 10 = 9, G_{0} = γ^{2} \cdot 10 = 8.1$ Why this step? Discounting multiplies each future reward by $γ$ per step of delay, so the earlier you stand, the more the far reward is shrunk.
Compare to $γ = 1$ . There all three equalled 10; discounting spreads them $8.1 < 9 < 10$ . Why this step? Shows $γ$ is a knob on patience: small $γ$ makes distant rewards nearly invisible to early actions.
Effect on updates. $a_{0}$ 's push is scaled by $8.1$ , $a_{2}$ 's by $10$ — the action nearest the reward gets the strongest credit. Why this step? Ties the number back to learning: discounting focuses credit near the payoff.

Example 8 — Cell H: real-world word problem (full episode, end to end)

Worked example Teaching a robot to grab a cup — one whole update

Three-step episode $τ$ . Discrete Softmax Policy over ${reach, grip, lift}$ ; at each step we track the single taken action's probability. Data:

$t$	action taken	$π_{θ} (a_{t})$	$r_{t}$
0	reach	0.5	0
1	grip	0.4	0
2	lift	0.3	+1

$γ = 0.9$ , baseline $b = 0.5$ at every step, $α = 0.1$ . Compute the reward-to-go $G_{t}$ , the advantage $A_{t} = G_{t} - b$ , the log-gradient factor for each taken action, and the resulting logit update.

Forecast: with $b = 0.5$ and small returns, will any action be pushed down?

Reward-to-go with $γ = 0.9$ . Only $r_{2} = + 1$ is nonzero: $G_{2} = 1, G_{1} = 0.9 \cdot 1 = 0.9, G_{0} = 0. 9^{2} \cdot 1 = 0.81$ Why this step? Same discounting as Ex 7; only the future matters to each action.
Subtract the baseline $b = 0.5$ to get the advantage: $A_{0} = 0.81 - 0.5 = 0.31, A_{1} = 0.9 - 0.5 = 0.40, A_{2} = 1 - 0.5 = 0.50$ Why this step? Centers each step's credit; all three beat the "average day" of 0.5.
Write the log-gradient factor for each taken action. For its own logit the softmax gradient is $1 - π_{θ} (a_{t})$ : $1 - π_{0} = 1 - 0.5 = 0.5, 1 - π_{1} = 1 - 0.4 = 0.6, 1 - π_{2} = 1 - 0.3 = 0.7$ Why this step? The update is not just the sign of $A_{t}$ — it is $α (1 - π_{θ} (a_{t})) A_{t}$ for each taken action's own logit (and $- α π_{θ} (a^{'}) A_{t}$ for every other logit). We show that full factor here.
Assemble each taken-logit update $Δ θ_{a_{t}} = α (1 - π_{t}) A_{t}$ : $Δ θ_{0} = 0.1 (0.5) (0.31) = 0.0155, Δ θ_{1} = 0.1 (0.6) (0.40) = 0.024, Δ θ_{2} = 0.1 (0.7) (0.50) = 0.035$ Why this step? All three $Δ θ > 0$ : because the episode succeeded, every action rises. Had $b$ been $0.9$ , then $A_{0} = 0.81 - 0.9 < 0$ and reach's logit would drop — that is exactly the "forecast" trap.

Example 9 — Cell I: exam twist — a baseline that hurts

Bias check. Any baseline independent of the action leaves the gradient unbiased: $E [\nabla lo g π \cdot b] = b \nabla \sum_{a} π = b \cdot 0 = 0$ . So $b = - 100$ is still unbiased. Why this step? Reconfirms the theorem from the parent note — bias depends on independence from the action, not on the value.
Effective weights. With $b = - 100$ : every advantage becomes $1 - (- 100) = 101$ . The estimator now multiplies each $\nabla lo g π$ by 101 instead of by the centered value $0$ . Why this step? The good baseline $b = 1$ (the mean) gives advantages $0, 0, 0, 0$ — minimal-variance. The bad baseline inflates them to 101, blowing up variance.
Conclusion. Correct direction (unbiased) but terrible variance — learning becomes jittery and slow. Why this step? Drives home the real purpose of a baseline: it's a variance tool, and the near-optimal choice is $b \approx \hat{V} (s) =$ mean return, per Value Function V(s).

Wrap-up recall

Recall Which cell is each question?

Positive $G$ , discrete, action goes up ::: Cell A (Ex 1) $π (a) = 1$ already, update is zero ::: Cell C (Ex 3, saturation) Gaussian mean slides toward rewarded action, spread widens ::: Cell D (Ex 4) Two positive returns, only one should rise ::: Cell E, baseline (Ex 5) Past reward $r_{0}$ dropped from $G_{1}$ ::: Cell F, causality (Ex 6) Far reward shrinks by $γ$ per step ::: Cell G (Ex 7) Wrong-value baseline stays unbiased but jitters ::: Cell I (Ex 9)

See also: Policy Gradient Methods for the family this belongs to, and Hinglish version.