Intuition The one core ideaREINFORCE tries out actions, watches what reward follows, and then adjusts the knobs of a decision-maker so that actions followed by lots of reward become more likely and actions followed by little reward become less likely. That is the whole story — everything below is just the vocabulary you need to say that story precisely and turn it into math.
This page assumes nothing . Before you touch the REINFORCE algorithm derivation on the parent note, you must be able to read every symbol in it on sight. We build them one at a time, each picture leaning on the last. If a term appears in the parent, it appears here first, defined and drawn.
Reinforcement learning has an agent living inside an environment . Look at the loop below.
s state s Picture: one particular square on a game board, or the current screen of a video game.
Why the topic needs it: the policy reads s
a action a Picture: the arrow the agent picks — move Left, move Right, jump.
Why the topic needs it: REINFORCE's whole job is to change how likely each action is .
r r t reward r t r t time-stamp — the reward received at step number t 0 , 1 , 2 , … Picture: a treat dropping out of a machine after you press a button.
Why the topic needs it: reward is the only teaching signal. There is no "correct answer" label — only "how much reward followed".
Before we can say "make this action more likely ", we need the word likely as a number.
A probability is a number between 0 1 0 1 0.5 1 Picture: slicing a pie into wedges — each wedge is one action, its slice-size is its probability, the whole pie is 1
Intuition Why not just pick the single best action?A firm "always do X" is a point ; a probability spread is a dial you can nudge smoothly . REINFORCE improves by nudging — so it needs the smooth, probabilistic version. That smoothness is exactly why the topic prefers a probability distribution over a hard choice.
Now we assemble state + action + probability into the central object.
π policy π s pi (the letter "p", for policy / probability ).
π ( a ∣ s ) ∣ "given" . So π ( a ∣ s ) a given that we are in state s Picture: feed a state-photo into a box; out comes the pie-chart of action probabilities from §1.
θ π θ θ tunable numbers — the knobs inside the decision box (for a neural network, these are its weights). Writing π θ θ Picture: a mixing desk full of sliders. Slide them one way, the pie-chart tilts toward Left; slide them another, it tilts toward Right.
Why the topic needs it: REINFORCE never edits probabilities by hand — it edits θ θ
Two standard ways to build π θ Softmax Policy (for a handful of discrete actions — turns raw scores into a pie-chart) and the Gaussian Policy (for continuous actions — outputs a bell-curve of where to aim). Both are just concrete recipes for the box.
τ trajectory τ τ = ( s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , … , s T ) Picture: the full replay of a single game, frame by frame.
T final time step — the length of the episode.
τ ∼ π θ ∼ "is sampled from" / "is generated by". So τ ∼ π θ π θ Why the topic needs it: the trajectory is random — same policy, different luck, different τ
Rewards arrive over time. We need one number summarizing a whole play-through, and a way to say "sooner is worth a bit more than later."
Definition Discount factorγ γ [ 0 , 1 ] k γ k Picture: a treat seen far down the hallway looks smaller. γ = 1 γ = 0.5
R ( τ ) return is the total discounted reward of a whole trajectory:
R ( τ ) = ∑ t = 0 T γ t r t ∑ "add up" : run t 0 T γ t r t Picture: the height of the final stacked bar in the figure above.
G t G t only from step t :
G t = ∑ k = t T γ k − t r k k = t 0 k − t t γ 0 = 1 Picture: stand at step t forward only , and total up what still lies ahead.
Why the topic needs it: an action can only affect the future . Rewards banked before it are noise for judging it. Using G t R ( τ )
R ( τ ) G t Why it feels right: both are "the reward of the episode."
Fix: R ( τ ) G t depends on t — it shrinks as t G 0 γ = 1 R ( τ ) G t
J ( θ ) J ( θ ) average return we'd get if we played many episodes with policy π θ It is a single number measuring how good the current knob-setting is.
Picture: your long-run average score in a game. Turn the θ J Why the topic needs it: this is the mountain REINFORCE climbs. "Learning" = "make J ( θ )
The return is random (different τ
E [ ⋅ ] E Expected value ) means the probability-weighted average of the thing inside the brackets: run it infinitely many times and take the mean.
The subscript names the source of randomness. E τ ∼ π θ [ R ( τ )] R ( τ ) π θ Picture: roll a die a thousand times, average the faces — you land near 3.5 3.5
So the parent's objective reads, in plain words: "J ( θ ) π θ
J ( θ ) = E τ ∼ π θ [ R ( τ ) ]
Intuition Why estimate the average by sampling?We can never actually average over all trajectories — there are too many. Instead we play a handful of episodes and average those. This "average a few samples to guess the true mean" is exactly the Monte Carlo idea, and it is why REINFORCE is called a Monte Carlo policy-gradient method.
To increase J
∇ θ J ∇ θ for each knob in θ J Bundle all those sensitivities into one arrow.
Picture: stand on a hillside (the hill = J θ straight uphill , steepest direction.
Definition Gradient ascentThe update θ ← θ + α ∇ θ J α learning rate — the step size.
Picture: repeatedly step uphill until you reach the top (max J Descent would step downhill ; that's why the parent flips a minus sign in the code — optimizers descend, so we hand them − J
Definition Log and the log-derivative tricklog product into a sum : log ( x ⋅ y ) = log x + log y Log-Derivative Trick then uses ∇ θ log p = p ∇ θ p can sample. You will meet the full derivation on the parent; here just know log
Definition Value functionV ( s ) Value Function V(s) V ^ ( s ) expected return starting from state s — "how good is being here, on average?"
Picture: a number written on each board square saying "typical score from here on."
b ( s ) A t baseline b ( s ) G t "was this better or worse than usual?" A natural choice is b ( s ) = V ^ ( s ) advantage A t = G t − V ^ ( s t ) Picture: grading on a curve — an action scoring 105 100 110 Why the topic needs it: subtracting the baseline recentres the signal without changing the true direction — the parent's variance-reduction step. Feeding a learned baseline is the first step toward Actor-Critic Methods , and the pattern grows out of Policy Gradient Methods .
Policy pi_theta of a given s
Parameters theta the knobs
Trajectory tau one playthrough
Return and reward-to-go G_t
Objective J theta average return
Expectation E average over luck
Gradient nabla ascent uphill
Log turns product into sum
Cover the right side; can you answer before revealing?
What does π θ ( a ∣ s ) A probability for each action, given the state s θ
What is the vertical bar ∣ "given" — a conditional probability.
What is a trajectory τ The full recorded sequence s 0 , a 0 , r 0 , s 1 , … , s T
What does γ Discounts future rewards by γ k [ 0 , 1 ]
Difference between R ( τ ) G t R ( τ ) G t t
What is J ( θ ) The average (expected) return when policy π θ
What does E [ ⋅ ] Probability-weighted average; trajectories are random, so we average over them.
What does ∇ θ J The steepest-uphill direction in J θ
Why gradient ascent not descent? We want to maximize J
What single algebra fact makes log log ( x y ) = log x + log y
What is the advantage A t G t − V ^ ( s t )
Why subtract a baseline at all? To recentre the signal and cut variance, without biasing the gradient direction.
