Actor-critic methods: Worked examples — AI-ML deep dive

Intuition Read this first

The whole actor-critic engine turns on ONE number: the TD error $δ_{t} = r_{t + 1} + γ V_{w} (s_{t + 1}) - V_{w} (s_{t}) .$ Here $r_{t + 1}$ is the reward we just collected, $γ$ (gamma, a number between $0$ and $1$ ) is how much we care about the future, $V_{w} (s)$ is the critic's guess of "how much total reward I expect starting from state $s$ ", and $s_{t} \to s_{t + 1}$ is the step we took. If $δ_{t}$ confuses you, revisit the parent note and Temporal Difference Learning.

Notation note (read once): the subscript $t$ on $δ_{t}$ just tags "the TD error at time step $t$ ". Since every example below looks at a single step, there is no other step to confuse it with, so we write plain $δ$ from now on. Whenever you see $δ$ , mentally read $δ_{t}$ for the step being discussed.

This page does one thing: we walk through every kind of situation $δ$ can be in, so you never meet a case in the wild you haven't already seen worked out here.

Definition The five symbols you need on THIS page (restated so it stands alone)

Actor $π_{θ} (a ∣ s)$ : the policy — a rule that outputs the probability of taking action $a$ in state $s$ . It has tunable numbers $θ$ (the network weights). " $π$ " is just the name of that probability function; " $θ$ " (theta) are the knobs we turn.
Critic $V_{w} (s)$ : a second function, with its own knobs $w$ , guessing the total future reward from state $s$ — averaged over whatever actions the policy would take there.
==Action-value $Q (s, a)$ ==: the total future reward if we commit to action $a$ now in state $s$ , then follow the policy afterward. Difference from $V$ : $V$ averages over the action we might take; $Q$ pins down one specific action first. So $Q (s, a)$ answers "how good is this exact move", while $V (s)$ answers "how good is this situation on average".
Advantage $A (s, a) = Q (s, a) - V (s)$ : "how much better than average was action $a$ in state $s$ " — literally (value of this specific move) minus (value of the average move). Its cheap one-sample estimate is exactly the TD error $δ$ , because $Q (s, a) \approx r + γ V (s^{'})$ (one observed reward plus the discounted future) and $V (s)$ is the baseline we subtract.
Logits $h_{θ} (s, a)$ : for a discrete action set, raw unnormalized scores (one per action). Softmax turns them into probabilities: $π_{θ} (a ∣ s) = \frac{e ^{h_{θ} (s, a)}}{\sum _{a^{'}} e ^{h_{θ} (s, a^{'})}}$ . Bigger logit ⇒ bigger probability.

The scenario matrix

Every actor-critic step reduces to computing $δ$ and asking "was this better or worse than the critic expected?". The table below lists every distinct cell — every sign, every degenerate input, every limit. Each of the worked examples below is tagged with the cell(s) it covers.

Cell	Situation	What is special	Sign of $δ$	Example
C1	Reward good and future looks up	ordinary "good" step	$δ > 0$	Ex 1
C2	Reward positive but future collapses	good reward, bad news	$δ < 0$	Ex 2
C3	Reward negative but future brightens	bad reward, good news	$δ > 0$	Ex 3
C4	Critic already perfect	expectation met exactly	$δ = 0$	Ex 4
C5	Terminal state ( $V (s^{'}) = 0$ )	no future term	either sign	Ex 5
C6	$γ = 0$ (myopic) vs $γ \to 1$ (far-sighted)	limiting values of $γ$	changes with $γ$	Ex 6
C7	Softmax actor direction	which logits move, how much	driven by $δ$	Ex 7
C8	Real-world word problem	translate to $δ$	either	Ex 8
C9	Exam twist: backprop-through-target trap	why stop-gradient	—	Ex 9
C10	Multi-step / $n$ -step return vs 1-step	bias–variance trade	either	Ex 10

Worked examples

Example 1 — the ordinary "good" step (Cell C1)

Worked example Statement

Critic says $V_{w} (s) = 5$ . We take action $a$ , receive reward $r = 1$ , land in $s^{'}$ where $V_{w} (s^{'}) = 6$ . Discount $γ = 0.9$ . Find $δ$ and say what the actor does.

Forecast: before reading on, guess the sign of $δ$ . We got positive reward and moved to a state the critic likes more — do you expect a pleasant surprise or a disappointment?

Step 1 — write the target. The critic's expectation for this step is $r + γ V_{w} (s^{'}) = 1 + 0.9 (6) = 6.4$ . Why this step? $r + γ V_{w} (s^{'})$ is a one-sample bootstrap estimate of "true value of being in $s$ and doing $a$ " (that is, an estimate of $Q (s, a)$ ) — it is what we actually observed, blended with the critic's opinion of the future.

Step 2 — subtract the old belief. $δ = 6.4 - V_{w} (s) = 6.4 - 5 = 1.4$ . Why this step? $V_{w} (s)$ was our belief before acting. The difference is the "news": how much reality exceeded belief. $δ = 1.4 > 0$ .

Step 3 — read off the actor update. Since $δ > 0$ , the actor moves $θ$ so $lo g π_{θ} (a ∣ s)$ increases → action $a$ becomes more likely. The critic also nudges $V_{w} (s)$ up (it under-predicted). Why this step? The policy-gradient step is $θ \leftarrow θ + α_{θ} δ \nabla_{θ} lo g π_{θ} (a ∣ s)$ (from the restated formula box). The positive scalar $δ$ points the update along "make $a$ more probable"; the learning rate $α_{θ}$ only sets how far we step, never the direction — direction is $δ$ 's job.

Verify: Sanity — we predicted $5$ , effectively realized $6.4$ , so we were too pessimistic by $1.4$ . Sign of "under-predicted" ⇒ raise both the action's probability and $V_{w} (s)$ . Consistent. ✓

Figure below (s01): a value "thermometer" for this exact example. The chalk-blue mark is the critic's belief $V_{w} (s) = 5$ ; the pale-yellow mark is the realized target $6.4$ ; the pink arrow between them is $δ = + 1.4$ . Read the figure as: " $δ$ is literally the vertical gap between what we believed and what we observed — and its arrow points UP, so the action's probability goes UP."

Example 2 — good reward, bad news (Cell C2)

Worked example Statement

Same $V_{w} (s) = 5$ , $γ = 0.9$ , reward $r = 1$ — but this time we land in $s^{'}$ with $V_{w} (s^{'}) = 2$ .

Forecast: reward is positive. Naive intuition says "reward the action". Do you agree? Compute first.

Step 1 — target. $r + γ V_{w} (s^{'}) = 1 + 0.9 (2) = 2.8$ . Why this step? The future term $γ V_{w} (s^{'})$ collapsed from $5.4$ (Ex 1) to $1.8$ . The observed value of the step is dragged down by the bleak future.

Step 2 — subtract belief. $δ = 2.8 - 5 = - 2.2 < 0$ . Why this step? We expected $5$ but the transition only "cashes out" to $2.8$ . The reward being positive is irrelevant — what matters is expectation vs reality.

Step 3 — actor update. $δ < 0$ ⇒ decrease $lo g π_{θ} (a ∣ s)$ : action $a$ becomes less likely, even though $r > 0$ . Why this step? This is the entire point of using advantage instead of raw reward: we compare against the baseline $V_{w} (s)$ , so "positive reward that leads somewhere worse than average" is correctly punished.

Verify: Contrast with Ex 1 — same reward $r = 1$ , same start $V_{w} (s) = 5$ , opposite $δ$ sign purely because the future changed. This proves reward alone is not the signal. ✓

Example 3 — bad reward, good news (Cell C3)

Worked example Statement

$V_{w} (s) = 3$ , $γ = 0.9$ . We take a penalty: $r = - 2$ . But we land in $s^{'}$ with $V_{w} (s^{'}) = 8$ .

Forecast: negative reward — punish the action? Compute before deciding.

Step 1 — target. $r + γ V_{w} (s^{'}) = - 2 + 0.9 (8) = - 2 + 7.2 = 5.2$ . Why this step? A short-term cost of $- 2$ opened the door to a state worth $8$ . The bootstrap correctly bundles cost and future promise.

Step 2 — subtract belief. $δ = 5.2 - 3 = 2.2 > 0$ .

Step 3 — actor update. $δ > 0$ ⇒ increase the action's probability despite the negative reward. Why this step? This is the mirror image of Ex 2. The critic teaches the actor to accept short-term pain for long-term gain — impossible if you used raw reward as the signal.

Verify: Symmetry check — Ex 2 had $r > 0, δ < 0$ ; Ex 3 has $r < 0, δ > 0$ . The sign of $r$ and the sign of $δ$ are decoupled exactly as the Advantage function promises. ✓

Example 4 — the critic is already perfect (Cell C4)

Worked example Statement

$V_{w} (s) = 10$ , $γ = 0.5$ , $r = 4$ , and $s^{'}$ has $V_{w} (s^{'}) = 12$ .

Forecast: what happens to learning when the critic's forecast is exactly right?

Step 1 — target. $r + γ V_{w} (s^{'}) = 4 + 0.5 (12) = 4 + 6 = 10$ .

Step 2 — subtract belief. $δ = 10 - 10 = 0$ . Why this step? The realized bootstrap equals the prior belief — no surprise.

Step 3 — updates. Actor step $= α_{θ} \cdot 0 \cdot \nabla lo g π = 0$ . Critic step $= α_{w} \cdot 0 \cdot \nabla V = 0$ . Nothing moves (the learning rates $α_{θ}, α_{w}$ can't matter when they multiply zero). Why this step? $δ$ multiplies both update rules, so $δ = 0$ is a fixed point — the algorithm has learned this state and leaves it alone. This is the healthy end-state, not a bug.

Verify: A degenerate/zero input must produce a zero update or the algorithm would drift away from a correct critic. It does. ✓

Example 5 — terminal state, no future (Cell C5)

Worked example Statement

The episode ends after this step. By convention a terminal next-state has $V (s_{terminal}) = 0$ . Say $V_{w} (s) = 7$ , $γ = 0.99$ , final reward $r = 3$ .

Forecast: does the $γ V (s^{'})$ term matter here?

Step 1 — kill the future term. Because $s^{'}$ is terminal, $V_{w} (s^{'}) = 0$ , so the target is just $r + γ \cdot 0 = r = 3$ . Why this step? There is nothing after termination — the total future reward from a terminal state is $0$ by definition. Forgetting this and plugging a nonzero $V (s^{'})$ is a classic bug.

Step 2 — compute. $δ = 3 - 7 = - 4 < 0$ . Why this step? We hoped this state was worth $7$ but it only delivered $3$ before the curtain fell → disappointment.

Step 3 — updates. Actor lowers $π_{θ} (a ∣ s)$ ; critic lowers $V_{w} (s)$ toward $3$ .

Verify: With the (wrong) inclusion of, say, $V_{w} (s^{'}) = 5$ : $δ$ would be $3 + 0.99 (5) - 7 = 0.95 > 0$ — the opposite sign! Handling the terminal case correctly flips the learning direction. ✓

Figure below (s03): the terminal-state convention made visual. The left path (correct) multiplies the future value by a "done" mask of $0$ , so only the reward $r = 3$ survives and the pink arrow points down ( $δ = - 4$ ). The right path (buggy) leaks a nonzero $V (s^{'}) = 5$ through, flipping the arrow up ( $δ = + 0.95$ ). The picture makes the sign flip literal: forgetting the mask reverses what the actor learns.

Common mistake The stop-at-terminal one-liner (reinforcement of Ex 5)

In code the fix is a done mask $(1 - done)$ multiplying the future term:

# done = 1.0 if s_next is terminal else 0.0
target = r + gamma * (1.0 - done) * V(s_next)
delta  = target - V(s)

When done == 1, the whole gamma * V(s_next) term vanishes — exactly the convention $V (s_{terminal}) = 0$ . This single (1 - done) factor is the entire bug fix.

Example 6 — the two limits of $γ$ (Cell C6)

Worked example Statement

Fixed transition: $V_{w} (s) = 5$ , $r = 1$ , $V_{w} (s^{'}) = 100$ . Compute $δ$ for the myopic limit $γ = 0$ and the far-sighted limit $γ = 1$ .

Forecast: which $γ$ makes the huge future value $100$ dominate?

Step 1 — myopic, $γ = 0$ . $δ = 1 + 0 \cdot 100 - 5 = 1 - 5 = - 4$ . Why this step? $γ = 0$ means "I only care about the immediate reward". The future ( $100$ ) is completely ignored, so we see disappointment ( $1 < 5$ ).

Step 2 — far-sighted, $γ = 1$ . $δ = 1 + 1 \cdot 100 - 5 = 101 - 5 = 96$ . Why this step? $γ = 1$ fully counts the future, and the future is glorious ( $100$ ), so massive positive surprise.

Step 3 — interpret. The same transition is "bad" or "great" depending purely on the discount. $γ$ is a modelling choice about horizon, not a fact about the environment. Why this step? It shows $γ$ interpolates continuously; the sign of $δ$ crosses zero at the $γ$ where $1 + 100 γ = 5$ , i.e. $γ = 0.04$ .

Verify: Solve $1 + 100 γ - 5 = 0 \Rightarrow γ = 0.04$ . For $γ < 0.04$ , $δ < 0$ ; for $γ > 0.04$ , $δ > 0$ . Matches the two endpoints $(- 4)$ and $(96)$ . ✓

Figure below (s02): $δ$ plotted against $γ$ for this fixed transition. Follow the yellow curve from left (myopic, $δ = - 4$ ) to right (far-sighted, $δ = 96$ ); the pink dot marks the sign-flip at $γ = 0.04$ . The lesson the figure makes visible: your choice of horizon $γ$ can turn the same real experience from "punish this action" into "strongly reward it".

Example 7 — softmax actor, which logits move (Cell C7)

Worked example Statement

Two actions ${a_{1}, a_{2}}$ with logits (raw scores before softmax) $h_{1} = 2, h_{2} = 0$ . We picked $a_{1}$ and got $δ = + 1.5$ . Assume the logits are the parameters $θ$ and we take one step with learning rate $α_{θ} = 0.5$ . Show the resulting new probabilities.

Forecast: guess $π (a_{1})$ before the update. Softmax of $(2, 0)$ — is it closer to $0.9$ or $0.5$ ?

Step 1 — current policy. Using the softmax definition from the definition box, $π (a_{1}) = \frac{e ^{2}}{e ^{2} + e ^{0}} = \frac{e ^{2}}{e ^{2} + 1} \approx 0.8808$ , so $π (a_{2}) \approx 0.1192$ . Why this step? Softmax turns scores into probabilities; we need the "before" picture to see the direction of change.

Step 2 — the score-function gradient. Plug into the restated identity $\nabla_{θ} lo g π = \nabla_{θ} h (s, a) - \sum_{a^{'}} π (a^{'}) \nabla_{θ} h (s, a^{'})$ . Since $\nabla_{θ} h (s, a_{i})$ is the indicator of slot $i$ : $\nabla_{θ} lo g π (a_{1} ∣ s) = raise chosen (1, 0) - lower all, by prob (π (a_{1}), π (a_{2})) = (1 - 0.8808, - 0.1192) = (0.1192, - 0.1192) .$ Why this step? The identity says: raise the chosen logit, pull all logits down in proportion to their current probability — a competition.

Step 3 — apply $δ$ and the learning rate. New logits $= (h_{1}, h_{2}) + α_{θ} δ (0.1192, - 0.1192) = (2, 0) + 0.5 \times 1.5 \times (0.1192, - 0.1192) = (2 + 0.0894, 0 - 0.0894) = (2.0894, - 0.0894)$ . Why this step? The magnitude of the nudge is proportional to $δ$ : a bigger surprise = a bigger shove. A negative $δ$ would reverse both signs (demote the chosen action).

Step 4 — new probabilities. Softmax of the new logits: $π_{new} (a_{1}) = \frac{e ^{2.0894}}{e ^{2.0894} + e ^{- 0.0894}} \approx 0.8992$ , so $π_{new} (a_{2}) \approx 0.1008$ . Why this step? This is the payoff: because $δ > 0$ , the chosen action's probability rose from $0.8808$ to $\approx 0.8992$ — exactly the "raise the red logit, push the blue one down" motion the figure for Ex 1 drew as an upward arrow. The actor became more confident in the surprisingly-good action.

Verify: The gradient components sum to $0.1192 - 0.1192 = 0$ — as they must, since probabilities always sum to $1$ and the update can't create probability mass, only move it. And indeed $π_{new} (a_{1}) + π_{new} (a_{2}) = 0.8992 + 0.1008 = 1$ . ✓

Example 8 — real-world word problem (Cell C8)

Worked example Statement

A delivery drone in state $s$ (battery $60%$ , 2 km from depot). The critic values this state at $V_{w} (s) = 40$ (expected future reward units). The drone chooses "take the shortcut over the river". It earns $r = - 5$ (extra risk penalty) and reaches $s^{'}$ where the critic says $V_{w} (s^{'}) = 52$ . Use $γ = 0.95$ . Should the drone learn to prefer the shortcut?

Forecast: the immediate reward is negative. Reward-only thinking says "avoid the shortcut". Trust that?

Step 1 — translate to $δ$ . $δ = r + γ V_{w} (s^{'}) - V_{w} (s) = - 5 + 0.95 (52) - 40$ . Why this step? Word problems are just $δ$ in disguise — identify $r$ , $V_{w} (s^{'})$ , $V_{w} (s)$ , $γ$ and substitute.

Step 2 — arithmetic. $0.95 \times 52 = 49.4$ ; $δ = - 5 + 49.4 - 40 = 4.4 > 0$ .

Step 3 — decision. $δ > 0$ ⇒ the actor increases the probability of "take the shortcut". The small penalty was worth it because the resulting position is much stronger ( $52$ vs the starting $40$ ). Why this step? Exactly Cell C3 in a costume: short-term pain, long-term gain, correctly rewarded.

Verify: Units — all quantities are in "future-reward units", $γ$ dimensionless, so $δ$ is in reward units, comparable across steps. Value $4.4$ is a modest positive surprise, plausible for one shortcut. ✓

Example 9 — exam twist: the stop-gradient trap (Cell C9)

Worked example Statement

A student writes the critic loss as $L = (r + γ V_{w} (s^{'}) - V_{w} (s))^{2}$ and differentiates all occurrences of $w$ , including inside $V_{w} (s^{'})$ . Show the correct critic gradient, define what "stop-gradient" concretely means, and demonstrate numerically why the extra term is wrong.

Forecast: how many terms does the correct gradient of $δ^{2}$ w.r.t. $w$ have — one or two?

Step 1 — what "stop-gradient" concretely means. "Freezing the target" means we compute the number $y = r + γ V_{w} (s^{'})$ using the current weights, then treat it as a plain constant with no dependence on $w$ when we differentiate. In code this is literally one call: PyTorch y = (r + gamma * V(s_next)).detach(), TensorFlow y = tf.stop_gradient(r + gamma * V(s_next)). Autodiff then sees $y$ as a fixed label, exactly like a supervised-learning target. Why this step? The TD derivation defines the target from a bootstrap sample of data — it is something we regress toward, not a function to be optimized through.

Step 2 — the correct (semi-)gradient. With $y$ frozen, $L = (y - V_{w} (s))^{2}$ and $\nabla_{w} L = 2 (y - V_{w} (s)) \cdot (- \nabla_{w} V_{w} (s)) = - 2 δ \nabla_{w} V_{w} (s) .$ Taking the descent step $w \leftarrow w - \frac{1}{2} α_{w} \nabla_{w} L$ gives the update rule from the formula box: $w \leftarrow w + α_{w} δ \nabla_{w} V_{w} (s)$ . One term, one job.

Step 3 — the wrong "full gradient", expanded. If instead we (incorrectly) let $y$ depend on $w$ , the chain rule adds a second term $+ 2 δ γ \nabla_{w} V_{w} (s^{'})$ : $\nabla_{w} L_{wrong} = - 2 δ \nabla_{w} V_{w} (s) + 2 δ γ \nabla_{w} V_{w} (s^{'}) = - 2 δ (\nabla_{w} V_{w} (s) - γ \nabla_{w} V_{w} (s^{'})) .$ Why this step? This "residual gradient" optimizes a different objective; the critic now chases a target it is simultaneously editing, which empirically slows and destabilizes learning.

Step 4 — numeric toy. Take $δ = 1.4$ , $γ = 0.9$ , and suppose both slopes equal $1$ ( $\nabla_{w} V_{w} (s) = 1, \nabla_{w} V_{w} (s^{'}) = 1$ ). Correct update magnitude $= δ \cdot 1 = 1.4$ . Wrong update magnitude $= δ (1 - γ \cdot 1) = 1.4 \times 0.1 = 0.14$ . Why this step? A concrete $10 \times$ gap proves these are genuinely different algorithms, not a rounding nuance — the "full gradient" would move the critic ten times less (and in general in a different direction).

Verify: Correct $= 1.4$ , wrong $= 0.14$ , ratio $= (1 - γ) = 0.1$ . Matches the factor $(1 - γ)$ predicted by the expanded formula in Step 3. ✓

Example 10 — 1-step vs n-step return (Cell C10)

Worked example Statement

A 3-step rollout gives rewards $r_{1} = 1, r_{2} = 1, r_{3} = 1$ , starting $V_{w} (s_{0}) = 5$ , ending at $s_{3}$ with $V_{w} (s_{3}) = 6$ , $γ = 0.9$ . Compare the 1-step advantage $δ_{0}$ (using $V_{w} (s_{1}) = 4.5$ ) with the 3-step advantage.

Forecast: which estimate leans more on the (possibly wrong) critic, and which leans more on real rewards?

Step 1 — 1-step advantage. $δ_{0} = r_{1} + γ V_{w} (s_{1}) - V_{w} (s_{0}) = 1 + 0.9 (4.5) - 5 = 1 + 4.05 - 5 = 0.05$ . Why this step? The 1-step estimate bootstraps immediately off $V_{w} (s_{1})$ — low variance (short random sum) but biased if the critic is wrong. (Here we do keep the subscript on $δ_{0}$ precisely because two different-horizon estimates for the same start-step coexist.)

Step 2 — 3-step advantage. Use three real rewards then bootstrap once: $A^{(3)} = r_{1} + γ r_{2} + γ^{2} r_{3} + γ^{3} V_{w} (s_{3}) - V_{w} (s_{0}) .$ $= 1 + 0.9 (1) + 0.81 (1) + 0.729 (6) - 5 = 1 + 0.9 + 0.81 + 4.374 - 5 = 2.084$ . Why this step? Using more real rewards trusts the critic less — lower bias but higher variance (longer random sum). This is the bias–variance dial that Generalized Advantage Estimation (GAE) tunes continuously, and that A3C and A2C / Proximal Policy Optimization (PPO) exploit.

Step 3 — interpret the gap. The 1-step estimate ( $0.05$ ) is nearly neutral because it trusted a possibly-low $V_{w} (s_{1}) = 4.5$ ; the 3-step estimate ( $2.084$ ) saw the actual reward stream and the strong ending state, revealing a much bigger positive surprise. So C10 genuinely can have either sign relative to the 1-step version — here both are positive, but the 3-step correction can flip a near-zero 1-step signal into a decisive one.

Verify: Discount powers: $γ^{0} = 1, γ^{1} = 0.9, γ^{2} = 0.81, γ^{3} = 0.729$ . Sum of reward terms $1 + 0.9 + 0.81 = 2.71$ ; plus $0.729 \times 6 = 4.374$ ; minus $5$ gives $2.084$ . ✓

Active recall

Recall Same reward, opposite update — how?

Reward sign ( $r$ ) does not determine the update sign; $δ = r + γ V (s^{'}) - V (s)$ does. A positive reward with a collapsing future (Ex 2) gives $δ < 0$ ; a negative reward with a brightening future (Ex 3) gives $δ > 0$ .

Recall What is

$V (s^{'})$ at a terminal state, and why does it matter? It is $0$ by definition (no future after the episode ends). In code, a (1 - done) mask multiplies the future term. Forgetting this and plugging a nonzero value can flip the sign of $δ$ and corrupt learning (Ex 5).

Recall What is the difference between

$Q (s, a)$ and $V (s)$ , and how does the advantage use both? $Q (s, a)$ = value of committing to this specific action; $V (s)$ = value of the average action in $s$ . Advantage $A = Q - V$ is "how much better than average this move is", and its one-sample estimate is $δ = r + γ V (s^{'}) - V (s)$ .

Recall Why does

$δ = 0$ mean "no update"? $δ$ multiplies both the actor and critic update rules. If the critic's forecast was exactly met, there is no surprise, so the correct behaviour is to leave everything unchanged — a healthy fixed point (Ex 4).

Recall What does "stop-gradient on the target" mean in code?

Compute $y = r + γ V_{w} (s^{'})$ with current weights, then .detach() / tf.stop_gradient it so autodiff treats $y$ as a constant. This yields the correct one-term semi-gradient $- 2 δ \nabla_{w} V_{w} (s)$ ; differentiating through the target adds a spurious $(1 - γ)$ -scaled residual term (Ex 9).

Mnemonic

Connections

Actor-critic methods — parent; this page instantiates every case of its $δ_{t}$ .
Advantage function — Ex 2/3 show advantage decouples from raw reward.
Temporal Difference Learning — the bootstrap target behind $δ$ .
Generalized Advantage Estimation (GAE) — the n-step dial of Ex 10.
Proximal Policy Optimization (PPO), A3C and A2C — use these advantage estimates in practice.
Policy Gradient Theorem, REINFORCE — the high-variance ancestors the critic tames.