Temporal Difference learning: Worked examples

Recall the single machine we keep feeding inputs into. After observing $S_{t} R_{t + 1} S_{t + 1}$ :

Before any numbers, four plain-word reminders so no symbol is unearned:

$R_{t + 1}$ — the reward you actually receive on the one step you are currently processing. A real observed number, not a guess. In the worked examples below we shorten $R_{t + 1}$ to just $R$ — same thing, fewer subscripts — because each example looks at a single transition, so there is only ever one reward to name.
$γ$ (gamma) — the discount factor, a dial in $[0, 1]$ that says "how much do rewards later count compared to now?" $γ = 0$ means "ignore the future entirely"; $γ = 1$ means "the future counts as much as the present."
$α$ (alpha) — the learning rate, a dial in $(0, 1]$ saying "how big a step do I take toward the target?" $α = 1$ means "throw away the old estimate completely."
Terminal state — a state where the episode ends (goal reached, game over). By definition it has no future rewards, so its value is fixed at $V (terminal) = 0$ . This is not learned, it is a boundary condition we set. Whenever a transition lands in a terminal state, the bootstrapped future term $γ V (S_{t + 1})$ collapses to $0$ and the target is reward only.

The picture below fixes the whole machine in your mind before we touch a single number — three quantities on a number line and the nudge that connects them.

The scenario matrix

Every TD transition you will ever compute falls into one of these cells. The examples below are labelled with the cell they cover.

#	Case class	What makes it special	Covered by
C1	Positive TD error ( $δ > 0$ )	target beats old estimate → value goes up	Ex 1
C2	Negative TD error ( $δ < 0$ )	target below old estimate → value goes down	Ex 2
C3	Zero TD error ( $δ = 0$ )	already consistent → no change	Ex 3
C4	Terminal transition	$V (S_{t + 1}) = 0$ (defined above), target = reward only	Ex 4
C5	$γ = 0$ (myopic)	future term vanishes, TD = one-step reward chasing	Ex 5
C6	$γ = 1$ (undiscounted)	full future counts; back-propagation cascade	Ex 6
C7	$α = 1$ instability	old estimate overwritten by noisy target	Ex 7
C8	Stochastic reward	$δ$ never reaches 0, only its mean does	Ex 8
C9	Real-world word problem	build state/reward from a story	Ex 9
C10	Exam twist: multi-step order	update order changes intermediate numbers	Ex 10

We build on Bellman Equations (the recursion the target comes from), contrast with Monte Carlo Methods and Dynamic Programming, and the noise cell touches Bias-Variance Tradeoff and Bootstrapping.

Worked examples

Worked example Example 1 — Positive TD error (cell C1)

Statement. States $A \to B$ , reward on this step $R = 0$ (i.e. $R_{t + 1} = 0$ ), $γ = 1$ , $α = 0.1$ . Current estimates $V (A) = 0.5$ , $V (B) = 0.8$ . Update $V (A)$ after observing $A \to B$ .

Forecast: Does $V (A)$ rise or fall? (Guess before reading — $B$ turned out better than $A$ 's estimate, so...)

Compute the TD target $= R + γ V (B) = 0 + 1 (0.8) = 0.8$ . Why this step? The target is our improved estimate of $A$ 's worth: the reward we saw plus the discounted value of where we landed.
Compute the TD error $δ = 0.8 - V (A) = 0.8 - 0.5 = + 0.3$ . Why this step? $δ$ measures how wrong the old estimate was. Positive means the old estimate was too low.
Apply the update $V (A) \leftarrow 0.5 + 0.1 (0.3) = 0.53$ . Why this step? We move a fraction $α = 0.1$ of the way toward the better estimate — not all the way, because one sample is only weak evidence.

Verify: New value $0.53$ sits between old $0.5$ and target $0.8$ — exactly where a fractional nudge should land. Rose, as forecast. ✓

Worked example Example 2 — Negative TD error (cell C2)

Statement. Grid world, state $(0, 0)$ , reward per step $R = - 1$ , $γ = 0.9$ , $α = 0.1$ . All values start at $0$ . Agent moves right into $(0, 1)$ with $V (0, 1) = 0$ . Update $V (0, 0)$ .

Forecast: We paid a cost of $- 1$ to reach a state we value at $0$ . Was $(0, 0)$ 's optimistic value of $0$ too high or too low?

TD target $= - 1 + 0.9 (0) = - 1$ . Why this step? Even though the future value is $0$ , the step itself cost us $- 1$ ; the target captures that pain.
TD error $δ = - 1 - 0 = - 1$ . Why this step? Negative → our estimate of $0$ was too optimistic.
Update $V (0, 0) \leftarrow 0 + 0.1 (- 1) = - 0.1$ . Why this step? We drop the value slightly to reflect that reaching $(0, 0)$ implies future cost.

Verify: $- 0.1$ is between old $0$ and target $- 1$ , on the negative side. Value fell, as forecast. ✓

Worked example Example 4 — Terminal transition (cell C4)

Statement. State $B \to C$ , where $C$ is terminal with reward $R = + 1$ . $γ = 1$ , $α = 0.1$ , $V (B) = 0.8$ . Update $V (B)$ .

Forecast: For a terminal next state, what is $V (C)$ — and therefore what does the target simplify to?

Use the terminal rule $V (C) = 0$ . Why this step? As we defined up front, a terminal state has no future rewards, so its value is exactly $0$ — this is not a guess, it is a boundary condition.
TD target $= 1 + 1 (0) = 1$ . Why this step? The bootstrapped term dies, leaving only the observed reward — here TD momentarily equals Monte Carlo, because there is no future to estimate.
TD error $δ = 1 - 0.8 = 0.2$ , update $V (B) \leftarrow 0.8 + 0.1 (0.2) = 0.82$ . Why this step? $B$ leads straight to a $+ 1$ payoff, so its value should climb toward $1$ ; we take the first step there.

Verify: $0.82$ moved toward $1$ . Repeated visits would push $V (B) \to 1$ , the true value of a state one deterministic step from a $+ 1$ terminal. ✓

Worked example Example 5 — Myopic agent,

$γ = 0$ (cell C5) Statement. State $P \to Q$ , $R = + 4$ , $γ = 0$ , $α = 0.5$ . $V (P) = 1$ , $V (Q) = 100$ (huge). Update $V (P)$ .

Forecast: $V (Q) = 100$ is enormous — but with $γ = 0$ , does it matter at all?

TD target $= 4 + 0 \cdot (100) = 4$ . Why this step? $γ = 0$ zeroes the future term completely — the agent is myopic, caring only about the immediate reward. The huge $V (Q)$ is multiplied by $0$ and disappears.
TD error $δ = 4 - 1 = 3$ . Why this step? Measures how far the old value $1$ sits below the myopic target $4$ ; positive, so $V (P)$ was too pessimistic about its immediate reward.
Update $V (P) \leftarrow 1 + 0.5 (3) = 2.5$ . Why this step? We step halfway ( $α = 0.5$ ) from $1$ toward $4$ . With $γ = 0$ , $V (P)$ can only ever learn the expected immediate reward from $P$ , never anything downstream.

Verify: The giant $V (Q) = 100$ had zero influence — target is $4$ , not $54$ . This confirms $γ = 0$ throws away the future entirely. ✓

Worked example Example 6 — Undiscounted back-propagation,

$γ = 1$ (cell C6) Statement. A chain $s_{97} \to s_{98} \to s_{99} \to goal (+ 10)$ . $γ = 1$ , $α = 0.5$ . Just after the first episode, $V (s_{99})$ was updated toward the goal; suppose $V (s_{99}) = 5$ now, and $V (s_{98}) = 0$ , $V (s_{97}) = 0$ , all intra-chain rewards $= 0$ . In episode 2 we traverse $s_{98} \to s_{99}$ . Update $V (s_{98})$ .

Forecast: With no discount, how much of $s_{99}$ 's value flows back into $s_{98}$ ?

TD target $= 0 + 1 (5) = 5$ . Why this step? $γ = 1$ passes the entire value of $s_{99}$ back with no shrinkage — that is what "undiscounted" means.
TD error $δ = 5 - 0 = 5$ . Why this step? $s_{98}$ 's current value of $0$ is far below the freshly-informed target $5$ .
Update $V (s_{98}) \leftarrow 0 + 0.5 (5) = 2.5$ . Why this step? The $+ 10$ signal, first learned at $s_{99}$ , is now cascading one step further back — the bootstrapping cascade the parent note described.

Verify: $V (s_{98}) = 2.5$ became nonzero because $V (s_{99})$ was nonzero — no discount let it flow freely. The figure below traces this backward flow. ✓

Reading the figure: Four nodes sit along the chain, start on the left and the $+ 10$ goal on the right (orange). The straight ink arrows show the agent walking forward $s_{97} \to s_{98} \to s_{99} \to$ goal. The curved plum arrow shows the opposite: value flowing backward from $s_{99}$ into $s_{98}$ . Under each node the label reads its value before and after this update — only $s_{98}$ changes ( $0 \to 2.5$ ), because it is the only state whose transition we processed this episode. Run more episodes and this " $0 \to$ nonzero" event marches one node left each time: that marching wave is the value gradient.

Worked example Example 7 —

$α = 1$ instability (cell C7) Statement. Stochastic reward: from state $W$ , half the time $R = + 10$ , half the time $R = 0$ ; next state terminal ( $V = 0$ ), $γ = 1$ . True value $V (W) = 5$ . Start $V (W) = 5$ (already correct!). We observe two draws: first $R = 10$ , then $R = 0$ , using $α = 1$ . Track $V (W)$ .

Forecast: Old value is already the true $5$ . With $α = 1$ , will it stay near $5$ ?

Draw 1 ( $R = 10$ ): target $= 10 + 0 = 10$ , $δ = 10 - 5 = 5$ , update $V (W) \leftarrow 5 + 1 (5) = 10$ . Why this step? $α = 1$ discards the old estimate entirely and copies the noisy target.
Draw 2 ( $R = 0$ ): target $= 0$ , $δ = 0 - 10 = - 10$ , update $V (W) \leftarrow 10 + 1 (- 10) = 0$ . Why this step? Again the estimate is overwritten by a single noisy sample.

Verify: The estimate slammed from $5 \to 10 \to 0$ , never settling — despite starting exactly right. This is why $α = 1$ is unusable with noise; a small $α$ would have averaged the draws toward $5$ . ✓

Worked example Example 8 — Stochastic reward, error never zero (cell C8)

Statement. Same slot-machine state $W$ : $R = + 10$ w.p. $0.5$ , else $0$ ; terminal next state, $γ = 1$ . Suppose $V (W) = 5$ (the true value). What is the TD error on each of the two possible outcomes, and its average?

Forecast: If our value is perfectly correct, is the per-step TD error zero?

Payout outcome ( $R = 10$ ): $δ = 10 + 0 - 5 = + 5$ . Why this step? We compute the error for the lucky draw; it is a single realised sample, not the mean, so it lands above zero.
No-payout outcome ( $R = 0$ ): $δ = 0 + 0 - 5 = - 5$ . Why this step? Now the unlucky draw; again a single realised sample, this one below zero — showing individual errors swing to both signs even with a perfect value.
Expected error $E [δ] = 0.5 (+ 5) + 0.5 (- 5) = 0$ . Why this step? The correct diagnostic of convergence is that $E [δ] \to 0$ , not that individual $δ$ vanish.

Verify: Individual errors are $\pm 5$ (never zero) but average exactly $0$ — the variance persists while the mean is unbiased. Matches Mistake 2 in the parent note. ✓

Worked example Example 9 — Real-world word problem (cell C9)

Statement. You model your commute. State = "at the on-ramp." Reward = negative minutes elapsed each leg. You currently estimate the total remaining time from the on-ramp as $V (ramp) = - 30$ (i.e. 30 min). Today you spend $8$ min on the ramp segment ( $R = - 8$ ), and arrive at "midtown," which you estimate takes $V (midtown) = - 18$ more minutes. Use $γ = 1$ , $α = 0.2$ . Update $V (ramp)$ .

Forecast: Observed leg + remaining estimate $= 8 + 18 = 26$ min, but you had guessed $30$ . Should your ramp estimate get less negative (better) or more negative?

Build the target $= R + γ V (midtown) = - 8 + 1 (- 18) = - 26$ . Why this step? Reward is negative time; the target is "time actually just spent + estimated time still to go," a fresher estimate of total minutes.
TD error $δ = - 26 - (- 30) = + 4$ . Why this step? Positive because the trip is turning out shorter (less negative) than the $- 30$ we feared.
Update $V (ramp) \leftarrow - 30 + 0.2 (4) = - 29.2$ . Why this step? We nudge our commute estimate from 30 min toward 26 min — updating during the drive, exactly the parent note's opening intuition.

Verify: $- 29.2$ moved toward $- 26$ (shorter trip). Units: minutes throughout; $α$ dimensionless. Sign sensible: good news → less negative. ✓

Worked example Example 10 — Exam twist: update order matters (cell C10)

Statement. One episode visits $A \to B \to terminal (+ 1)$ , rewards $0$ then $+ 1$ , $γ = 1$ , $α = 0.5$ . Initial $V (A) = 0$ , $V (B) = 0$ . Perform the two online updates in the order they occur and report final $V (A)$ , $V (B)$ . Then say what $V (A)$ would have been if $V (B)$ had been updated first.

Forecast: When we update $V (A)$ , does it use the old $V (B) = 0$ or the already-updated one?

Update $A \to B$ first (this happens first in time): target $= 0 + 1 (0) = 0$ , $δ = 0$ , $V (A) \leftarrow 0 + 0.5 (0) = 0$ . Why this step? Online TD updates a state the moment its transition is observed, using whatever $V (B)$ is right now — which is still $0$ .
Update $B \to$ terminal: target $= 1 + 1 (0) = 1$ , $δ = 1$ , $V (B) \leftarrow 0 + 0.5 (1) = 0.5$ . Why this step? $B$ 's value only learns about the reward on this step; $A$ has already been processed and does not see the new $V (B)$ this episode.
Counterfactual: had $B$ updated first ( $V (B) = 0.5$ ), then $A$ 's target would be $0 + 1 (0.5) = 0.5$ , giving $V (A) = 0.25$ . Why this step? This shows the offline/batch or reversed-order variant propagates value one episode faster — the exam trap is assuming $V (A)$ "sees" $B$ 's new value in the same forward pass.

Verify: Forward online: $V (A) = 0$ , $V (B) = 0.5$ . Reversed order: $V (A) = 0.25$ . The intermediate number genuinely differs by update order.

Why this matters for TD(λ): plain TD(0) needs a whole extra episode to carry the reward one more state back (as Example 6 showed, one node per episode). TD(λ) fixes exactly this lag: it keeps an eligibility trace — a fading memory of recently-visited states — so that when the $+ 1$ arrives at $B$ , the error is broadcast backward within the same episode to $A$ (and earlier states), scaled by how recently each was visited. So the "order matters, and forward order is slow" lesson here is precisely the pain that eligibility traces were invented to remove. ✓

Recall Quick self-test

Positive $δ$ means the old value was too ::: low (target beat it), so the value rises With $γ = 0$ , the TD target reduces to ::: just the immediate reward $R_{t + 1}$ For a terminal next state the target is ::: the reward only, because $V (terminal) = 0$ Why does $α = 1$ fail on stochastic rewards? ::: it overwrites the estimate with a single noisy sample instead of averaging If all transitions give $δ = 0$ , learning ::: stops — you are at a fixed point (Bellman-consistent) In online TD(0), when updating $V (A)$ on $A \to B$ , you use ::: the current $V (B)$ , not any value $B$ gets later this episode