Temporal Difference learning: Foundations — AI-ML deep dive

This page assumes you have seen nothing. Every letter in the parent note's equations is unpacked here, one at a time, each building on the last. Read top to bottom; do not skip.

0. The picture the whole topic lives inside

Everything in Temporal Difference learning is about an agent walking through states, collecting rewards. Look at the figure: a little dot (the agent) sits in a box (a state $s$ ), takes a step (an action $a$ ), lands in a new box, and a number pops out (a reward $r$ ). That is the entire world. Every symbol below labels one part of this picture.

1. State — where you are

Picture: one box in the grid of figure s01. Standing in that box is all you need to know to decide what to do next.

Why the topic needs it: value learning answers the question "how good is it to be here?" — and "here" is a state. Without naming the state, there is nothing to attach a value to.

$S_{t}$ ::: the state at step $t$ (a variable — could be any box)
$s$ ::: one specific state you point at
terminal state ::: the last box of an episode; nothing happens after it, so its value is $0$

2. Time-step $t$ — the ticking clock

Picture: the numbers written under each footprint as the dot walks in figure s01.

Why the topic needs it: TD's whole trick is comparing consecutive moments — $t$ versus $t + 1$ . The clock is what lets us say "the next step."

3. Action $A_{t}$ and Reward $R_{t + 1}$

Picture: the arrow leaving a box (action) and the number floating above the arrow (reward) in figure s01.

Why the topic needs it: rewards are the only feedback the agent ever gets. Learning "how good a state is" means adding up rewards you expect to collect from it.

$A_{t}$ ::: the action taken at step $t$
$R_{t + 1}$ ::: the reward received after that action (a real number, can be negative)

4. Policy $π$ — the habit that picks actions

Picture: an arrow pre-drawn out of every box saying "if you're here, go this way."

Why the topic needs it: the value of a state only makes sense relative to a behaviour. "How good is this box?" depends on what you'll do afterward. $π$ fixes that behaviour. See SARSA and Q-Learning for what happens when we also improve $π$ .

5. Discount factor $γ$ — future money is worth less

Picture: figure s02 shows a row of $+ 1$ rewards at steps $1, 2, 3, \dots$ ; the bars shrink as $γ^{1}, γ^{2}, γ^{3}$ . Near rewards are tall, far rewards are short.

Why this tool and not just "add everything"? If we added infinitely many rewards with equal weight, the sum could blow up to infinity and never settle. $γ < 1$ guarantees the total is a finite number, and it also encodes impatience — "a reward now beats the same reward later."

All cases of $γ$ :
- $γ = 0$ ::: agent is totally short-sighted, only the immediate reward matters
- $0 < γ < 1$ ::: future rewards count but fade (the normal case)
- $γ = 1$ ::: no discounting — safe only when episodes are guaranteed to end

6. The Return $G_{t}$ — the whole prize from here on

Reading it in words: the reward you get next, plus $γ$ times the one after, plus $γ^{2}$ times the one after that, forever (or until the episode ends).

Picture: add up all the shrinking bars in figure s02 — that single grand total is $G_{t}$ .

Why the topic needs it: $G_{t}$ is the true quantity we wish we knew. But it lives in the future, so we can't see it until later. This is exactly the gap TD is built to bridge.

7. Expectation $E$ — the average over randomness

Picture: roll the same dice a thousand times, take the mean — that mean is $E$ . In figure s02, if the rewards were random, $E$ is the height each bar would settle to on average.

Why the topic needs it: the world is noisy. The same action can lead to different rewards or different next states. We don't want the value to jump around with luck — we want the average outcome. That is why value is defined as an expectation.

8. Value function $V^{π} (s)$ — the number we're chasing

Picture: write a single number inside each box of figure s01 — "the average total prize you'll collect from here." That grid of numbers is the value function.

Why the topic needs it: this is the entire goal. Learning $V^{π}$ tells the agent which states are good and, indirectly, which moves are worth making.

$V^{π} (s)$ ::: the true value under policy $π$
$V (s)$ ::: our current estimate of it — a guess we keep improving

9. Bellman equation — value defined by the next value

Why the second term became $V^{π} (S_{t + 1})$ : the average of the return from the next state is, by definition, the value of the next state. That's the swap that turns an infinite sum into one clean reference to a neighbour. Full treatment in Bellman Equations.

Why the topic needs it: this says "a state's value is defined in terms of its neighbour's value." That self-referencing structure is what lets us learn by bootstrapping — improving one guess using another guess. Compare with Dynamic Programming, which uses this same equation but needs a full model, and Monte Carlo Methods, which ignores it and waits for full episodes.

10. The moving pieces of the TD update

Figure s03 lays the three quantities on a number line: the old guess $V (S_{t})$ , the TD target (a better-informed guess), and the small step $α \times$ (the gap between them) that pulls old toward target.

$\leftarrow$ ::: "gets replaced by" — an assignment, not an equals sign
TD target $R_{t + 1} + γ V (S_{t + 1})$ ::: real reward you just saw + your current guess of the future
TD error $δ_{t} = target - old guess$ ::: how wrong the old guess was (an arrow in figure s03)
$α$ — the learning rate (next section)

11. Learning rate $α$ — how big a nudge

Picture: in figure s03, $α$ is how far along the arrow we slide from old guess toward target.

All cases of $α$ :
- $α \to 0$ ::: barely move — very slow, very stable learning
- $α \approx 0.1$ ::: gentle nudges (typical)
- $α = 1$ ::: throw away the old guess entirely, adopt the noisy target — unstable, jumps around
- $α > 1$ or $α \leq 0$ ::: forbidden — you'd overshoot or never learn

12. Prerequisite map

Read it bottom-up: states, actions, rewards, and discount feed the return; averaging the return under a policy gives the value function; the value function's self-referencing Bellman form enables bootstrapping, which — plus a learning rate — becomes the TD update.

Equipment checklist

Self-test: can you answer each before revealing?

What does a state $S_{t}$ represent? ::: A complete description of the agent's current situation — the box the agent occupies at step $t$ .
Why is the reward written $R_{t + 1}$ and not $R_{t}$ ? ::: It is the reward received on arriving at the next state, so it carries the next step's subscript.
What does the discount factor $γ$ do and why is it needed? ::: It shrinks far-future rewards by $γ^{k}$ , keeping the infinite sum finite and encoding impatience.
Write the return $G_{t}$ and its one-step recursion. ::: $G_{t} = R_{t + 1} + γ R_{t + 2} + \dots = R_{t + 1} + γ G_{t + 1}$ .
What does $E_{π} [\cdot ∣ S_{t} = s]$ mean in plain words? ::: The average value, over all randomness while following policy $π$ , given you started in state $s$ .
Define the value function $V^{π} (s)$ . ::: The expected (average) return starting from state $s$ and following policy $π$ .
State the Bellman equation for $V^{π}$ . ::: $V^{π} (s) = E_{π} [R_{t + 1} + γ V^{π} (S_{t + 1}) ∣ S_{t} = s]$ .
What is the TD target and the TD error? ::: Target $= R_{t + 1} + γ V (S_{t + 1})$ ; error $δ_{t} = target - V (S_{t})$ .
What does the learning rate $α$ control, and what goes wrong at $α = 1$ ? ::: How far we move from old guess toward the target; $α = 1$ discards the old estimate entirely, making learning unstable and noisy.
Why can TD learn without a model? ::: It observes the next state and reward from real interaction rather than predicting them from transition dynamics.

Continue to: Temporal Difference learning · related: TD(λ), Value Function Approximation, 5.1.11 Temporal Difference learning (Hinglish)