Temporal Difference learning: Question bank — AI-ML deep dive

Before we start, a tiny glossary so every symbol below is earned:

True or false — justify

TD(0) needs the environment's transition probabilities

P (s^{'} ∣ s, a)

to compute its target

False. It never computes the next state — it observes

S_{t + 1}

by acting. The target uses the observed

S_{t + 1}

, so no model is required; this is what makes TD model-free.

If our value estimates are already perfectly correct, every individual TD error

δ_{t}

becomes exactly

0

False. With stochastic rewards/transitions, single-step

δ_{t}

still swings above and below zero; only its expectation

E [δ_{t}]

is zero at the true values. See variance.

TD is unbiased once it converges but has high variance

Half-true and mislabelled. TD reduces variance (it samples only one step of randomness) but is biased while

V (S_{t + 1})

is still wrong. That biased-but-low-variance tradeoff is TD's signature, opposite to Monte Carlo.

Setting

γ = 0

turns TD into a method that only ever cares about the immediate reward

True. With

γ = 0

the target collapses to

R_{t + 1}

, so

V (s)

learns the expected next reward and ignores the future entirely — a myopic estimator.

Setting

γ = 1

is always safe

False.

γ = 1

is fine for guaranteed-terminating (episodic) tasks but makes the return infinite/divergent for continuing tasks, so

V

may not converge.

TD(0) can update a state's value before the episode ends

True. That is its whole point — it updates online after each transition, unlike Monte Carlo which waits for the final return

G_{t}

Terminal states must be initialized to

V = 0

True by definition. A terminal state has no future rewards, so its true value is

0

; using anything else corrupts every TD target that bootstraps off it.

The TD update always moves

V (S_{t})

closer to the true value

False. A single noisy sample can push it the wrong way; correctness is only guaranteed in expectation / over many updates with a decaying

α

Bootstrapping means "learning from the final outcome of the whole episode"

False. That is Monte Carlo. Bootstrapping means updating one guess (

V (S_{t})

) toward another guess (

V (S_{t + 1})

) — see Bootstrapping and DP, which also bootstraps.

The Bellman equation and the TD target are unrelated ideas

False. The TD target is a one-sample estimate of the Bellman right-hand side

E [R_{t + 1} + γ V^{π} (S_{t + 1})]

; TD replaces the expectation with an observed sample.

Spot the error

"TD error should shrink to exactly zero as training finishes, otherwise learning failed."

Wrong: individual errors never vanish under stochasticity. The right diagnostic is whether the average error and the change in $V$ shrink, not each

δ_{t}

"Use

α = 1

so we learn as fast as possible."

Wrong:

α = 1

throws away the old estimate and replaces it with one noisy sample, causing wild oscillation. Small (often decaying)

α

trades speed for stability.

V (S_{t + 1})

in the target is the true value of the next state."

Wrong: it is our current estimate, which is itself imperfect. That is exactly the source of TD's bias.

"Because TD bootstraps from DP, it also needs the full model like DP does."

Wrong: TD inherits DP's bootstrapping idea but not DP's model requirement. It samples transitions instead of sweeping over known dynamics.

"A positive TD error means the agent did something bad and

V

must go down."

Wrong: positive

δ_{t}

means the outcome was better than predicted, so

V (S_{t})

is nudged up. Positive = pleasant surprise.

"Since MC and TD both estimate

V^{π}

, after one episode they give the same estimates."

Wrong: MC updates only states actually visited using the real return; TD updates each state toward a bootstrapped one-step target. Their intermediate estimates differ sharply, especially early.

"If a reward is stochastic, TD is broken because

δ_{t}

keeps flipping sign."

Wrong: sign-flipping is expected and healthy — the flips average out. TD converges in expectation despite noisy per-step errors.

Why questions

Why does TD have lower variance than Monte Carlo?

MC's target is a sum of many random rewards

R_{t + 1} + γ R_{t + 2} + \dots

, accumulating randomness at every step; TD replaces that whole tail with

γ V (S_{t + 1})

, so only one step of randomness enters the target.

Why does TD have bias that MC does not?

TD's target leans on

V (S_{t + 1})

, an estimate that starts wrong, so early targets are systematically off. MC uses the actual return, which is an unbiased sample of

V^{π}

Why does value "propagate backward" faster in TD than MC?

Once a state near the goal learns a good value, its predecessor immediately bootstraps off it next visit. MC must wait for whole episodes to feed the real return back, so a 100-step chain needs roughly 100 episodes to fill in.

Why do we still use a learning rate

α

instead of jumping straight to the TD target?

Because a single target is noisy;

α

makes the update a running average, blending old estimate with new evidence so noise cancels over time.

Why is TD called "model-free" even though it uses

V (S_{t + 1})

Because it never asks "which states could follow?"; it uses the one

S_{t + 1}

the environment actually handed it. No transition probabilities are ever queried.

Why is the TD target considered "more informed" than the old estimate

V (S_{t})

Because it contains a piece of real experienced data — the actual reward

R_{t + 1}

— that the old estimate was made without.

Why must terminal states be pinned at

V = 0

rather than learned?

Because their true value is defined as zero (no future); learning them would let noise leak in and bias every target that bootstraps through the episode's end.

Edge cases

What is the TD target when

S_{t + 1}

is terminal?

It becomes simply

R_{t + 1} + γ \cdot 0 = R_{t + 1}

, since terminal states carry value

0

. The bootstrap term vanishes at the boundary.

What happens to learning if

α \to 0

Updates become vanishingly small, so

V

barely moves and effectively stops learning — stable but stuck. There is a sweet spot, not a "smaller is always safer" rule.

What does

δ_{t}

look like for a fair coin-flip reward of

{+ 10, 0}

once

V = 5

is correct?

On a

+ 10

draw

δ_{t} = + 5

, on a

0

draw

δ_{t} = - 5

; the errors are large and permanent per step but average to

0

. Convergence lives in the expectation, not the individual

δ_{t}

If two different states happen to share the same true value, will TD treat them identically?

Not necessarily during learning — each has its own estimate updated only when visited, so visitation frequency and neighbours make their learning trajectories differ even if they converge to the same number.

What if the initial

V (s)

values are wildly wrong (say all

+ 1000

TD still converges; the surprising

δ_{t}

values simply drive large early corrections. Bad initialization slows the start but does not break the method — that is why "initialize arbitrarily" is allowed.

In a continuing (never-ending) task with

γ = 1

, what breaks?

The return has no finite value to converge to, so targets can grow unboundedly and

V

diverges. Discounting with

γ < 1

(or an average-reward formulation) is needed to keep values finite.

Recall Quick self-test

One TD idea that is shared with DP but not with MC ::: Bootstrapping (updating a guess from another guess). One TD idea that is shared with MC but not with DP ::: Learning from sampled experience, model-free. The single quantity that carries all the "surprise" in a TD update ::: The TD error $δ_{t}$ .