Visual walkthrough — Temporal Difference learning

Before any symbols: imagine you are a robot standing on a tile in a room. From wherever you stand, you care about one number — "how good is it to be standing here?" That number is what we are about to build, name, and learn to correct.

---

Step 1 — What "being somewhere" is worth: the return

WHAT. Picture a chain of tiles the robot walks across. Each time it leaves a tile it collects a little payout — a number that can be positive (good) or negative (a cost). Call the payout collected on the first move $R_{t+1}$, the next $R_{t+2}$, and so on. The little subscript $t$ just means "the current moment"; $t+1$ means "one step later".

WHY. To decide "how good is standing here?" we must add up all the payouts we will collect from here onward. But a payout far in the future should count for less — the room might change, we might not get there. So we shrink each future payout by a factor before adding.

PICTURE.

Look at the shrinking bars. The first bar $R_{t+1}$ is full height. The next is multiplied by $\gamma$ (a number between 0 and 1 — think $0.9$), the one after by $\gamma \times \gamma ={\gamma }^2$, and so on. That fading is the discount factor $\gamma$ at work.

Adding the faded bars gives the return:

$G_t=\underset{\text{payout now}}{\underbrace{R_{t+1}}}+\underset{\text{next, faded once}}{\underbrace{\gamma R_{t+2}}}+\underset{\text{faded twice}}{\underbrace{{\gamma }^2{R}_{t+3}}}+\cdots$

---

Step 2 — The value is the average return

WHAT. The robot's world is a little random — the same tile might send it left one day and right another, and payouts wiggle. So $G_t$ is not one fixed number; it is a lottery of possible totals.

WHY. We want one trustworthy number per tile, not a lottery. The honest summary of a lottery is its average — the expectation. The symbol ${\mathbb{E}}_{\pi }[\cdot ]$ means "average over everything that could happen if we follow behaviour rule $\pi$".

PICTURE.

Each faint grey line is one possible walk (one possible $G_t$). The bold coloured line is their average — that average height is the value.

$V^{\pi }(s)={\mathbb{E}}_{\pi }[\underset{\text{the lottery}}{\underbrace{G_t}}\mid \underset{\text{given we start on tile }s}{\underbrace{S_t=s}}]$

---

Step 3 — Peel off the first payout (the recursive trick)

WHAT. Look again at the return sum. Every term after the first shares a common factor of $\gamma$. Factor it out.

WHY. We want to separate what we can actually observe right now ($R_{t+1}$) from everything else. The "everything else" is going to turn out to be a return again — one step later — and that is the door to bootstrapping.

PICTURE.

The figure snips the first bar off the front and boxes the rest. The boxed part, once you pull out $\gamma$, is exactly the return starting from the next tile:

$G_t=R_{t+1}+\gamma \left(\underset{=G_{t+1}}{\underbrace{R_{t+2}+\gamma R_{t+3}+\cdots }}\right)=\underset{\text{seen now}}{\underbrace{R_{t+1}}}+\gamma \underset{\text{return from next tile}}{\underbrace{G_{t+1}}}$

This is the single most important line on the page. Read it as: "the whole future = this step's payout, plus a discounted copy of the future starting one tile over."

---

Step 4 — Turn the recursion into the Bellman equation

WHAT. Take the average of both sides of the Step 3 identity.

WHY. Averaging is linear — the average of a sum is the sum of the averages. That lets us split the right-hand side into two clean pieces we can name.

PICTURE.

$V^{\pi }(s)={\mathbb{E}}_{\pi }[R_{t+1}+\gamma G_{t+1}\mid S_t=s]=\underset{\text{avg immediate payout}}{\underbrace{{\mathbb{E}}_{\pi }[R_{t+1}\mid S_t=s]}}+\gamma \underset{\text{avg future return}}{\underbrace{{\mathbb{E}}_{\pi }[G_{t+1}\mid S_t=s]}}$

Now the magic: the second average — "average return starting from wherever we land next" — is by definition the value of the next tile. So:

$\boxed{V^{\pi }(s)={\mathbb{E}}_{\pi }[\underset{\text{payout}}{\underbrace{R_{t+1}}}+\gamma \underset{\text{value of next tile}}{\underbrace{V^{\pi }(S_{t+1})}}\mid S_t=s]}$

This is the Bellman equation (see Bellman Equations). It says a tile's value equals the payout for leaving it, plus the discounted value of where you land — averaged over the randomness. Notice we replaced an infinite tail ($R_{t+2}+\gamma R_{t+3}+\cdots$) with a single number $V^{\pi }(S_{t+1})$. That swap is called bootstrapping (see Bootstrapping).

---

Step 5 — From averages we can't compute to one sample we can

WHAT. The Bellman equation contains ${\mathbb{E}}_{\pi }[\cdot ]$ — a true average over all futures. We do not know that average; if we did, we would already be done. But we can take one real step and watch what happens.

WHY. A single observed transition $S_t\stackrel{R_{t+1}}{\to }{S}_{t+1}$ is a sample of that average. We drop the $\mathbb{E}$ and use the one thing we actually saw. This is exactly the Monte-Carlo idea of "estimate an average with samples", but applied to only one step instead of the whole episode.

PICTURE.

$\underset{\text{the TD target}}{\underbrace{R_{t+1}+\gamma V(S_{t+1})}}$

We call this the TD target. It is our new, better-informed guess for $V(S_t)$, because it contains one piece of real data ($R_{t+1}$, the payout we truly received) glued to our current estimate of the rest.

---

Step 6 — Measure how wrong we were: the TD error

WHAT. Compare the fresh target to the old stored number $V(S_t)$. Their difference is the TD error.

WHY. A guess is only worth correcting by how much it was off. Subtracting old from new gives a signed size-of-mistake: positive means "better than I thought", negative means "worse than I thought".

PICTURE.

Two dots on a number line: old estimate $V(S_t)$ and the target. The arrow between them is the error.

${\delta }_t=\underset{\text{target (new)}}{\underbrace{R_{t+1}+\gamma V(S_{t+1})}}-\underset{\text{estimate (old)}}{\underbrace{V(S_t)}}$

---

Step 7 — Step, don't jump: the learning rate

WHAT. We do not overwrite $V(S_t)$ with the target. We nudge it a fraction $\alpha$ of the way there.

WHY. The target came from one noisy sample. If we jumped fully onto it (that would be $\alpha =1$), one unlucky payout would erase everything we learned. A small $\alpha$ averages many noisy targets over time — trading a little slowness for stability. This is the Bias-Variance Tradeoff showing up as a knob.

PICTURE.

The dot slides only part-way along the error arrow — a distance $\alpha \times {\delta }_t$.

$V(S_t)\leftarrow \underset{\text{old}}{\underbrace{V(S_t)}}+\underset{\text{step size }(0,1]}{\underbrace{\alpha }}\cdot \underset{\text{full surprise}}{\underbrace{{\delta }_t}}$

Unfolding ${\delta }_t$ gives the box from the parent note — now every symbol has been earned:

$\boxed{V(S_t)\leftarrow V(S_t)+\alpha [R_{t+1}+\gamma V(S_{t+1})-V(S_t)]}$

---

Step 8 — Degenerate cases: check the arrow still makes sense

WHAT / WHY / PICTURE. A good derivation must survive its extremes.

• Terminal next state. If $S_{t+1}$ is terminal, there is no future, so $V(S_{t+1})=0$ by definition. Target collapses to just $R_{t+1}$: $V(S_t)\leftarrow V(S_t)+\alpha [R_{t+1}-V(S_t)]$. Pure real data, no bootstrap.
• $\gamma =0$ (myopic). The whole $\gamma V(S_{t+1})$ term vanishes; the target is $R_{t+1}$ alone. Each tile only ever learns its immediate payout — no propagation.
• $\gamma =1$ (undiscounted). The next tile's value counts fully. Fine for episodes that always end; can diverge for endless loops (the return sum has no shrinking to keep it finite).
• ${\delta }_t=0$. Target equals old estimate → no update. This does not mean "learned"; a noisy environment produces nonzero ${\delta }_t$ even at the true value. What must vanish is the average $\mathbb{E}[{\delta }_t]$, not each individual one.

Recall Quick self-check on the edges

If $S_{t+1}$ is terminal, what is the TD target? ::: Just $R_{t+1}$, because $V(\text{terminal})=0$. With $\gamma =0$, what does each tile learn? ::: Only its immediate reward $R_{t+1}$; nothing propagates backward. Does ${\delta }_t=0$ on one step mean learning is finished? ::: No — noise keeps single errors nonzero; only $\mathbb{E}[{\delta }_t]\to 0$ signals convergence.

---

The one-picture summary

Everything compresses into one loop: observe a payout $R_{t+1}$ and next tile $S_{t+1}$ → form the target $R_{t+1}+\gamma V(S_{t+1})$ → subtract the old $V(S_t)$ to get the surprise ${\delta }_t$ → slide the estimate a fraction $\alpha$ toward the target → repeat.

Recall Feynman retelling — the whole walkthrough in plain words

I want to know how good it is to stand on a tile. "Good" means the total of all payouts I'll collect from here, with far-off payouts faded down by a shrink-factor $\gamma$ — that faded total is the return. But payouts are random, so I take the average return; that average is the tile's value.

Here's the clever peel: the whole future equals this step's payout plus a faded copy of the future starting one tile over. Averaging both sides, the "future from the next tile" is just the next tile's value. So a tile's value = payout for leaving + faded value of where I land. That's Bellman.

I can't compute the true average, but I can take one real step and watch the payout. Real payout + my stored guess of the next tile = a slightly smarter guess, the target. How wrong was my old number? Target minus old = the surprise $\delta$. I don't leap onto the target — one step is noisy — I only slide a fraction $\alpha$ of the way. Slide, walk, repeat. That sliding arrow is TD(0), and it's the seed of Q-Learning, SARSA, and TD(λ).

---

Connected ideas: Bellman Equations · Monte Carlo Methods · Dynamic Programming · Bootstrapping · Bias-Variance Tradeoff · Q-Learning · SARSA · TD(λ) · Value Function Approximation