Visual walkthrough — State-value and action-value functions

This page rebuilds the single most important identity in state-value and action-value functions — the Bellman expectation equation — one picture at a time. We assume you know nothing: not what a "value" is, not what $\gamma$ means, not what a "policy" does. Every symbol is earned before it is used.

By the end you will see, in one picture, why:

$V^{\pi }(s)={\sum }_a\pi (a\mid s){\sum }_{s^{\prime },r}p(s^{\prime },r\mid s,a)\left[r+\gamma V^{\pi }(s^{\prime })\right]$

Don't worry that this looks like alphabet soup. We build it letter by letter.

---

Step 1 — What is a reward, and why sum the future?

WHAT. An agent lives in states (situations) and takes actions. After each action the world hands it a number called a reward — think of it as points. We write the reward that arrives at time step $t+1$ as $R_{t+1}$. The little subscript is just a clock reading: "this reward showed up on tick $t+1$."

WHY. A single reward tells you almost nothing. Grabbing $+1$ now might trap you somewhere that pays $-100$ forever after. So the quantity we actually care about is the whole future stream of rewards added up, starting right after the current moment $t$:

$G_t=R_{t+1}+R_{t+2}+R_{t+3}+\cdots$

• $G_t$ ::: the return — total future points collected from time $t$ onward
• each $R$ ::: one payout, one tick apart on the clock

PICTURE. Look at the timeline below. The agent sits at $t$ (the magenta dot). Every arrow forward drops a coin $R_{t+1},R_{t+2},\dots$ into the bucket labelled $G_t$.

---

Step 2 — Why discount? Introducing $\gamma$

WHAT. The naive sum $R_{t+1}+R_{t+2}+\cdots$ has a problem: if rewards keep coming forever, the sum can blow up to infinity, and infinity can't be compared to infinity. We fix this by shrinking each future reward a little more the further away it is, using a knob $\gamma$ (Greek letter "gamma"), a number between $0$ and $1$:

$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots ={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$

• $\gamma$ ::: the discount factor, how much we care about the next step's future ($0=$ ignore future, $1=$ future matters as much as now)
• ${\gamma }^k$ ::: a reward $k$ steps away is worth only ${\gamma }^k$ of its face value
• ${\sum }_{k=0}^{\infty }$ ::: "add up over all future steps $k=0,1,2,\dots$"

WHY $\gamma$ and not, say, subtracting a constant? Multiplying by $\gamma <1$ each step makes the coins geometrically smaller, so even an infinite stream adds to a finite number (a geometric series). Subtraction wouldn't guarantee that. It also captures a real idea: points sooner are worth more than points later.

PICTURE. The same coins as Step 1, but now each is drawn smaller — scaled by ${\gamma }^0,{\gamma }^1,{\gamma }^2,\dots$ — a shrinking staircase.

---

Step 3 — Why average? Introducing the policy $\pi$ and $\mathbb{E}$

WHAT. Two things are uncertain: (a) which action the agent picks, and (b) where the world sends it afterward. The rule for picking actions is the policy $\pi$. We write $\pi (a\mid s)$ = "the probability of choosing action $a$ when in state $s$."

Because outcomes are random, $G_t$ is a random number — different every time you play. So we take its expected value (its long-run average over many playthroughs), written ${\mathbb{E}}_{\pi }[\cdot ]$.

WHY average and not the best case? The agent doesn't get to pick the luckiest future; it gets the typical one under its own habits $\pi$. The honest "how good is here?" number is the mean return, not the max.

PICTURE. From one state, three possible futures fan out with probabilities $\pi (a\mid s)$. Each future has its own bucket $G_t$. The expectation is the probability-weighted blend of all buckets.

---

Step 4 — The one trick that makes it recursive: peel off the first reward

WHAT. Look again at the return. The very first coin $R_{t+1}$ is special — it's the only one not multiplied by any $\gamma$. Factor $\gamma$ out of everything after it:

$G_t=R_{t+1}+\gamma \left(\underset{\text{this is just }{G}_{t+1}}{\underbrace{R_{t+2}+\gamma R_{t+3}+\cdots }}\right)=R_{t+1}+\gamma G_{t+1}$

• $R_{t+1}$ ::: the immediate reward — the payout for this one step
• $\gamma G_{t+1}$ ::: the entire rest of the future, discounted once and viewed from the next state

WHY this peel? It turns an infinite sum into a two-piece object: now + (discounted) later. And "later" has exactly the same shape as the thing we started with — that self-similarity is what lets us write an equation for $V$ in terms of $V$ itself.

PICTURE. The coin staircase splits with a scissors cut after the first coin: a lone coin $R_{t+1}$, then the whole remaining staircase compressed into one box labelled $\gamma G_{t+1}$.

---

Step 5 — Where do you land next? The environment $p(s^{\prime },r\mid s,a)$

WHAT. After you pick action $a$ in state $s$, the world decides two things at once: the reward $r$ it pays, and the next state $s^{\prime }$ it drops you into. It does this randomly, with probability

$p(s^{\prime },r\mid s,a).$

• $s^{\prime }$ ::: the next state (read "s prime")
• $r$ ::: the reward that comes with that landing
• $p(s^{\prime },r\mid s,a)$ ::: probability the world sends you to $s^{\prime }$ paying $r$, given you did $a$ in $s$

WHY do we need this? In Step 4 we said "the rest of the future is viewed from the next state." But which next state? The environment answers that — and it may be random. To take the average return we must average over all the places $s^{\prime }$ we might land, weighted by $p$.

PICTURE. From the pair $(s,a)$ (a small square) the world branches to several $(s^{\prime },r)$ outcomes, each edge labelled with its probability $p$.

---

Step 6 — Assemble $Q$: immediate reward + discounted value of where you land

WHAT. Start from $Q^{\pi }(s,a)=\mathbb{E}[R_{t+1}+\gamma G_{t+1}\mid s,a]$ (Step 4's peel, applied to $Q$). Split the average into "now" and "later," then average both over the landings from Step 5. The key fact — the Markov property — is that once you know you landed in $s^{\prime }$, the future no longer cares how you got there, so $\mathbb{E}[G_{t+1}\mid S_{t+1}=s^{\prime }]$ is exactly $V^{\pi }(s^{\prime })$. This gives:

$Q^{\pi }(s,a)={\sum }_{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma V^{\pi }(s^{\prime })]$

Term by term, reading the picture:

• ${\sum }_{s^{\prime },r}$ ::: "average over every place you might land"
• $p(s^{\prime },r\mid s,a)$ ::: the weight (probability) of that landing
• $r$ ::: the immediate coin for that landing
• $\gamma V^{\pi }(s^{\prime })$ ::: the discounted value of all future coins, viewed from $s^{\prime }$

WHY split like this? Because the immediate reward $r$ is knowable now, but everything after landing is summarised by the number $V^{\pi }(s^{\prime })$ we already defined. We never re-simulate the whole future — we reuse $V$. That reuse is called bootstrapping.

PICTURE. Each branch from Step 5 now carries a label $[r+\gamma V^{\pi }(s^{\prime })]$; multiply by $p$, add the branches.

---

Step 7 — Assemble $V$: average $Q$ over the policy's action choices

WHAT. In state $s$ the policy picks an action randomly with weights $\pi (a\mid s)$. Each possible action has its own quality $Q^{\pi }(s,a)$. The state's value is the probability-weighted blend:

$V^{\pi }(s)={\sum }_a\pi (a\mid s)Q^{\pi }(s,a)$

• ${\sum }_a$ ::: "average over every action you might take"
• $\pi (a\mid s)$ ::: the weight of action $a$ under your policy
• $Q^{\pi }(s,a)$ ::: how good that action is (Step 6)

WHY this exact average? $V$ is "how good is the state," and a state is only ever experienced through the actions you choose there. The honest answer is the mean over those choices, weighted by how often you make them.

PICTURE. A hub $s$ with action-spokes; each spoke has thickness $\pi (a\mid s)$ and a value tag $Q^{\pi }(s,a)$. The blended result flows back into $s$.

---

Step 8 — Substitute and get the Bellman expectation equation

WHAT. Now nest Step 6 inside Step 7. Replace each $Q^{\pi }(s,a)$ in the $V$-average with its immediate-plus-discounted form:

$\boxed{V^{\pi }(s)=\sum _a\pi (a\mid s)\sum _{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma V^{\pi }(s^{\prime })]}$

Reading the two layers:

• outer sum ${\sum }_a\pi (a\mid s)$ ::: average over my action choices
• inner sum ${\sum }_{s^{\prime },r}p$ ::: average over the world's landings
• $r+\gamma V^{\pi }(s^{\prime })$ ::: this step's coin plus the discounted value of next step's whole future

WHY is this profound? $V^{\pi }(s)$ appears on the left and $V^{\pi }(s^{\prime })$ appears on the right. The value of a state is pinned down by the values of its neighbours. Solving this self-consistent web is what policy evaluation does exactly, Monte-Carlo estimates by sampling whole returns, and temporal-difference learning approximates by nudging $V(s)$ toward $r+\gamma V(s^{\prime })$.

PICTURE. The full two-layer tree: state $s$ → action layer (policy weights) → outcome layer (environment weights) → successor states $s^{\prime }$, each feeding its own $V$ back up.

---

Step 9 — Edge & degenerate cases (never let the reader hit an unshown scenario)

WHAT. Three boundary situations that break naive intuition:

1. Terminal (goal) state. There is no future, so $V^{\pi }(\text{terminal})=0$. The sum has no successors; the recursion stops here. This is the anchor that makes back-substitution work in the gridworld below.
2. $\gamma =0$. The future term $\gamma V^{\pi }(s^{\prime })$ vanishes entirely: $V^{\pi }(s)={\sum }_a\pi (a\mid s){\sum }_{s^{\prime },r}p\cdot r$ — pure immediate reward, a myopic agent.
3. Deterministic transition. If action $a$ always leads to one $s^{\prime }$ with one $r$, then $p(s^{\prime },r\mid s,a)=1$ and the inner sum collapses to a single term: $Q^{\pi }(s,a)=r+\gamma V^{\pi }(s^{\prime })$. No averaging needed.

WHY show these? Every real computation lives in one of these regimes. Miss the terminal case and your recursion never bottoms out (infinite loop). Miss $\gamma =0$ and you can't explain a myopic agent.

PICTURE. Three mini-panels: a terminal state with a flat $V=0$ tag; a $\gamma =0$ tree with the future branch greyed out; a deterministic single-arrow transition.

---

The one-picture summary

The whole derivation in one diagram: from state $s$, the policy chooses an action (magenta weights $\pi (a\mid s)$), the environment chooses a landing (orange weights $p$), each landing pays $r$ and hands you the discounted value $\gamma V^{\pi }(s^{\prime })$ of the next state's future — and averaging both layers gives back $V^{\pi }(s)$.

Recall Feynman retelling — say it like you'd explain to a friend

Imagine standing on a tile in a game. You want one number that says "how good is standing here, playing the way I usually play?" That number is $V$. To get it, you ask two questions and take averages. First: "what might I do?" — you don't always pick the same move, so you average over your habits, weighting each move by how often you make it. Second, for each move: "where might the world throw me, and what does it pay?" — again random, so you average over landings, weighting by their probabilities. On each landing you collect the immediate coin $r$, and you also get "how good is the new tile," shrunk by $\gamma$ because it's one step in the future. Add coin plus shrunken next-tile-value, blend over the world's dice, blend over your own dice — and you're back to the value of the tile you started on. That circular sentence is the Bellman equation. And it terminates because a goal tile has no future, so its value is just zero — the wall the whole recursion leans against.

---

Quick self-check