Visual walkthrough — Bellman equations

This is the visual companion to Bellman equations. We build the state-value Bellman expectation equation — the parent's central boxed result — from absolute scratch.

---

Step 1 — What is an agent, a state, and a reward?

WHAT: Look at the board. You sit on a square, pick an arrow, slide to a new square, and a number pops out.

WHY: Everything else is built from this loop: state → action → reward → new state. If we cannot picture this single tick of time, no equation later will mean anything.

PICTURE: In the figure, the chalk-blue dot is you in state $s$. The pale-yellow arrow is one action $a$. The pink number $r$ is the reward the move pays. The dot lands in a new state we will call $s^{\prime }$ (read "s-prime", just "the next state").

---

Step 2 — The return: adding up all future reward

WHAT: One reward is not enough. We care about the whole future stream of rewards. Stack them:

$G=R_1+R_2+R_3+R_4+\cdots$

Here $R_1$ is the reward from the first move, $R_2$ from the second, and so on forever. The capital letter $G$ is the return — the grand total.

WHY: A move that pays $+1$ now but traps you in a $-100$ pit later is a bad move. Only the total tells the truth, so we must sum the whole tail.

PICTURE: The figure is a chalk timeline. Each tick drops one reward brick $R_1,R_2,R_3,\dots$ into a growing stack on the right. The stack never stops — that is the problem we fix next.

---

Step 3 — The discount factor: why far-off reward counts less

$G=R_1+\gamma R_2+{\gamma }^2{R}_3+{\gamma }^3{R}_4+\cdots$

Term by term:

• $R_1$ — this step's reward, full strength (multiplied by ${\gamma }^0=1$).
• $\gamma R_2$ — next step's reward, shrunk once.
• ${\gamma }^2{R}_3$ — two steps away, shrunk twice.

WHY this tool and not a plain sum? Two reasons, both visible in the picture. (1) A shrinking geometric stack has a finite total even if it runs forever — so values become comparable. (2) It bakes in "reward now beats reward later," which matches real impatience. See Discount Factor.

PICTURE: Same timeline as Step 2, but each brick is now scaled by a shrinking pale-yellow bar ${\gamma }^0,{\gamma }^1,{\gamma }^2,\dots$. The bricks visibly melt toward zero, so the total stack has a ceiling (the dashed line).

Recall

Why does $\gamma <1$ make the infinite sum finite? ::: Because $1+\gamma +{\gamma }^2+\cdots =\frac{1}{1-\gamma }$, a finite number whenever $\gamma <1$. What does $\gamma =0$ mean in words? ::: Only the immediate reward matters; the agent is fully short-sighted.

---

Step 4 — Expectation: the world rolls dice

WHAT: The same action can land you in different squares (a windy world, a random policy). So the return $G$ is not one fixed number — it's a bag of possible totals, each with a probability. We take the average of that bag.

WHY the average? We must commit to a single "how good is this state?" number before the dice are rolled. The fair single number is the probability-weighted mean — that is exactly $\mathbb{E}$.

PICTURE: From state $s$ the figure fans out three chalk paths with probabilities $0.5,0.3,0.2$ and totals $G=4,G=1,G=-2$. The pink dashed line marks the weighted average $\mathbb{E}[G]=0.5(4)+0.3(1)+0.2(-2)=1.9$.

Now we can finally name our quantity:

The letter $\pi$ here is just a name for a strategy; see Markov Decision Process for the full setup.

---

Step 5 — The one clever cut: peel off the first reward

WHAT: Split the discounted stack into the first brick plus everything after it:

$G=\underset{\text{now}}{\underbrace{R_1}}+\underset{\text{the future}}{\underbrace{\gamma R_2+{\gamma }^2{R}_3+\cdots }}$

Now factor one $\gamma$ out of the whole future group:

$G=R_1+\gamma \left(\underset{\text{this is just the return, restarted at step 2}}{\underbrace{R_2+\gamma R_3+{\gamma }^2{R}_4+\cdots }}\right)$

WHY this exact cut? Stare at the parenthesis. It is a return again — the very same shape as $G$, only starting one step later, from the next state. That self-similarity is the whole trick: the future-from-here contains a smaller copy of the same problem.

PICTURE: The figure snaps the reward stack into two chalk boxes: a yellow box holding lone $R_1$, and a blue box holding the rest — and the blue box is drawn as a miniature copy of the entire original stack, sitting on the next state $s^{\prime }$.

---

Step 6 — Fold the tail into $V^{\pi }(s^{\prime })$

WHAT: Take the expectation of both sides of the Step 5 cut. Because the parenthesis is the return starting from the next state $S_1$, its expectation is $V^{\pi }(S_1)$:

$V^{\pi }(s)={\mathbb{E}}_{\pi }[R_1\mid S_0=s]+\gamma {\mathbb{E}}_{\pi }[V^{\pi }(S_1)\mid S_0=s]$

Term by term:

• $\mathbb{E}[R_1\mid S_0=s]$ — average reward for the first move out of $s$.
• $\gamma$ — shrink the future by one step.
• $\mathbb{E}[V^{\pi }(S_1)]$ — average value of wherever we land.

WHY: We replaced an infinite sum with a finite expression that mentions $V^{\pi }$ once on the left and once on the right. The infinity is gone; a self-referencing rule remains.

PICTURE: The figure shows the equation as a chalk machine: $s$ enters, the yellow box emits one reward, the blue box (labelled $V^{\pi }(s^{\prime })$) feeds its own value back in scaled by $\gamma$ — an arrow literally curving back on itself.

---

Step 7 — Open the expectation: sum over actions and outcomes

WHAT: An expectation is just "each outcome times its probability, added up." Two dice are rolled: the policy picks the action ($\pi (a\mid s)$), then the world picks where you land and what you're paid ($p(s^{\prime },r\mid s,a)$). Sum over both:

$\boxed{V^{\pi }(s)=\sum _a\pi (a\mid s)\sum _{s^{\prime },r}p(s^{\prime },r\mid s,a)[r+\gamma V^{\pi }(s^{\prime })]}$

Reading it left to right, exactly where each symbol sits:

• ${\sum }_a\pi (a\mid s)$ — average over your choices: for every action $a$, weight it by how often policy $\pi$ picks it.
• ${\sum }_{s^{\prime },r}p(s^{\prime },r\mid s,a)$ — average over the world's dice: for every possible landing spot $s^{\prime }$ and payout $r$, weight by its probability.
• $r$ — the immediate payout of that branch.
• $\gamma V^{\pi }(s^{\prime })$ — the discounted value of that landing spot.

WHY split into two sums? Because two independent random choices happen: yours ($\pi$) and the environment's ($p$). Each needs its own weighted average, nested.

PICTURE: A two-layer chalk tree. Layer 1: from $s$, one yellow branch per action, labelled with $\pi (a\mid s)$. Layer 2: each action fans into blue branches labelled $p(s^{\prime },r\mid s,a)$, each leaf carrying $r+\gamma V^{\pi }(s^{\prime })$.

This is the parent's central boxed result. It also seeds Dynamic Programming (solve it iteratively) and Temporal Difference Learning (solve it from samples).

---

Step 8 — Edge and degenerate cases

WHAT & WHY: A derivation you can trust must survive its extreme inputs. Four corners:

1. Terminal state. A terminal state has no future — it never moves. So $V^{\pi }(s_{\text{term}})=0$. The blue future box in the figure is empty; only the reward-on-entry survives.
2. $\gamma =0$ (fully short-sighted). The $\gamma V^{\pi }(s^{\prime })$ term vanishes: $V^{\pi }(s)={\sum }_a\pi (a\mid s){\sum }_{s^{\prime },r}p[r]$ — pure one-step reward. The self-loop arrow is cut.
3. Deterministic world. Each inner sum collapses to a single branch (probability $1$): $V^{\pi }(s)={\sum }_a\pi (a\mid s)[r+\gamma V^{\pi }(s^{\prime })]$. The blue fan becomes one twig.
4. Greedy (deterministic) policy. The outer ${\sum }_a\pi (a\mid s)$ collapses to the one chosen action: the yellow fan becomes a single stalk — the doorway to the optimality equations and to Q-Learning.

PICTURE: Four small chalk panels, each showing which part of the Step-7 tree gets pruned in that corner (pruned branches drawn as faded dashed chalk).

Recall

In case 1 (terminal), what is the value and why? ::: Zero — a terminal state has no future rewards to discount. Which sum disappears when the policy is deterministic (greedy)? ::: The outer sum over actions, ${\sum }_a\pi (a\mid s)$ — only one action has probability 1.

---

Step 9 — Numbers on the parent's grid-world

We reuse the parent's Example 1: uniform policy (each of 4 actions has $\pi =0.25$), $r=-1$ per step, $\gamma =0.9$, and by symmetry the two neighbours share value $V$; two actions bump walls and stay in $s_1$.

$V^{\pi }(s_1)=0.25[-1+0.9V]+0.25[-1+0.9V]+0.25[-1+0.9V^{\pi }(s_1)]+0.25[-1+0.9V^{\pi }(s_1)]$

Collect terms:

$V^{\pi }(s_1)=-1+0.9[0.5V+0.5V^{\pi }(s_1)]\Rightarrow 0.55V^{\pi }(s_1)=-1+0.45V$

If we take $V=-5.6$ (a converged neighbour value), then $V^{\pi }(s_1)=\frac{-1+0.45(-5.6)}{0.55}=-6.40$.

Bellman-optimality flavour (Example 2): with $r=5$, $\gamma =0.9$, ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })=12$, learning rate $\alpha =0.1$, starting from $Q(s,a)=10$:

$\text{target}=5+0.9(12)=15.8,Q\leftarrow 10+0.1(15.8-10)=10.58$

---

The one-picture summary

The whole page in a single chalk diagram: state $s$ pays an immediate reward $r$ (yellow), then discounts the value of the next state $s^{\prime }$ by $\gamma$ (blue) — and that next value obeys the same rule, so the arrow loops back into itself. Infinite future, one-step law.

Recall Feynman retelling — say it like you'd explain to a friend

Imagine you're standing on a square and you want a single grade for "how good is standing here?" The honest grade is all the reward you'll ever collect from here on, but future reward matters a bit less each step, so you shrink it with $\gamma$. That total looks infinite and scary. Then you notice a trick: peel off just the first reward, and everything left over is the exact same kind of grade, only for the square you land on next. So "value here" = "reward now" + "$\gamma$ times value of the next square." Because the world and your own choices are both random, you average over every action you might pick and every place you might land. That's the Bellman equation — a rule that quietly points back at itself, which is why a computer can solve it by keeping estimates and improving them until they stop changing.

---

Connections

• Parent: Bellman equations · 5.1.05 Bellman equations (Hinglish)
• Uses: Discount Factor · Bootstrapping · Markov Decision Process
• Leads to: Dynamic Programming · Temporal Difference Learning · Q-Learning · Policy Gradient Methods