Worked examples — State-value and action-value functions

This page is a drill. The parent note built the tools $V^{\pi }(s)$ and $Q^{\pi }(s,a)$. Here we throw every kind of situation at them — every sign of reward, every degenerate policy, the two limits $\gamma =0$ and $\gamma \to 1$, a word problem, and an exam trap. If you can do all ten, no gridworld can surprise you.

Before we start, one line of vocabulary insurance so nothing here is assumed:

The two engines we will crank over and over. Notice the $\pi$ superscript is never optional: it names which policy's values we mean, and every $V$ and $Q$ on this page carries it.

---

The scenario matrix

Every value-function problem is really a combination of a few knobs. Here is the full board. Each of the ten examples below is stamped with the cell it lives in.

#	Cell class	The twist it tests	Example
C1	Deterministic transition, negative reward	value goes down as goal recedes	Ex 1
C2	Stochastic transition, mixed-sign rewards	probability-weighting outcomes	Ex 2
C3	Stochastic policy, known $Q$	$V$ as a weighted average	Ex 3
C4	$\gamma =0$ (myopic limit)	future collapses to zero	Ex 4
C5	$\gamma \to 1$ (far-sighted limit)	infinite sum must still converge	Ex 5
C6	Terminal / degenerate state	$V=0$, the recursion's floor	Ex 6
C7	Self-loop (agent can get stuck)	geometric series inside one state	Ex 7
C8	$V\to Q$ round trip	consistency check both directions	Ex 8
C9	Real-world word problem	translating English into $p,r,\gamma$	Ex 9
C10	Exam twist: "best $Q$ ≠ $V$"	why $V^{\pi }<{\max }_a{Q}^{\pi }$	Ex 10

---

[!example] Example 1 — Cell C1: deterministic path, negative rewards

A $3\times 3$ grid. We label each cell by $(\text{row},\text{col})$ using zero-based indexing: the top-left cell is $(0,0)$, rows increase downward, columns increase rightward. So the top-right corner is $(0,2)$ and the exact centre is $(1,1)$. The goal is the top-right corner $(0,2)$; each step costs $r=-1$; reaching the goal is terminal. $\gamma =0.9$. The policy is deterministic: "go right if you can, else go up." We want $V^{\pi }(1,1)$ (the centre), given it is two moves from the goal.

Forecast: guess — will $V^{\pi }(1,1)$ be positive or negative, and roughly how big?

The figure below draws this grid. The green cell (top-right) is the goal with $V^{\pi }=0$; the blue cell is our start $(1,1)$; the orange cell is the intermediate $(1,2)$. The two red arrows are the forced policy path, each labelled with its $r=-1$ cost.

The key insight the figure carries: the red arrows point toward the goal — that is the direction the agent moves — but the numbers printed in the cells get filled in the opposite direction, starting from the green cell and flowing back to the blue one. Value and motion run in opposite directions. That is the single most important habit in all of dynamic programming: you can only compute a state's value once you already know the value of everything downstream of it, so you always start at the terminal cell (where value is known to be $0$) and work backward against the arrows.

1. From $(1,1)$ the policy moves right to $(1,2)$ (the blue-to-orange red arrow), collecting $r=-1$. Why this step? $V^{\pi }$ under a deterministic policy is just the value of the single path it forces — no averaging needed, so we walk the path.
2. From $(1,2)$ the policy moves up to the goal $(0,2)$ (the orange-to-green red arrow), collecting $r=-1$; the goal is terminal so $V^{\pi }(0,2)=0$. Why this step? Terminal states have no future, so their value is the recursion's floor (see C6).
3. Back-substitute from the goal outward (exactly the backward flow the figure highlights): $V^{\pi }(1,2)=-1+0.9\cdot 0=-1$ $V^{\pi }(1,1)=-1+0.9\cdot (-1)=-1.9$ Why this step? The Bellman equation propagates value backward from where reward is known.

Verify: two steps, each $-1$, second one discounted once: $-1+0.9(-1)=-1.9$. Sign is negative (every move is a cost) and magnitude $<2$ (discount shrinks the far cost). Matches the forecast if you guessed "a bit less than $-2$." ✓

---

[!example] Example 2 — Cell C2: risky vs safe action, mixed signs

State $s$, two actions. $\gamma =1$ (undiscounted).

• $a_1$ (safe): $p=1$ → $s^{\prime }$, $r=1$, with $V^{\pi }(s^{\prime })=10$.
• $a_2$ (risky): $p=0.7$ → $s^{\prime \prime }$, $r=5$, $V^{\pi }(s^{\prime \prime })=15$; $p=0.3$ → $s^{\prime \prime \prime }$, $r=-3$, $V^{\pi }(s^{\prime \prime \prime })=2$.

Forecast: which action has the higher $Q^{\pi }$? The safe one feels comfortable — trust that feeling?

1. Safe action collapses to one term (probability 1): $Q^{\pi }(s,a_1)=1\cdot [1+1\cdot 10]=11$ Why this step? When $p(s^{\prime },r\mid s,a)=1$ the sum ${\sum }_{s^{\prime },r}$ has a single surviving term.
2. Risky action weights each branch by its $p$: $Q^{\pi }(s,a_2)=0.7[5+15]+0.3[-3+2]=0.7(20)+0.3(-1)=14-0.3=13.7$ Why this step? Expectation = probability-weighted sum of outcomes; the $-3$ reward is real but rare, so it barely dents the total.

Verify: $Q^{\pi }(s,a_2)=13.7>Q^{\pi }(s,a_1)=11$. The risky action wins on expectation even though it can go badly. Sanity: $13.7$ sits between the good branch total ($20$) and the bad branch total ($-1$), closer to $20$ because that branch is more likely. ✓ This is exactly the seed of policy improvement.

---

[!example] Example 3 — Cell C3: $V$ from $Q$ under a stochastic policy

State $s$ with $Q^{\pi }(s,a_1)=5$, $Q^{\pi }(s,a_2)=8$, $Q^{\pi }(s,a_3)=3$. Policy $\pi (a_1)=0.5,\pi (a_2)=0.3,\pi (a_3)=0.2$.

Forecast: $a_2$ is best at $8$. Will $V^{\pi }(s)$ equal $8$? Above it? Below it?

1. Apply $V^{\pi }(s)={\sum }_a\pi (a\mid s)Q^{\pi }(s,a)$: $V^{\pi }(s)=0.5(5)+0.3(8)+0.2(3)$ Why this step? $V^{\pi }$ is what you actually get by following $\pi$, and $\pi$ is a lottery over actions — so average the $Q^{\pi }$s with the policy's odds.
2. Add up: $2.5+2.4+0.6=5.5$.

Verify: the weights sum to $0.5+0.3+0.2=1$ (a valid distribution), and $5.5$ lies strictly between the worst $Q^{\pi }$ ($3$) and the best $Q^{\pi }$ ($8$). A weighted average must fall inside that range — it does. ✓

---

[!example] Example 4 — Cell C4: the myopic limit $\gamma =0$

Same setup as Example 2, but now $\gamma =0$. Recompute both $Q^{\pi }$s.

Forecast: with $\gamma =0$ the future term vanishes. What survives?

1. Plug $\gamma =0$ into the $Q^{\pi }$ formula — the $\gamma V^{\pi }(s^{\prime })$ term dies: $Q^{\pi }(s,a_1)=1\cdot [1+0\cdot 10]=1$ Why this step? $\gamma =0$ literally means "ignore everything after the next reward."
2. Risky action, immediate rewards only: $Q^{\pi }(s,a_2)=0.7(5)+0.3(-3)=3.5-0.9=2.6$

Verify: now the risky action still wins ($2.6>1$) but the margin shrank because the landing values (10, 15, 2) no longer count. A myopic agent judges by immediate reward alone. Sanity: $2.6$ is exactly the expected immediate reward $0.7\cdot 5+0.3\cdot (-3)$. ✓

---

[!example] Example 5 — Cell C5: the far-sighted limit $\gamma \to 1$ on an infinite reward stream

An agent sits in a chain and receives a constant reward $r=2$ forever (no terminal state). The return is $G=2+2\gamma +2{\gamma }^2+\dots$. Compute $V^{\pi }$ for $\gamma =0.9$, and describe $\gamma \to 1$.

Forecast: an infinite sum of $2$s — does it blow up, or settle to a finite number?

1. Recognise a geometric series: $V^{\pi }={\sum }_{k=0}^{\infty }2{\gamma }^k=\frac{2}{1-\gamma }(\gamma <1).$ Why this step? This is why we discount: $\gamma <1$ makes the tail shrink fast enough that the infinite sum converges to a finite value.
2. Evaluate at $\gamma =0.9$: $V^{\pi }=\frac{2}{1-0.9}=\frac{2}{0.1}=20$.
3. The limit $\gamma \to 1$: $\frac{2}{1-\gamma }\to \infty$. A never-ending stream of positive reward has infinite value once we stop discounting. Why this step? It shows the boundary case: $\gamma$ strictly less than $1$ is what keeps infinite-horizon values finite.

Verify: $2/0.1=20$. Cross-check the first few terms: $2+1.8+1.62+1.458+\dots$ climbs toward $20$ and never passes it. ✓

---

[!example] Example 6 — Cell C6: terminal / degenerate state

The goal cell in Example 1. No actions lead anywhere; the episode ends.

Forecast: what is $V^{\pi }(\text{terminal})$?

1. A terminal state has no future rewards, so the return from it is the empty sum: $V^{\pi }(s_{\text{terminal}})=0.$ Why this step? Every recursion needs a base case. This is it — without it, the Bellman back-substitution in Example 1 would never bottom out.

Verify: substitute $V^{\pi }=0$ back into the goal's Bellman line: $0=0$. Consistent. This zero is what made Example 1's $V^{\pi }(1,2)=-1$ come out clean. ✓

---

[!example] Example 7 — Cell C7: a self-loop (the agent can get stuck)

State $s$ has one action. With probability $0.8$ it stays in $s$ (self-loop, reward $r=1$); with probability $0.2$ it exits to a terminal state ($r=10$, then value $0$). $\gamma =1$. Find $V^{\pi }(s)$.

Forecast: the loop keeps paying $1$ per stay. Since $\gamma =1$, will this diverge — or does the escape probability save us?

The figure draws the two-node world: the blue circle is $s$ (with its printed answer $V^{\pi }=14$), the curved orange arrow is the self-loop $p=0.8,r=1$, and the straight red arrow is the escape $p=0.2,r=10$ into the green terminal node.

The key insight the figure carries: notice the orange arrow leaves $s$ and comes right back to $s$. That returning arrow is the visual reason $V^{\pi }(s)$ shows up on both sides of its own equation below — you cannot "walk the path" like in Example 1 because the path folds back on itself. The escape arrow (red, weight $0.2$) is the only way out, so intuitively the agent stays for about $1/0.2=5$ visits on average before leaving; the loop is finite precisely because that red exit always eventually fires. A picture with no red arrow would be a genuinely infinite loop with (for $\gamma =1$) infinite value.

1. Write the Bellman equation with $s$ on both sides (the orange self-loop term keeps $V^{\pi }$ on the right): $V^{\pi }(s)=0.8[1+1\cdot V^{\pi }(s)]+0.2[10+1\cdot 0].$ Why this step? A self-loop puts $V^{\pi }(s)$ inside its own equation — that is legal and we solve for it algebraically.
2. Expand and collect $V^{\pi }(s)$: $V^{\pi }(s)=0.8+0.8V^{\pi }(s)+2\Rightarrow V^{\pi }(s)-0.8V^{\pi }(s)=2.8\Rightarrow 0.2V^{\pi }(s)=\mathrm{2.8.}$ Why this step? We gather every $V^{\pi }(s)$ onto one side so the single unknown can be isolated — the standard move for a linear equation where the unknown appears twice.
3. Solve: $V^{\pi }(s)=2.8/0.2=14$. Why this step? Dividing by the coefficient $0.2$ isolates $V^{\pi }(s)$; that $0.2$ escape probability acts like an effective discount, guaranteeing the loop ends eventually, so even with $\gamma =1$ the value is finite.

Verify: plug $V^{\pi }(s)=14$ back: $0.8(1+14)+0.2(10)=0.8(15)+2=12+2=14$. ✓ (Intuition: about $1/0.2=5$ expected stays plus the exit reward.)

---

[!example] Example 8 — Cell C8: the $V\to Q\to V$ round trip

State $s$ with policy $\pi (a_1)=0.6,\pi (a_2)=0.4$. We are told $Q^{\pi }(s,a_1)=4$ and $Q^{\pi }(s,a_2)=9$. Compute $V^{\pi }(s)$, then reconstruct both action-values from a hypothetical one-step model to confirm nothing contradicts.

Forecast: will the round trip land back on the same number for every action, not just one?

1. Down direction, $V^{\pi }$ from $Q^{\pi }$: $V^{\pi }(s)=0.6(4)+0.4(9)=2.4+3.6=\mathrm{6.0.}$ Why this step? This is the averaging equation — the canonical way to collapse action-values into a state-value.
2. Up direction for $a_1$ (consistency): suppose $a_1$ deterministically ($p=1$) leads to a state $s_1^{\prime }$ with $V^{\pi }(s_1^{\prime })=3$, $r=1$, $\gamma =1$. Then $Q^{\pi }(s,a_1)=1\cdot [1+1\cdot 3]=4.$ Why this step? We rebuild the given $Q^{\pi }(s,a_1)=4$ from the one-step formula.
3. Up direction for $a_2$ (full check): suppose $a_2$ deterministically leads to a state $s_2^{\prime }$ with $V^{\pi }(s_2^{\prime })=6$, $r=3$, $\gamma =1$. Then $Q^{\pi }(s,a_2)=1\cdot [3+1\cdot 6]=9.$ Why this step? Reconstructing only $a_1$ would leave $a_2$ untested; the Bellman equation must hold for every action, so we verify $a_2$ too before trusting the model.

Verify: both reconstructed values match their givens ($4$ and $9$), and $V^{\pi }(s)=6.0$ lies between $4$ and $9$ as any convex combination must. ✓ This two-way traffic is exactly what policy evaluation iterates.

---

[!example] Example 9 — Cell C9: real-world word problem

A delivery drone in "Charging" state can, each minute: (i) keep charging — 90% it stays Charging (reward $0$), 10% it becomes Ready (reward $0$); or (ii) launch early — always goes to Flying, but reward $-5$ for low battery. Once Ready, launching gives reward $+20$ and ends the task (terminal). $\gamma =0.9$. The policy always charges until Ready, then launches. What is the value of the Charging state?

Forecast: patience (waiting to charge) should beat launching early — will the value be near $+20$ or dragged down by the wait?

1. Value of Ready under this policy: launch → terminal, so $V^{\pi }(\text{Ready})=+20+0.9\cdot 0=20.$ Why this step? Ready leads straight to the big reward; it's the anchor we back up from.
2. Bellman for Charging (self-loop with escape to Ready): $V^{\pi }(\text{Charging})=0.9[0+0.9V^{\pi }(\text{Charging})]+0.1[0+0.9\cdot 20].$ Why this step? We translate the English probabilities into $p(s^{\prime },r\mid s,a)$: the $0.9$ stay-loop and the $0.1$ escape, each carrying the discount $\gamma =0.9$.
3. Solve for $V^{\pi }(\text{Charging})$ (write $V_C$ for short): $V_C=0.81V_C+0.1(18)\Rightarrow V_C(1-0.81)=1.8\Rightarrow 0.19V_C=\mathrm{1.8.}$ $V_C=\frac{1.8}{0.19}=\frac{180}{19}=9.4736\dots \approx \mathrm{9.47.}$ Why this step? The self-loop again puts $V_C$ on both sides; we isolate it. The exact value is the fraction $180/19$; we round to $9.47$ only for reporting, and we keep $180/19$ for the check so no rounding error creeps in.

Verify: plug the exact $V_C=180/19$ back into step 2: $0.9(0.9\cdot \frac{180}{19})+0.1(18)=0.81\cdot \frac{180}{19}+1.8=\frac{145.8}{19}+\frac{34.2}{19}=\frac{180}{19}=V_C.✓$ The value $9.47$ is well below $20$: the drone must survive many discounted, reward-less charging minutes before cashing in.

---

[!example] Example 10 — Cell C10: the exam trap "best $Q^{\pi }$ ≠ $V^{\pi }$"

Given $Q^{\pi }(s,a_1)=5$, $Q^{\pi }(s,a_2)=8$, $Q^{\pi }(s,a_3)=3$ and a policy $\pi (a_1)=0.5,\pi (a_2)=0.3,\pi (a_3)=0.2$ (same numbers as Example 3). A student writes "$V^{\pi }(s)=8$ because $a_2$ is the best action." Diagnose and correct.

Forecast: what is the one word in the policy that breaks the student's claim?

1. Spot the error: the student used ${\max }_a{Q}^{\pi }$, but $V^{\pi }$ averages over the policy's actual action probabilities, not the best one. Why this step? $V^{\pi }(s)={\max }_a{Q}^{\pi }(s,a)$ holds only for the optimal/greedy policy — see optimal value functions. Here $\pi$ is stochastic, so it isn't greedy.
2. Correct computation (from Example 3): $V^{\pi }(s)=0.5(5)+0.3(8)+0.2(3)=5.5$. Why this step? We follow the actual policy weights; $a_2$'s $8$ only counts $30\%$ of the time.
3. State the general fact: $V^{\pi }(s)={\sum }_a\pi (a\mid s)Q^{\pi }(s,a)\le {\max }_a{Q}^{\pi }(s,a),$ with equality iff $\pi$ puts all its probability on a maximising action. Why this step? A weighted average of numbers can never exceed the largest of them — that inequality is the whole lesson.

Verify: $5.5\le 8$, and $5.5=8$ would require $\pi (a_2)=1$ — which it isn't. The gap $8-5.5=2.5$ is the price of a policy that sometimes chooses $a_1$ or $a_3$. ✓

---

Recall Quick self-test

How does $V^{\pi }(s)$ under a stochastic policy relate to ${\max }_a{Q}^{\pi }(s,a)$? ::: $V^{\pi }(s)\le {\max }_a{Q}^{\pi }(s,a)$, with equality only when $\pi$ is greedy on a maximiser. Why does a $\gamma =1$ self-loop still give a finite value in Ex 7? ::: The $0.2$ escape probability guarantees eventual exit, acting like an effective discount. What is $V^{\pi }$ of any terminal state? ::: $0$ — the base case of the Bellman recursion. Sum of a constant reward $r$ forever with discount $\gamma$? ::: $r/(1-\gamma )$, which diverges as $\gamma \to 1$. What does $p(s^{\prime },r\mid s,a)$ stand for? ::: The world's rulebook: probability of landing in $s^{\prime }$ with reward $r$ after doing $a$ in $s$.

Next stops that use these values: estimating them from samples in Monte-Carlo prediction and temporal-difference learning.