Worked examples — Policies and value functions

This page is the drill ground for the parent topic. We do not re-derive the theory here — we stress-test it. Every kind of situation a value-function problem can throw at you gets its own fully-worked example, and we label each with the exact cell of the scenario matrix it covers.

If any symbol below feels unfamiliar, that means you skipped ahead — go read the parent note and 5.1.02-Markov-Decision-Processes first, because we assume you already met:

• a state $s$ (the situation the agent is in),
• an action $a$ (a choice the agent makes),
• a reward $r$ (a number the environment hands back),
• the discount $\gamma$ (how much we shrink future rewards, a number between $0$ and $1$),
• the policy $\pi (a\mid s)$ (probability of choosing action $a$ in state $s$),
• and the two value functions $V^{\pi }(s)$ and $Q^{\pi }(s,a)$.

---

The scenario matrix

Think of every value-function problem as living in a grid. The rows are the structural traps — the shapes an MDP can take. If you can solve one example in every row, nothing on an exam can surprise you.

#	Cell (what makes it tricky)	Where it bites	Covered by
A	Absorbing / terminal state (you get stuck and keep collecting)	Geometric series, $\frac{r}{1-\gamma }$	Ex 1
B	Self-loop with a mix (partly stay, partly leave)	Coupled linear equations	Ex 2
C	Multiple actions → compute $Q$, pick greedy	$Q$ vs $V$, $\arg \max$	Ex 3
D	Stochastic policy (agent flips a coin)	Weighted average over actions	Ex 4
E	Degenerate: $\gamma =0$ (total impatience)	Value collapses to immediate reward	Ex 5
F	Limiting: $\gamma \to 1$ (perfect patience)	Divergence vs finite horizon	Ex 6
G	Negative rewards / a "trap" state	Signs, avoiding bad states	Ex 7
H	Optimal policy $V^{\ast }$ with a $\max$	Bellman optimality, not expectation	Ex 8
I	Real-world word problem	Translating English → MDP	Ex 9
J	Exam twist: which action does ${\pi }^{\ast }$ pick?	Reading off $\arg {\max }_a{Q}^{\ast }$	Ex 10

We reuse a tiny running world so the numbers stay checkable by hand, and change one knob per example so you see exactly what each knob does.

---

Warm-up geometry: what a value even is

Before the examples, one picture. The value of a state is a sum of shrinking rewards stretching into the future. The discount $\gamma$ is what makes each later reward count for less.

Recall Why does

$\frac{r}{1-\gamma }$ work — where does it come from? Let $S=r+\gamma r+{\gamma }^{2}r+\cdots$. Multiply by $\gamma$: $\gamma S=\gamma r+{\gamma }^{2}r+\cdots$. Subtract: $S-\gamma S=r$, so $S(1-\gamma )=r$, giving $S=\frac{r}{1-\gamma }$. It converges only because $\gamma <1$. ::: The trick is "subtract a shifted copy of the same series."

---

Ex 1 — Cell A: the absorbing state

Forecast: Guess a number before reading. Is it $2$? $20$? $200$? Write it down.

1. Write the Bellman expectation equation. Since the reward is deterministic ($+2$) and the next state is always $s_2$ (i.e. $p(s_2\mid s_2,a)=1$): $V^{\pi }(s_2)=2+0.9V^{\pi }(s_2).$ Why this step? An absorbing state feeds into itself, so its value appears on both sides — this is what makes it solvable directly.

2. Solve the linear equation. $V^{\pi }(s_2)-0.9V^{\pi }(s_2)=2\Rightarrow 0.1V^{\pi }(s_2)=2\Rightarrow V^{\pi }(s_2)=20.$ Why this step? Collecting the $V$ terms turns "value defined by itself" into ordinary algebra.

Verify: Use the geometric formula directly: $\frac{r}{1-\gamma }=\frac{2}{1-0.9}=\frac{2}{0.1}=20.$ ✓ Both roads agree. (If you guessed $2$, you forgot the forever; if $200$, you over-shrunk.)

---

Ex 2 — Cell B: a self-loop that partly leaves

Forecast: $s_1$ pays less immediately (+1 vs +2) and sometimes traps you in the low-reward room. So expect $V^{\pi }(s_1)$ below $20$. How far below?

1. Write the Bellman equation, averaging over the two next states. $V^{\pi }(s_1)=1+0.9[0.5V^{\pi }(s_1)+0.5V^{\pi }(s_2)].$ Why this step? The environment is stochastic, so the future value is an expectation — weight each destination by its transition probability $p(s^{\prime }\mid s_1,a)$.

2. Substitute the known value $V^{\pi }(s_2)=20$. $V^{\pi }(s_1)=1+0.9[0.5V^{\pi }(s_1)+10]=1+0.45V^{\pi }(s_1)+9.$ Why this step? We already solved $s_2$, so it becomes a constant here — one unknown left.

3. Solve. $V^{\pi }(s_1)-0.45V^{\pi }(s_1)=10\Rightarrow 0.55V^{\pi }(s_1)=10\Rightarrow V^{\pi }(s_1)=\frac{10}{0.55}\approx \mathrm{18.18.}$ Why this step? The unknown $V^{\pi }(s_1)$ sits on both sides because of the self-loop; gathering its terms onto one side isolates it, and dividing by the leftover coefficient $0.55$ finally frees the single number we want.

Verify: $18.18<20$ ✓ (matches the forecast). Sanity check by plugging back: $1+0.9(0.5\cdot 18.18+0.5\cdot 20)=1+0.9(19.09)=1+17.18=\mathrm{18.18.}$ ✓

---

Ex 3 — Cell C: two actions, compute Q, act greedy

Forecast: $a_1$ pays more now and sends you to the good room. Bet on $a_1$.

1. Apply the $Q$ Bellman relation $Q^{\pi }(s,a)=r+\gamma V^{\pi }(s^{\prime })$ (deterministic transitions here). $Q^{\pi }(s_1,a_1)=1+0.9\cdot V^{\pi }(s_2)=1+0.9\cdot 20=19.$ Why this step? $Q$ answers "commit to this action first, then behave normally" — so it uses the immediate reward of that action, not the policy-averaged one.

2. Same for $a_2$ (it loops back into $s_1$): $Q^{\pi }(s_1,a_2)=0.5+0.9\cdot V^{\pi }(s_1)=0.5+0.9\cdot 18.18=0.5+16.36=\mathrm{16.86.}$ Why this step? Action $a_2$'s immediate reward is $+0.5$, and afterward it returns to $s_1$, so its future value is the state value $V^{\pi }(s_1)$ we were given — we plug that in rather than re-deriving it.

3. Compare. $19>16.86$, so $\arg {\max }_a{Q}^{\pi }(s_1,a)=a_1$. Why this step? Comparing $Q$-values at a fixed state is exactly how you improve a policy — pick the higher column.

Verify: $Q^{\pi }(s_1,a_1)=19>Q^{\pi }(s_1,a_2)=16.86$ ✓, so the greedy choice is $a_1$, exactly as forecast. ✓

---

Ex 4 — Cell D: a genuinely stochastic policy

Forecast: Mostly picks the good action ($80\%$), so $V^{\pi }(s_1)$ should sit close to $19$ but pulled slightly down by the $20\%$ chance of $a_2$.

1. Write the two $Q$ equations — but $Q^{\pi }(s_1,a_2)$ depends on $V^{\pi }(s_1)$, which depends on both $Q$'s. This coupling is the whole point. $Q^{\pi }(s_1,a_1)=1+0.9\cdot 20=19.$ $Q^{\pi }(s_1,a_2)=0.5+0.9V^{\pi }(s_1).$ Why this step? $a_2$ loops back to $s_1$, so its Q needs the state's own value — a self-reference.

2. Use the linking identity $V^{\pi }(s_1)={\sum }_a\pi (a\mid s_1)Q^{\pi }(s_1,a)$: $V^{\pi }(s_1)=0.8\cdot 19+0.2\cdot Q^{\pi }(s_1,a_2)=15.2+0.2Q^{\pi }(s_1,a_2).$ Why this step? A state's value is the policy-weighted average of its action-values — this is the bridge between $Q$ and $V$.

3. Substitute and solve the coupled system. $Q^{\pi }(s_1,a_2)=0.5+0.9(15.2+0.2Q^{\pi }(s_1,a_2))=0.5+13.68+0.18Q^{\pi }(s_1,a_2).$ $0.82Q^{\pi }(s_1,a_2)=14.18\Rightarrow Q^{\pi }(s_1,a_2)\approx \mathrm{17.29.}$ Why this step? We have two equations in two unknowns ($V^{\pi }(s_1)$ and $Q^{\pi }(s_1,a_2)$). Substituting step 2 into step 1's second line eliminates $V^{\pi }(s_1)$, leaving a single equation where $Q^{\pi }(s_1,a_2)$ appears on both sides — then we gather its terms and divide, exactly like Ex 2.

4. Back out $V^{\pi }(s_1)$: $V^{\pi }(s_1)=15.2+0.2\cdot 17.29=15.2+3.46=\mathrm{18.66.}$ Why this step? Now that $Q^{\pi }(s_1,a_2)$ is a known number, the linking identity from step 2 immediately hands us the state value — no more unknowns remain.

Verify: $V^{\pi }(s_1)=18.66$ lies between $17.29$ and $19$ (the two $Q$'s), as a weighted average must ✓, and closer to $19$ because $a_1$ has the bigger weight ✓.

---

Ex 5 — Cell E: the degenerate discount $\gamma =0$

Forecast: With $\gamma =0$ the future is worth nothing. So the value should collapse to just the one immediate reward — guess $2$.

1. Apply the geometric formula at $\gamma =0$. $V^{\pi }(s)=\frac{r}{1-\gamma }=\frac{2}{1-0}=2.$ Why this step? $\gamma =0$ multiplies every future reward $R_{t+2},R_{t+3},\dots$ by $0$; only the immediate reward $R_{t+1}$ survives. Value literally equals the next reward — the agent is completely myopic.

2. Cross-check by expanding the sum. $V^{\pi }(s)=2+0\cdot 2+0^2\cdot 2+\cdots =2+0+0+\cdots =2.$ Why this step? Writing out the series confirms the formula didn't hide anything — every term past the first is annihilated.

Verify: $\frac{2}{1-0}=2$ ✓, matching the forecast. This is the degenerate endpoint: value cannot depend on any future.

---

Ex 6 — Cell F: the limiting discount $\gamma \to 1$

Forecast: Near-total patience adds up many rewards, so expect a big number. As $\gamma$ hits exactly $1$, expect trouble.

1. Evaluate at $\gamma =0.99$. $V^{\pi }(s)=\frac{2}{1-0.99}=\frac{2}{0.01}=200.$ Why this step? A tiny gap $1-\gamma =0.01$ in the denominator makes the value huge — the agent now cares about roughly the next hundred rewards. Notice a small change in $\gamma$ caused a large change in value: a warning when tuning discounts.

2. Take the limit $\gamma \to 1$. ${\lim }_{\gamma \to 1}\frac{2}{1-\gamma }=+\infty .$ Why this step? At exactly $\gamma =1$ nothing shrinks the future, so summing a constant positive reward forever diverges. The infinite-horizon value $V^{\pi }(s)$ is undefined here — you would need a terminal state or an average-reward formulation instead.

Verify: $\frac{2}{0.01}=200$ ✓, and the value grows without bound as $\gamma \to 1$ ✓ (see the green curve in the figure below shooting up near the red dashed line).

---

Ex 7 — Cell G: negative rewards and a trap

Forecast: Both values are negative. The trap should be worse than $s_0$ (which has one step of grace).

1. Value of the trap (geometric sum of a negative reward): $V^{\pi }(s_{\text{trap}})=\frac{-5}{1-0.8}=\frac{-5}{0.2}=-25.$ Why this step? The formula doesn't care about the reward's sign — a constant $-5$ forever piles up to $-25$.

2. Value of $s_0$: $V^{\pi }(s_0)=0+0.8\cdot V^{\pi }(s_{\text{trap}})=0+0.8\cdot (-25)=-20.$ Why this step? $s_0$ delays the pain by one discounted step, so it's less bad than the trap itself.

Verify: $-20>-25$ ✓ — $s_0$ is less negative, i.e. better, exactly as forecast. Units check: rewards are dimensionless "points," so values are points too. ✓

---

Ex 8 — Cell H: the optimal value with a max

Forecast: Optimal play never chooses the worse door, so $V^{\ast }(s_1)$ should equal the better of the two action-values.

1. Write the Bellman optimality equation — note the $\max$, not a policy-weighted sum: $V^{\ast }(s_1)={\max }_a[r(s_1,a)+\gamma V^{\ast }(s^{\prime })].$ Why this step? An optimal agent is greedy: it takes the single best action, so we maximize instead of average.

2. Evaluate each branch. $a_1:1+0.9\cdot 20=19,a_2:0.5+0.9V^{\ast }(s_1).$ For $a_2$ to ever win it would need $0.5+0.9V^{\ast }(s_1)>19$, i.e. $V^{\ast }(s_1)>20.6$ — impossible since the max is at most $19$ from branch $a_1$. So $a_1$ wins. Why this step? We check the self-referential branch can't beat the clean one before committing.

3. Take the max: $V^{\ast }(s_1)=19$, and ${\pi }^{\ast }(s_1)=a_1$. Why this step? Once we know $a_1$ dominates $a_2$ at every feasible value, the $\max$ simply selects $a_1$'s number as $V^{\ast }(s_1)$, and the optimal policy is defined to pick whichever action attains that max — here $a_1$.

Verify: Plug the greedy choice back: $\max (19,0.5+0.9\cdot 19)=\max (19,17.6)=19$ ✓ (consistent fixed point).

---

Ex 9 — Cell I: a real-world word problem

Forecast: Broken is a negative absorbing state → clearly negative. Working earns well but risks breaking → large positive.

1. Translate English to symbols. "Working" and "Broken" are states; the daily profit is the reward; "stays Working with prob 0.9" is a transition, i.e. $p(\text{W}\mid \text{W},a)=0.9$ and $p(\text{Broken}\mid \text{W},a)=0.1$. This is just Ex 2's structure with real labels. Why this step? Word problems are trivial once you name the states, rewards, and transition probabilities.

2. Write and solve the Bellman equation for the absorbing Broken state. Broken pays $-4$ and always returns to Broken, so: $V^{\pi }(\text{Broken})=-4+0.95V^{\pi }(\text{Broken}).$ Gathering terms: $V^{\pi }(\text{Broken})(1-0.95)=-4$, i.e. $0.05V^{\pi }(\text{Broken})=-4$, so $V^{\pi }(\text{Broken})=\frac{-4}{0.05}=-80.$ Why this step? Broken is absorbing, so its value depends only on itself — the same one-line algebra as Ex 1, now with a negative reward.

3. Write and solve the Bellman equation for Working (a self-loop that partly leaks into Broken): $V^{\pi }(\text{W})=10+0.95[0.9V^{\pi }(\text{W})+0.1V^{\pi }(\text{Broken})].$ Substitute $V^{\pi }(\text{Broken})=-80$: $V^{\pi }(\text{W})=10+0.95[0.9V^{\pi }(\text{W})+0.1\cdot (-80)]=10+0.855V^{\pi }(\text{W})-\mathrm{7.6.}$ Gather the $V^{\pi }(\text{W})$ terms: $V^{\pi }(\text{W})(1-0.855)=2.4\Rightarrow 0.145V^{\pi }(\text{W})=2.4\Rightarrow V^{\pi }(\text{W})\approx \mathrm{16.55.}$ Why this step? Working feeds partly into itself and partly into Broken; averaging over both destinations (weighted by the transition probabilities $p(s^{\prime }\mid \text{W},a)$) and then isolating $V^{\pi }(\text{W})$ is exactly the coupled-equation move from Ex 2 — same math, business clothes.

Verify: $V^{\pi }(\text{Broken})=-80<0$ ✓ and $V^{\pi }(\text{W})\approx 16.55>0$ ✓. Plug back: $10+0.95(0.9\cdot 16.55+0.1\cdot (-80))=10+0.95(14.90-8)=10+0.95\cdot 6.90=10+6.55=16.55$ ✓.

---

Ex 10 — Cell J: the exam twist ("which action?")

Forecast: $V^{\ast }$ is the biggest number; the optimal action is whichever hits that max — but here there's a tie.

1. $V^{\ast }(s)={\max }_a{Q}^{\ast }(s,a)=\max (3.2,5.1,5.1)=\mathrm{5.1.}$ Why this step? By definition $V^{\ast }(s)={\max }_a{Q}^{\ast }(s,a)$ — the value of a state under optimal play is the best action-value available.

2. ${\pi }^{\ast }(s)=\arg {\max }_a{Q}^{\ast }(s,a)$. Both RIGHT and STAY achieve $5.1$, so the optimal policy is any choice among the tied maximizers — RIGHT, STAY, or any mix of them. Why this step? The optimal policy need not be unique; ties mean multiple optimal policies exist, all equally good.

Verify: $5.1=5.1$ ✓ (the two maximizers agree), and LEFT ($3.2$) is strictly worse so it is never chosen ✓. This "read off the argmax" is precisely how 5.3.01-Q-Learning and 5.3.03-PolicyGradient-Methods recover a policy from learned values.

---

Recall One-line summary of each cell

A: absorbing → $\frac{r}{1-\gamma }$. B: self-loop → coupled linear eqn. C: two actions → compare $Q$. D: stochastic policy → weighted average. E: $\gamma =0$ → value = immediate reward. F: $\gamma \to 1$ → diverges. G: negative rewards → value stays negative. H: optimality → $\max$ not average. I: word problem → name states first. J: read ${\pi }^{\ast }$ = $\arg {\max }_a{Q}^{\ast }$. ::: If you can reproduce all ten, you have covered every scenario this topic offers.

Next stop: turn these hand-solved equations into an algorithm at 5.1.04-Bellman-Equations.