State-value and action-value functions: Exercises

This page is a self-test. Each problem is graded by cognitive level:

L1 Recognition — can you read the notation and pick the right definition?
L2 Application — can you plug numbers into one formula?
L3 Analysis — can you chain formulas or compare cases?
L4 Synthesis — can you build a small system and solve it?
L5 Mastery — can you reason about edge cases and limits?

Every solution hides inside a collapsible [!recall]- callout. Try first, then reveal. All the tools you need were built in the parent note State-value and action-value functions; here we only use them.

Here $γ \in [0, 1]$ is the discount factor: a knob that says how much a reward one step in the future is worth compared to a reward right now. $γ = 1$ means "the future counts fully"; $γ = 0$ means "I only care about the very next reward".

L1 — Recognition

Exercise 1.1

You are handed the number $Q^{π} (s, a) = 7$ . In one sentence, what real-world quantity does the $7$ measure? And which of $V$ or $Q$ lets you compare two different actions in the same state?

Recall Solution

WHAT the 7 means: if the agent is in state $s$ , commits to action $a$ right now, and thereafter behaves according to policy $π$ , then the average total discounted reward it will collect from this moment onward is $7$ .

Which one compares actions: $Q^{π} (s, a)$ . It carries an action slot $a$ , so you can hold $s$ fixed and vary $a$ : compare $Q^{π} (s, a_{1})$ vs $Q^{π} (s, a_{2})$ . $V^{π} (s)$ has no action slot — it has already averaged the actions away, so it cannot separate them.

Exercise 1.2

True or false: " $V^{π} (s)$ can be larger than the best $Q^{π} (s, a)$ available at $s$ ." Justify.

Recall Solution

False. Because $V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)$ is a weighted average of the $Q$ -values (the weights $π (a ∣ s)$ are probabilities that sum to 1). An average can never exceed the largest thing you are averaging. So $V^{π} (s) \leq max_{a} Q^{π} (s, a),$ with equality only if the policy puts all its probability on the best action.

L2 — Application

Exercise 2.1

A state $s$ has three actions with known Q-values $Q (s, a_{1}) = 4$ , $Q (s, a_{2}) = 10$ , $Q (s, a_{3}) = - 2$ . The policy picks them with probabilities $0.2, 0.5, 0.3$ . Compute $V^{π} (s)$ .

Recall Solution

Apply the averaging formula $V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)$ : $V^{π} (s) = 0.2 (4) + 0.5 (10) + 0.3 (- 2) = 0.8 + 5.0 - 0.6 = 5.2.$ Sanity check: $5.2$ sits between the smallest Q ( $- 2$ ) and the largest ( $10$ ) — exactly as an average must.

Exercise 2.2

Action $a$ in state $s$ is deterministic: it always lands in $s^{'}$ and pays reward $r = 3$ . You know $V^{π} (s^{'}) = 20$ and $γ = 0.9$ . Compute $Q^{π} (s, a)$ .

Recall Solution

With a deterministic transition, the environment-sum collapses to a single term (probability $1$ ): $Q^{π} (s, a) = 1 \cdot [r + γ V^{π} (s^{'})] = 3 + 0.9 (20) = 3 + 18 = 21.$

Exercise 2.3

Same idea, but the action is stochastic. Action $a$ from $s$ :

$60%$ : land in $s_{1}$ , reward $2$
$40%$ : land in $s_{2}$ , reward $- 5$

Given $V^{π} (s_{1}) = 10$ , $V^{π} (s_{2}) = 0$ , $γ = 1$ . Compute $Q^{π} (s, a)$ .

Recall Solution

Weight each branch by its probability: $Q^{π} (s, a) = 0.6 [2 + 1 \cdot 10] + 0.4 [- 5 + 1 \cdot 0]$ $= 0.6 (12) + 0.4 (- 5) = 7.2 - 2.0 = 5.2.$ WHY: "reward now + discounted value of where you land" is computed inside each outcome, then averaged. Do not average the states first and the rewards separately — average the whole bracket.

L3 — Analysis

Exercise 3.1 — chaining $V$ and $Q$

State $s$ has two actions, each deterministic:

$a_{1} \to s_{1}$ , reward $1$
$a_{2} \to s_{2}$ , reward $4$

Policy: $π (a_{1} ∣ s) = 0.5, π (a_{2} ∣ s) = 0.5$ . Values $V^{π} (s_{1}) = 6$ , $V^{π} (s_{2}) = 2$ , $γ = 0.5$ . Compute $Q^{π} (s, a_{1})$ , $Q^{π} (s, a_{2})$ , and then $V^{π} (s)$ two ways to confirm they agree.

Recall Solution

Step 1 — the two Q's (environment-average, here deterministic so one term each): $Q^{π} (s, a_{1}) = 1 + 0.5 (6) = 1 + 3 = 4,$ $Q^{π} (s, a_{2}) = 4 + 0.5 (2) = 4 + 1 = 5.$ Step 2 — $V$ via the averaging formula $V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)$ : $V^{π} (s) = 0.5 (4) + 0.5 (5) = 2 + 2.5 = 4.5.$ Step 3 — $V$ via the full Bellman equation $V^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}, r} p (\cdot) [r + γ V (s^{'})]$ : $V^{π} (s) = 0.5 [1 + 0.5 (6)] + 0.5 [4 + 0.5 (2)] = 0.5 (4) + 0.5 (5) = 4.5. ✓$ Both routes give $4.5$ — as they must, since the Bellman- $V$ equation is literally the averaging formula with each $Q$ expanded.

Exercise 3.2 — which action does policy improvement prefer?

Using the $Q$ 's from 3.1, if you were allowed to make the policy greedy at $s$ (put all probability on the best action), which action wins, and what would $V$ become?

Recall Solution

Compare the action-values: $Q^{π} (s, a_{2}) = 5 > Q^{π} (s, a_{1}) = 4$ , so greedy picks $a_{2}$ . The new greedy policy has $π^{'} (a_{2} ∣ s) = 1$ , so $V^{π^{'}} (s) = 1 \cdot Q^{π} (s, a_{2}) = 5 \geq 4.5 = V^{π} (s) .$ Going greedy cannot decrease the value — this one step is the seed of the policy-improvement theorem in 5.1.05-Optimal-policiesand-optimal-value-functions.

L4 — Synthesis

Exercise 4.1 — a 1-D corridor, solve it end to end

A corridor of cells $s_{0} - s_{1} - s_{2} - G$ . The agent only ever moves right (policy is deterministic, one action). Each move pays $r = - 1$ ; reaching goal $G$ is terminal with $V^{π} (G) = 0$ . Discount $γ = 0.9$ .

Figure — State-value and action-value functions

Compute $V^{π} (s_{2})$ , $V^{π} (s_{1})$ , $V^{π} (s_{0})$ by backing up from the goal.

Recall Solution

Work right-to-left — you can only evaluate a state once you know the value of where it leads. From $s_{2}$ (one step to $G$ ): $V^{π} (s_{2}) = - 1 + 0.9 V^{π} (G) = - 1 + 0.9 (0) = - 1.$ From $s_{1}$ (one step to $s_{2}$ ): $V^{π} (s_{1}) = - 1 + 0.9 V^{π} (s_{2}) = - 1 + 0.9 (- 1) = - 1.9.$ From $s_{0}$ (one step to $s_{1}$ ): $V^{π} (s_{0}) = - 1 + 0.9 V^{π} (s_{1}) = - 1 + 0.9 (- 1.9) = - 1 - 1.71 = - 2.71.$ Pattern: the farther from the goal, the more negative — each cell "inherits" its neighbour's discounted pain plus its own $- 1$ step cost. This backward sweep is exactly 5.2.01-Dynamic-programming-policy-evaluation on a chain.

Exercise 4.2 — closed form for an infinite corridor

Suppose the corridor never ends: every step pays $r = - 1$ forever, $γ = 0.9$ . What is $V^{π} (s)$ for any state? Use the geometric-series shortcut.

Recall Solution

The return is $G_{t} = - 1 - 0.9 - 0. 9^{2} - 0. 9^{3} - \dots = - \sum_{k = 0}^{\infty} 0. 9^{k}$ . WHY a geometric series: each further step's cost is the previous one multiplied by the constant ratio $γ = 0.9$ , and the classic sum $\sum_{k = 0}^{\infty} γ^{k} = \frac{1}{1 - γ}$ applies whenever $0 \leq γ < 1$ . $V^{π} (s) = - \frac{1}{1 - 0.9} = - \frac{1}{0.1} = - 10.$ Consistency check with Bellman: guess $V = v$ constant, then $v = - 1 + 0.9 v \Rightarrow 0.1 v = - 1 \Rightarrow v = - 10. ✓$

L5 — Mastery

Exercise 5.1 — the $γ \to 1$ limit

In the finite 3-step corridor of 4.1, what happens to $V^{π} (s_{0})$ as $γ \to 1$ ? Contrast this with the infinite corridor of 4.2 as $γ \to 1$ .

Recall Solution

Finite corridor (3 steps from $s_{0}$ to $G$ ): the return is just the sum of three discounted $- 1$ 's, $V^{π} (s_{0}) = - 1 - γ - γ^{2} .$ As $γ \to 1$ this tends to $- 1 - 1 - 1 = - 3$ : a finite answer, because there are only finitely many rewards. (Check with $γ = 0.9$ : $- 1 - 0.9 - 0.81 = - 2.71$ , matching 4.1.) Infinite corridor: $V = - \frac{1}{1 - γ}$ . As $γ \to 1^{-}$ , $1 - γ \to 0^{+}$ , so $V \to - \infty$ . The lesson: discounting is what keeps an endless stream of rewards from summing to infinity. With a guaranteed-terminating task, you can afford $γ = 1$ ; with a non-terminating one you generally need $γ < 1$ for the value to even exist.

Exercise 5.2 — $γ = 0$ degenerate case

Take the stochastic action of Exercise 2.3 but set $γ = 0$ . Recompute $Q^{π} (s, a)$ and explain in words what $γ = 0$ turns the agent into.

Recall Solution

With $γ = 0$ the future-value term $γ V (s^{'})$ vanishes in every branch: $Q^{π} (s, a) = 0.6 [2 + 0 \cdot 10] + 0.4 [- 5 + 0 \cdot 0] = 0.6 (2) + 0.4 (- 5) = 1.2 - 2.0 = - 0.8.$ What $γ = 0$ means: the agent becomes myopic — it only values the immediate reward $R_{t + 1}$ and is completely blind to consequences. $Q$ reduces to the expected next reward alone. This is the extreme opposite of the far-sighted $γ \to 1$ agent.

Exercise 5.3 — self-consistency of a proposed value function

Someone claims that for a two-state loop $s_{A} \leftrightarrow s_{B}$ (from $s_{A}$ you always go to $s_{B}$ with reward $0$ ; from $s_{B}$ you always go to $s_{A}$ with reward $2$ ; $γ = 0.5$ , single action each), the values are $V^{π} (s_{A}) = ?$ and $V^{π} (s_{B}) = ?$ . Set up and solve the coupled Bellman equations, then verify they are self-consistent.

Recall Solution

Set up the two equations (each state deterministic, one term each): $V_{A} = 0 + 0.5 V_{B}, V_{B} = 2 + 0.5 V_{A} .$ Substitute the first into the second: $V_{B} = 2 + 0.5 (0.5 V_{B}) = 2 + 0.25 V_{B} \Rightarrow 0.75 V_{B} = 2 \Rightarrow V_{B} = \frac{2}{0.75} = \frac{8}{3} \approx 2.667.$ Then $V_{A} = 0.5 V_{B} = \frac{4}{3} \approx 1.333.$ Verify self-consistency by plugging back: $V_{A} = 0 + 0.5 (\frac{8}{3}) = \frac{4}{3} ✓, V_{B} = 2 + 0.5 (\frac{4}{3}) = 2 + \frac{2}{3} = \frac{8}{3} ✓ .$ Both hold, so the value function is the unique fixed point of the Bellman operator on this loop — the same fixed point that iterative methods like 5.3.01-Monte-Carlo-prediction and 5.4.01-Temporal-difference-learning converge to from samples.

Recall One-line self-quiz (reveal to check)

Why can $V^{π} (s)$ never exceed $max_{a} Q^{π} (s, a)$ ? ::: Because $V^{π} (s)$ is a probability-weighted average of the $Q$ -values, and an average never exceeds its maximum term. What single quantity should you compare to choose between actions? ::: $Q^{π} (s, a)$ — it already folds reward, discount, and next-state value into one number. What role does $γ < 1$ play for a non-terminating task? ::: It makes the infinite discounted reward sum converge to a finite value.