State-value and action-value functions: Question bank

This is a conceptual stress-test for the parent topic on $V^\pi$ and $Q^\pi$. No heavy arithmetic here — every item targets a misconception, a boundary case, or a why. Cover the reveal, say your answer out loud with a reason, then check.

Before we start, the symbols every item below leans on — pinned here so you never meet them undefined:

Definition The pieces, in plain words

$V^{π} (s)$ — the expected total future reward if you start in state $s$ and keep obeying policy $π$ forever.
$Q^{π} (s, a)$ — the expected total future reward if you start in $s$ , take action $a$ once (even if $π$ wouldn't), then obey $π$ forever.
$γ$ — the discount factor in $[0, 1]$ that shrinks each future reward by $γ$ per step — "how much I care about later versus now".
$R_{t + 1}$ — the reward (a random number) the environment pays you one step after being in $s$ ; the lowercase $r$ is a particular value that $R_{t + 1}$ might take.
$G_{t}$ — the return: the whole discounted stream of future rewards, $G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots$ . Value functions are just expectations of $G_{t}$ .
$p (s^{'}, r ∣ s, a)$ — the transition probability: the chance the environment sends you to next-state $s^{'}$ and pays reward $r$ , given you took action $a$ in state $s$ . It is how "the world" responds to you.

Three pictures to keep in mind

Most of the traps below dissolve the moment you see the mechanism. These three figures are the visual backbone for the whole bank — glance back at them whenever an item feels slippery.

Figure 1 — the $Q \to V$ averaging fan. $V$ is not a new quantity; it is the centre of gravity of the action-values, pulled toward whichever action the policy favours.

Figure — State-value and action-value functions

Figure 2 — unfolding a trajectory to split the Bellman backup. The single identity $G_{t} = R_{t + 1} + γ G_{t + 1}$ is what lets the expectation split into "reward now" plus "discounted value of where I land".

Figure 3 — why $γ < 1$ contracts. Each Bellman sweep multiplies the error between your estimate and the truth by at most $γ$ ; the gap shrinks geometrically to zero — the fixed-point (Banach) picture.

True or false — justify

V^{π} (s)

can be larger than the largest

Q^{π} (s, a)

in that state.

False.

V^{π}

is a probability-weighted average of the

Q

values (see Figure 1), and an average never exceeds its maximum term — so

V^{π} (s) \leq max_{a} Q^{π} (s, a)

always.

If a policy is deterministic (picks one action

a^{⋆}

with probability 1), then

V^{π} (s) = Q^{π} (s, a^{⋆})

True. The averaging sum

V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)

collapses to the single term with weight 1, so

V

and that one

Q

coincide — the fan of Figure 1 shrinks to a single arrow.

Q^{π} (s, a)

only makes sense for actions that

π

actually takes in

s

False.

Q^{π}

is defined for every action — you force action

a

for one step and then follow

π

; that's exactly why

Q

lets us compare "off-policy" alternatives during improvement.

Two different policies can share the same

V^{π} (s)

at a state

s

True. Value only measures expected return; two policies that differ elsewhere but funnel state

s

into equally-rewarding futures give the same number there.

A state with reward

+ 100

on the very next step must have a high

V^{π}

False. If every path after that reward drags you into large penalties, the discounted sum

r + γ V^{π} (s^{'})

can still be low — value is about the whole future, not one step.

With

γ = 0

V^{π} (s)

equals the expected immediate reward only.

True. Every future term carries a factor

γ^{k} = 0

for

k \geq 1

, so only

R_{t + 1}

survives; the agent becomes purely myopic.

With

γ = 1

in an episode that never terminates,

V^{π} (s)

is guaranteed finite.

False. An infinite sum of undiscounted rewards can diverge;

γ < 1

(or guaranteed termination) is what keeps the geometric series bounded.

The Bellman equation is an assumption we impose on value functions.

False. It's a consequence — any correctly-defined

V^{π}

automatically satisfies it, because splitting

G_{t} = R_{t + 1} + γ G_{t + 1}

(Figure 2) and taking expectations is just algebra plus the Markov property.

Q^{π} (s, a)

can be negative even when all rewards are non-negative.

False. A weighted sum of non-negative rewards (all with non-negative weights

γ^{k}

) can't be negative; you need at least one negative reward somewhere reachable.

Spot the error

V^{π} (s) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]

" — what's wrong?

The action-averaging is missing.

V^{π}

must sum over actions weighted by

π (a ∣ s)

first (the fan of Figure 1); as written it fixes a single

a

, which is actually

Q^{π} (s, a)

, not

V^{π} (s)

"To get

Q

from

V

, use

Q^{π} (s, a) = r + γ V^{π} (s^{'})

." — what's missing?

The expectation over the environment. Rewards and next states are random, so you need

\sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [\dots]

; a single

(r, s^{'})

only works if the transition is deterministic.

"Since

a_{2}

has the highest Q-value,

V^{π} (s)

equals

Q^{π} (s, a_{2})

." — why wrong?

Only if the policy picks

a_{2}

with probability 1. A stochastic policy blends in the worse actions too, so

V^{π} (s) < Q^{π} (s, a_{2})

whenever

π

ever chooses something else.

"The goal state has huge value because you get

+ 10

there." — why wrong?

A terminal goal has

V^{π} = 0

: no future reward flows from it. The

+ 10

shows up in the value of the states that lead into the goal, via the

r

term of their Bellman backup.

"Value functions store the actual reward sequence you'll receive." — why wrong?

They store an expectation, a single averaged number over all possible random futures — not any specific realised trajectory.

π (a ∣ s)

are Q-values." — why wrong?

π (a ∣ s)

are probabilities of choosing actions (they sum to 1 over

a

);

Q^{π} (s, a)

are expected returns (unbounded reals). They multiply together in

V

, but they are different objects.

Why questions

Why does

Q

carry more information than

V

at the same state?

Q

resolves the value per action (each arrow in Figure 1), whereas

V

has already averaged those away; you can recover

V

from

Q

(average over

π

) but not

Q

from

V

without the transition model.

Why is the Markov property essential to the Bellman derivation?

It lets us write

E_{π} [G_{t + 1} ∣ S_{t + 1} = s^{'}] = V^{π} (s^{'})

— the future depends only on where you are (

s^{'}

), not on the history of how you got there (see the Markov definition above), so the recursion (Figure 2) closes.

Why do we call solving Bellman equations "bootstrapping"?

Each value is defined using other values (

V^{π} (s)

in terms of

V^{π} (s^{'})

), so we update estimates from estimates rather than waiting for full returns — the idea behind TD learning.

Why does the

- 1

per-step reward in Gridworld push values below zero near the goal?

Each step costs

- 1

and only the terminal goal contributes

0

; so states are penalised by the (discounted) number of steps still needed to arrive.

Why can we compute

V^{π}

exactly by iterating the Bellman equation?

It's a fixed-point equation with a

γ

-contraction (Figure 3), so repeated application converges to the unique solution — this is exactly policy evaluation.

Why does a smaller

γ

make an agent "short-sighted"?

Distant rewards get multiplied by

γ^{k}

, which decays fast for small

γ

, so they barely register and the value is dominated by near-term rewards.

Why is

V^{π}

tied to a specific

π

rather than being a property of the state alone?

The future reward you collect from

s

depends entirely on which actions you'll take; change the policy and the same state's expected return changes — the optimal case is treated in optimal value functions.

Why does

γ < 1

guarantee convergence, not just help it?

Two consecutive estimates can never differ by more than

γ

times their previous difference (Figure 3); a factor below 1 forces the error to shrink each sweep, so any starting guess is dragged to the unique fixed point.

Why can Monte-Carlo prediction estimate

V^{π}

without knowing

p (s^{'}, r ∣ s, a)

It averages actual observed returns

G_{t}

from complete episodes, so the environment's randomness is sampled rather than modelled — no transition table needed.

Edge cases

What is

V^{π}

of a terminal (absorbing, zero-reward) state?

Exactly

0

— there is no future reward to accumulate, and it anchors every backward Bellman computation.

If a state has only one available action, how do

V^{π} (s)

and

Q^{π} (s, a)

relate?

They are equal: the single action carries all the probability mass, so the action-averaging sum has just one term.

With

γ = 1

but the environment guaranteed to terminate in finite steps, is

V^{π}

well-defined?

Yes — the return

G_{t}

is a finite sum, so it converges even without discounting; divergence only threatens infinite-horizon undiscounted problems.

For a deterministic environment, does the

\sum_{s^{'}, r} p (s^{'}, r ∣ s, a)

in the

Q

formula still apply?

Yes, but it collapses to a single term because exactly one

(s^{'}, r)

has probability 1 and the rest are 0 — the sum is still correct, just trivial.

If two actions have identical

Q^{π} (s, a)

, does the choice between them change

V^{π} (s)

No — any split of probability between equal-valued actions gives the same weighted average, so

V^{π} (s)

is unchanged regardless of how

π

mixes them.

How large can the gap

∣ V^{π} (s) - V^{π} (s^{'}) ∣

between a state and its deterministic successor be?

From the backup

V^{π} (s) = r + γ V^{π} (s^{'})

, subtract

V^{π} (s^{'})

to get

V^{π} (s) - V^{π} (s^{'}) = r + (γ - 1) V^{π} (s^{'})

; now take absolute values and apply the triangle inequality

∣ x + y ∣ \leq ∣ x ∣ + ∣ y ∣

, giving

∣ V^{π} (s) - V^{π} (s^{'}) ∣ \leq ∣ r ∣ + (1 - γ) ∣ V^{π} (s^{'}) ∣

. So a big one-step reward

r

or a small

γ

allows a big jump, while

γ \to 1

with modest

r

keeps neighbours close.

Can

V^{π} (s)

and

V^{π} (s^{'})

for a neighbour differ by more than the immediate reward alone?

Yes — the extra slack is the

(1 - γ) ∣ V^{π} (s^{'}) ∣

term from that triangle-inequality bound; only when

γ = 1

does the gap reduce to exactly the immediate reward

∣ r ∣

Recall Quick self-check

Cover every answer above and re-derive the reason — if you can only recall "true/false" you have not learned the item.