Policies and value functions: Question bank — AI-ML deep dive

Before we start, a few symbols this whole page leans on — make sure each is a picture in your head, not just a letter:

Definition The characters and notation

$t$ (time step) — the clock reading right now. Picture the agent's life as a ticking sequence $t = 0, 1, 2, \dots$ ; " $t$ " always names the current moment, and $t + 1, t + 2, \dots$ are the steps that come after it. A subscript like $R_{t + 1}$ means "the reward that arrives one tick after the current one."
$π$ (policy) — the actor. Give it a state $s$ , it hands back an action (or a dice-roll over actions). Picture a strategy card: "in this situation, do this." $π (a ∣ s)$ is the probability the policy picks action $a$ in state $s$ .
$V^{π} (s)$ (state-value) — the critic scoring a situation. One number: "how much total reward do I expect from here on, if I keep obeying $π$ ?"
$Q^{π} (s, a)$ (action-value) — the critic scoring a move. "If I first do action $a$ , then obey $π$ forever, how much reward total?"
$γ$ (discount) — the patience dial, between $0$ and $1$ . Multiplies each future reward; $γ^{k}$ shrinks the reward $k$ steps away.
$R_{t + 1}, R_{t + 2}, \dots$ — the rewards the environment pays out on the steps after the current step $t$ ; $R_{m a x}$ is a bound on their size ( $∣ R ∣ \leq R_{m a x}$ ).
$G_{t}$ (the return) — the whole future reward stacked up, measured from the current step $t$ , discounted: $G_{t} = R_{t + 1} + γ R_{t + 2} + γ^{2} R_{t + 3} + \dots$ . Its one-step recursive form is $G_{t} = R_{t + 1} + γ G_{t + 1}$ — "reward now, plus a discounted copy of the same problem starting next step."
$E [\cdot]$ / $E_{π} [\cdot]$ — the average over all possible futures (the $π$ subscript means "when actions follow policy $π$ "). Picture rolling every dice — the agent's and the world's — many times and averaging the result.
$p (s^{'} ∣ s, a)$ — the transition probability: chance the world moves you to next state $s^{'}$ when you do $a$ in $s$ . $p (s^{'}, r ∣ s, a)$ is the joint chance of landing in $s^{'}$ and receiving reward $r$ . (See 5.1.02-Markov-Decision-Processes for the full dynamics.)

True or false — justify

TF1.

V^{π} (s)

is just the reward you get in state

s

False.

V^{π}

is the expected sum of all future discounted rewards (the return

G_{t}

) from

s

onward, not the single immediate reward. The immediate reward is only the first term

R_{t + 1}

inside the sum.

TF2. A deterministic policy is always at least as good as a stochastic one.

False in general. In partially observable settings or adversarial games (mixed strategies), a randomized policy can strictly beat every deterministic one. It's only in a fully observable MDP that a deterministic optimal policy is guaranteed to exist.

TF3. If

γ = 0

, the agent maximizes only its immediate reward.

True. With

γ = 0

, every future term

γ^{k} R_{t + k + 1}

for

k \geq 1

vanishes, leaving

V^{π} (s) = E [R_{t + 1}]

. The agent becomes totally myopic.

TF4. Two different policies can have exactly the same value function

V^{π}

True. Nothing forbids two distinct strategies from producing equal expected returns in every state — e.g. if they only differ on actions that lead to identical downstream outcomes. "Different policy" does not imply "different value."

TF5. There is always exactly one optimal policy.

False. When two actions tie in optimal action-value, breaking the tie either way (or randomizing over them) is still optimal, so several distinct optimal policies can share the single optimal value function

V^{*}

TF6.

Q^{π} (s, a)

requires action

a

to be one the policy

π

would actually take.

False.

Q^{π} (s, a)

is defined for any action

a

— you force

a

on the first step, then follow

π

. That's exactly why

Q

is useful: it lets you ask "what if I deviated once?"

TF7.

V^{*} (s) = max_{a} Q^{*} (s, a)

True. Under the optimal policy you take the best action, so the state's value equals the value of its best action. (Contrast with

V^{π} (s) = \sum_{a} π (a ∣ s) Q^{π} (s, a)

, an average not a max.)

TF8. The Bellman equation is an assumption we impose on

V^{π}

False. It's a consequence — it follows from the recursive split of the return

G_{t} = R_{t + 1} + γ G_{t + 1}

(defined above) plus taking expectations. Any valid

V^{π}

automatically satisfies it.

TF9. "Increasing

γ

toward

1

always makes values larger" — is this statement, exactly as written, true?

False as written.

γ

raises the weight on future rewards; if those future rewards are negative, a larger

γ

can lower the value. The only defensible claim is conditional — for strictly non-negative rewards a larger

γ

cannot decrease value — so the unqualified "always" is wrong.

Spot the error

SE1. "

V^{π} (s) = R (s) + γ V^{π} (s)

, so the same state's value on both sides."

The right side must reference the next state

s^{'}

, not

s

itself, and average over what the world does:

V^{π} (s) = \sum_{a} π (a ∣ s) \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]

. Only when the environment is guaranteed to stay in

s

do the two coincide.

SE2. "

π^{*} (s) = ar g max_{a} V^{*} (s)

You can't

ar g max

over

a

something that doesn't depend on

a

—

V^{*} (s)

has no action argument. The correct greedy rule is

π^{*} (s) = ar g max_{a} Q^{*} (s, a)

, choosing the best action-value.

SE3. "Since the policy is stochastic, we drop the expectation and just use the sampled reward."

The whole point of

E_{π} [\cdot]

is to average over the policy's randomness and the environment's randomness. Using a single sample gives a noisy estimate, not the value function.

SE4. "

Q^{π} (s, a) = r + γ V^{π} (s)

Two errors: the reward and next-state value must be averaged over transition outcomes

p (s^{'}, r ∣ s, a)

, and the future value is

V^{π} (s^{'})

(the state you land in), not

V^{π} (s)

Q^{π} (s, a) = \sum_{s^{'}, r} p (s^{'}, r ∣ s, a) [r + γ V^{π} (s^{'})]

SE5. "The optimal policy maximizes reward at every single step."

It maximizes the long-run discounted return

G_{t}

, which sometimes means accepting a small immediate reward now for a large one later. Greedy-per-step is generally not optimal — that confusion is the whole reason we need value functions.

SE6. "

V^{π} (s) = \sum_{a} max_{a} Q^{π} (s, a)

For a fixed policy

π

, the state value is the policy-weighted average,

\sum_{a} π (a ∣ s) Q^{π} (s, a)

— not a max. The max only appears for the optimal value function

V^{*}

SE7. "Discounting is only a convergence trick with no real meaning."

It has three real meanings: uncertainty about the far future, economic impatience, and guaranteeing a finite sum for bounded rewards (

∣ R ∣ \leq R_{m a x}

). Dismissing it as "just a trick" hides the modelling choice it encodes.

Why questions

WHY1. Why average (take expectation) instead of just summing rewards along one path?

Because both the policy (if stochastic) and the transitions

p (s^{'} ∣ s, a)

can be random, so from one state many futures are possible. The value is the average over all of them, weighted by their probabilities.

WHY2. Why do we ever need

Q

if we already have

V

V

only rates situations; to improve a policy you must compare actions, which is exactly what

Q (s, a)

ranks. With

Q^{*}

you read off the best action directly by

ar g max_{a}

, no environment model required.

WHY3. Why must

γ < 1

(strictly) for infinite-horizon problems?

With bounded reward

∣ R ∣ \leq R_{m a x}

, the sum

\sum_{k} γ^{k} R_{t + k + 1}

is finite only if

γ < 1

(geometric series). At

γ = 1

the sum can diverge unless episodes are guaranteed to terminate.

WHY4. Why does the Bellman equation let us solve for values rather than simulate forever?

It turns an infinite sum into a finite system of linear equations — one equation per state, each referencing only immediate reward plus neighbours' values. We solve the system instead of rolling out infinite trajectories.

WHY5. Why does a stochastic environment force

V^{π}

to depend on

p (s^{'} ∣ s, a)

Because even after fixing the action, where you land next is uncertain. The value must weight each possible next state

s^{'}

by its transition probability, then add its downstream value

γ V^{π} (s^{'})

WHY6. Why is being greedy w.r.t.

Q^{*}

optimal, but being greedy w.r.t. an arbitrary

Q^{π}

generally isn't?

Q^{*}

already encodes the best possible future, so acting greedily on it is self-consistent. Being greedy on a sub-optimal

Q^{π}

gives a different, usually better policy — this one improvement step is the engine of 5.2.01-Policy-Iteration.

WHY7. Why can we split

V^{π} (s)

into "action from policy, then next state from dynamics"?

Because the process is Markov: the future depends only on the current state and chosen action, letting us factor the expectation as

\sum_{a} π (a ∣ s) \sum_{s^{'}} p (s^{'} ∣ s, a) [\dots]

— first the agent's dice, then the world's dice.

Edge cases

EC1. What is

V^{π}

of an absorbing terminal state that gives reward

0

forever?

It's

0

: every future reward is

0

, so the discounted sum is

0

regardless of

γ

. Terminal states anchor the recursion — their values are known without any equation to solve.

EC2. What if a state has only one available action?

Then

π (a ∣ s) = 1

for that action and

V^{π} (s) = Q^{π} (s, a)

— the average over actions collapses to that single action's value. The policy choice is trivial there.

EC3. What happens to

Q^{π} (s, a)

for an action the policy takes with probability

0

It's still perfectly well-defined —

Q

forces the action once, then follows

π

. A zero-probability action can even have the highest

Q

, which is a signal to change the policy toward it.

EC4. If two actions have exactly equal

Q^{*}

values, is the optimal policy undefined?

No — any tie-breaking (including a random mix over the tied actions) is optimal. This is precisely how multiple optimal policies can share one

V^{*}

EC5. With

γ

exactly

1

in a guaranteed-terminating (episodic) task, is

V^{π}

still finite?

Yes. If every episode ends in finitely many steps, the sum has finitely many nonzero terms, so undiscounted return is finite even at

γ = 1

. The divergence worry only bites for non-terminating infinite horizons.

EC6. What is

V^{π} (s)

if the immediate reward is always the constant

c

and the agent never leaves the region?

The geometric sum gives

V^{π} (s) = \frac{c}{1 - γ}

(for

γ < 1

). For example, a state that pays

+ 2

forever with

γ = 0.9

has value

\frac{2}{1 - 0.9} = 20

EC7. Can

V^{π} (s)

be negative?

Yes — if rewards can be negative (penalties/costs), the expected discounted sum can be negative. Values are not probabilities; nothing forces them to be non-negative.

Recall Quick self-test before you leave

One-max-vs-average distinction ::: $V^{π} = \sum_{a} π (a ∣ s) Q^{π}$ (average), but $V^{*} = max_{a} Q^{*}$ (max). The greedy rule that recovers the optimal policy ::: $π^{*} (s) = ar g max_{a} Q^{*} (s, a)$ , never $ar g max$ of $V^{*}$ . Why discounting keeps sums finite ::: $γ < 1$ makes $\sum_{k} γ^{k}$ a convergent geometric series for bounded rewards.