Question bank — Bellman equations

False — it holds for any fixed policy $\pi$, optimal or terrible. It is just a self-consistency statement: value now = expected reward + discounted value next. Optimality adds a $\max$; consistency does not require it.

False — $V^{\pi }$ averages over actions with weights $\pi (a\mid s)$, while $V^{\ast }$ takes ${\max }_a$. Same-looking bracket $[r+\gamma V(s^{\prime })]$, but "average" vs "best" are different operators.

True — with $\gamma =0$ every future term ${\gamma }^t{R}_{t+1}$ for $t\ge 1$ vanishes, so $V^{\pi }(s)={\mathbb{E}}_{\pi }[R_1\mid s]$. The agent becomes myopic, caring only about the next step.

False — it is written as an expectation precisely to absorb randomness: ${\sum }_{s^{\prime },r}p(s^{\prime },r\mid s,a)$ handles stochastic transitions and rewards. Determinism just collapses each sum to a single term.

True — for a fixed $\pi$ the equations $V^{\pi }=R^{\pi }+\gamma P^{\pi }{V}^{\pi }$ are linear in the unknowns $V^{\pi }(s)$, so they can be solved by matrix inversion. The optimality equation, with its $\max$, is not linear.

False — $V^{\ast }$ is unique (it is the fixed point of a contraction for $\gamma <1$). There can be many optimal policies, but they all share the same $V^{\ast }$ and $Q^{\ast }$.

False — $Q^{\ast }(s,a)$ is defined for every action: it is the value of taking $a$ now (possibly a bad choice) and acting optimally thereafter. Comparing these values is exactly how you recover the greedy optimal action.

False — Bootstrapping means we reuse current value estimates for successor states, but each update still consumes a real reward $r$. It mixes estimate-based and sample-based information.

Two errors — the discount $\gamma$ is missing (returns can diverge), and there is no expectation, so it silently assumes a single deterministic successor. Correct: $V^{\pi }(s)=\mathbb{E}[R_1+\gamma V^{\pi }(S_1)\mid s]$.

The optimal policy is deterministic and greedy, so ${\pi }^{\ast }$ is a $\max$, not an average. Writing it as a weighted sum over actions is the $V^{\pi }$ form; the optimality form is $V^{\ast }(s)={\max }_a{\sum }_{s^{\prime },r}p[r+\gamma V^{\ast }(s^{\prime })]$.

The first line is fine, but the substitution for $V^{\ast }(s^{\prime })$ is wrong: at optimality $V^{\ast }(s^{\prime })={\max }_{a^{\prime }}{Q}^{\ast }(s^{\prime },a^{\prime })$, not a policy-weighted average. Mixing the expectation backup into the optimality equation breaks it.

For $\gamma =1$ over an infinite horizon the geometric bound $R_{\max }/(1-\gamma )$ blows up. Boundedness of the return requires $\gamma <1$ (or a guaranteed terminal state).

The optimality-based target uses ${\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })$ over the next state's actions, not the previous action $a$. Reusing $a$ would be the SARSA/expectation flavour, not the optimality backup that Q-Learning uses.

It is over both the policy's action choices and the environment's transition/reward randomness. Dropping the policy randomness gives the wrong value for any stochastic $\pi$.

One step is enough to expose the recursion: everything after the first reward is itself a discounted value $\gamma V^{\pi }(S_1)$. Splitting further would just re-derive the same self-reference with extra clutter.

Strict inequality makes the geometric series $\sum {\gamma }^t$ converge and makes the Bellman operator a contraction, which is what guarantees a unique fixed point that iteration converges to.

$Q^{\ast }$ already accounts for acting optimally forever after, so picking its argmax cannot be improved. A general $Q^{\pi }$ assumes you keep following the possibly-suboptimal $\pi$, so its greedy action may just be the best of a bad set.

$V(s)$ ranks states but hides which action produced the value; to choose without a model you need $Q(s,a)$, which ranks actions directly. That is why model-free methods like Q-Learning learn $Q$.

The recursion assumes the future depends on the present state alone (the Markov property). If history mattered, $V^{\pi }(s)$ would not be a well-defined function of $s$, and the one-step decomposition would be invalid.

It lets Temporal Difference Learning update from a single step without waiting for an episode to end (strength), but it propagates errors in the current estimate into the target, so bad estimates can reinforce themselves (risk).

They optimise the policy parameters by ascending an expected-return gradient rather than iterating a value backup. Bellman values often appear inside them (as baselines/critics), but the core update is gradient-based, not a $\max$ fixed point.

By convention it is $0$: no future rewards can accrue once the episode ends, so there is nothing to discount forward. The recursion terminates there instead of referencing another successor.

Self-reference $V=r+\gamma V$ solves to $V=r/(1-\gamma )$, the discounted geometric sum of getting $r$ forever. This is finite exactly because $\gamma <1$.

Yes — with no distinguishing choice, the $\max$ and the average collapse to the same single value, so the expectation and optimality equations coincide at that state.

The $\max$ is still well-defined (both give the same value), so $V^{\ast }$ is unique. Multiple optimal policies exist; any tie-breaking rule among the maximisers is optimal.

No — $V^{\pi }(s)$ is defined as the return if you were placed in $s$ and then followed $\pi$, regardless of whether $\pi$ ever reaches $s$ on its own. Visitation frequency affects learning, not the definition.

It equals just $r$, since $V^{\ast }(\text{terminal})=0$ makes the discounted future term vanish. The action's value is exactly its immediate payoff.

The averaging over $s^{\prime },r$ collapses to one term, but the expectation over the policy's actions remains for $V^{\pi }$ (unless $\pi$ is also deterministic). Only when both policy and dynamics are deterministic does every sum become a single term.