Question bank — Reward shaping and sparse rewards

False. Magnitude is not the issue — shape is. A tiny per-step bonus $+0.1$ for hugging a wall still lets an infinite loop accumulate unbounded reward, so the agent maximizes the loop, not the task. Only potential-based form guarantees safety, no matter the size.

False. PBRS guarantees the optimal policy is unchanged, but speed depends entirely on whether $\Phi$ resembles the true value $V^{\ast }$. A useless or misleading $\Phi$ (say $\Phi \equiv 0$, or one pointing away) preserves correctness while giving no learning boost — or even a temporarily misleading gradient.

False. The policy is unchanged, but the value is shifted: $V^{\prime \pi }(s)=V^{\pi }(s)-\Phi (s)$. The shift is a per-state constant over actions, which is exactly why $\arg {\max }_a$ survives while the numbers do not.

False in general. With discount $\gamma <1$, $F=\gamma \Phi (s^{\prime })-\Phi (s)=(\gamma -1)\Phi (s)\ne 0$ unless $\Phi (s)=0$. The equality only forces $F=0$ when $\gamma =1$. This is exactly why the $\gamma$ cannot be dropped.

False. PBRS routinely makes some transitions negative (moving away from goal) and others positive, yet the optimal policy is provably identical because the whole shaping term telescopes to a start-state constant.

False. HER relabels goals on already-collected transitions; it never injects extra reward into a loop. There is no free reward to milk, so it cannot create reward hacking — it just densifies the learning signal.

False. It changes the distribution of goals / start states over training, not $R$ itself. The reward stays the true task reward; only the difficulty of the tasks the agent faces is scheduled.

False. A sparse-but-correct reward defines the true objective exactly. A dense reward is only better if it is faithful; a badly designed dense reward invites specification gaming. Sparse is safe but slow; dense is fast but risky.

The $\gamma$ is missing. Without it the telescoping leaves residual terms proportional to $(1-\gamma )\Phi$, so the shaping does not collapse to a start-state constant and the optimal policy can shift. Correct form is $\gamma \Phi (s^{\prime })-\Phi (s)$.

Illegitimate. The invariance proof requires $\Phi$ to depend on the state only; a $\Phi (s,a)$ term does not telescope and behaves like arbitrary shaping, so it can change the optimal policy. Steer via state potentials only.

This is per-visit reward, not potential-based. Re-entering $s_B$ pays again every loop, so the agent learns to oscillate through $s_B$ forever. The fix is $\gamma \Phi (s_B)-\Phi (s_A)$ so entering is cancelled by leaving.

If you don't know where the goal is you cannot build a useful $\Phi$, so shaping gives no real gradient. You still need exploration — an intrinsic curiosity bonus or HER — that needs no domain knowledge.

The telescope collapses everything except the very first term to zero; the last potential is annihilated by the discount factor of the term after it (in the infinite-horizon / absorbing sum). Only $-\Phi (s_0)$ survives, so $G_0^{\prime }=G_0-\Phi (s_0)$.

Backwards. Curiosity rewards novelty, so it must decrease with visits: $F=\beta /\sqrt{N(s)}$. As written it rewards revisiting familiar states and kills exploration.

Almost every observed $r=0$, so the return $G_t$ is $0$ for essentially all visited states. With near-zero returns everywhere the agent has been, no action's estimated value exceeds another, so there is nothing to be greedy about.

Reaching the goal by random exploration means hitting a specific sequence of choices; each step multiplies the success probability by a factor below 1, so an $n$-step goal has success probability roughly $p^n$, which vanishes fast in $n$.

If entry and exit cancel, then any closed loop sums the shaping to zero, so looping yields no extra reward. That kills the incentive to repeat a cycle, which is precisely the mechanism of reward hacking.

It makes the shaped optimal value roughly zero everywhere, so from step one the shaped reward already distinguishes good moves (small loss) from bad ones (large loss). The agent starts life with a near-perfect gradient, so learning becomes almost trivial.

$\Phi (s)$ depends on the state, not the action, so it is the same constant subtracted from every action's $Q(s,\cdot )$. Subtracting a shared constant never changes which action is largest.

The wall bonus is per-step and non-potential, so a wall-hugging loop banks it repeatedly. The distance term written as $\gamma \Phi (s^{\prime })-\Phi (s)$ telescopes to zero over any loop, so no cyclic exploit exists.

It relabels the intended goal to the goal the agent actually reached, so a trajectory that failed at $g$ becomes a genuine success at $g^{\prime }$. The transitions and true reward are unchanged — only the goal input is swapped, so no reward is fabricated.

Yes, provided the potential of any absorbing/terminal state is fixed (commonly $\Phi =0$ there). With $\gamma =1$ the shaping is $\Phi (s^{\prime })-\Phi (s)$ and still telescopes to $-\Phi (s_0)$ over a finite episode, so invariance holds.

Then $F=\gamma c-c=(\gamma -1)c$, a fixed per-step reward. It is a legal potential (policy still invariant) but adds a uniform bias each step — useful as a "living penalty" ($c>0,\gamma <1$ gives $F<0$) yet it provides no directional gradient.

$F=\gamma \Phi (s)-\Phi (s)=(\gamma -1)\Phi (s)$. For $\gamma <1$ and $\Phi (s)>0$ this is negative, discouraging idling — and it can never be positive-plus-repeatable in a way that beats the task, preserving safety.

Set $\Phi (s_T)=0$ (or treat the episode as ending, so no next-state potential is added after $s_T$). Otherwise a nonzero terminal potential leaks an uncancelled term into the return and can distort the policy.

Only if $\Phi$ was built from prior knowledge (e.g. a distance heuristic), because then $\Phi$ supplies a gradient before any goal contact. If $\Phi$ was to be learned from experience, it is uninformative until the goal is found — so you still need exploration first.

Perfectly fine. Signs of $\Phi$ are irrelevant to invariance; only the telescoping form matters. Negative-distance potentials are the standard "closer is better" heuristic and change nothing about the guarantee.

No. A constant shift adds $(\gamma -1)c$ to every transition uniformly — itself a valid potential shaping — so both preserve the same optimal policy. The added constant washes out of $\arg {\max }_a$.