Reward shaping and sparse rewards: Worked examples

This page is the exhaustive drill for the parent topic. We take the potential-based shaping formula $F (s, a, s^{'}) = γ Φ (s^{'}) - Φ (s)$ and run it through every kind of situation that can appear: moving toward or away from a goal, standing still, looping, terminal states, the discount $γ$ set to different values, a non-potential (dangerous) shaping, and a real-world word problem. If you have never seen this notation, read the parent first — but every symbol used here is re-anchored below.

Definition The symbols we will reuse, in plain words

$Φ (s)$ (read "phi of s") — a number you assign to each state $s$ . Think of it as height on a hill: high $Φ$ = "good place to be". You invent it.
$γ$ (read "gamma") — the discount, a number between $0$ and $1$ . It says how much we shrink future rewards. $γ = 1$ means "the future counts fully"; $γ = 0.99$ means "next step counts $99%$ as much".
$F (s, a, s^{'})$ — the shaping bonus we hand the agent for moving from state $s$ to state $s^{'}$ . Positive = "nice move", negative = "bad move".
$R$ (or $R (s, a, s^{'})$ ) — the true environment reward, the real payoff the task hands out (e.g. $+ 1$ for reaching the goal). This is not invented by us; the environment defines it. The shaped reward is $R^{'} = R + F$ .
$V^{π} (s)$ — the value of state $s$ under policy $π$ : the total discounted reward you expect to collect from $s$ onward if you follow $π$ . Read "how good is it to be here".
$Q^{π} (s, a)$ — the action-value: how good it is to take action $a$ in state $s$ , then follow $π$ . Read "how good is this move".
$V^{'}$ and $Q^{'}$ — the same two quantities but measured in the shaped problem (using $R^{'}$ instead of $R$ ). The prime $^{'}$ always means "in the shaped world".

The scenario matrix

Here is every case class this topic can throw at you. Each later example is tagged with the cell it covers.

Cell	Situation	What we test
A	Move toward goal ( $Φ$ increases)	$F > 0$ ?
B	Move away from goal ( $Φ$ decreases)	$F < 0$ ?
C	Stay put ( $s^{'} = s$ )	sign of $F$ when $Φ$ unchanged
D	Full loop back to start	does $F$ sum to $0$ ?
E	Terminal / degenerate state (episode ends)	how $Φ$ (terminal) is handled
F	Limiting $γ \to 1$ vs $γ$ small	how the $γ$ factor bites
G	Non-potential shaping (the trap)	policy corruption
H	Real-world word problem	translate a task into $Φ$
I	Exam twist — recover $V$ from $V^{'}$	policy-invariance algebra

The examples below cover A–I.

Worked examples

Worked example Example 1 — Cell A: moving toward the goal

Goal at origin. $Φ (s) = - d (s)$ where $d$ is Manhattan distance to goal (so closer = higher $Φ$ , since $- 4 > - 5$ ). Discount $γ = 0.99$ . The agent steps from a state with $d = 5$ to one with $d = 4$ . Find $F$ .

Forecast: getting closer should feel good — guess $F$ is positive and roughly $+ 1$ .

Steps:

Write the potentials: $Φ (s) = - 5$ , $Φ (s^{'}) = - 4$ . Why this step? We must convert "distance" into "height on the hill" before plugging in.
Apply the formula: $F = γ Φ (s^{'}) - Φ (s) = 0.99 (- 4) - (- 5)$ . Why this step? This is the definition of PBRS — the ONLY shaping that keeps the policy safe.
Arithmetic: $0.99 (- 4) = - 3.96$ ; then $- 3.96 - (- 5) = - 3.96 + 5 = 1.04$ . Why this step? Subtracting a negative flips it to addition.

Answer: $F = + 1.04$ .

Verify: the sign matches our forecast (positive). Sanity: without discount it would be exactly $Φ (s^{'}) - Φ (s) = - 4 - (- 5) = + 1$ ; the extra $+ 0.04$ comes from $γ$ shrinking the negative next-state height, which nudges it up. Consistent. ✔

Worked example Example 2 — Cell B: moving away from the goal

Same setup ( $Φ = - d$ , $γ = 0.99$ ). Now the agent moves from $d = 5$ to $d = 6$ . Find $F$ .

Forecast: going backwards should sting — guess $F$ is negative, near $- 1$ .

Steps:

$Φ (s) = - 5$ , $Φ (s^{'}) = - 6$ . Why this step? Farther away = lower height, so $Φ$ drops.
$F = 0.99 (- 6) - (- 5) = - 5.94 + 5$ . Why this step? Same formula, only the numbers change — we never switch formulas per case.
$- 5.94 + 5 = - 0.94$ . Why this step? We add $+ 5$ (from cancelling the minus of $- (- 5)$ ) to $- 5.94$ ; since $5.94 > 5$ , the result stays negative, giving $- 0.94$ .

Answer: $F = - 0.94$ .

Verify: negative as forecast. Note it is not the exact mirror of Example 1's $+ 1.04$ ; the $γ$ factor makes forward/backward slightly asymmetric ( $1.04 \neq = 0.94$ ). That asymmetry is real and expected in discounted settings. ✔

Worked example Example 5 — Cell D: a full loop nets zero

$γ = 1$ . Potentials $Φ (s_{A}) = 2, Φ (s_{B}) = 5, Φ (s_{C}) = 1$ . The agent loops $s_{A} \to s_{B} \to s_{C} \to s_{A}$ . Sum the shaping bonuses over the loop.

Forecast: PBRS is supposed to make loops free of reward. Guess the total.

Steps (see figure):

Edge $A \to B$ : $F_{1} = 1 \cdot Φ (s_{B}) - Φ (s_{A}) = 5 - 2 = + 3$ .
Edge $B \to C$ : $F_{2} = 1 \cdot Φ (s_{C}) - Φ (s_{B}) = 1 - 5 = - 4$ .
Edge $C \to A$ : $F_{3} = 1 \cdot Φ (s_{A}) - Φ (s_{C}) = 2 - 1 = + 1$ . Why these steps? Every edge uses the same $F = γ Φ (s^{'}) - Φ (s)$ ; we just walk the cycle edge by edge.
Sum: $F_{1} + F_{2} + F_{3} = 3 - 4 + 1 = 0$ . Why this step? Around any closed loop, each $Φ$ is added once (as $s^{'}$ ) and subtracted once (as $s$ ), so all cancel — telescoping on a cycle.

Figure — what you are looking at: three lavender circles are the states $s_{A}$ , $s_{B}$ , $s_{C}$ , drawn at heights equal to their potentials ( $Φ = 2, 5, 1$ on the vertical "potential" axis). Three coral arrows are the moves: $A \to B$ climbing labelled $F = + 3$ , $B \to C$ dropping labelled $F = - 4$ , and a curved arrow $C \to A$ labelled $F = + 1$ . The mint box on the right sums them: $3 - 4 + 1 = 0$ . The picture makes visible that whatever height you gain going up an edge, you give back closing the loop.

Answer: total shaping over the loop $= 0$ .

Verify: the coral arrows carry $+ 3, - 4, + 1$ ; adding these three heights returns to zero. No free reward for looping ⇒ no reward hacking. ✔

Worked example Example 6 — Cell G: the trap (non-potential shaping)

A designer, ignoring PBRS, gives $F = + 0.1$ for every step spent adjacent to a wall (not of the form $γ Φ (s^{'}) - Φ (s)$ ). The agent finds a 4-step loop that stays wall-adjacent, $γ = 1$ . How much shaping reward does one loop pay, and is the policy corrupted?

Forecast: guess whether this loop pays $0$ (safe) or something positive (corrupt).

Steps:

Each of the 4 steps pays $+ 0.1$ : loop total $= 4 \times 0.1 = + 0.4$ . Why this step? $F$ here depends only on the current state's label ("near wall"), not on any potential difference, so nothing cancels.
Ask: can $F = + 0.1$ be written as $γ Φ (s^{'}) - Φ (s)$ for some $Φ$ ? Over a closed loop that form MUST telescope to $0$ , but here it sums to $+ 0.4 \neq = 0$ . Why this step? A quick contradiction test: if it were potential-based, Example 5's cycle-cancellation would force the loop sum to zero.
Since the loop pays $+ 0.4$ forever, the agent maximizes shaped return by looping ⇒ policy corrupted.

Answer: loop pays $+ 0.4$ ; policy IS corrupted — this is reward hacking.

Verify: $4 \times 0.1 = 0.4 \neq = 0$ , confirming non-potential shaping breaks the loop-cancellation guarantee. Contrast Example 5 (potential, sum $= 0$ ). ✔

Worked example Example 7 — Cell E: terminal / degenerate state

Grid task, $γ = 0.99$ , $Φ (s) = - d (s)$ . The agent is at $d = 1$ and steps onto the goal (terminal), where $d = 0$ so $Φ (goal) = 0$ . The environment also pays the true reward $R = + 1$ for reaching the goal (recall $R$ is the real task payoff, not something we invented). Find the total shaped reward $R^{'} = R + F$ on this final step.

Forecast: guess whether the goal step feels more or less rewarding than a plain +1.

Steps:

$Φ (s) = - 1$ (at $d = 1$ ), $Φ (goal) = 0$ . Why this step? Terminal states get a real $Φ$ value too — the convention that matters for policy invariance is $Φ (terminal) = 0$ .
$F = γ Φ (goal) - Φ (s) = 0.99 (0) - (- 1) = + 1$ . Why this step? The final-step bonus is exactly the leftover $- Φ (s_{0})$ style term; with $Φ (terminal) = 0$ it reduces to $- Φ (s)$ .
Total: $R^{'} = R + F = 1 + 1 = 2$ . Why this step? Shaped reward on any step is always true reward plus shaping.

Answer: $R^{'} = + 2$ on the winning step.

Verify: the shaping added exactly $- Φ (s) = + 1$ , so a nearby state gets a strong final nudge, yet by policy invariance the ordering of policies is untouched. Setting $Φ (terminal) = 0$ is the standard convention that keeps the telescoped constant clean. ✔

Worked example Example 8 — Cell H: real-world word problem

A warehouse robot must carry a box to a dock. Battery drains each step. You have a rough map giving straight-line distance $ρ (s)$ (metres) from the robot to the dock. Sparse reward: $+ 10$ only when the box reaches the dock. Design $Φ$ and compute the shaping for a step from $ρ = 8 m$ to $ρ = 6 m$ with $γ = 0.98$ .

Forecast: guess the shape of $Φ$ and whether the step earns a positive bonus.

Steps:

Choose $Φ (s) = - ρ (s)$ so that being closer to the dock is a higher (less negative) potential — mirrors $Φ \approx V^{*}$ since closer states are genuinely better. Why this step? PBRS works best when $Φ$ tracks the true value; "negative distance to goal" is a cheap, effective heuristic.
$Φ (s) = - 8, Φ (s^{'}) = - 6$ . Why this step? We evaluate the potential we just chose at the two distances: $Φ (s) = - ρ (s) = - 8$ and $Φ (s^{'}) = - ρ (s^{'}) = - 6$ . Note the box got closer ( $8 \to 6$ m), so its height rose from $- 8$ to $- 6$ .
$F = 0.98 (- 6) - (- 8) = - 5.88 + 8 = 2.12$ . Why this step? Same universal formula; units of $F$ are "reward", not metres — $Φ$ converts metres into reward-height.

Answer: $F = + 2.12$ for that 2-metre progress.

Verify: positive, as forecast — real progress toward the dock is rewarded, while any wandering loop still nets zero by Example 5's argument. The +10 terminal reward is untouched, so the true task ("reach dock") remains optimal. ✔

Worked example Example 9 — Cell I: exam twist (recover

$V$ from $V^{'}$ ) The parent proved $V^{' π} (s) = V^{π} (s) - Φ (s)$ (shaped value = true value minus potential). In a shaped problem you measured $V^{' π} (s_{0}) = 3.5$ at start state $s_{0}$ with $Φ (s_{0}) = - 2$ . Also, action $a_{1}$ has $Q^{'} (s_{0}, a_{1}) = 3.5$ and $a_{2}$ has $Q^{'} (s_{0}, a_{2}) = 4.1$ (recall $Q^{'}$ = action-value in the shaped world). Find the true $V^{π} (s_{0})$ and the greedy action.

Forecast: guess whether shaping changed which action is best.

Steps:

Invert the relation: $V^{π} (s_{0}) = V^{' π} (s_{0}) + Φ (s_{0}) = 3.5 + (- 2) = 1.5$ . Why this step? Rearranging $V^{'} = V - Φ$ gives $V = V^{'} + Φ$ .
True Q-values: $Q (s_{0}, a) = Q^{'} (s_{0}, a) + Φ (s_{0})$ , so $Q (s_{0}, a_{1}) = 3.5 - 2 = 1.5$ , $Q (s_{0}, a_{2}) = 4.1 - 2 = 2.1$ . Why this step? The same offset $Φ (s_{0})$ is added to every action's Q at this state.
Greedy action: $ar g max$ is $a_{2}$ in both the shaped and true problems ( $4.1 > 3.5$ and $2.1 > 1.5$ ). Why this step? A constant offset over actions never changes the $ar g max$ — that is policy invariance in action.

Answers: $V^{π} (s_{0}) = 1.5$ ; greedy action $= a_{2}$ (unchanged by shaping).

Verify: the offset $Φ (s_{0}) = - 2$ shifted both Q-values equally by $- 2$ , preserving their gap of $4.1 - 3.5 = 2.1 - 1.5 = 0.6$ . Same best action ⇒ policy invariance confirmed numerically. ✔

Recall Quick self-test across the matrix

Toward-goal step sign of $F$ ? ::: Positive (Example 1, $+ 1.04$ ). Stay-put $F$ when $γ < 1$ and $Φ (s) = - 5$ ? ::: $(γ - 1) Φ (s)$ , e.g. $+ 0.05$ at $γ = 0.99$ — small, not zero (Example 3). Sum of $F$ around any closed loop under PBRS? ::: Exactly $0$ (Example 5). Does a $+ 0.1$ -per-wall bonus corrupt the policy? ::: Yes — a 4-step loop pays $+ 0.4$ , it is not potential-based (Example 6). Recover true $V$ from shaped $V^{'}$ ? ::: $V = V^{'} + Φ$ (Example 9).

Related tools when a good $Φ$ is unavailable: Hindsight Experience Replay, Curriculum Learning, and Exploration vs Exploitation bonuses. The invariance proof rests on Value Functions and Bellman Equations over Markov Decision Processes.