Visual walkthrough — Reward shaping and sparse rewards

This is the visual companion to Reward shaping and sparse rewards. If you have not met the ideas of state, reward, return, or policy, we build them here too — assume nothing.

---

Step 0 — The vocabulary, drawn before it is used

Before any algebra, a handful of words. Each is a picture.

The symbol $\gamma$ (gamma) is the discount factor. Why does it exist? A reward now is worth more than the same reward later, so each step into the future we shrink its worth by multiplying by $\gamma$. After $t$ steps the shrink factor is ${\gamma }^t$.

In the figure the reward $r_2$ sitting two steps away is drawn short because ${\gamma }^2{r}_2$ is a shrunk copy of it. That shrinking is the entire meaning of "discounted".

Recall

What does ${\gamma }^t$ do to a reward $t$ steps in the future? ::: Shrinks it — the further away, the smaller its contribution to the return. Under what assumptions do the infinite sums converge? ::: $\gamma \in [0,1)$ with bounded rewards and bounded $\Phi$.

---

Step 1 — The potential $\Phi$: a height map over the states

We now invent one new object, the potential.

WHAT: a number per state, nothing more. WHY invent it: we want a "you're getting warmer" signal, and altitude is the cleanest such signal — going downhill toward the goal feels like progress. PICTURE: a landscape where the goal sits in a valley.

The teal surface is $\Phi$. Notice: $\Phi$ is defined before we know the agent's plan. It depends only on where you are, never on how you got there. Hold onto that — it is the key.

---

Step 2 — Turn altitude into a step reward (the shaping term $F$)

We cannot hand the agent an altitude directly; RL only understands rewards per step. So we convert a change in altitude into a reward, using the successor-state notation $s^{\prime }$ from Step 0.

Term by term, standing on the picture:

• $\Phi (s)$ — the height of the square you are leaving. You "pay" this.
• $\Phi (s^{\prime })$ — the height of the successor square you are entering. You "collect" this.
• $\gamma$ in front of $\Phi (s^{\prime })$ — because the collected part is one step in the future, it gets the same shrink every future thing gets. This $\gamma$ is not decoration; Step 5 dies without it.

WHY this exact shape and not just $\Phi (s^{\prime })-\Phi (s)$? Because the collected altitude is enjoyed next step, so to stay honest with the return's bookkeeping it must be discounted by $\gamma$. We will literally watch the mismatch cancel only when the $\gamma$ is present.

---

Step 3 — Write the shaped return, then split it

The shaped reward at each step is $r_t^{\prime }=r_t+F_t$ where $F_t=\gamma \Phi (s_{t+1})-\Phi (s_t)$. Feed that into the return.

WHAT we just did: replaced $r_t$ by $r_t+F_t$ inside the definition of return. WHY: the agent optimizes this quantity now, so we must understand it exactly.

Is it legal to split this infinite sum? Yes — and only because of Step 0. With $\gamma \in [0,1)$ and $|r_t|,|\Phi |$ bounded, every series here is dominated by a convergent geometric series ${\sum }_t{\gamma }^t(R_{\max }+2{\Phi }_{\max })<\infty$, so it converges absolutely. Absolute convergence is the precise permission slip that lets us add term-by-term and rearrange without changing the value. So we split:

$G_0^{\prime }=\underset{G_0(\text{the TRUE return})}{\underbrace{\sum _{t=0}^{\infty }{\gamma }^t{r}_t}}+\underset{T(\text{all the shaping})}{\underbrace{\sum _{t=0}^{\infty }{\gamma }^t(\gamma \Phi (s_{t+1})-\Phi (s_t))}}$

Our whole job reduces to one question: what is $T$?

---

Step 4 — The telescope: watch every middle term die

Write $T$ out for the first few steps, and watch the altitudes cancel.

$$T=(\gamma \Phi (s_1)-\Phi (s_0))+\gamma (\gamma \Phi (s_2)-\Phi (s_1))+{\gamma }^2(\gamma \Phi (s_3)-\Phi (s_2))+\cdots$$

Now collect every appearance of a single altitude, say $\Phi (s_1)$:

• It shows up $+\gamma \Phi (s_1)$ from the first bracket.
• It shows up $-\gamma \cdot \Phi (s_1)=-\gamma \Phi (s_1)$ from the second bracket.
• Sum: $\gamma \Phi (s_1)-\gamma \Phi (s_1)=0$. Gone.

The identical thing happens for $\Phi (s_2)$ (as ${\gamma }^2$ from one place, $-{\gamma }^2$ from the next), for $\Phi (s_3)$, forever. Every interior altitude is collected once with $+$ and once with $-$, and they annihilate.

More carefully, the partial sum up to horizon $T-1$ telescopes to ${\sum }_{t=0}^{T-1}{\gamma }^t(\gamma \Phi (s_{t+1})-\Phi (s_t))={\gamma }^T\Phi (s_T)-\Phi (s_0).$ The tail term ${\gamma }^T\Phi (s_T)$ must vanish for the infinite telescope to leave only $-\Phi (s_0)$. It does: since $|\Phi |\le {\Phi }_{\max }$ and $\gamma \in [0,1)$, we have $|{\gamma }^T\Phi (s_T)|\le {\gamma }^T{\Phi }_{\max }\to 0$ as $T\to \infty$. This is exactly why we demanded bounded $\Phi$ and $\gamma <1$ in Step 0 — without either, this tail could survive and the clean result would fail.

This is also where the leading $\gamma$ earns its keep: the "collect" coefficient $\gamma \cdot {\gamma }^{t-1}={\gamma }^t$ must match the "pay" coefficient ${\gamma }^t$ from the next step. Drop the leading $\gamma$ and the two coefficients become ${\gamma }^{t-1}$ vs ${\gamma }^t$ — they no longer cancel, leaving a leftover proportional to $(1-\gamma )\Phi$. The telescope is what forces the $\gamma$ to be there.

Taking $T\to \infty$:

---

Step 5 — Collect: the shaped return is the true return minus a start constant

Put Step 3 and Step 4 together:

Term by term:

• $G_0^{\prime }$ — what the agent now tries to maximize (with our helper).
• $G_0$ — the real return we actually care about.
• $\Phi (s_0)$ — altitude of the start state only. It does not depend on the actions taken, only on where the episode began.

To lift this from single trajectories to values, we need one more object: the policy.

Averaging the boxed identity over all trajectories that a policy $\pi$ produces, the constant $\Phi$ simply rides along:

$V^{\prime \pi }(s)=V^{\pi }(s)-\Phi (s),Q^{\prime \pi }(s,a)=Q^{\pi }(s,a)-\Phi (s)$

See Value Functions and Bellman Equations for where $V$ and $Q$ come from, and Markov Decision Processes for the full formal setting of policies and trajectories.

---

Step 6 — Why the policy cannot change (the geometry of "same shift")

The agent picks the action with the biggest $Q$. Compare shaped vs true:

$\arg {\max }_a{Q}^{\prime }(s,a)=\arg {\max }_a(Q(s,a)-\Phi (s))=\arg {\max }_{a}Q(s,a)$

WHY the middle equals the right: at a fixed state $s$, the term $\Phi (s)$ is the same number for every action $a$. Subtracting the same constant from a list of numbers lowers the whole list equally — the tallest one is still the tallest.

The orange bars (true $Q$) and the teal bars (shaped $Q$) differ by one uniform drop $\Phi (s)$. The winner does not move. This is policy invariance. The helper made the values different but left the decision untouched — exactly what we wanted. See Exploration vs Exploitation for why a better-shaped $Q$ still speeds learning even though the optimum is unchanged.

---

Step 7 — Edge cases: $\gamma =1$, loops, zero potential, and non-potential shaping

A proof you can trust must survive its corners.

(a) The $\gamma =1$ (undiscounted) case and loops. When $\gamma =1$ the partial-sum telescope becomes $\Phi (s_T)-\Phi (s_0)$, and the tail $\Phi (s_T)$ no longer shrinks. So the clean infinite-horizon result needs the setting to be episodic — the agent reaches a terminal state $s_T$ in finite time, where by convention $\Phi (s_T)=0$, giving $T=-\Phi (s_0)$ again. In that episodic $\gamma =1$ world consider a closed loop $s_A\to s_B\to s_A$ with $\Phi (s_A)=2,\Phi (s_B)=5$: step one gives $\Phi (s_B)-\Phi (s_A)=+3$; step two gives $\Phi (s_A)-\Phi (s_B)=-3$; total $0$. A loop returns you to where you started, so its shaping telescopes to $\Phi (\text{start})-\Phi (\text{start})=0$ — no free reward for looping ⇒ no way to hack the shaping. (In the discounted $\gamma <1$ case a non-terminating loop is fine too: the tail ${\gamma }^T\Phi \to 0$ handles it automatically.)

(b) The degenerate $\gamma =0$ and $\Phi \equiv 0$ cases. If $\gamma =0$ the return is just $r_0$ and $F=0\cdot \Phi (s^{\prime })-\Phi (s)=-\Phi (s)$ — still a constant over $a$, so still no policy change (and the telescope trivially gives $-\Phi (s_0)$). If instead $\Phi \equiv 0$ then $F\equiv 0$ and $G_0^{\prime }=G_0$ — no shaping at all. Both extremes leave the theorem intact; a correct proof should quietly contain "do nothing" as a case.

(c) Non-potential shaping breaks everything. Suppose $F(s)=+0.1$ just for hugging a wall — this is not of the form $\gamma \Phi (s^{\prime })-\Phi (s)$. Now the loop in (a) accumulates $+0.1$ every pass with no cancelling partner, so $T\ne -\Phi (s_0)$, the telescope fails, and $\arg \max$ can move. The agent hugs walls forever. This is Reward Hacking and Specification Gaming in one line.

---

The one-picture summary

The final figure compresses the whole chain and is labelled by the step that produced each panel. Reading left to right:

• (Steps 2–3) raw per-step shaping bonuses $F_t$ stack up along a trajectory (top row);
• (Step 4) the interior altitudes pairwise annihilate via the telescope, and the tail ${\gamma }^T\Phi (s_T)\to 0$ under our bounded-$\Phi$, $\gamma <1$ assumption (middle);
• (Steps 5–6) only a single start-state offset $-\Phi (s_0)$ survives, which shifts every action's value equally and therefore cannot change the winner (bottom).

Recall Feynman retelling (optional recap — skip if the seven steps already landed)

This box adds no new result; it is a plain-words re-run of the walkthrough for a second pass. Imagine painting every square of a map with a height. The helper reward pays you the height of the square you step onto (shrunk by $\gamma$ because it's one step in the future) and charges you the height of the square you step off of. Walk any path and add up all these payments-and-charges: every square you pass through gets paid for once and charged for once — those cancel like $+3$ and $-3$. Because heights are bounded and future amounts keep shrinking (that's $\gamma <1$), even the far-future leftover fades to nothing. The only thing that never gets cancelled is the charge for your very first square, fixed the moment you start and independent of your choices. Same fixed amount for every choice ⇒ it can't change which choice is best. That's the magic: the helper changes the numbers but not the decision — it guides without ever tricking you into looping forever. And if the helper isn't built from heights this way, the cancellation fails, a loop pays real money every lap, and the agent will happily spin in circles collecting it.

Related paths from here: relabel failed goals into successes with Hindsight Experience Replay; grow goals from near to far with Curriculum Learning; the full MDP scaffolding lives in Markov Decision Processes. For the same walkthrough in Hindi-English, see 5.2.13 Reward shaping and sparse rewards (Hinglish).