Agent, environment, state, action, reward: Worked examples

This page is the "grind through every case" companion to the parent note. We take the two ideas that the parent introduced — the reward $r_{t}$ (a single number the world hands you at step $t$ ) and the return $G_{t}$ (the whole future added up) — and we hammer them against every situation the maths can throw at us.

Before we compute anything, let's recall the two objects in plain words, because every example below is just these two, dressed differently.

Recall What is

$r_{t}$ vs $G_{t}$ ? (open me first) $r_{t}$ ::: the reward at one step — a single scalar (one number) the environment emits right after action $a_{t}$ . $G_{t}$ ::: the return — the sum of this reward plus all future rewards, each future one shrunk by a factor $γ$ per step into the future. $γ$ ::: the discount factor, a dial between $0$ and $1$ that says "how much do I care about the future?"

The scenario matrix

Every return calculation you will ever meet is one of the cells below. The examples that follow are labelled with the cell they cover, and together they hit every cell.

Cell	What makes it special	Covered by
A. Finite horizon, plain sum	episode ends at step $T$ , $γ$ ignored (or $= 1$ )	Ex 1
B. Infinite horizon, constant reward	same $r$ forever, geometric series	Ex 2
C. Mixed signs	some rewards $+$ , some $-$ (sacrifice now for later)	Ex 3
D. Degenerate $γ = 0$	myopic agent, future ignored entirely	Ex 4
E. Limiting $γ \to 1^{-}$	far-sighted, sum blows toward its cap	Ex 4
F. Recursive / bootstrapped	compute $G_{t}$ from $G_{t + 1}$	Ex 5
G. Stochastic reward → expectation	reward is random, we need the average	Ex 6
H. Real-world word problem	translate a story into $r_{t}$ , then sum	Ex 7
I. Exam twist	reward AFTER episode end / self-consistency check	Ex 8

Figure — Agent, environment, state, action, reward

The figure above is our mental picture for the whole page: a timeline of steps, each with a reward bar, and the discount curtain $γ^{k}$ fading the far-away bars. Look at how the plum curve $γ^{k}$ drops — a reward far in the future contributes almost nothing.

Example 1 — Finite horizon, plain sum (Cell A)

Forecast: guess before reading — three penalties then a payoff. Positive or negative total?

Write the finite-horizon sum. With $γ = 1$ the return is just addition: $G_{0} = r_{0} + r_{1} + r_{2} + r_{3} .$ Why this step? When an episode has a clear end, there is no infinite tail to worry about — the "sum forever" collapses to a plain finite sum.
Plug the numbers. $G_{0} = (- 1) + (- 1) + (- 1) + 10 = 7.$ Why this step? Substitution — nothing subtle. Signs just add.

Verify: three steps cost $3$ units, the goal pays $10$ , net $10 - 3 = 7$ . Positive: the goal was worth the walk. Units: all rewards are the same abstract "score" unit, so adding is legal. ✓

Example 2 — Infinite horizon, constant reward (Cell B)

Forecast: the reward never stops. Will $G_{0}$ be infinite, or finite?

Substitute the constant into the sum. $G_{0} = \sum_{k = 0}^{\infty} γ^{k} r_{k} = 2 \sum_{k = 0}^{\infty} (0.9)^{k} .$ Why this step? Because $r_{k} = 2$ for all $k$ , we pull the constant out of the sum. This is why we need $γ < 1$ : the constant reward would give $\infty$ without the fading.
Use the geometric-series fact. For $∣ γ ∣ < 1$ , $\sum_{k = 0}^{\infty} γ^{k} = \frac{1}{1 - γ} .$ Why this step? This closed form is exactly why discounting was invented — it turns an infinite process into one clean number.
Evaluate. $G_{0} = 2 \cdot \frac{1}{1 - 0.9} = 2 \cdot 10 = 20.$ Why this step? $1 - 0.9 = 0.1$ , and $1/0.1 = 10$ .

Verify: the cap from the parent note is $\frac{R _{ma x}}{1 - γ} = \frac{2}{0.1} = 20$ . We hit the cap exactly, which is correct because every reward equals $R_{ma x}$ . ✓ (See the Bellman Equations deep dive for why constant reward saturates the bound.)

Example 3 — Mixed signs, sacrifice now for later (Cell C)

Forecast: immediate reward is negative. Is the return still positive?

Write the discounted finite sum. $G_{0} = r_{0} + γ r_{1} + γ^{2} r_{2} + γ^{3} r_{3} .$ Why this step? Mixed signs change nothing structurally — we still discount each future term by its own $γ^{k}$ .
Plug in. $G_{0} = - 3 + 0.9 (0) + 0.81 (0) + 0.729 (9) .$ Why this step? $γ^{3} = 0. 9^{3} = 0.729$ ; the two zeros vanish.
Compute. $G_{0} = - 3 + 6.561 = 3.561.$

Verify: the immediate reward alone ( $- 3$ ) would tell a greedy agent to never sacrifice. But the return is $+ 3.561 > 0$ , so the sacrifice pays. This is the exact "reward $\neq =$ return" lesson from the parent's [!mistake] callout. ✓

Example 4 — The two extreme dials: $γ = 0$ and $γ \to 1^{-}$ (Cells D & E)

Forecast: the big prize $100$ is one step away. Which $γ$ "sees" it?

Case (a), $γ = 0$ . The formula becomes $G_{0} = r_{0} + 0 \cdot r_{1} = 1.$ Why this step? $γ^{0} = 1$ always, but $γ^{1} = 0$ when $γ = 0$ . Every future term is multiplied by zero — the agent is myopic, blind to anything past now.
Case (b), $γ = 0.99$ . $G_{0} = 1 + 0.99 (100) = 1 + 99 = 100.$ Why this step? $γ$ near $1$ keeps almost the full value of the future reward — the agent is far-sighted.

Look at the figure: as $γ$ slides from $0$ to nearly $1$ , the value of the $100$ -reward rises from $0$ to $\approx 99$ . The teal curve is $G_{0} (γ)$ ; the dashed line is the myopic value $1$ .

Verify: at $γ = 0$ we get exactly $r_{0} = 1$ (the definition of myopic). At $γ = 0.99$ we get $1 + 99 = 100$ . As $γ \to 1^{-}$ the return $\to 1 + 100 = 101$ , its finite-horizon cap. The tuning of $γ$ is what the parent called "the control knob." ✓ (Deeper choices of $γ$ live in Markov Decision Process (MDP).)

Example 5 — Recursive / bootstrapped form (Cell F)

Forecast: can you get $G_{0}$ from $G_{1}$ with one multiply and one add?

Use the recursion. From the parent note, $G_{0} = r_{0} + γ G_{1} .$ Why this step? The infinite tail starting at step $1$ is $G_{1}$ by definition; discounting it by one $γ$ folds it into $G_{0}$ . This is bootstrapping — reusing a computed value instead of re-summing.
Plug in. $G_{0} = 5 + 0.9 \cdot 30 = 5 + 27 = 32.$

Verify: unroll one more layer to sanity-check the logic — $G_{0} = r_{0} + γ (r_{1} + γ G_{2})$ , and $r_{1} + γ G_{2}$ must equal $G_{1} = 30$ for consistency. It does by assumption, so $G_{0} = 32$ . This recursion is the literal seed of the Bellman Equations. ✓

Example 6 — Stochastic reward → expectation (Cell G)

Forecast: the reward is random. What single number summarises it?

The return of one step is just its reward. Since the episode ends immediately, $G_{0} = r_{0}$ . Why this step? No future terms exist, so no discounting is needed.
Take the expectation because $r_{0}$ is random. Expectation = "probability-weighted average of outcomes": $E [G_{0}] = 0.3 \cdot 10 + 0.7 \cdot 0.$ Why this step? When rewards are stochastic, the agent maximises the average return, not any single lucky draw. This is exactly the setting of Multi-Armed Bandits.
Compute. $E [G_{0}] = 3 + 0 = 3.$

Verify: the answer must lie between the smallest ( $0$ ) and largest ( $10$ ) possible reward — and $0 \leq 3 \leq 10$ . ✓ It is closer to $0$ because the $0$ -outcome is more likely ( $0.7$ ). The tension between "try the risky $+ 10$ " and "the average is only $3$ " is Exploration vs Exploitation.

Example 7 — Real-world word problem (Cell H)

Forecast: all rewards are negative (penalties). Is $G_{0}$ negative? By how much?

Compute $r_{0}$ . Substitute $T_{0} = 18, a_{0} = 0.8$ : $r_{0} = - (18 - 21)^{2} - 0.01 (0.8)^{2} = - 9 - 0.0064 = - 9.0064.$ Why this step? The squared-error term punishes being $3^{\circ}$ off target; the tiny $a^{2}$ term punishes energy use.
Compute $r_{1}$ . Substitute $T_{1} = 19.5, a_{1} = 0.6$ : $r_{1} = - (19.5 - 21)^{2} - 0.01 (0.6)^{2} = - 2.25 - 0.0036 = - 2.2536.$ Why this step? Closer to target ( $1. 5^{\circ}$ off), so a smaller penalty — good, the controller is working.
Discount and sum. $G_{0} = r_{0} + γ r_{1} = - 9.0064 + 0.9 (- 2.2536) = - 9.0064 - 2.02824 = - 11.03464.$ Why this step? Standard two-term discounted return.

Verify: both rewards are negative, so $G_{0}$ must be negative — and $- 11.03 < 0$ . ✓ The magnitude is dominated by the first step's large $9^{\circ}^{2}$ error, exactly as the reward design intended: penalise deviation heavily, energy lightly. (Continuous actions like the heater power sit in Model-Free vs Model-Based RL control tasks.)

Example 8 — Exam twist: reward after the horizon, and self-consistency (Cell I)

Forecast: a reward listed after the episode ends is a trap. How much does it wrongly add?

(a) Respect the horizon $T = 2$ . Only $r_{0}, r_{1}, r_{2}$ count: $G_{0}^{correct} = 2 + 0.5 (2) + 0.25 (2) = 2 + 1 + 0.5 = 3.5.$ Why this step? Once the episode ends, no further rewards exist — the sum stops at $k = T - t = 2$ .
(b) The student's mistaken sum adds the phantom $r_{3}$ at $γ^{3} = 0.125$ : $G_{0}^{wrong} = 3.5 + 0.125 (2) = 3.5 + 0.25 = 3.75.$ Why this step? We just append the extra discounted term to see the size of the error.
The inflation is $3.75 - 3.5 = 0.25$ . Why this step? Subtracting isolates exactly the phantom term's contribution.

Verify: the phantom term is $γ^{3} r_{3} = 0.125 \cdot 2 = 0.25$ , matching the inflation exactly. ✓ Lesson: always stop the sum at the true episode end. Partial-information versions of "when does the episode end?" are studied in Partial Observability (POMDP). Policy quality across such returns is the subject of Policy and Value Functions.

Recall Rapid self-test

Constant reward $r = 2$ , $γ = 0.9$ , infinite horizon — $G_{0}$ ? ::: $2/ (1 - 0.9) = 20$ . $γ = 0$ makes the agent... ::: myopic — it only sees the immediate reward. Given $G_{1} = 30, r_{0} = 5, γ = 0.9$ , then $G_{0}$ ? ::: $5 + 0.9 (30) = 32$ . A reward listed after the episode ends should be... ::: ignored — the sum stops at the horizon $T$ .