Agent, environment, state, action, reward: Visual walkthrough

This page builds ONE idea from nothing: why the agent chases the sum of future rewards, discounted, and why that sum is a finite, well-behaved number. We start with a single arrow of feedback and end with the recursive rule that seeds the Bellman Equations.

Everything here refines the loop from the parent note Agent, environment, state, action, reward. If a symbol appears, it was drawn first.

Step 1 — One tick of the clock: what actually happens

WHAT. At a single moment in time we have four things in a chain: the agent is in a situation, it makes a choice, the world changes, and a single number comes back.

WHY. Before summing anything, we must agree on what one reward even is and where it comes from. Everything later is just this one tick, repeated.

PICTURE. Look at the figure. The purple box is the agent (the chooser). The coral box is the environment (the world). The three labelled arrows are the entire transaction of one time step.

Figure — Agent, environment, state, action, reward

Read the arrows left to right: see $s_{t}$ → choose $a_{t}$ → receive $r_{t}$ and land in $s_{t + 1}$ . That " $+ 1$ " in $s_{t + 1}$ just means "the next tick's state."

Step 2 — Chaining ticks into a trajectory

WHAT. We paste many single ticks end to end. The state a tick lands in becomes the state the next tick starts from.

WHY. A reward alone tells you this instant was good or bad. But learning to ride a bike is about a whole ride. To talk about "how good was the whole thing," we need the full chain.

PICTURE. Each grey capsule is one Step-1 tick. The mint arrow shows how $s_{t + 1}$ of one capsule is the $s_{t}$ of the next — the ticks interlock like train cars.

The single number we care about — how good was the whole ride — must be built out of these $r_{0}, r_{1}, r_{2}, \dots$ . That's Step 3.

Step 3 — The naive answer: just add the rewards (finite horizon)

WHAT. If the ride ends — a game is won, a robot reaches the goal at some final tick $T$ — the honest "total goodness from now on" is simply the sum of the rewards left.

WHY. Nothing fancy is needed when there's a finish line. Every reward counts once, fully. We call this total the return.

PICTURE. Bars are individual rewards $r_{t}, r_{t + 1}, \dots, r_{T}$ . Stacking them (mint total on the right) is the return $G_{t}$ . Notice the sum starts at the current tick $t$ , not at $0$ — we always ask "from here on."

Step 4 — The problem: what if the ride never ends?

WHAT. Many tasks have no finish line — a thermostat runs forever, a trading agent never "wins." Then $T = \infty$ and the naive sum can blow up to infinity.

WHY. If every future reward is, say, $+ 1$ , then $1 + 1 + 1 + \dots = \infty$ . Every policy would score "infinity," so we could never say one is better. A useless objective. We need a total that stays finite yet still prefers good behaviour.

PICTURE. Left panel: equal bars marching off to the right forever — the area (total) is unbounded, the number breaks. This is the wall we must get past.

Step 5 — The fix: discounting (why the exponential, not something else)

WHAT. Multiply each future reward by a shrinking weight before adding: the reward $k$ ticks away is worth only $γ^{k}$ of its face value, where $γ$ (the Greek letter gamma) is a number between $0$ and $1$ .

WHY this exact tool — the power $γ^{k}$ , not a fixed cap or a linear fade? Three reasons stack up:

It converges (Step 6 proves it) — the total stays finite.
It's consistent across time. One tick of waiting always costs the same factor $γ$ , no matter where you are in the ride. A linear fade like $(1 - k /100)$ would treat tick 50 and tick 51 differently from tick 1 and tick 2, and would even go negative. Only the exponential $γ^{k}$ has "each step of delay costs a fixed fraction" — the same reason compound interest uses powers.
It's a dial. $γ \to 0$ = only care about right now (myopic); $γ \to 1$ = care about the far future (far-sighted).

PICTURE. The same infinite bars from Step 4, but now each is scaled by the mint curve $γ^{k}$ . The far-off bars are crushed toward zero, so the shaded total is finite — a filled shape with a real area.

Step 6 — The edge case that must not break: does the sum stay finite?

WHAT. We prove that even with infinitely many terms, $G_{t}$ is a finite number — provided rewards don't themselves run to infinity.

WHY. Step 5 claimed convergence; a claim isn't a proof. We must guarantee the agent's objective is a real, comparable number in the worst case (every reward as large as allowed).

PICTURE. The bound in one image: replace every real reward with the largest possible one, $R_{m a x}$ (grey dashed ceiling). The discounted stack of these ceilings is the tallest the return could ever be — and even that stack is a finite geometric total (the mint area saturates to a line).

Now bound the return term by term: $∣ G_{t} ∣ = \sum_{k = 0}^{\infty} γ^{k} r_{t + k} \leq \sum_{k = 0}^{\infty} γ^{k} ∣ r_{t + k} ∣ \leq \sum_{k = 0}^{\infty} γ^{k} R_{m a x} = R_{m a x} \sum_{k = 0}^{\infty} γ^{k}$

First $\leq$ : the size of a sum is at most the sum of sizes (triangle inequality).
Second $\leq$ : swap each $∣ r_{t + k} ∣$ for its ceiling $R_{m a x}$ .

The leftover $\sum_{k = 0}^{\infty} γ^{k}$ is a geometric series — each term is the previous times $γ$ . Its closed form: $\sum_{k = 0}^{\infty} γ^{k} = \frac{1}{1 - γ} valid for ∣ γ ∣ < 1.$

This is the degenerate boundary made explicit: $γ = 1$ ⇒ budget $\infty$ ⇒ return may diverge (back to Step 4's wall). $γ = 0$ ⇒ budget $1$ ⇒ $G_{t} = r_{t}$ , pure greed. The safe operating range is $0 \leq γ < 1$ .

Step 7 — The payoff: the return folds into itself

WHAT. Pull the very first reward out of the discounted sum. What remains is the same kind of return, one tick later, scaled by $γ$ .

WHY. This turns an infinite sum into a tiny two-piece rule — "now's reward, plus a discounted copy of the future." It's the seed from which Policy and Value Functions and the Bellman Equations grow, and it's what lets algorithms bootstrap (estimate a value from a neighbouring value).

PICTURE. Split the stack: the bottom brick is $r_{t}$ at full height; the entire rest of the stack, lifted out and re-labelled, is $G_{t + 1}$ — and it enters scaled by $γ$ (mint bracket).

$G_{t} = r_{t} + γ r_{t + 1} + γ^{2} r_{t + 2} + \dots = r_{t} + γ this is exactly G_{t + 1} (r_{t + 1} + γ r_{t + 2} + \dots)$

Every term inside the bracket had one factor of $γ$ pulled out front, and the leftover is the definition of the return starting at $t + 1$ .

The one-picture summary

Every idea on this page, in a single frame: one tick becomes a chain; the chain's rewards become a discounted stack; the stack is provably finite and folds into $r_{t} + γ G_{t + 1}$ .

Recall Feynman retelling — say it like you'd explain to a friend

The world hands the agent one number at a time — a reward. Nobody cares about a single number; we care about the whole trip's worth, which is just all the future rewards added up. But if the trip never ends, "add them all" can be infinity, and infinity beats infinity, so nothing would ever look better than anything else. Fix: shrink each future reward by a fixed fraction $γ$ per step of delay — the reward $k$ steps away is worth $γ^{k}$ of itself. Distant rewards politely fade to nothing, and (because each is at most $R_{m a x}$ and the weights add to $1/ (1 - γ)$ ) the grand total is always a finite number. The magic bonus: pull the first reward out and what's left is the very same total, one step later, so $G_{t} = r_{t} + γ G_{t + 1}$ . That little loop is the whole engine of RL.

Recall Quick self-test

What is $G_{t}$ in words? ::: The discounted sum of all rewards from time $t$ onward — the total future goodness the agent tries to maximise. Why must $γ < 1$ (not $= 1$ )? ::: At $γ = 1$ the weight budget $\frac{1}{1 - γ}$ is infinite, so the sum can diverge; $γ < 1$ keeps $G_{t}$ finite. What does $γ = 0$ make the agent do? ::: Only care about the immediate reward $r_{t}$ — purely greedy/myopic. State the recursive return. ::: $G_{t} = r_{t} + γ G_{t + 1}$ . What ceiling bounds the return? ::: $∣ G_{t} ∣ \leq \frac{R _{m a x}}{1 - γ}$ .

Where this goes next: replace the actual rewards with their expected values under a strategy and $G_{t}$ becomes a value — the doorway to Policy and Value Functions, the Bellman Equations, and (once we ask how the agent gathers rewards it hasn't seen) Exploration vs Exploitation. When the state is only partly visible, see Partial Observability (POMDP); the fully-formalised loop is the Markov Decision Process (MDP).