Agent, environment, state, action, reward: Question bank

Before we start, two pictures to anchor the vocabulary. The first separates the four things learners most often blur — the true state, the observation the agent sees, the action it emits, and the reward the environment hands back:

Figure — Agent, environment, state, action, reward

Look at the red observation channel: it is only a slice of the true state box on the left. That gap is the whole story of partial observability — the agent decides using the red thing, not the full box.

The second picture is the single most-confused pair on this page — a reward ( $r_{t}$ , one bar at one time step) versus a return ( $G_{t}$ , the whole discounted pile from $t$ onward):

Each grey bar is one reward $r_{t}$ ; the red bracket is the return $G_{0}$ — every future reward, each shrunk by another factor of $γ$ . Notice the far bars barely count: that is discounting doing its job.

Now the symbols, pinned down in plain language so no later line surprises you:

Definition The symbols we will lean on

State $s_{t}$ : a complete description of the situation the world is in at step $t$ — for a grid robot, its cell $(x, y)$ ; for a thermostat, the current temperature. The environment state is the true one; when the agent sees it perfectly we just write $s_{t}$ .
Observation $o_{t}$ : what the agent actually perceives at step $t$ through its sensors. In a fully observable world $o_{t} = s_{t}$ ; when sensors are partial or noisy, $o_{t}$ is only a slice of $s_{t}$ .
Action $a_{t}$ : the choice the agent makes at step $t$ — the only thing it directly controls.
Transition probability $P (s^{'} ∣ s, a)$ : the environment's rulebook — read "the probability the next state is $s^{'}$ given we were in state $s$ and took action $a$ ." This is the environment's dynamics; the agent does not set it.
Reward function $R (s, a, s^{'})$ : a rule, written by the task designer, that assigns a number to the event "in state $s$ we took action $a$ and landed in $s^{'}$ ." The realized reward is that number for the step actually taken: $r_{t} = R (s_{t}, a_{t}, s_{t + 1})$ — a single scalar handed to the agent, nothing more.
Episode / finite horizon: an episode is one complete run that is guaranteed to terminate — it hits a terminal state (goal reached, game over) after at most $T$ steps. A task with such a guaranteed end is called finite-horizon. A task that can run forever with no terminal state is infinite-horizon.
Return $G_{t}$ : the whole future pile of realized rewards from step $t$ onward, $G_{t} = r_{t} + γ r_{t + 1} + γ^{2} r_{t + 2} + \dots$ , where the ==discount factor $γ$ == shrinks far-off rewards. We use $γ \in [0, 1)$ for infinite-horizon tasks; $γ = 1$ is only allowed in the finite-horizon case above, where the sum has finitely many terms and cannot blow up.
Reward bound $R_{m a x}$ : a number such that every reward stays inside $[- R_{m a x}, R_{m a x}]$ , i.e. $∣ r_{t} ∣ \leq R_{m a x}$ for all $t$ . We assume rewards are bounded this way.
Policy $π$ : the agent's rule for turning what it sees into an action. Two flavours, and two settings:
- Deterministic: a fixed map $a = π (s)$ — same input, same action.
- Stochastic: a dice-roll $a \sim π (a ∣ s)$ — sample the action from a probability distribution.
- When the world is only partially observable, the policy reads the observation, not the state, so we write $a = π (o)$ or $a \sim π (a ∣ o)$ instead of $s$ .

The recursive shape of the return — one reward now plus the discounted future — is worth seeing on its own, because it is the seed of the Bellman equation:

The red arrow shows $G_{t}$ folding into " $r_{t}$ plus $γ$ times the next return $G_{t + 1}$ " — the same shape nested inside itself.

Everything below tests whether you can tell reward, return, and policy apart, keep state $s$ , observation $o$ , transition $P$ and reward function $R$ straight, and handle their edge cases.

True or false — justify

TF1. "Maximizing reward at every single step is the same as maximizing return."

False. A greedy step can spoil the future — sacrificing a chess piece (negative

r_{t}

) to force checkmate (huge later reward) beats the short-sighted grab. Return sums the whole future, not one step.

TF2. "The reward is chosen by the environment's physics."

False. The reward function

R (s, a, s^{'})

that produces each realized

r_{t} = R (s_{t}, a_{t}, s_{t + 1})

is written by the task designer to encode the goal; the environment merely evaluates it. Change

R

and you change what "good" means.

TF3. "If

γ = 0

, the agent completely ignores the future."

True. With

γ = 0

G_{t} = r_{t} + 0 \cdot (\dots) = r_{t}

, so the agent optimizes only the immediate reward — fully myopic.

TF4. "The discount factor

γ

may equal exactly

1

in the infinite-horizon return."

False. For an infinite sum we need

γ < 1

so the geometric series converges to

\frac{R _{m a x}}{1 - γ}

; at

γ = 1

the bound blows up. As stated in the symbol list,

γ = 1

is only safe in the finite-horizon case, where the sum has finitely many terms.

TF5. "A stochastic policy means the environment is random."

False. Stochasticity in

π

lives in the agent's action choice (

a \sim π (a ∣ s)

). A deterministic agent can still face a random environment via the transition

P (s^{'} ∣ s, a)

— they are independent sources of randomness.

TF6. "In a fully observable environment, the observation equals the state, so

o_{t} = s_{t}

True. Full observability is defined by the agent's perception

o_{t}

matching the true state

s_{t}

. When they differ you are in POMDP territory and the policy must read

o_{t}

, i.e.

π (a ∣ o)

TF7. "The agent can directly set the next state."

False. The agent controls only its action

a_{t}

. The next state is drawn by the environment's transition rule

s_{t + 1} \sim P (s^{'} ∣ s_{t}, a_{t})

— the agent nudges through

a_{t}

, the world decides

s_{t + 1}

TF8. "Because rewards are scalars, you can always convert a two-goal task (fast AND safe) into a single reward, with no loss."

False. You must collapse everything into one scalar

r_{t}

to run standard RL, typically by a weighted sum — but the weights are a design choice that silently fixes the trade-off, and that collapse can lose information. So "with no loss" is what makes the statement false.

TF9. "The Markov property says the past doesn't matter at all."

Half-true. It says the past doesn't matter given the current state:

P (s_{t + 1} ∣ s_{t}, a_{t}) = P (s_{t + 1} ∣ s_{t}, a_{t}, whole history)

. If the state is poorly designed, the history does carry needed info — then it isn't really Markov.

Spot the error

SE1. "I set

r_{t} = 0

for every non-goal cell and

r_{t} = + 10

only at the goal, so the agent will learn the shortest path."

Error: with zero step-cost, wandering forever costs nothing, so nothing rewards speed. Add a per-step penalty (e.g.

r_{t} = - 1

on non-goal steps) so lingering hurts and the shortest path wins.

SE2. "To make the walking robot go forward, I reward it for not falling over."

Error: "not falling" is maximized by lying down and never moving. Reward the forward distance (the actual goal), not a proxy that has a cheap degenerate solution.

SE3. "My return is

G_{t} = r_{t} + γ r_{t + 1} + γ r_{t + 2} + \dots

Error: the discount's exponent must carry the time offset and grow with it — term

k

steps ahead gets

γ^{k}

, giving

γ^{0} r_{t} + γ^{1} r_{t + 1} + γ^{2} r_{t + 2} + \dots

. The written version reuses

γ^{1}

on every future term, dropping the growing exponent, so distant rewards are never down-weighted and (for

γ < 1

) the sum's convergence bound no longer applies.

SE4. "The agent state must contain everything in the environment state."

Error: the environment's true state

s_{t}

can be huge and hidden; the agent only ever holds its observation

o_{t}

— what its sensors reveal. Demanding

o_{t} = s_{t}

always defeats the whole point of partial observability.

SE5. "Since actions are continuous here, I'll pick five discrete throttle levels — same thing."

Error: discretizing a naturally continuous action (throttle

\in [0, 1]

) throws away precision and can miss the optimal value that sits between your buckets. It's an approximation, not an equivalence.

SE6. "The recursive return is

G_{t} = r_{t} + G_{t + 1}

Error: the future pile must itself be discounted:

G_{t} = r_{t} + γ G_{t + 1}

. Dropping the

γ

double-counts distant rewards at full strength.

SE7. "The environment boundary is a law of physics I cannot move."

Error: the agent–environment boundary is a design choice. Motor controllers can sit inside the agent or inside the environment (which changes what counts as

a_{t}

versus what the transition

P (s^{'} ∣ s, a)

handles); what matters is that everything the agent can't perfectly control is "environment."

Why questions

WHY1. Why maximize return instead of the sum of undiscounted rewards forever?

Because an infinite undiscounted sum can be infinite (no best policy) and treats a reward now the same as one in a billion years. Discounting makes the objective finite and prefers sooner, more certain rewards.

WHY2. Why does the geometric series

\sum_{k = 0}^{\infty} γ^{k} = \frac{1}{1 - γ}

matter for RL?

With the reward bound

∣ r_{t} ∣ \leq R_{m a x}

, it proves the return is bounded by

\frac{R _{m a x}}{1 - γ}

when

γ < 1

, so "maximize return" is a well-posed question with a finite answer.

WHY3. Why is the recursive form

G_{t} = r_{t} + γ G_{t + 1}

so important?

It expresses the whole future in terms of one reward plus next step's future — this self-reference is exactly the seed of the Bellman equation, which makes value estimation tractable.

WHY4. Why insist the reward be a single scalar and not a vector?

Because "maximize" needs a total order — you can compare two numbers but not two vectors without extra rules. The scalar forces the designer to state the trade-offs up front.

WHY5. Why is the Markov property worth engineering the state for?

s_{t}

summarizes all decision-relevant history, the agent can act on the present alone, shrinking the problem from "remember everything" to "look at now" — the basis of the MDP formulation.

WHY6. Why can't the agent just be told the right action, like in supervised learning?

RL gives only a scalar reward

r_{t}

, never the labeled correct action. The agent must infer which actions were good from delayed, noisy feedback — this is why it must explore.

WHY7. Why does a tiny energy penalty (

- 0.01 a_{t}^{2}

) get added in the thermostat reward?

To break ties toward efficiency: among policies that hold the target temperature, the one that also uses less power scores slightly higher, nudging the agent to stop over-heating.

Edge cases

EC1. What is the return of a trajectory that ends immediately at

t

(episode of length one)?

Just

G_{t} = r_{t}

— the sum has a single term, no future to discount. A perfectly valid, if trivial, episode.

EC2. What happens to the objective if every reward is

0

everywhere?

Every policy has return

0

; the agent has no gradient of preference and learns nothing useful. Reward must vary to communicate a goal.

EC3. A designer sets

γ

extremely close to

1

(say

0.9999

) in an infinite task. Any risk?

The return converges but the bound

\frac{R _{m a x}}{1 - γ}

becomes enormous, so tiny reward differences barely change return — learning becomes slow and sensitive. It's the far-sighted extreme, not free.

EC4. What is the return if all rewards are negative but bounded (so

- R_{m a x} \leq r_{t} \leq 0

Still finite:

G_{t} \geq - \frac{R _{m a x}}{1 - γ}

. "Maximizing" then means making the return least negative — reaching a terminal reward or minimizing accumulated penalty.

EC5. The agent's sensor gives the same observation

o_{t}

in two genuinely different environment states

s_{t}

. Which regime?

This is aliasing under partial observability — the observation is not a Markov state, so a memoryless policy

π (a ∣ o)

can't distinguish them. You've stepped into a POMDP.

EC6. Single state, no transitions, only a choice of arms with random payoffs — is that still this framework?

Yes, it's the degenerate one-state case: the multi-armed bandit. With no state dynamics, return collapses to the reward of the action you pick.

EC7. The environment's transition

P (s^{'} ∣ s, a)

is unknown to the agent. Can it still learn a policy?

Yes — that's the model-free setting, where the agent learns straight from experienced

(s, a, r, s^{'})

tuples without ever writing down

P

EC8. A "stochastic policy" that puts probability

1

on one action in every state — what is it really?

A deterministic policy in disguise. Determinism is the special case of a stochastic policy with a degenerate (point-mass) distribution.

Recall Self-check

Return vs reward — which one does the agent maximize? ::: The return $G_{t}$ (discounted sum of future realized rewards), never a single reward $r_{t}$ . What single condition makes $o_{t} = s_{t}$ ? ::: Full observability — the agent's observation $o_{t}$ equals the true environment state $s_{t}$ . Why must $γ < 1$ for infinite horizons? ::: So the geometric series converges and the return stays finite (bounded by $R_{m a x} / (1 - γ)$ ); $γ = 1$ is only safe when the episode is guaranteed to end. How do you write a policy when the world is only partially observable? ::: Over observations: $a = π (o)$ or $a \sim π (a ∣ o)$ , since the agent cannot see the true state.